LeaderRebalancerTest.RestartTserver: Unrecognized error type. Please see the error log for more information.

[==========] Running 15 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 13 tests from LeaderRebalancerTest
[ RUN      ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260808 01:47:49.869086  7831 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.7.165.254:36421
I20260808 01:47:49.872728  7831 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 01:47:49.874742  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:47:49.902133  7838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:47:49.902169  7837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:47:49.902271  7840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:47:49.903569  7831 server_base.cc:1061] running on GCE node
I20260808 01:47:51.297952  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:47:51.298364  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:47:51.298611  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786153671298576 us; error 0 us; skew 500 ppm
I20260808 01:47:51.299971  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:47:51.307065  7831 webserver.cc:533] Webserver started at http://127.7.165.254:35751/ using document root <none> and password file <none>
I20260808 01:47:51.308688  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:47:51.309010  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:47:51.309700  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:47:51.318332  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/master-0-root/instance:
uuid: "45f1bafd07d4419b8e68b4d0d13a6759"
format_stamp: "Formatted at 2026-08-08 01:47:51 on dist-test-slave-8tdl"
I20260808 01:47:51.330506  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.011s	user 0.005s	sys 0.004s
I20260808 01:47:51.339223  7846 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:47:51.341498  7831 fs_manager.cc:730] Time spent opening block manager: real 0.007s	user 0.003s	sys 0.000s
I20260808 01:47:51.342052  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/master-0-root
uuid: "45f1bafd07d4419b8e68b4d0d13a6759"
format_stamp: "Formatted at 2026-08-08 01:47:51 on dist-test-slave-8tdl"
I20260808 01:47:51.342545  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:47:51.407172  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:47:51.409257  7831 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 01:47:51.410017  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:47:51.507840  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.254:36421
I20260808 01:47:51.507951  7899 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.254:36421 every 8 connection(s)
I20260808 01:47:51.522688  7900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:51.552059  7900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759: Bootstrap starting.
I20260808 01:47:51.561745  7900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:51.564582  7900 log.cc:826] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759: Log is configured to *not* fsync() on all Append() calls
I20260808 01:47:51.570982  7900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759: No bootstrap required, opened a new log
I20260808 01:47:51.596069  7900 raft_consensus.cc:359] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f1bafd07d4419b8e68b4d0d13a6759" member_type: VOTER }
I20260808 01:47:51.596753  7900 raft_consensus.cc:385] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:51.597092  7900 raft_consensus.cc:740] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f1bafd07d4419b8e68b4d0d13a6759, State: Initialized, Role: FOLLOWER
I20260808 01:47:51.598572  7900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f1bafd07d4419b8e68b4d0d13a6759" member_type: VOTER }
I20260808 01:47:51.599242  7900 raft_consensus.cc:399] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 01:47:51.599583  7900 raft_consensus.cc:493] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 01:47:51.600031  7900 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:51.604837  7900 raft_consensus.cc:515] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f1bafd07d4419b8e68b4d0d13a6759" member_type: VOTER }
I20260808 01:47:51.606061  7900 leader_election.cc:304] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45f1bafd07d4419b8e68b4d0d13a6759; no voters: 
I20260808 01:47:51.608227  7900 leader_election.cc:290] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260808 01:47:51.608752  7903 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:47:51.611521  7903 raft_consensus.cc:697] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [term 1 LEADER]: Becoming Leader. State: Replica: 45f1bafd07d4419b8e68b4d0d13a6759, State: Running, Role: LEADER
I20260808 01:47:51.612844  7903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f1bafd07d4419b8e68b4d0d13a6759" member_type: VOTER }
I20260808 01:47:51.614130  7900 sys_catalog.cc:565] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 01:47:51.628410  7905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45f1bafd07d4419b8e68b4d0d13a6759. Latest consensus state: current_term: 1 leader_uuid: "45f1bafd07d4419b8e68b4d0d13a6759" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f1bafd07d4419b8e68b4d0d13a6759" member_type: VOTER } }
I20260808 01:47:51.628778  7904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45f1bafd07d4419b8e68b4d0d13a6759" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f1bafd07d4419b8e68b4d0d13a6759" member_type: VOTER } }
I20260808 01:47:51.629369  7904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [sys.catalog]: This master's current role is: LEADER
I20260808 01:47:51.629369  7905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [sys.catalog]: This master's current role is: LEADER
I20260808 01:47:51.633173  7913 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260808 01:47:51.649868  7913 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260808 01:47:51.652940  7831 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260808 01:47:51.674731  7913 catalog_manager.cc:1383] Generated new cluster ID: 9034a8654d114a8b9b732fc68b81c33a
I20260808 01:47:51.675139  7913 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260808 01:47:51.696200  7913 catalog_manager.cc:1406] Generated new certificate authority record
I20260808 01:47:51.698835  7913 catalog_manager.cc:1540] Loading token signing keys...
I20260808 01:47:51.720508  7913 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759: Generated new TSK 0
I20260808 01:47:51.721884  7913 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260808 01:47:51.786742  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:47:51.800056  7923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:47:51.801707  7924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:47:51.807735  7831 server_base.cc:1061] running on GCE node
W20260808 01:47:51.807708  7926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:47:51.809191  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:47:51.809528  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:47:51.809762  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786153671809728 us; error 0 us; skew 500 ppm
I20260808 01:47:51.810708  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:47:51.816046  7831 webserver.cc:533] Webserver started at http://127.7.165.193:39685/ using document root <none> and password file <none>
I20260808 01:47:51.817142  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:47:51.817461  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:47:51.817836  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:47:51.819769  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/ts-0-root/instance:
uuid: "b822036e833c464b81cf37b24ec55c88"
format_stamp: "Formatted at 2026-08-08 01:47:51 on dist-test-slave-8tdl"
I20260808 01:47:51.828053  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.002s
I20260808 01:47:51.834126  7932 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:47:51.835253  7831 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.001s
I20260808 01:47:51.835681  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/ts-0-root
uuid: "b822036e833c464b81cf37b24ec55c88"
format_stamp: "Formatted at 2026-08-08 01:47:51 on dist-test-slave-8tdl"
I20260808 01:47:51.836189  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:47:51.875640  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:47:51.877221  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:47:51.880134  7831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:47:51.885195  7831 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:47:51.885576  7831 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:47:51.885880  7831 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:47:51.886096  7831 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:47:52.022013  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.193:40863
I20260808 01:47:52.022260  7996 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.193:40863 every 8 connection(s)
I20260808 01:47:52.031102  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:47:52.047338  8001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:47:52.060149  8002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:47:52.062726  8004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:47:52.067149  7831 server_base.cc:1061] running on GCE node
I20260808 01:47:52.068624  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:47:52.069316  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:47:52.069578  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786153672069554 us; error 0 us; skew 500 ppm
I20260808 01:47:52.070400  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:47:52.075387  7997 heartbeater.cc:344] Connected to a master server at 127.7.165.254:36421
I20260808 01:47:52.076354  7997 heartbeater.cc:461] Registering TS with master...
I20260808 01:47:52.078328  7997 heartbeater.cc:507] Master 127.7.165.254:36421 requested a full tablet report, sending...
I20260808 01:47:52.079279  7831 webserver.cc:533] Webserver started at http://127.7.165.194:43095/ using document root <none> and password file <none>
I20260808 01:47:52.080130  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:47:52.080440  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:47:52.080812  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:47:52.082543  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/ts-1-root/instance:
uuid: "282686b2e6f746d3b2eb16cf77647c1b"
format_stamp: "Formatted at 2026-08-08 01:47:52 on dist-test-slave-8tdl"
I20260808 01:47:52.083868  7863 ts_manager.cc:194] Registered new tserver with Master: b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863)
I20260808 01:47:52.087556  7863 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43414
I20260808 01:47:52.091377  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.007s	sys 0.000s
I20260808 01:47:52.097903  8010 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:47:52.099246  7831 fs_manager.cc:730] Time spent opening block manager: real 0.006s	user 0.005s	sys 0.001s
I20260808 01:47:52.099685  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/ts-1-root
uuid: "282686b2e6f746d3b2eb16cf77647c1b"
format_stamp: "Formatted at 2026-08-08 01:47:52 on dist-test-slave-8tdl"
I20260808 01:47:52.100203  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:47:52.118167  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:47:52.119848  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:47:52.123096  7831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:47:52.126454  7831 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:47:52.126736  7831 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:47:52.127022  7831 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:47:52.127195  7831 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:47:52.240718  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.194:34507
I20260808 01:47:52.241004  8073 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.194:34507 every 8 connection(s)
I20260808 01:47:52.253862  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:47:52.266649  8077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:47:52.269443  8074 heartbeater.cc:344] Connected to a master server at 127.7.165.254:36421
I20260808 01:47:52.270015  8074 heartbeater.cc:461] Registering TS with master...
I20260808 01:47:52.271402  8074 heartbeater.cc:507] Master 127.7.165.254:36421 requested a full tablet report, sending...
I20260808 01:47:52.275346  7863 ts_manager.cc:194] Registered new tserver with Master: 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507)
W20260808 01:47:52.278009  8078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:47:52.279469  7863 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43424
I20260808 01:47:52.282308  7831 server_base.cc:1061] running on GCE node
W20260808 01:47:52.282401  8080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:47:52.283641  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:47:52.283916  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:47:52.284128  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786153672284111 us; error 0 us; skew 500 ppm
I20260808 01:47:52.284698  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:47:52.288252  7831 webserver.cc:533] Webserver started at http://127.7.165.195:40803/ using document root <none> and password file <none>
I20260808 01:47:52.288864  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:47:52.289134  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:47:52.289464  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:47:52.290632  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/ts-2-root/instance:
uuid: "4a640ba2733143fb9807a7675ed337e3"
format_stamp: "Formatted at 2026-08-08 01:47:52 on dist-test-slave-8tdl"
I20260808 01:47:52.296401  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.003s
I20260808 01:47:52.300973  8087 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:47:52.302043  7831 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260808 01:47:52.302801  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/ts-2-root
uuid: "4a640ba2733143fb9807a7675ed337e3"
format_stamp: "Formatted at 2026-08-08 01:47:52 on dist-test-slave-8tdl"
I20260808 01:47:52.303444  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786153669593643-7831-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:47:52.346349  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:47:52.347913  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:47:52.349627  7831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:47:52.353070  7831 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:47:52.353334  7831 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:47:52.353605  7831 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:47:52.353798  7831 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:47:52.478920  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.195:36653
I20260808 01:47:52.479708  8154 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.195:36653 every 8 connection(s)
I20260808 01:47:52.499717  8155 heartbeater.cc:344] Connected to a master server at 127.7.165.254:36421
I20260808 01:47:52.500162  8155 heartbeater.cc:461] Registering TS with master...
I20260808 01:47:52.501358  8155 heartbeater.cc:507] Master 127.7.165.254:36421 requested a full tablet report, sending...
I20260808 01:47:52.504004  7863 ts_manager.cc:194] Registered new tserver with Master: 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:52.504566  7831 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016900265s
I20260808 01:47:52.505685  7863 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43426
I20260808 01:47:52.571559  7863 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43440:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260808 01:47:52.576643  7863 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260808 01:47:52.637639  7914 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:47:52.733424  8036 tablet_service.cc:1511] Processing CreateTablet for tablet 6e6647c3e5e146d185e0230bd07689e4 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260808 01:47:52.731328  8039 tablet_service.cc:1511] Processing CreateTablet for tablet 2d3db31373fa48e7a8e8a5a6e0851624 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260808 01:47:52.734462  8037 tablet_service.cc:1511] Processing CreateTablet for tablet 31d230e1f50b4c888c85615b35087706 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260808 01:47:52.733623  8035 tablet_service.cc:1511] Processing CreateTablet for tablet 8064f9a1744648c4963af166d4b67cfa (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260808 01:47:52.742846  8036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e6647c3e5e146d185e0230bd07689e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.742345  8034 tablet_service.cc:1511] Processing CreateTablet for tablet f441f7e95cba45ed9cb394a38cb4b4a2 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260808 01:47:52.749516  8033 tablet_service.cc:1511] Processing CreateTablet for tablet f98266402a6c4c82ae25f5fa9faaff51 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260808 01:47:52.759261  8118 tablet_service.cc:1511] Processing CreateTablet for tablet 2d3db31373fa48e7a8e8a5a6e0851624 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260808 01:47:52.765062  8035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8064f9a1744648c4963af166d4b67cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.759440  8116 tablet_service.cc:1511] Processing CreateTablet for tablet 6e6647c3e5e146d185e0230bd07689e4 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260808 01:47:52.769747  8115 tablet_service.cc:1511] Processing CreateTablet for tablet 8064f9a1744648c4963af166d4b67cfa (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260808 01:47:52.767371  8118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d3db31373fa48e7a8e8a5a6e0851624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.773846  7962 tablet_service.cc:1511] Processing CreateTablet for tablet 2d3db31373fa48e7a8e8a5a6e0851624 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260808 01:47:52.764099  8117 tablet_service.cc:1511] Processing CreateTablet for tablet 31d230e1f50b4c888c85615b35087706 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260808 01:47:52.784540  8114 tablet_service.cc:1511] Processing CreateTablet for tablet f441f7e95cba45ed9cb394a38cb4b4a2 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260808 01:47:52.785606  7961 tablet_service.cc:1511] Processing CreateTablet for tablet 31d230e1f50b4c888c85615b35087706 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260808 01:47:52.788705  8113 tablet_service.cc:1511] Processing CreateTablet for tablet f98266402a6c4c82ae25f5fa9faaff51 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260808 01:47:52.816996  7960 tablet_service.cc:1511] Processing CreateTablet for tablet 6e6647c3e5e146d185e0230bd07689e4 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260808 01:47:52.818333  8115 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8064f9a1744648c4963af166d4b67cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.823462  7962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d3db31373fa48e7a8e8a5a6e0851624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.821147  7959 tablet_service.cc:1511] Processing CreateTablet for tablet 8064f9a1744648c4963af166d4b67cfa (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260808 01:47:52.827392  8032 tablet_service.cc:1511] Processing CreateTablet for tablet cc6dc97c4b8e482683facaeb4e261128 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260808 01:47:52.831529  8037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31d230e1f50b4c888c85615b35087706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.832619  7958 tablet_service.cc:1511] Processing CreateTablet for tablet f441f7e95cba45ed9cb394a38cb4b4a2 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260808 01:47:52.830477  8112 tablet_service.cc:1511] Processing CreateTablet for tablet cc6dc97c4b8e482683facaeb4e261128 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260808 01:47:52.837930  7957 tablet_service.cc:1511] Processing CreateTablet for tablet f98266402a6c4c82ae25f5fa9faaff51 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260808 01:47:52.845124  7961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31d230e1f50b4c888c85615b35087706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.846745  7955 tablet_service.cc:1511] Processing CreateTablet for tablet 68c2bac20b264a42adee938c783e6102 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260808 01:47:52.848928  7960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e6647c3e5e146d185e0230bd07689e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.851876  8111 tablet_service.cc:1511] Processing CreateTablet for tablet 68c2bac20b264a42adee938c783e6102 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260808 01:47:52.856381  7955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68c2bac20b264a42adee938c783e6102. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.850761  8039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d3db31373fa48e7a8e8a5a6e0851624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.844094  8114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f441f7e95cba45ed9cb394a38cb4b4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.861263  7958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f441f7e95cba45ed9cb394a38cb4b4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.864144  8032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc6dc97c4b8e482683facaeb4e261128. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.866958  7957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f98266402a6c4c82ae25f5fa9faaff51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.866135  8113 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f98266402a6c4c82ae25f5fa9faaff51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.876030  8033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f98266402a6c4c82ae25f5fa9faaff51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.875900  8031 tablet_service.cc:1511] Processing CreateTablet for tablet 68c2bac20b264a42adee938c783e6102 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260808 01:47:52.877916  8031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68c2bac20b264a42adee938c783e6102. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.869614  8030 tablet_service.cc:1511] Processing CreateTablet for tablet 82541e7deca64da19bc6dd1e54b4a3c8 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260808 01:47:52.859162  7956 tablet_service.cc:1511] Processing CreateTablet for tablet cc6dc97c4b8e482683facaeb4e261128 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260808 01:47:52.885566  7959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8064f9a1744648c4963af166d4b67cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.894626  8110 tablet_service.cc:1511] Processing CreateTablet for tablet 82541e7deca64da19bc6dd1e54b4a3c8 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260808 01:47:52.903146  7954 tablet_service.cc:1511] Processing CreateTablet for tablet 82541e7deca64da19bc6dd1e54b4a3c8 (DEFAULT_TABLE table=test-workload [id=31b02a0fa9cb40559ba2da73a660a276]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260808 01:47:52.902956  8112 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc6dc97c4b8e482683facaeb4e261128. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.902091  8030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82541e7deca64da19bc6dd1e54b4a3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.913152  8117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31d230e1f50b4c888c85615b35087706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.909452  7956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc6dc97c4b8e482683facaeb4e261128. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.921797  8034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f441f7e95cba45ed9cb394a38cb4b4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.928766  8111 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68c2bac20b264a42adee938c783e6102. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.948115  7954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82541e7deca64da19bc6dd1e54b4a3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:52.949354  8110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82541e7deca64da19bc6dd1e54b4a3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:53.024315  8116 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e6647c3e5e146d185e0230bd07689e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:47:53.092445  7997 heartbeater.cc:499] Master 127.7.165.254:36421 was elected leader, sending a full tablet report...
I20260808 01:47:53.141129  8173 tablet_bootstrap.cc:492] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b: Bootstrap starting.
I20260808 01:47:53.211026  8174 tablet_bootstrap.cc:492] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88: Bootstrap starting.
I20260808 01:47:53.234186  8175 tablet_bootstrap.cc:492] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3: Bootstrap starting.
I20260808 01:47:53.245393  8173 tablet_bootstrap.cc:654] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:53.275450  8174 tablet_bootstrap.cc:654] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:53.283404  8074 heartbeater.cc:499] Master 127.7.165.254:36421 was elected leader, sending a full tablet report...
I20260808 01:47:53.300985  8173 tablet_bootstrap.cc:492] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b: No bootstrap required, opened a new log
I20260808 01:47:53.301556  8173 ts_tablet_manager.cc:1403] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b: Time spent bootstrapping tablet: real 0.185s	user 0.013s	sys 0.027s
I20260808 01:47:53.309926  8173 raft_consensus.cc:359] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.310889  8173 raft_consensus.cc:385] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:53.311275  8173 raft_consensus.cc:740] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Initialized, Role: FOLLOWER
I20260808 01:47:53.312137  8173 consensus_queue.cc:260] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.320567  8175 tablet_bootstrap.cc:654] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:53.327527  8173 ts_tablet_manager.cc:1434] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b: Time spent starting tablet: real 0.026s	user 0.009s	sys 0.010s
I20260808 01:47:53.332291  8174 tablet_bootstrap.cc:492] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88: No bootstrap required, opened a new log
I20260808 01:47:53.332859  8174 ts_tablet_manager.cc:1403] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88: Time spent bootstrapping tablet: real 0.126s	user 0.013s	sys 0.016s
I20260808 01:47:53.329751  8173 tablet_bootstrap.cc:492] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b: Bootstrap starting.
I20260808 01:47:53.344715  8174 raft_consensus.cc:359] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.347472  8174 raft_consensus.cc:385] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:53.347795  8174 raft_consensus.cc:740] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Initialized, Role: FOLLOWER
I20260808 01:47:53.348719  8174 consensus_queue.cc:260] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.349972  8175 tablet_bootstrap.cc:492] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3: No bootstrap required, opened a new log
I20260808 01:47:53.350636  8173 tablet_bootstrap.cc:654] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:53.350739  8175 ts_tablet_manager.cc:1403] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3: Time spent bootstrapping tablet: real 0.118s	user 0.013s	sys 0.008s
I20260808 01:47:53.356256  8175 raft_consensus.cc:359] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.357631  8175 raft_consensus.cc:385] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:53.358213  8175 raft_consensus.cc:740] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Initialized, Role: FOLLOWER
I20260808 01:47:53.359375  8175 consensus_queue.cc:260] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.365396  8174 ts_tablet_manager.cc:1434] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88: Time spent starting tablet: real 0.032s	user 0.014s	sys 0.009s
I20260808 01:47:53.367321  8174 tablet_bootstrap.cc:492] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88: Bootstrap starting.
I20260808 01:47:53.378095  8175 ts_tablet_manager.cc:1434] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3: Time spent starting tablet: real 0.027s	user 0.011s	sys 0.014s
I20260808 01:47:53.379177  8155 heartbeater.cc:499] Master 127.7.165.254:36421 was elected leader, sending a full tablet report...
I20260808 01:47:53.379380  8173 tablet_bootstrap.cc:492] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b: No bootstrap required, opened a new log
I20260808 01:47:53.380149  8173 ts_tablet_manager.cc:1403] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b: Time spent bootstrapping tablet: real 0.051s	user 0.004s	sys 0.021s
I20260808 01:47:53.383431  8175 tablet_bootstrap.cc:492] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3: Bootstrap starting.
I20260808 01:47:53.390959  8173 raft_consensus.cc:359] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.391703  8174 tablet_bootstrap.cc:654] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:53.391870  8173 raft_consensus.cc:385] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:53.392320  8173 raft_consensus.cc:740] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Initialized, Role: FOLLOWER
I20260808 01:47:53.397289  8175 tablet_bootstrap.cc:654] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:53.402308  8173 consensus_queue.cc:260] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.405453  8173 ts_tablet_manager.cc:1434] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b: Time spent starting tablet: real 0.025s	user 0.009s	sys 0.002s
I20260808 01:47:53.406812  8173 tablet_bootstrap.cc:492] T 82541e7deca64da19bc6dd1e54b4a3c8 P 282686b2e6f746d3b2eb16cf77647c1b: Bootstrap starting.
I20260808 01:47:53.409713  8174 tablet_bootstrap.cc:492] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88: No bootstrap required, opened a new log
I20260808 01:47:53.410344  8174 ts_tablet_manager.cc:1403] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88: Time spent bootstrapping tablet: real 0.043s	user 0.017s	sys 0.010s
I20260808 01:47:53.414590  8175 tablet_bootstrap.cc:492] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3: No bootstrap required, opened a new log
I20260808 01:47:53.415393  8175 ts_tablet_manager.cc:1403] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3: Time spent bootstrapping tablet: real 0.032s	user 0.007s	sys 0.016s
I20260808 01:47:53.417855  8174 raft_consensus.cc:359] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.418774  8175 raft_consensus.cc:359] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.419425  8173 tablet_bootstrap.cc:654] T 82541e7deca64da19bc6dd1e54b4a3c8 P 282686b2e6f746d3b2eb16cf77647c1b: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:53.420032  8175 raft_consensus.cc:385] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:53.418802  8174 raft_consensus.cc:385] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:53.420665  8175 raft_consensus.cc:740] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Initialized, Role: FOLLOWER
I20260808 01:47:53.420759  8174 raft_consensus.cc:740] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Initialized, Role: FOLLOWER
I20260808 01:47:53.421669  8175 consensus_queue.cc:260] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.421921  8174 consensus_queue.cc:260] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.424361  8175 ts_tablet_manager.cc:1434] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260808 01:47:53.425448  8175 tablet_bootstrap.cc:492] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3: Bootstrap starting.
I20260808 01:47:53.426095  8174 ts_tablet_manager.cc:1434] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88: Time spent starting tablet: real 0.015s	user 0.009s	sys 0.000s
I20260808 01:47:53.429883  8174 tablet_bootstrap.cc:492] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88: Bootstrap starting.
I20260808 01:47:53.432516  8175 tablet_bootstrap.cc:654] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:53.438892  8174 tablet_bootstrap.cc:654] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:53.449862  8173 tablet_bootstrap.cc:492] T 82541e7deca64da19bc6dd1e54b4a3c8 P 282686b2e6f746d3b2eb16cf77647c1b: No bootstrap required, opened a new log
I20260808 01:47:53.450425  8173 ts_tablet_manager.cc:1403] T 82541e7deca64da19bc6dd1e54b4a3c8 P 282686b2e6f746d3b2eb16cf77647c1b: Time spent bootstrapping tablet: real 0.044s	user 0.012s	sys 0.002s
I20260808 01:47:53.452869  8173 raft_consensus.cc:359] T 82541e7deca64da19bc6dd1e54b4a3c8 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.455937  8173 raft_consensus.cc:385] T 82541e7deca64da19bc6dd1e54b4a3c8 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:53.456629  8173 raft_consensus.cc:740] T 82541e7deca64da19bc6dd1e54b4a3c8 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Initialized, Role: FOLLOWER
I20260808 01:47:53.457480  8173 consensus_queue.cc:260] T 82541e7deca64da19bc6dd1e54b4a3c8 P 282686b2e6f746d3b2eb16cf77647c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.458765  8175 tablet_bootstrap.cc:492] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3: No bootstrap required, opened a new log
I20260808 01:47:53.459271  8175 ts_tablet_manager.cc:1403] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3: Time spent bootstrapping tablet: real 0.034s	user 0.016s	sys 0.003s
I20260808 01:47:53.459990  8173 ts_tablet_manager.cc:1434] T 82541e7deca64da19bc6dd1e54b4a3c8 P 282686b2e6f746d3b2eb16cf77647c1b: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.003s
I20260808 01:47:53.461225  8173 tablet_bootstrap.cc:492] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b: Bootstrap starting.
I20260808 01:47:53.462394  8175 raft_consensus.cc:359] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.463370  8175 raft_consensus.cc:385] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:53.463716  8175 raft_consensus.cc:740] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Initialized, Role: FOLLOWER
I20260808 01:47:53.464663  8175 consensus_queue.cc:260] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.467259  8175 ts_tablet_manager.cc:1434] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260808 01:47:53.468283  8175 tablet_bootstrap.cc:492] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3: Bootstrap starting.
I20260808 01:47:53.468677  8174 tablet_bootstrap.cc:492] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88: No bootstrap required, opened a new log
I20260808 01:47:53.469228  8174 ts_tablet_manager.cc:1403] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88: Time spent bootstrapping tablet: real 0.040s	user 0.012s	sys 0.013s
I20260808 01:47:53.479175  8175 tablet_bootstrap.cc:654] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:53.486176  8173 tablet_bootstrap.cc:654] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:53.488096  8175 tablet_bootstrap.cc:492] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3: No bootstrap required, opened a new log
I20260808 01:47:53.488727  8175 ts_tablet_manager.cc:1403] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3: Time spent bootstrapping tablet: real 0.021s	user 0.010s	sys 0.006s
I20260808 01:47:53.482597  8174 raft_consensus.cc:359] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.492430  8175 raft_consensus.cc:359] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.493173  8175 raft_consensus.cc:385] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260808 01:47:53.493418  8156 tablet.cc:2406] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:47:53.493567  8175 raft_consensus.cc:740] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Initialized, Role: FOLLOWER
I20260808 01:47:53.494402  8175 consensus_queue.cc:260] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.497208  8175 ts_tablet_manager.cc:1434] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260808 01:47:53.498294  8175 tablet_bootstrap.cc:492] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3: Bootstrap starting.
I20260808 01:47:53.499636  8173 tablet_bootstrap.cc:492] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b: No bootstrap required, opened a new log
I20260808 01:47:53.500190  8173 ts_tablet_manager.cc:1403] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b: Time spent bootstrapping tablet: real 0.039s	user 0.004s	sys 0.015s
I20260808 01:47:53.492916  8174 raft_consensus.cc:385] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:53.502225  8174 raft_consensus.cc:740] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Initialized, Role: FOLLOWER
I20260808 01:47:53.503484  8174 consensus_queue.cc:260] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.507459  8175 tablet_bootstrap.cc:654] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:53.509089  8174 ts_tablet_manager.cc:1434] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88: Time spent starting tablet: real 0.039s	user 0.010s	sys 0.010s
I20260808 01:47:53.504316  8173 raft_consensus.cc:359] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.510017  8173 raft_consensus.cc:385] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:53.510510  8174 tablet_bootstrap.cc:492] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88: Bootstrap starting.
I20260808 01:47:53.510421  8173 raft_consensus.cc:740] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Initialized, Role: FOLLOWER
I20260808 01:47:53.515127  8178 raft_consensus.cc:493] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:53.516393  8178 raft_consensus.cc:515] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.520025  8173 consensus_queue.cc:260] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.521486  8178 leader_election.cc:290] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:53.556879  8174 tablet_bootstrap.cc:654] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:53.559080  8173 ts_tablet_manager.cc:1434] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b: Time spent starting tablet: real 0.059s	user 0.004s	sys 0.010s
I20260808 01:47:53.560927  8173 tablet_bootstrap.cc:492] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b: Bootstrap starting.
I20260808 01:47:53.575497  8180 raft_consensus.cc:493] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:53.577683  8180 raft_consensus.cc:515] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.585197  8173 tablet_bootstrap.cc:654] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:53.584162  8180 leader_election.cc:290] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:53.593189  8175 tablet_bootstrap.cc:492] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3: No bootstrap required, opened a new log
I20260808 01:47:53.599867  8175 ts_tablet_manager.cc:1403] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3: Time spent bootstrapping tablet: real 0.102s	user 0.004s	sys 0.027s
I20260808 01:47:53.616297  8174 tablet_bootstrap.cc:492] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88: No bootstrap required, opened a new log
I20260808 01:47:53.625702  8174 ts_tablet_manager.cc:1403] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88: Time spent bootstrapping tablet: real 0.115s	user 0.017s	sys 0.003s
I20260808 01:47:53.628674  8175 raft_consensus.cc:359] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.629526  8175 raft_consensus.cc:385] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:53.629835  8175 raft_consensus.cc:740] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Initialized, Role: FOLLOWER
I20260808 01:47:53.633487  8183 raft_consensus.cc:493] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:53.630649  8175 consensus_queue.cc:260] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.636847  8183 raft_consensus.cc:515] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
W20260808 01:47:53.642320  7914 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:47:53.648234  8173 tablet_bootstrap.cc:492] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b: No bootstrap required, opened a new log
I20260808 01:47:53.648947  8174 raft_consensus.cc:359] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.649718  8183 leader_election.cc:290] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507)
I20260808 01:47:53.648963  8173 ts_tablet_manager.cc:1403] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b: Time spent bootstrapping tablet: real 0.089s	user 0.013s	sys 0.014s
I20260808 01:47:53.657357  8174 raft_consensus.cc:385] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:53.661453  8174 raft_consensus.cc:740] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Initialized, Role: FOLLOWER
I20260808 01:47:53.661921  8129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a640ba2733143fb9807a7675ed337e3" is_pre_election: true
I20260808 01:47:53.665396  8174 consensus_queue.cc:260] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.678988  8180 raft_consensus.cc:493] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:53.679960  7972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b822036e833c464b81cf37b24ec55c88" is_pre_election: true
I20260808 01:47:53.680779  8180 raft_consensus.cc:515] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.680898  8173 raft_consensus.cc:359] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.684504  8180 leader_election.cc:290] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
W20260808 01:47:53.686563  8014 leader_election.cc:343] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:47:53.687129  8180 raft_consensus.cc:493] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:53.687852  8180 raft_consensus.cc:515] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.684666  8173 raft_consensus.cc:385] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:53.691148  8173 raft_consensus.cc:740] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Initialized, Role: FOLLOWER
I20260808 01:47:53.692344  8173 consensus_queue.cc:260] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.681120  8174 ts_tablet_manager.cc:1434] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88: Time spent starting tablet: real 0.054s	user 0.010s	sys 0.002s
I20260808 01:47:53.696779  8174 tablet_bootstrap.cc:492] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88: Bootstrap starting.
I20260808 01:47:53.680658  8129 raft_consensus.cc:2468] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 0.
I20260808 01:47:53.707072  8173 ts_tablet_manager.cc:1434] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b: Time spent starting tablet: real 0.046s	user 0.002s	sys 0.009s
I20260808 01:47:53.708477  8173 tablet_bootstrap.cc:492] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b: Bootstrap starting.
I20260808 01:47:53.710295  8180 leader_election.cc:290] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:53.756069  8173 tablet_bootstrap.cc:654] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:53.756732  8183 raft_consensus.cc:493] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:53.760912  8183 raft_consensus.cc:515] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.765367  8175 ts_tablet_manager.cc:1434] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3: Time spent starting tablet: real 0.165s	user 0.016s	sys 0.035s
I20260808 01:47:53.766522  8013 leader_election.cc:304] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, 4a640ba2733143fb9807a7675ed337e3; no voters: b822036e833c464b81cf37b24ec55c88
I20260808 01:47:53.767503  8175 tablet_bootstrap.cc:492] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3: Bootstrap starting.
I20260808 01:47:53.768373  8193 raft_consensus.cc:2804] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:47:53.771288  8193 raft_consensus.cc:493] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:47:53.771792  8193 raft_consensus.cc:3060] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:53.775125  8174 tablet_bootstrap.cc:654] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:53.783110  8183 leader_election.cc:290] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507)
I20260808 01:47:53.790084  8206 raft_consensus.cc:493] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:53.792909  8206 raft_consensus.cc:515] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.805317  8129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d3db31373fa48e7a8e8a5a6e0851624" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a640ba2733143fb9807a7675ed337e3" is_pre_election: true
I20260808 01:47:53.804787  8049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d3db31373fa48e7a8e8a5a6e0851624" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b" is_pre_election: true
I20260808 01:47:53.805882  8048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc6dc97c4b8e482683facaeb4e261128" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b" is_pre_election: true
I20260808 01:47:53.810721  8129 raft_consensus.cc:2468] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 0.
I20260808 01:47:53.809013  8047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82541e7deca64da19bc6dd1e54b4a3c8" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b" is_pre_election: true
I20260808 01:47:53.817626  8206 leader_election.cc:290] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507)
I20260808 01:47:53.818768  8047 raft_consensus.cc:2468] T 82541e7deca64da19bc6dd1e54b4a3c8 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 0.
I20260808 01:47:53.824640  7934 leader_election.cc:304] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:47:53.814857  8048 raft_consensus.cc:2468] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 0.
I20260808 01:47:53.829676  8178 raft_consensus.cc:493] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:53.830387  8178 raft_consensus.cc:515] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.830821  8209 raft_consensus.cc:2804] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260808 01:47:53.837352  7934 leader_election.cc:343] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:47:53.820832  8193 raft_consensus.cc:515] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.843243  8209 raft_consensus.cc:493] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:47:53.846318  8048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b" is_pre_election: true
I20260808 01:47:53.848304  8129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82541e7deca64da19bc6dd1e54b4a3c8" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a640ba2733143fb9807a7675ed337e3" is_pre_election: true
I20260808 01:47:53.845949  8128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc6dc97c4b8e482683facaeb4e261128" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a640ba2733143fb9807a7675ed337e3" is_pre_election: true
I20260808 01:47:53.851411  8209 raft_consensus.cc:3060] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:53.849138  7934 leader_election.cc:304] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:47:53.859826  8178 leader_election.cc:290] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:53.855895  8047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f441f7e95cba45ed9cb394a38cb4b4a2" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b" is_pre_election: true
I20260808 01:47:53.856616  8049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8064f9a1744648c4963af166d4b67cfa" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b" is_pre_election: true
I20260808 01:47:53.860275  7972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68c2bac20b264a42adee938c783e6102" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b822036e833c464b81cf37b24ec55c88" is_pre_election: true
I20260808 01:47:53.860631  8048 raft_consensus.cc:2393] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a640ba2733143fb9807a7675ed337e3 in current term 1: Already voted for candidate 282686b2e6f746d3b2eb16cf77647c1b in this term.
I20260808 01:47:53.859398  8128 raft_consensus.cc:2468] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 0.
I20260808 01:47:53.865221  8180 raft_consensus.cc:2804] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:47:53.867491  7935 leader_election.cc:304] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3, b822036e833c464b81cf37b24ec55c88; no voters: 282686b2e6f746d3b2eb16cf77647c1b
W20260808 01:47:53.869961  8014 leader_election.cc:343] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:47:53.869963  8180 raft_consensus.cc:493] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:47:53.871797  8180 raft_consensus.cc:3060] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:47:53.871831  8089 leader_election.cc:343] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:47:53.883325  8173 tablet_bootstrap.cc:492] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b: No bootstrap required, opened a new log
W20260808 01:47:53.878754  8089 leader_election.cc:343] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:47:53.883687  7972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b822036e833c464b81cf37b24ec55c88"
I20260808 01:47:53.887715  8128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68c2bac20b264a42adee938c783e6102" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a640ba2733143fb9807a7675ed337e3" is_pre_election: true
W20260808 01:47:53.888319  8014 leader_election.cc:343] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:47:53.886982  8173 ts_tablet_manager.cc:1403] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b: Time spent bootstrapping tablet: real 0.179s	user 0.013s	sys 0.014s
W20260808 01:47:53.891008  8013 leader_election.cc:343] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:47:53.891971  8013 leader_election.cc:304] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b; no voters: 4a640ba2733143fb9807a7675ed337e3, b822036e833c464b81cf37b24ec55c88
I20260808 01:47:53.892514  8175 tablet_bootstrap.cc:654] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:53.894163  8178 raft_consensus.cc:2749] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:47:53.891573  7935 leader_election.cc:343] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:47:53.892998  8193 leader_election.cc:290] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 1 election: Requested vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:53.900447  8211 raft_consensus.cc:2804] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:47:53.896898  8128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
I20260808 01:47:53.903112  8128 raft_consensus.cc:3060] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:53.907341  8180 raft_consensus.cc:515] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.903991  8211 raft_consensus.cc:493] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:47:53.910327  8211 raft_consensus.cc:3060] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:53.915093  7972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b822036e833c464b81cf37b24ec55c88" is_pre_election: true
I20260808 01:47:53.917192  8174 tablet_bootstrap.cc:492] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88: No bootstrap required, opened a new log
I20260808 01:47:53.917919  7972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8064f9a1744648c4963af166d4b67cfa" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b822036e833c464b81cf37b24ec55c88" is_pre_election: true
I20260808 01:47:53.918263  8174 ts_tablet_manager.cc:1403] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88: Time spent bootstrapping tablet: real 0.222s	user 0.018s	sys 0.032s
W20260808 01:47:53.925343  8092 leader_election.cc:343] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:47:53.925467  7972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f441f7e95cba45ed9cb394a38cb4b4a2" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b822036e833c464b81cf37b24ec55c88" is_pre_election: true
I20260808 01:47:53.926421  8049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc6dc97c4b8e482683facaeb4e261128" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
I20260808 01:47:53.927071  8049 raft_consensus.cc:3060] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:53.926188  7972 raft_consensus.cc:2468] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 0.
W20260808 01:47:53.926198  8092 leader_election.cc:343] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:47:53.928519  8092 leader_election.cc:304] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3; no voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88
I20260808 01:47:53.930773  8180 leader_election.cc:290] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 election: Requested vote from peers 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:53.937947  8206 raft_consensus.cc:2749] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:47:53.934311  8175 tablet_bootstrap.cc:492] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3: No bootstrap required, opened a new log
I20260808 01:47:53.939091  8175 ts_tablet_manager.cc:1403] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3: Time spent bootstrapping tablet: real 0.172s	user 0.024s	sys 0.020s
I20260808 01:47:53.937948  8173 raft_consensus.cc:359] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.939878  8173 raft_consensus.cc:385] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:53.937261  8128 raft_consensus.cc:2468] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 1.
I20260808 01:47:53.940207  8173 raft_consensus.cc:740] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Initialized, Role: FOLLOWER
I20260808 01:47:53.938203  8174 raft_consensus.cc:359] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.942849  8211 raft_consensus.cc:515] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.943084  8049 raft_consensus.cc:2468] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 1.
I20260808 01:47:53.945647  8174 raft_consensus.cc:385] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:53.946414  8129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc6dc97c4b8e482683facaeb4e261128" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
I20260808 01:47:53.945019  8209 raft_consensus.cc:515] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.951001  8129 raft_consensus.cc:3060] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:53.949872  8174 raft_consensus.cc:740] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Initialized, Role: FOLLOWER
I20260808 01:47:53.945894  8092 leader_election.cc:304] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3, b822036e833c464b81cf37b24ec55c88; no voters: 282686b2e6f746d3b2eb16cf77647c1b
I20260808 01:47:53.947069  8173 consensus_queue.cc:260] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.954391  8206 raft_consensus.cc:2804] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:47:53.955274  8178 raft_consensus.cc:493] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:53.955660  8206 raft_consensus.cc:493] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:47:53.956074  8206 raft_consensus.cc:3060] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:53.955796  8178 raft_consensus.cc:515] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.956622  8173 ts_tablet_manager.cc:1434] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b: Time spent starting tablet: real 0.066s	user 0.011s	sys 0.000s
I20260808 01:47:53.953972  7934 leader_election.cc:304] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:47:53.959129  8174 consensus_queue.cc:260] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.960815  8173 tablet_bootstrap.cc:492] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b: Bootstrap starting.
I20260808 01:47:53.962024  8180 raft_consensus.cc:2804] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:47:53.963147  8013 leader_election.cc:304] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, 4a640ba2733143fb9807a7675ed337e3; no voters: b822036e833c464b81cf37b24ec55c88
I20260808 01:47:53.968094  8193 raft_consensus.cc:2804] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:47:53.970250  8049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d3db31373fa48e7a8e8a5a6e0851624" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
I20260808 01:47:53.970992  8178 leader_election.cc:290] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:53.972139  8128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f98266402a6c4c82ae25f5fa9faaff51" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a640ba2733143fb9807a7675ed337e3" is_pre_election: true
I20260808 01:47:53.972139  7972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f98266402a6c4c82ae25f5fa9faaff51" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b822036e833c464b81cf37b24ec55c88" is_pre_election: true
I20260808 01:47:53.975099  7972 raft_consensus.cc:2468] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 0.
W20260808 01:47:53.973269  7934 leader_election.cc:343] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:47:53.978788  8175 raft_consensus.cc:359] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.979110  8193 raft_consensus.cc:697] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 LEADER]: Becoming Leader. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Running, Role: LEADER
I20260808 01:47:53.979607  8175 raft_consensus.cc:385] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:53.979941  8175 raft_consensus.cc:740] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Initialized, Role: FOLLOWER
I20260808 01:47:53.980715  8175 consensus_queue.cc:260] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.980425  8193 consensus_queue.cc:237] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:53.984445  8175 ts_tablet_manager.cc:1434] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3: Time spent starting tablet: real 0.045s	user 0.005s	sys 0.005s
I20260808 01:47:53.985394  8175 tablet_bootstrap.cc:492] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3: Bootstrap starting.
I20260808 01:47:53.978138  8211 leader_election.cc:290] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 election: Requested vote from peers 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:53.987970  8129 raft_consensus.cc:2468] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 1.
I20260808 01:47:53.977510  8209 leader_election.cc:290] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 election: Requested vote from peers 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:53.977550  8180 raft_consensus.cc:697] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Becoming Leader. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Running, Role: LEADER
I20260808 01:47:53.987995  8049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82541e7deca64da19bc6dd1e54b4a3c8" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
W20260808 01:47:53.992403  8013 leader_election.cc:343] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:47:53.994901  8049 raft_consensus.cc:3060] T 82541e7deca64da19bc6dd1e54b4a3c8 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:53.995704  8128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82541e7deca64da19bc6dd1e54b4a3c8" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
I20260808 01:47:53.996555  8127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d3db31373fa48e7a8e8a5a6e0851624" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
I20260808 01:47:54.000782  8127 raft_consensus.cc:3060] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:53.997098  8180 consensus_queue.cc:237] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
W20260808 01:47:54.001686  7935 leader_election.cc:343] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:47:54.010216  8174 ts_tablet_manager.cc:1434] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88: Time spent starting tablet: real 0.090s	user 0.008s	sys 0.022s
I20260808 01:47:54.014930  8174 tablet_bootstrap.cc:492] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88: Bootstrap starting.
I20260808 01:47:54.015571  8206 raft_consensus.cc:515] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.024641  8206 leader_election.cc:290] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 election: Requested vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507)
I20260808 01:47:54.026124  7972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f441f7e95cba45ed9cb394a38cb4b4a2" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b822036e833c464b81cf37b24ec55c88"
I20260808 01:47:54.026790  7972 raft_consensus.cc:3060] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:54.029217  8047 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f441f7e95cba45ed9cb394a38cb4b4a2" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
I20260808 01:47:54.029032  8127 raft_consensus.cc:2468] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 1.
I20260808 01:47:54.032202  7935 leader_election.cc:304] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3, b822036e833c464b81cf37b24ec55c88; no voters: 282686b2e6f746d3b2eb16cf77647c1b
I20260808 01:47:54.033751  8214 raft_consensus.cc:2804] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:47:54.033469  8175 tablet_bootstrap.cc:654] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:54.034847  8214 raft_consensus.cc:697] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Becoming Leader. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Running, Role: LEADER
I20260808 01:47:54.036715  8214 consensus_queue.cc:237] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.041465  8049 raft_consensus.cc:2468] T 82541e7deca64da19bc6dd1e54b4a3c8 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 1.
I20260808 01:47:54.044516  7934 leader_election.cc:304] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88; no voters: 4a640ba2733143fb9807a7675ed337e3
I20260808 01:47:54.047883  8180 raft_consensus.cc:2804] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:47:54.050503  7972 raft_consensus.cc:2468] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 1.
I20260808 01:47:54.051970  8173 tablet_bootstrap.cc:654] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:54.052434  8180 raft_consensus.cc:697] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Becoming Leader. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Running, Role: LEADER
I20260808 01:47:54.052496  8092 leader_election.cc:304] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:47:54.055490  8206 raft_consensus.cc:2804] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:47:54.056035  8206 raft_consensus.cc:697] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 1 LEADER]: Becoming Leader. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Running, Role: LEADER
I20260808 01:47:54.056978  8206 consensus_queue.cc:237] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.058251  8180 consensus_queue.cc:237] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.065219  8174 tablet_bootstrap.cc:654] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:54.078712  8173 tablet_bootstrap.cc:492] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b: No bootstrap required, opened a new log
I20260808 01:47:54.080188  8173 ts_tablet_manager.cc:1403] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b: Time spent bootstrapping tablet: real 0.120s	user 0.019s	sys 0.011s
I20260808 01:47:54.092012  8175 tablet_bootstrap.cc:492] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3: No bootstrap required, opened a new log
I20260808 01:47:54.092583  8175 ts_tablet_manager.cc:1403] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3: Time spent bootstrapping tablet: real 0.107s	user 0.014s	sys 0.022s
I20260808 01:47:54.092519  8173 raft_consensus.cc:359] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.093266  8173 raft_consensus.cc:385] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:54.093547  8173 raft_consensus.cc:740] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Initialized, Role: FOLLOWER
I20260808 01:47:54.093731  8174 tablet_bootstrap.cc:492] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88: No bootstrap required, opened a new log
I20260808 01:47:54.094285  8174 ts_tablet_manager.cc:1403] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88: Time spent bootstrapping tablet: real 0.080s	user 0.021s	sys 0.003s
I20260808 01:47:54.095016  8173 consensus_queue.cc:260] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.096838  8173 ts_tablet_manager.cc:1434] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b: Time spent starting tablet: real 0.016s	user 0.003s	sys 0.003s
I20260808 01:47:54.096498  8175 raft_consensus.cc:359] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.097261  8175 raft_consensus.cc:385] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:54.097620  8175 raft_consensus.cc:740] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Initialized, Role: FOLLOWER
I20260808 01:47:54.097836  8173 tablet_bootstrap.cc:492] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b: Bootstrap starting.
I20260808 01:47:54.097900  8174 raft_consensus.cc:359] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.099052  8174 raft_consensus.cc:385] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:54.098439  8175 consensus_queue.cc:260] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.099712  8174 raft_consensus.cc:740] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Initialized, Role: FOLLOWER
I20260808 01:47:54.101855  8174 consensus_queue.cc:260] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.104275  8175 ts_tablet_manager.cc:1434] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.000s
I20260808 01:47:54.105492  8174 ts_tablet_manager.cc:1434] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.003s
I20260808 01:47:54.106981  8174 tablet_bootstrap.cc:492] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88: Bootstrap starting.
I20260808 01:47:54.109285  8014 leader_election.cc:304] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88; no voters: 4a640ba2733143fb9807a7675ed337e3
I20260808 01:47:54.110461  8173 tablet_bootstrap.cc:654] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b: Neither blocks nor log segments found. Creating new log.
W20260808 01:47:54.110646  8014 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:40863, user_credentials={real_user=slave}} blocked reactor thread for 118273us
I20260808 01:47:54.111716  8193 raft_consensus.cc:2804] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:47:54.116405  8193 raft_consensus.cc:493] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:47:54.116770  8174 tablet_bootstrap.cc:654] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:54.116852  8193 raft_consensus.cc:3060] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:54.115688  8089 leader_election.cc:304] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3; no voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88
I20260808 01:47:54.119411  8206 raft_consensus.cc:2749] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:47:54.123517  8089 leader_election.cc:343] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:47:54.137506  8175 tablet_bootstrap.cc:492] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3: Bootstrap starting.
I20260808 01:47:54.144693  8193 raft_consensus.cc:515] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.141179  7861 catalog_manager.cc:5697] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 reported cstate change: term changed from 0 to 1, leader changed from <none> to b822036e833c464b81cf37b24ec55c88 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "b822036e833c464b81cf37b24ec55c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20260808 01:47:54.149277  8127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f98266402a6c4c82ae25f5fa9faaff51" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
I20260808 01:47:54.150125  8127 raft_consensus.cc:3060] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:54.150753  7972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f98266402a6c4c82ae25f5fa9faaff51" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b822036e833c464b81cf37b24ec55c88"
I20260808 01:47:54.151506  7972 raft_consensus.cc:3060] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:54.161320  7972 raft_consensus.cc:2468] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 1.
I20260808 01:47:54.162874  8014 leader_election.cc:304] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:47:54.164109  8216 raft_consensus.cc:2804] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:47:54.164659  8216 raft_consensus.cc:697] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 LEADER]: Becoming Leader. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Running, Role: LEADER
I20260808 01:47:54.165614  8216 consensus_queue.cc:237] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.168838  8175 tablet_bootstrap.cc:654] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:54.147758  8193 leader_election.cc:290] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 1 election: Requested vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:54.164667  8127 raft_consensus.cc:2468] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 1.
I20260808 01:47:54.178478  8174 tablet_bootstrap.cc:492] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88: No bootstrap required, opened a new log
I20260808 01:47:54.179200  8174 ts_tablet_manager.cc:1403] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88: Time spent bootstrapping tablet: real 0.073s	user 0.024s	sys 0.032s
I20260808 01:47:54.185835  8174 raft_consensus.cc:359] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.186967  8173 tablet_bootstrap.cc:492] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b: No bootstrap required, opened a new log
I20260808 01:47:54.187516  8173 ts_tablet_manager.cc:1403] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b: Time spent bootstrapping tablet: real 0.090s	user 0.005s	sys 0.022s
I20260808 01:47:54.186844  8174 raft_consensus.cc:385] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:54.188661  8174 raft_consensus.cc:740] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Initialized, Role: FOLLOWER
I20260808 01:47:54.189993  8174 consensus_queue.cc:260] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.195015  8174 ts_tablet_manager.cc:1434] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88: Time spent starting tablet: real 0.015s	user 0.010s	sys 0.000s
I20260808 01:47:54.195614  8175 tablet_bootstrap.cc:492] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3: No bootstrap required, opened a new log
I20260808 01:47:54.196148  8174 tablet_bootstrap.cc:492] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88: Bootstrap starting.
I20260808 01:47:54.196354  8175 ts_tablet_manager.cc:1403] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3: Time spent bootstrapping tablet: real 0.059s	user 0.020s	sys 0.003s
I20260808 01:47:54.198134  8173 raft_consensus.cc:359] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.198872  8173 raft_consensus.cc:385] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:54.199155  8173 raft_consensus.cc:740] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Initialized, Role: FOLLOWER
I20260808 01:47:54.199508  8175 raft_consensus.cc:359] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.200269  8175 raft_consensus.cc:385] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:54.199856  8173 consensus_queue.cc:260] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.200675  8175 raft_consensus.cc:740] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Initialized, Role: FOLLOWER
I20260808 01:47:54.202486  8173 ts_tablet_manager.cc:1434] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260808 01:47:54.203447  8173 tablet_bootstrap.cc:492] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b: Bootstrap starting.
I20260808 01:47:54.203032  8175 consensus_queue.cc:260] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.204881  8174 tablet_bootstrap.cc:654] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:54.207666  8175 ts_tablet_manager.cc:1434] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260808 01:47:54.208684  8175 tablet_bootstrap.cc:492] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3: Bootstrap starting.
I20260808 01:47:54.210260  8173 tablet_bootstrap.cc:654] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:54.217136  7863 catalog_manager.cc:5697] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "4a640ba2733143fb9807a7675ed337e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: HEALTHY } } }
I20260808 01:47:54.220738  8175 tablet_bootstrap.cc:654] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:54.228363  8174 tablet_bootstrap.cc:492] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88: No bootstrap required, opened a new log
I20260808 01:47:54.229077  8174 ts_tablet_manager.cc:1403] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88: Time spent bootstrapping tablet: real 0.033s	user 0.005s	sys 0.017s
I20260808 01:47:54.234184  8174 raft_consensus.cc:359] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.235292  8174 raft_consensus.cc:385] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:54.235680  8174 raft_consensus.cc:740] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Initialized, Role: FOLLOWER
I20260808 01:47:54.236266  8206 raft_consensus.cc:493] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:54.236817  8174 consensus_queue.cc:260] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.236299  7860 catalog_manager.cc:5697] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194). New cstate: current_term: 1 leader_uuid: "282686b2e6f746d3b2eb16cf77647c1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20260808 01:47:54.240296  8174 ts_tablet_manager.cc:1434] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.001s
I20260808 01:47:54.235661  7861 catalog_manager.cc:5697] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 reported cstate change: term changed from 0 to 1, leader changed from <none> to b822036e833c464b81cf37b24ec55c88 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "b822036e833c464b81cf37b24ec55c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20260808 01:47:54.236956  8206 raft_consensus.cc:515] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.241775  8174 tablet_bootstrap.cc:492] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88: Bootstrap starting.
I20260808 01:47:54.242600  7861 catalog_manager.cc:5697] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 reported cstate change: term changed from 0 to 1, leader changed from <none> to b822036e833c464b81cf37b24ec55c88 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "b822036e833c464b81cf37b24ec55c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20260808 01:47:54.243626  8206 leader_election.cc:290] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507)
I20260808 01:47:54.245275  7972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d230e1f50b4c888c85615b35087706" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b822036e833c464b81cf37b24ec55c88" is_pre_election: true
I20260808 01:47:54.248852  7972 raft_consensus.cc:2468] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 0.
I20260808 01:47:54.245298  8048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d230e1f50b4c888c85615b35087706" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b" is_pre_election: true
I20260808 01:47:54.249897  8048 raft_consensus.cc:2468] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 0.
I20260808 01:47:54.250286  8175 tablet_bootstrap.cc:492] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3: No bootstrap required, opened a new log
I20260808 01:47:54.250741  8175 ts_tablet_manager.cc:1403] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3: Time spent bootstrapping tablet: real 0.042s	user 0.018s	sys 0.004s
I20260808 01:47:54.252395  8092 leader_election.cc:304] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:47:54.253237  8206 raft_consensus.cc:2804] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:47:54.253603  8206 raft_consensus.cc:493] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:47:54.253961  8206 raft_consensus.cc:3060] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:54.261651  8180 raft_consensus.cc:493] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:54.261329  8206 raft_consensus.cc:515] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.262146  8180 raft_consensus.cc:515] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.263540  8206 leader_election.cc:290] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 election: Requested vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507)
I20260808 01:47:54.264300  8180 leader_election.cc:290] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:54.264681  8175 raft_consensus.cc:359] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.265394  8175 raft_consensus.cc:385] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:54.265658  8175 raft_consensus.cc:740] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Initialized, Role: FOLLOWER
I20260808 01:47:54.268114  8048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8064f9a1744648c4963af166d4b67cfa" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b" is_pre_election: true
I20260808 01:47:54.266292  8175 consensus_queue.cc:260] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.267709  8049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d230e1f50b4c888c85615b35087706" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
I20260808 01:47:54.268905  8049 raft_consensus.cc:3060] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:47:54.269927  7934 leader_election.cc:343] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:47:54.270208  8175 ts_tablet_manager.cc:1434] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.000s
I20260808 01:47:54.271319  8173 tablet_bootstrap.cc:492] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b: No bootstrap required, opened a new log
I20260808 01:47:54.271785  8173 ts_tablet_manager.cc:1403] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b: Time spent bootstrapping tablet: real 0.069s	user 0.011s	sys 0.008s
I20260808 01:47:54.272495  7972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d230e1f50b4c888c85615b35087706" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b822036e833c464b81cf37b24ec55c88"
I20260808 01:47:54.273095  7972 raft_consensus.cc:3060] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:54.275518  8206 raft_consensus.cc:493] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:54.275831  8127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8064f9a1744648c4963af166d4b67cfa" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a640ba2733143fb9807a7675ed337e3" is_pre_election: true
I20260808 01:47:54.276046  8206 raft_consensus.cc:515] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.276515  8127 raft_consensus.cc:2468] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 0.
I20260808 01:47:54.278143  8206 leader_election.cc:290] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507)
I20260808 01:47:54.277607  7863 catalog_manager.cc:5697] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194). New cstate: current_term: 1 leader_uuid: "282686b2e6f746d3b2eb16cf77647c1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20260808 01:47:54.280154  8048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68c2bac20b264a42adee938c783e6102" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b" is_pre_election: true
I20260808 01:47:54.279994  8173 raft_consensus.cc:359] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.280710  8173 raft_consensus.cc:385] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:54.280809  8048 raft_consensus.cc:2468] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 0.
I20260808 01:47:54.281016  8173 raft_consensus.cc:740] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Initialized, Role: FOLLOWER
I20260808 01:47:54.281812  8173 consensus_queue.cc:260] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.282439  8089 leader_election.cc:304] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, 4a640ba2733143fb9807a7675ed337e3; no voters: 
I20260808 01:47:54.283257  8206 raft_consensus.cc:2804] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:47:54.283636  8206 raft_consensus.cc:493] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:47:54.283967  8206 raft_consensus.cc:3060] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:54.285301  7935 leader_election.cc:304] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3, b822036e833c464b81cf37b24ec55c88; no voters: 282686b2e6f746d3b2eb16cf77647c1b
I20260808 01:47:54.285501  7971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68c2bac20b264a42adee938c783e6102" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b822036e833c464b81cf37b24ec55c88" is_pre_election: true
I20260808 01:47:54.286154  7971 raft_consensus.cc:2468] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 0.
I20260808 01:47:54.288951  8180 raft_consensus.cc:2804] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:47:54.289366  8180 raft_consensus.cc:493] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:47:54.289703  8180 raft_consensus.cc:3060] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:54.293259  8206 raft_consensus.cc:515] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.295680  8206 leader_election.cc:290] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 election: Requested vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507)
I20260808 01:47:54.297432  8173 ts_tablet_manager.cc:1434] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b: Time spent starting tablet: real 0.025s	user 0.004s	sys 0.004s
I20260808 01:47:54.298717  7971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68c2bac20b264a42adee938c783e6102" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b822036e833c464b81cf37b24ec55c88"
I20260808 01:47:54.299366  7971 raft_consensus.cc:3060] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:54.301612  8048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68c2bac20b264a42adee938c783e6102" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
I20260808 01:47:54.302227  8048 raft_consensus.cc:3060] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:54.306612  7971 raft_consensus.cc:2468] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 1.
I20260808 01:47:54.311337  8092 leader_election.cc:304] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:47:54.312191  8206 raft_consensus.cc:2804] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:47:54.315330  8048 raft_consensus.cc:2468] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 1.
I20260808 01:47:54.312680  8206 raft_consensus.cc:697] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 1 LEADER]: Becoming Leader. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Running, Role: LEADER
I20260808 01:47:54.318100  8206 consensus_queue.cc:237] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.322011  8049 raft_consensus.cc:2468] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 1.
I20260808 01:47:54.323890  8174 tablet_bootstrap.cc:654] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88: Neither blocks nor log segments found. Creating new log.
I20260808 01:47:54.324941  7972 raft_consensus.cc:2468] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 1.
I20260808 01:47:54.325601  8089 leader_election.cc:304] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, 4a640ba2733143fb9807a7675ed337e3; no voters: 
I20260808 01:47:54.326776  8183 raft_consensus.cc:2804] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:47:54.321946  8180 raft_consensus.cc:515] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.334002  8048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8064f9a1744648c4963af166d4b67cfa" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
I20260808 01:47:54.334929  8048 raft_consensus.cc:3060] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:54.335691  8183 raft_consensus.cc:697] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 1 LEADER]: Becoming Leader. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Running, Role: LEADER
I20260808 01:47:54.336961  8183 consensus_queue.cc:237] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.341463  8127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8064f9a1744648c4963af166d4b67cfa" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
I20260808 01:47:54.342358  8127 raft_consensus.cc:3060] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:54.355935  8174 tablet_bootstrap.cc:492] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88: No bootstrap required, opened a new log
I20260808 01:47:54.356359  8174 ts_tablet_manager.cc:1403] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88: Time spent bootstrapping tablet: real 0.115s	user 0.013s	sys 0.000s
I20260808 01:47:54.350816  8180 leader_election.cc:290] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 election: Requested vote from peers 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:54.359378  8127 raft_consensus.cc:2468] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 1.
I20260808 01:47:54.360298  8048 raft_consensus.cc:2468] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 1.
I20260808 01:47:54.361188  7935 leader_election.cc:304] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:47:54.362434  8180 raft_consensus.cc:2804] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:47:54.363058  8180 raft_consensus.cc:697] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Becoming Leader. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Running, Role: LEADER
I20260808 01:47:54.363812  8180 consensus_queue.cc:237] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.367534  7863 catalog_manager.cc:5697] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "4a640ba2733143fb9807a7675ed337e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: HEALTHY } } }
I20260808 01:47:54.370648  8174 raft_consensus.cc:359] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.371398  8174 raft_consensus.cc:385] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:47:54.371675  8174 raft_consensus.cc:740] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Initialized, Role: FOLLOWER
I20260808 01:47:54.372483  8174 consensus_queue.cc:260] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.374425  8174 ts_tablet_manager.cc:1434] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.000s
I20260808 01:47:54.376051  7860 catalog_manager.cc:5697] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 reported cstate change: term changed from 0 to 1, leader changed from <none> to b822036e833c464b81cf37b24ec55c88 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "b822036e833c464b81cf37b24ec55c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20260808 01:47:54.391031  7860 catalog_manager.cc:5697] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "4a640ba2733143fb9807a7675ed337e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: HEALTHY } } }
I20260808 01:47:54.402659  8216 raft_consensus.cc:493] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:54.403206  8216 raft_consensus.cc:515] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.405124  8216 leader_election.cc:290] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:54.406095  7972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f441f7e95cba45ed9cb394a38cb4b4a2" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b822036e833c464b81cf37b24ec55c88" is_pre_election: true
I20260808 01:47:54.406951  7972 raft_consensus.cc:2393] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 282686b2e6f746d3b2eb16cf77647c1b in current term 1: Already voted for candidate 4a640ba2733143fb9807a7675ed337e3 in this term.
I20260808 01:47:54.406741  8127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f441f7e95cba45ed9cb394a38cb4b4a2" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a640ba2733143fb9807a7675ed337e3" is_pre_election: true
I20260808 01:47:54.408562  8013 leader_election.cc:304] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b; no voters: 4a640ba2733143fb9807a7675ed337e3, b822036e833c464b81cf37b24ec55c88
I20260808 01:47:54.410362  8216 raft_consensus.cc:3060] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:54.414441  8180 consensus_queue.cc:1048] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:54.417534  8216 raft_consensus.cc:2749] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:47:54.431753  8211 consensus_queue.cc:1048] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:54.433573  8211 consensus_queue.cc:1048] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:47:54.445096  8129 raft_consensus.cc:3060] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:54.463860  8211 consensus_queue.cc:1048] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:54.467900  8049 raft_consensus.cc:3060] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:54.481520  8215 consensus_queue.cc:1048] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:54.495522  8215 raft_consensus.cc:493] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:47:54.496093  8215 raft_consensus.cc:515] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:54.499231  8215 leader_election.cc:290] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:54.499271  8048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b" is_pre_election: true
I20260808 01:47:54.503934  8216 consensus_queue.cc:1048] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:47:54.508003  8209 consensus_queue.cc:1048] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:47:54.508394  8129 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a640ba2733143fb9807a7675ed337e3" is_pre_election: true
I20260808 01:47:54.513841  7935 leader_election.cc:304] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b822036e833c464b81cf37b24ec55c88; no voters: 282686b2e6f746d3b2eb16cf77647c1b, 4a640ba2733143fb9807a7675ed337e3
I20260808 01:47:54.514961  8209 raft_consensus.cc:2749] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:47:54.524735  7972 raft_consensus.cc:3060] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:47:54.535135  8216 consensus_queue.cc:1048] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:54.546118  8183 consensus_queue.cc:1048] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:54.593097  8183 consensus_queue.cc:1048] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:47:54.719519  8193 consensus_queue.cc:1048] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:47:54.722183  8193 consensus_queue.cc:1048] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:47:54.790527  8127 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "31d230e1f50b4c888c85615b35087706"
dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
mode: GRACEFUL
new_leader_uuid: "b822036e833c464b81cf37b24ec55c88"
 from {username='slave'} at 127.0.0.1:51076
I20260808 01:47:54.792506  8127 raft_consensus.cc:606] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 1 LEADER]: Received request to transfer leadership to TS b822036e833c464b81cf37b24ec55c88
I20260808 01:47:54.817335  8183 consensus_queue.cc:1048] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:54.823904  8049 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4"
dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
mode: GRACEFUL
new_leader_uuid: "b822036e833c464b81cf37b24ec55c88"
 from {username='slave'} at 127.0.0.1:57550
I20260808 01:47:54.824824  8049 raft_consensus.cc:606] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 LEADER]: Received request to transfer leadership to TS b822036e833c464b81cf37b24ec55c88
I20260808 01:47:54.861127  8183 consensus_queue.cc:1048] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:54.861109  8239 consensus_queue.cc:1048] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:54.863749  8215 consensus_queue.cc:1048] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:54.874496  8127 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f441f7e95cba45ed9cb394a38cb4b4a2"
dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
mode: GRACEFUL
new_leader_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
 from {username='slave'} at 127.0.0.1:51076
I20260808 01:47:54.876053  8127 raft_consensus.cc:606] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 1 LEADER]: Received request to transfer leadership to TS 282686b2e6f746d3b2eb16cf77647c1b
I20260808 01:47:54.929344  8215 consensus_queue.cc:1048] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:54.945703  8183 consensus_queue.cc:1048] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:47:54.983394  7971 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cc6dc97c4b8e482683facaeb4e261128"
dest_uuid: "b822036e833c464b81cf37b24ec55c88"
mode: GRACEFUL
new_leader_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
 from {username='slave'} at 127.0.0.1:39486
I20260808 01:47:54.984210  7971 raft_consensus.cc:606] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Received request to transfer leadership to TS 282686b2e6f746d3b2eb16cf77647c1b
I20260808 01:47:54.989537  8206 raft_consensus.cc:993] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3: : Instructing follower b822036e833c464b81cf37b24ec55c88 to start an election
I20260808 01:47:54.990378  8183 raft_consensus.cc:1081] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 1 LEADER]: Signalling peer b822036e833c464b81cf37b24ec55c88 to start an election
I20260808 01:47:54.994027  7971 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "31d230e1f50b4c888c85615b35087706"
dest_uuid: "b822036e833c464b81cf37b24ec55c88"
 from {username='slave'} at 127.0.0.1:45138
I20260808 01:47:54.995078  7971 raft_consensus.cc:493] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:47:54.995671  7971 raft_consensus.cc:3060] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:55.011572  8128 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "68c2bac20b264a42adee938c783e6102"
dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
mode: GRACEFUL
new_leader_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
 from {username='slave'} at 127.0.0.1:51076
I20260808 01:47:55.013482  8128 raft_consensus.cc:606] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 1 LEADER]: Received request to transfer leadership to TS 282686b2e6f746d3b2eb16cf77647c1b
I20260808 01:47:55.020952  7971 raft_consensus.cc:515] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:55.029924  7972 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "82541e7deca64da19bc6dd1e54b4a3c8"
dest_uuid: "b822036e833c464b81cf37b24ec55c88"
mode: GRACEFUL
new_leader_uuid: "4a640ba2733143fb9807a7675ed337e3"
 from {username='slave'} at 127.0.0.1:39486
I20260808 01:47:55.030861  7972 raft_consensus.cc:606] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Received request to transfer leadership to TS 4a640ba2733143fb9807a7675ed337e3
I20260808 01:47:55.039247  8206 raft_consensus.cc:993] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3: : Instructing follower 282686b2e6f746d3b2eb16cf77647c1b to start an election
I20260808 01:47:55.039908  8206 raft_consensus.cc:1081] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 1 LEADER]: Signalling peer 282686b2e6f746d3b2eb16cf77647c1b to start an election
I20260808 01:47:55.043704  8048 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f441f7e95cba45ed9cb394a38cb4b4a2"
dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
 from {username='slave'} at 127.0.0.1:42642
I20260808 01:47:55.044701  8048 raft_consensus.cc:493] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:47:55.045228  8048 raft_consensus.cc:3060] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:55.030754  8049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d230e1f50b4c888c85615b35087706" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
I20260808 01:47:55.047299  8049 raft_consensus.cc:3060] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:55.048166  8216 raft_consensus.cc:993] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b: : Instructing follower b822036e833c464b81cf37b24ec55c88 to start an election
I20260808 01:47:55.049789  8216 raft_consensus.cc:1081] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 LEADER]: Signalling peer b822036e833c464b81cf37b24ec55c88 to start an election
I20260808 01:47:55.053725  7972 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4"
dest_uuid: "b822036e833c464b81cf37b24ec55c88"
 from {username='slave'} at 127.0.0.1:45130
I20260808 01:47:55.054607  7972 raft_consensus.cc:493] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:47:55.055277  7972 raft_consensus.cc:3060] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:55.057371  8128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d230e1f50b4c888c85615b35087706" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
I20260808 01:47:55.058226  7971 leader_election.cc:290] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 2 election: Requested vote from peers 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:55.058348  8128 raft_consensus.cc:3055] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:47:55.059927  8128 raft_consensus.cc:740] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Running, Role: LEADER
I20260808 01:47:55.067077  8049 raft_consensus.cc:2468] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 2.
I20260808 01:47:55.069331  7934 leader_election.cc:304] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:47:55.071182  8180 raft_consensus.cc:2804] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:47:55.072108  8180 raft_consensus.cc:697] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 2 LEADER]: Becoming Leader. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Running, Role: LEADER
I20260808 01:47:55.073783  8180 consensus_queue.cc:237] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:55.080730  8048 raft_consensus.cc:515] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:55.066669  8128 consensus_queue.cc:260] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:55.085585  8128 raft_consensus.cc:3060] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:55.086324  7971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f441f7e95cba45ed9cb394a38cb4b4a2" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b822036e833c464b81cf37b24ec55c88"
I20260808 01:47:55.087325  7971 raft_consensus.cc:3060] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:55.093078  7860 catalog_manager.cc:5697] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 reported cstate change: term changed from 1 to 2, leader changed from 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195) to b822036e833c464b81cf37b24ec55c88 (127.7.165.193). New cstate: current_term: 2 leader_uuid: "b822036e833c464b81cf37b24ec55c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20260808 01:47:55.104372  7972 raft_consensus.cc:515] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:55.108733  8128 raft_consensus.cc:2468] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 2.
I20260808 01:47:55.113473  8049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
I20260808 01:47:55.114235  8049 raft_consensus.cc:3055] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:47:55.114851  8049 raft_consensus.cc:740] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Running, Role: LEADER
I20260808 01:47:55.115001  8048 leader_election.cc:290] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 2 election: Requested vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:55.119030  8127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f441f7e95cba45ed9cb394a38cb4b4a2" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
I20260808 01:47:55.119915  8127 raft_consensus.cc:3055] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:47:55.120335  8127 raft_consensus.cc:740] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Running, Role: LEADER
I20260808 01:47:55.116146  8049 consensus_queue.cc:260] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:55.121856  8127 consensus_queue.cc:260] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:55.126293  8049 raft_consensus.cc:3060] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:55.127271  8127 raft_consensus.cc:3060] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:55.132130  7971 raft_consensus.cc:2468] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 2.
I20260808 01:47:55.135298  8014 leader_election.cc:304] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:47:55.139501  7972 leader_election.cc:290] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 2 election: Requested vote from peers 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:55.146880  8128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
I20260808 01:47:55.147908  8128 raft_consensus.cc:3060] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:55.138474  8216 raft_consensus.cc:2804] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:47:55.149070  8216 raft_consensus.cc:697] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 LEADER]: Becoming Leader. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Running, Role: LEADER
I20260808 01:47:55.150174  8216 consensus_queue.cc:237] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:55.152596  8127 raft_consensus.cc:2468] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 2.
I20260808 01:47:55.155506  8049 raft_consensus.cc:2468] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 2.
I20260808 01:47:55.157402  7934 leader_election.cc:304] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:47:55.162043  8180 raft_consensus.cc:2804] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:47:55.161317  8128 raft_consensus.cc:2468] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 2.
I20260808 01:47:55.162899  8180 raft_consensus.cc:697] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 2 LEADER]: Becoming Leader. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Running, Role: LEADER
I20260808 01:47:55.164379  8180 consensus_queue.cc:237] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:55.185838  7862 catalog_manager.cc:5697] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b reported cstate change: term changed from 1 to 2, leader changed from 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195) to 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194). New cstate: current_term: 2 leader_uuid: "282686b2e6f746d3b2eb16cf77647c1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20260808 01:47:55.195744  7860 catalog_manager.cc:5697] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 reported cstate change: term changed from 1 to 2, leader changed from 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194) to b822036e833c464b81cf37b24ec55c88 (127.7.165.193). New cstate: current_term: 2 leader_uuid: "b822036e833c464b81cf37b24ec55c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20260808 01:47:55.287652  8209 raft_consensus.cc:993] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88: : Instructing follower 282686b2e6f746d3b2eb16cf77647c1b to start an election
I20260808 01:47:55.288267  8180 raft_consensus.cc:1081] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Signalling peer 282686b2e6f746d3b2eb16cf77647c1b to start an election
I20260808 01:47:55.290833  8049 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cc6dc97c4b8e482683facaeb4e261128"
dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
 from {username='slave'} at 127.0.0.1:42638
I20260808 01:47:55.291371  8049 raft_consensus.cc:493] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:47:55.291702  8049 raft_consensus.cc:3060] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:55.302325  8049 raft_consensus.cc:515] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:55.305377  8049 leader_election.cc:290] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 2 election: Requested vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:55.306847  8128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc6dc97c4b8e482683facaeb4e261128" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
I20260808 01:47:55.306816  7972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc6dc97c4b8e482683facaeb4e261128" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b822036e833c464b81cf37b24ec55c88"
I20260808 01:47:55.307897  8128 raft_consensus.cc:3060] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:55.308661  7972 raft_consensus.cc:3055] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:47:55.309230  7972 raft_consensus.cc:740] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Running, Role: LEADER
I20260808 01:47:55.310407  7972 consensus_queue.cc:260] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:55.312320  7972 raft_consensus.cc:3060] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:55.319779  8128 raft_consensus.cc:2468] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 2.
I20260808 01:47:55.321794  8013 leader_election.cc:304] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, 4a640ba2733143fb9807a7675ed337e3; no voters: 
I20260808 01:47:55.323271  8216 raft_consensus.cc:2804] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:47:55.323611  7972 raft_consensus.cc:2468] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 2.
I20260808 01:47:55.324373  8216 raft_consensus.cc:697] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 LEADER]: Becoming Leader. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Running, Role: LEADER
I20260808 01:47:55.326367  8216 consensus_queue.cc:237] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:55.340523  7860 catalog_manager.cc:5697] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b reported cstate change: term changed from 1 to 2, leader changed from b822036e833c464b81cf37b24ec55c88 (127.7.165.193) to 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194). New cstate: current_term: 2 leader_uuid: "282686b2e6f746d3b2eb16cf77647c1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20260808 01:47:55.385627  8209 raft_consensus.cc:993] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88: : Instructing follower 4a640ba2733143fb9807a7675ed337e3 to start an election
I20260808 01:47:55.386286  8180 raft_consensus.cc:1081] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Signalling peer 4a640ba2733143fb9807a7675ed337e3 to start an election
I20260808 01:47:55.388965  8128 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "82541e7deca64da19bc6dd1e54b4a3c8"
dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
 from {username='slave'} at 127.0.0.1:45322
I20260808 01:47:55.390442  8128 raft_consensus.cc:493] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:47:55.391224  8128 raft_consensus.cc:3060] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:55.393719  8239 raft_consensus.cc:993] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3: : Instructing follower 282686b2e6f746d3b2eb16cf77647c1b to start an election
I20260808 01:47:55.396993  8183 raft_consensus.cc:1081] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 1 LEADER]: Signalling peer 282686b2e6f746d3b2eb16cf77647c1b to start an election
I20260808 01:47:55.401113  8049 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "68c2bac20b264a42adee938c783e6102"
dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
 from {username='slave'} at 127.0.0.1:42642
I20260808 01:47:55.402263  8049 raft_consensus.cc:493] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:47:55.402956  8049 raft_consensus.cc:3060] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:55.406178  8128 raft_consensus.cc:515] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:55.410250  8128 leader_election.cc:290] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 2 election: Requested vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507)
I20260808 01:47:55.411335  7972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82541e7deca64da19bc6dd1e54b4a3c8" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b822036e833c464b81cf37b24ec55c88"
I20260808 01:47:55.412381  7972 raft_consensus.cc:3055] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:47:55.412792  7972 raft_consensus.cc:740] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Running, Role: LEADER
I20260808 01:47:55.413758  7972 consensus_queue.cc:260] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:55.416447  8048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82541e7deca64da19bc6dd1e54b4a3c8" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
I20260808 01:47:55.417562  8048 raft_consensus.cc:3060] T 82541e7deca64da19bc6dd1e54b4a3c8 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:55.417706  8049 raft_consensus.cc:515] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:55.418161  7972 raft_consensus.cc:3060] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:55.421988  7971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68c2bac20b264a42adee938c783e6102" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b822036e833c464b81cf37b24ec55c88"
I20260808 01:47:55.422298  8049 leader_election.cc:290] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 2 election: Requested vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:47:55.422960  7971 raft_consensus.cc:3060] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:55.429845  8128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68c2bac20b264a42adee938c783e6102" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
I20260808 01:47:55.433182  7972 raft_consensus.cc:2468] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 2.
I20260808 01:47:55.434018  8128 raft_consensus.cc:3055] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:47:55.434607  8128 raft_consensus.cc:740] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Running, Role: LEADER
I20260808 01:47:55.435191  8092 leader_election.cc:304] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:47:55.436491  8239 raft_consensus.cc:2804] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:47:55.436095  8128 consensus_queue.cc:260] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:55.437438  8239 raft_consensus.cc:697] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 [term 2 LEADER]: Becoming Leader. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Running, Role: LEADER
I20260808 01:47:55.438086  8128 raft_consensus.cc:3060] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:55.439204  8239 consensus_queue.cc:237] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:55.443210  7971 raft_consensus.cc:2468] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 2.
I20260808 01:47:55.445426  8014 leader_election.cc:304] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:47:55.447744  8216 raft_consensus.cc:2804] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:47:55.447285  8048 raft_consensus.cc:2468] T 82541e7deca64da19bc6dd1e54b4a3c8 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 2.
I20260808 01:47:55.449306  8216 raft_consensus.cc:697] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 LEADER]: Becoming Leader. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Running, Role: LEADER
I20260808 01:47:55.451512  8216 consensus_queue.cc:237] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:55.458076  8128 raft_consensus.cc:2468] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 2.
I20260808 01:47:55.457695  7860 catalog_manager.cc:5697] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 reported cstate change: term changed from 1 to 2, leader changed from b822036e833c464b81cf37b24ec55c88 (127.7.165.193) to 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195). New cstate: current_term: 2 leader_uuid: "4a640ba2733143fb9807a7675ed337e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: HEALTHY } } }
I20260808 01:47:55.477362  7862 catalog_manager.cc:5697] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b reported cstate change: term changed from 1 to 2, leader changed from 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195) to 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194). New cstate: current_term: 2 leader_uuid: "282686b2e6f746d3b2eb16cf77647c1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20260808 01:47:55.584751  8128 raft_consensus.cc:1275] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Refusing update from remote peer b822036e833c464b81cf37b24ec55c88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:47:55.586668  8180 consensus_queue.cc:1048] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:47:55.606761  8048 raft_consensus.cc:1275] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Refusing update from remote peer b822036e833c464b81cf37b24ec55c88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:47:55.610157  8180 consensus_queue.cc:1048] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:47:55.621363  8128 raft_consensus.cc:1275] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Refusing update from remote peer b822036e833c464b81cf37b24ec55c88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:47:55.623626  8180 consensus_queue.cc:1048] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:47:55.640127  8048 raft_consensus.cc:1275] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Refusing update from remote peer b822036e833c464b81cf37b24ec55c88: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:47:55.642632  8180 consensus_queue.cc:1048] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:47:55.725446  8128 raft_consensus.cc:1275] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Refusing update from remote peer 282686b2e6f746d3b2eb16cf77647c1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:47:55.728294  8216 consensus_queue.cc:1048] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:47:55.746191  7971 raft_consensus.cc:1275] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Refusing update from remote peer 282686b2e6f746d3b2eb16cf77647c1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:47:55.754150  8279 consensus_queue.cc:1048] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:47:55.840457  7972 raft_consensus.cc:1275] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Refusing update from remote peer 282686b2e6f746d3b2eb16cf77647c1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:47:55.841989  8279 consensus_queue.cc:1048] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:47:55.852027  8128 raft_consensus.cc:1275] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Refusing update from remote peer 282686b2e6f746d3b2eb16cf77647c1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:47:55.853649  8279 consensus_queue.cc:1048] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:47:55.865669  7972 raft_consensus.cc:1275] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Refusing update from remote peer 282686b2e6f746d3b2eb16cf77647c1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:47:55.867230  8279 consensus_queue.cc:1048] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:47:55.881206  8128 raft_consensus.cc:1275] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Refusing update from remote peer 282686b2e6f746d3b2eb16cf77647c1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:47:55.883482  8279 consensus_queue.cc:1048] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:47:56.007290  8048 raft_consensus.cc:1275] T 82541e7deca64da19bc6dd1e54b4a3c8 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Refusing update from remote peer 4a640ba2733143fb9807a7675ed337e3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:47:56.009012  8239 consensus_queue.cc:1048] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:47:56.021251  7972 raft_consensus.cc:1275] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Refusing update from remote peer 4a640ba2733143fb9807a7675ed337e3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:47:56.024133  8287 consensus_queue.cc:1048] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:47:58.069753  7831 auto_leader_rebalancer-test.cc:402] The leader distribution is 
I20260808 01:47:58.070742  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:47:58.095376  7972 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2d3db31373fa48e7a8e8a5a6e0851624"
dest_uuid: "b822036e833c464b81cf37b24ec55c88"
mode: GRACEFUL
new_leader_uuid: "4a640ba2733143fb9807a7675ed337e3"
 from {username='slave'} at 127.0.0.1:39494
I20260808 01:47:58.095958  7972 raft_consensus.cc:606] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Received request to transfer leadership to TS 4a640ba2733143fb9807a7675ed337e3
I20260808 01:47:58.107630  8048 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f98266402a6c4c82ae25f5fa9faaff51"
dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
mode: GRACEFUL
new_leader_uuid: "4a640ba2733143fb9807a7675ed337e3"
 from {username='slave'} at 127.0.0.1:57560
I20260808 01:47:58.108220  8048 raft_consensus.cc:606] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 LEADER]: Received request to transfer leadership to TS 4a640ba2733143fb9807a7675ed337e3
I20260808 01:47:58.109755  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 01:47:58.110401  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:47:58.125665  8209 raft_consensus.cc:993] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88: : Instructing follower 4a640ba2733143fb9807a7675ed337e3 to start an election
I20260808 01:47:58.126150  8209 raft_consensus.cc:1081] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Signalling peer 4a640ba2733143fb9807a7675ed337e3 to start an election
I20260808 01:47:58.128532  8128 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2d3db31373fa48e7a8e8a5a6e0851624"
dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
 from {username='slave'} at 127.0.0.1:45322
I20260808 01:47:58.129154  8128 raft_consensus.cc:493] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:47:58.129515  8128 raft_consensus.cc:3060] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:58.134238  8128 raft_consensus.cc:515] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:58.135972  8128 leader_election.cc:290] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 2 election: Requested vote from peers 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507), b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863)
I20260808 01:47:58.136961  8048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d3db31373fa48e7a8e8a5a6e0851624" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
I20260808 01:47:58.137230  7972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d3db31373fa48e7a8e8a5a6e0851624" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b822036e833c464b81cf37b24ec55c88"
I20260808 01:47:58.137626  8048 raft_consensus.cc:3060] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:58.137773  7972 raft_consensus.cc:3055] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:47:58.138157  7972 raft_consensus.cc:740] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Running, Role: LEADER
I20260808 01:47:58.139075  7972 consensus_queue.cc:260] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:58.140340  7972 raft_consensus.cc:3060] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:58.144716  8048 raft_consensus.cc:2468] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 2.
I20260808 01:47:58.146437  8089 leader_election.cc:304] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, 4a640ba2733143fb9807a7675ed337e3; no voters: 
I20260808 01:47:58.147442  8239 raft_consensus.cc:2804] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:47:58.147708  7972 raft_consensus.cc:2468] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 2.
I20260808 01:47:58.148887  8239 raft_consensus.cc:697] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 2 LEADER]: Becoming Leader. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Running, Role: LEADER
I20260808 01:47:58.149720  8239 consensus_queue.cc:237] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:58.159888  7862 catalog_manager.cc:5697] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 reported cstate change: term changed from 1 to 2, leader changed from b822036e833c464b81cf37b24ec55c88 (127.7.165.193) to 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195). New cstate: current_term: 2 leader_uuid: "4a640ba2733143fb9807a7675ed337e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: HEALTHY } } }
I20260808 01:47:58.545763  7972 raft_consensus.cc:1275] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Refusing update from remote peer 4a640ba2733143fb9807a7675ed337e3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:47:58.547482  8239 consensus_queue.cc:1048] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:47:58.557940  8048 raft_consensus.cc:1275] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Refusing update from remote peer 4a640ba2733143fb9807a7675ed337e3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:47:58.559628  8296 consensus_queue.cc:1048] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:47:58.571738  8307 raft_consensus.cc:993] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b: : Instructing follower 4a640ba2733143fb9807a7675ed337e3 to start an election
I20260808 01:47:58.572292  8283 raft_consensus.cc:1081] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 LEADER]: Signalling peer 4a640ba2733143fb9807a7675ed337e3 to start an election
I20260808 01:47:58.573838  8128 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f98266402a6c4c82ae25f5fa9faaff51"
dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
 from {username='slave'} at 127.0.0.1:45316
I20260808 01:47:58.574290  8128 raft_consensus.cc:493] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:47:58.574621  8128 raft_consensus.cc:3060] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:58.578610  8128 raft_consensus.cc:515] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:58.580305  8128 leader_election.cc:290] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 2 election: Requested vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507)
I20260808 01:47:58.581223  7972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f98266402a6c4c82ae25f5fa9faaff51" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b822036e833c464b81cf37b24ec55c88"
I20260808 01:47:58.581733  8048 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f98266402a6c4c82ae25f5fa9faaff51" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
I20260808 01:47:58.581899  7972 raft_consensus.cc:3060] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:58.582356  8048 raft_consensus.cc:3055] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:47:58.582720  8048 raft_consensus.cc:740] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Running, Role: LEADER
I20260808 01:47:58.583493  8048 consensus_queue.cc:260] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:58.584684  8048 raft_consensus.cc:3060] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:47:58.588367  7972 raft_consensus.cc:2468] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 2.
I20260808 01:47:58.589605  8092 leader_election.cc:304] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:47:58.590342  8296 raft_consensus.cc:2804] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:47:58.590855  8296 raft_consensus.cc:697] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 2 LEADER]: Becoming Leader. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Running, Role: LEADER
I20260808 01:47:58.591248  8048 raft_consensus.cc:2468] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 2.
I20260808 01:47:58.591728  8296 consensus_queue.cc:237] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:47:58.599156  7860 catalog_manager.cc:5697] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 reported cstate change: term changed from 1 to 2, leader changed from 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194) to 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195). New cstate: current_term: 2 leader_uuid: "4a640ba2733143fb9807a7675ed337e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: HEALTHY } } }
I20260808 01:47:59.040262  7972 raft_consensus.cc:1275] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Refusing update from remote peer 4a640ba2733143fb9807a7675ed337e3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:47:59.041646  8296 consensus_queue.cc:1048] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:47:59.053828  8049 raft_consensus.cc:1275] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Refusing update from remote peer 4a640ba2733143fb9807a7675ed337e3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:47:59.055526  8296 consensus_queue.cc:1048] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:47:59.111630  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:47:59.117558  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:47:59.118111  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:00.119452  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:00.128298  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:00.128927  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:01.130381  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:01.136152  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:01.136703  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:02.137722  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:02.142719  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:02.143230  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:03.144320  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:03.149346  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:03.149873  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:04.151073  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:04.157846  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:04.158353  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:05.159477  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:05.165483  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:05.165979  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:06.167265  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:06.172871  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:06.173377  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:07.174600  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:07.180119  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:07.180631  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:08.220016  7831 auto_leader_rebalancer-test.cc:421] The leader distribution is 
I20260808 01:48:08.227386  8128 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2d3db31373fa48e7a8e8a5a6e0851624"
dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
mode: GRACEFUL
new_leader_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
 from {username='slave'} at 127.0.0.1:51076
I20260808 01:48:08.228478  8128 raft_consensus.cc:606] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 2 LEADER]: Received request to transfer leadership to TS 282686b2e6f746d3b2eb16cf77647c1b
I20260808 01:48:08.236714  7971 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "31d230e1f50b4c888c85615b35087706"
dest_uuid: "b822036e833c464b81cf37b24ec55c88"
mode: GRACEFUL
new_leader_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
 from {username='slave'} at 127.0.0.1:39486
I20260808 01:48:08.237421  7971 raft_consensus.cc:606] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 2 LEADER]: Received request to transfer leadership to TS 282686b2e6f746d3b2eb16cf77647c1b
I20260808 01:48:08.244387  7971 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4"
dest_uuid: "b822036e833c464b81cf37b24ec55c88"
mode: GRACEFUL
new_leader_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
 from {username='slave'} at 127.0.0.1:39486
I20260808 01:48:08.244949  7971 raft_consensus.cc:606] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 2 LEADER]: Received request to transfer leadership to TS 282686b2e6f746d3b2eb16cf77647c1b
I20260808 01:48:08.251920  7971 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8064f9a1744648c4963af166d4b67cfa"
dest_uuid: "b822036e833c464b81cf37b24ec55c88"
mode: GRACEFUL
new_leader_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
 from {username='slave'} at 127.0.0.1:39486
I20260808 01:48:08.252424  7971 raft_consensus.cc:606] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Received request to transfer leadership to TS 282686b2e6f746d3b2eb16cf77647c1b
I20260808 01:48:08.263387  8128 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f98266402a6c4c82ae25f5fa9faaff51"
dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
mode: GRACEFUL
new_leader_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
 from {username='slave'} at 127.0.0.1:51076
I20260808 01:48:08.264014  8128 raft_consensus.cc:606] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 2 LEADER]: Received request to transfer leadership to TS 282686b2e6f746d3b2eb16cf77647c1b
I20260808 01:48:08.266557  8319 raft_consensus.cc:993] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88: : Instructing follower 282686b2e6f746d3b2eb16cf77647c1b to start an election
I20260808 01:48:08.267500  8322 raft_consensus.cc:1081] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 2 LEADER]: Signalling peer 282686b2e6f746d3b2eb16cf77647c1b to start an election
I20260808 01:48:08.269388  8049 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4"
dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
 from {username='slave'} at 127.0.0.1:42638
I20260808 01:48:08.269910  8049 raft_consensus.cc:493] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:08.270309  8049 raft_consensus.cc:3060] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:08.276129  8049 raft_consensus.cc:515] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:08.278456  8049 leader_election.cc:290] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 3 election: Requested vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:48:08.279686  8128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
I20260808 01:48:08.280215  8128 raft_consensus.cc:3060] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:08.279428  7971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b822036e833c464b81cf37b24ec55c88"
I20260808 01:48:08.283000  7971 raft_consensus.cc:3055] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:48:08.283313  7971 raft_consensus.cc:740] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Running, Role: LEADER
I20260808 01:48:08.284075  7971 consensus_queue.cc:260] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:08.285305  7971 raft_consensus.cc:3060] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:08.285619  8128 raft_consensus.cc:2468] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 3.
I20260808 01:48:08.287257  8013 leader_election.cc:304] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, 4a640ba2733143fb9807a7675ed337e3; no voters: 
I20260808 01:48:08.288308  8283 raft_consensus.cc:2804] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:48:08.289615  8283 raft_consensus.cc:697] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 LEADER]: Becoming Leader. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Running, Role: LEADER
I20260808 01:48:08.290508  8283 consensus_queue.cc:237] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:08.291656  7971 raft_consensus.cc:2468] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 3.
I20260808 01:48:08.297495  7862 catalog_manager.cc:5697] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b reported cstate change: term changed from 2 to 3, leader changed from b822036e833c464b81cf37b24ec55c88 (127.7.165.193) to 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194). New cstate: current_term: 3 leader_uuid: "282686b2e6f746d3b2eb16cf77647c1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:08.322614  8319 raft_consensus.cc:993] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88: : Instructing follower 282686b2e6f746d3b2eb16cf77647c1b to start an election
I20260808 01:48:08.323204  8322 raft_consensus.cc:1081] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 2 LEADER]: Signalling peer 282686b2e6f746d3b2eb16cf77647c1b to start an election
I20260808 01:48:08.324810  8049 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "31d230e1f50b4c888c85615b35087706"
dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
 from {username='slave'} at 127.0.0.1:42638
I20260808 01:48:08.325313  8049 raft_consensus.cc:493] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:08.325625  8049 raft_consensus.cc:3060] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:08.330134  8049 raft_consensus.cc:515] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:08.331905  8049 leader_election.cc:290] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 3 election: Requested vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:48:08.332808  7971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d230e1f50b4c888c85615b35087706" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b822036e833c464b81cf37b24ec55c88"
I20260808 01:48:08.333012  8128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31d230e1f50b4c888c85615b35087706" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
I20260808 01:48:08.333370  7971 raft_consensus.cc:3055] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:48:08.333582  8128 raft_consensus.cc:3060] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:08.333740  7971 raft_consensus.cc:740] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Running, Role: LEADER
I20260808 01:48:08.334443  7971 consensus_queue.cc:260] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:08.335462  7971 raft_consensus.cc:3060] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:08.338022  8128 raft_consensus.cc:2468] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 3.
I20260808 01:48:08.339056  8013 leader_election.cc:304] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, 4a640ba2733143fb9807a7675ed337e3; no voters: 
I20260808 01:48:08.339764  8283 raft_consensus.cc:2804] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:48:08.340157  7971 raft_consensus.cc:2468] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 3.
I20260808 01:48:08.340231  8283 raft_consensus.cc:697] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 LEADER]: Becoming Leader. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Running, Role: LEADER
I20260808 01:48:08.341154  8283 consensus_queue.cc:237] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:08.348795  7862 catalog_manager.cc:5697] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b reported cstate change: term changed from 2 to 3, leader changed from b822036e833c464b81cf37b24ec55c88 (127.7.165.193) to 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194). New cstate: current_term: 3 leader_uuid: "282686b2e6f746d3b2eb16cf77647c1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:08.386567  8320 raft_consensus.cc:993] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3: : Instructing follower 282686b2e6f746d3b2eb16cf77647c1b to start an election
I20260808 01:48:08.387037  8320 raft_consensus.cc:1081] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 2 LEADER]: Signalling peer 282686b2e6f746d3b2eb16cf77647c1b to start an election
I20260808 01:48:08.388590  8049 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f98266402a6c4c82ae25f5fa9faaff51"
dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
 from {username='slave'} at 127.0.0.1:42642
I20260808 01:48:08.389065  8049 raft_consensus.cc:493] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:08.389393  8049 raft_consensus.cc:3060] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:08.394317  8049 raft_consensus.cc:515] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:08.395726  8049 leader_election.cc:290] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 3 election: Requested vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:48:08.396646  7971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f98266402a6c4c82ae25f5fa9faaff51" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b822036e833c464b81cf37b24ec55c88"
I20260808 01:48:08.396835  8128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f98266402a6c4c82ae25f5fa9faaff51" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
I20260808 01:48:08.397209  7971 raft_consensus.cc:3060] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:08.397431  8128 raft_consensus.cc:3055] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:48:08.397799  8128 raft_consensus.cc:740] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Running, Role: LEADER
I20260808 01:48:08.398509  8128 consensus_queue.cc:260] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:08.399581  8128 raft_consensus.cc:3060] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:08.401899  7971 raft_consensus.cc:2468] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 3.
I20260808 01:48:08.403098  8014 leader_election.cc:304] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:48:08.403877  8283 raft_consensus.cc:2804] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:48:08.404275  8283 raft_consensus.cc:697] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 LEADER]: Becoming Leader. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Running, Role: LEADER
I20260808 01:48:08.404312  8128 raft_consensus.cc:2468] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 3.
I20260808 01:48:08.405162  8283 consensus_queue.cc:237] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:08.411972  7862 catalog_manager.cc:5697] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b reported cstate change: term changed from 2 to 3, leader changed from 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195) to 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194). New cstate: current_term: 3 leader_uuid: "282686b2e6f746d3b2eb16cf77647c1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:08.425714  8330 raft_consensus.cc:993] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3: : Instructing follower 282686b2e6f746d3b2eb16cf77647c1b to start an election
I20260808 01:48:08.426258  8320 raft_consensus.cc:1081] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 2 LEADER]: Signalling peer 282686b2e6f746d3b2eb16cf77647c1b to start an election
I20260808 01:48:08.428146  8049 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2d3db31373fa48e7a8e8a5a6e0851624"
dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
 from {username='slave'} at 127.0.0.1:42642
I20260808 01:48:08.428555  8049 raft_consensus.cc:493] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:08.428785  8049 raft_consensus.cc:3060] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:08.432938  8049 raft_consensus.cc:515] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:08.434449  8049 leader_election.cc:290] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 3 election: Requested vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:48:08.435446  7971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d3db31373fa48e7a8e8a5a6e0851624" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b822036e833c464b81cf37b24ec55c88"
I20260808 01:48:08.435556  8128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d3db31373fa48e7a8e8a5a6e0851624" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
I20260808 01:48:08.436084  7971 raft_consensus.cc:3060] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:08.436144  8128 raft_consensus.cc:3055] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:48:08.436542  8128 raft_consensus.cc:740] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Running, Role: LEADER
I20260808 01:48:08.437310  8128 consensus_queue.cc:260] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:08.438424  8128 raft_consensus.cc:3060] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:08.442646  7971 raft_consensus.cc:2468] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 3.
I20260808 01:48:08.444414  8014 leader_election.cc:304] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:48:08.445156  8128 raft_consensus.cc:2468] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 3.
I20260808 01:48:08.445248  8283 raft_consensus.cc:2804] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:48:08.445773  8283 raft_consensus.cc:697] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 LEADER]: Becoming Leader. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Running, Role: LEADER
I20260808 01:48:08.447212  8283 consensus_queue.cc:237] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:08.454105  7862 catalog_manager.cc:5697] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b reported cstate change: term changed from 2 to 3, leader changed from 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195) to 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194). New cstate: current_term: 3 leader_uuid: "282686b2e6f746d3b2eb16cf77647c1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:08.640837  8319 raft_consensus.cc:993] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88: : Instructing follower 282686b2e6f746d3b2eb16cf77647c1b to start an election
I20260808 01:48:08.641343  8319 raft_consensus.cc:1081] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Signalling peer 282686b2e6f746d3b2eb16cf77647c1b to start an election
I20260808 01:48:08.643021  8049 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8064f9a1744648c4963af166d4b67cfa"
dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
 from {username='slave'} at 127.0.0.1:42638
I20260808 01:48:08.643522  8049 raft_consensus.cc:493] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:08.643797  8049 raft_consensus.cc:3060] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:08.648125  8049 raft_consensus.cc:515] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:08.650306  8049 leader_election.cc:290] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 2 election: Requested vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:48:08.651321  7971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8064f9a1744648c4963af166d4b67cfa" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b822036e833c464b81cf37b24ec55c88"
I20260808 01:48:08.651460  8127 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8064f9a1744648c4963af166d4b67cfa" candidate_uuid: "282686b2e6f746d3b2eb16cf77647c1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
I20260808 01:48:08.651816  7971 raft_consensus.cc:3055] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:48:08.652096  8127 raft_consensus.cc:3060] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:08.652158  7971 raft_consensus.cc:740] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Running, Role: LEADER
I20260808 01:48:08.652916  7971 consensus_queue.cc:260] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:08.653772  7971 raft_consensus.cc:3060] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:08.657835  7971 raft_consensus.cc:2468] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 2.
I20260808 01:48:08.657835  8127 raft_consensus.cc:2468] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 282686b2e6f746d3b2eb16cf77647c1b in term 2.
I20260808 01:48:08.659158  8014 leader_election.cc:304] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:48:08.659876  8283 raft_consensus.cc:2804] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:48:08.660308  8283 raft_consensus.cc:697] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b [term 2 LEADER]: Becoming Leader. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Running, Role: LEADER
I20260808 01:48:08.661005  8283 consensus_queue.cc:237] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:08.668471  7862 catalog_manager.cc:5697] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b reported cstate change: term changed from 1 to 2, leader changed from b822036e833c464b81cf37b24ec55c88 (127.7.165.193) to 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194). New cstate: current_term: 2 leader_uuid: "282686b2e6f746d3b2eb16cf77647c1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:08.761196  8127 raft_consensus.cc:1275] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Refusing update from remote peer 282686b2e6f746d3b2eb16cf77647c1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:08.762692  8283 consensus_queue.cc:1048] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:48:08.775575  7971 raft_consensus.cc:1275] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Refusing update from remote peer 282686b2e6f746d3b2eb16cf77647c1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:08.777858  8323 consensus_queue.cc:1048] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:48:08.883634  8127 raft_consensus.cc:1275] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Refusing update from remote peer 282686b2e6f746d3b2eb16cf77647c1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:08.885326  8323 consensus_queue.cc:1048] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:48:08.896508  7972 raft_consensus.cc:1275] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Refusing update from remote peer 282686b2e6f746d3b2eb16cf77647c1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:08.898597  8323 consensus_queue.cc:1048] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:48:08.917299  8127 raft_consensus.cc:1275] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Refusing update from remote peer 282686b2e6f746d3b2eb16cf77647c1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:08.918752  8323 consensus_queue.cc:1048] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:48:08.924542  8128 raft_consensus.cc:1275] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Refusing update from remote peer 282686b2e6f746d3b2eb16cf77647c1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:08.926061  8323 consensus_queue.cc:1048] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:48:08.930797  7972 raft_consensus.cc:1275] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Refusing update from remote peer 282686b2e6f746d3b2eb16cf77647c1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:08.935263  8283 consensus_queue.cc:1048] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:48:08.941469  7971 raft_consensus.cc:1275] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Refusing update from remote peer 282686b2e6f746d3b2eb16cf77647c1b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:08.943148  8338 consensus_queue.cc:1048] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260808 01:48:09.044013  8127 raft_consensus.cc:1275] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Refusing update from remote peer 282686b2e6f746d3b2eb16cf77647c1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:09.046329  8283 consensus_queue.cc:1048] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:48:09.057777  7971 raft_consensus.cc:1275] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Refusing update from remote peer 282686b2e6f746d3b2eb16cf77647c1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:09.061169  8323 consensus_queue.cc:1048] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:48:11.308848  7831 auto_leader_rebalancer-test.cc:436] The leader distribution is 
I20260808 01:48:11.309772  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:11.318091  8049 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2d3db31373fa48e7a8e8a5a6e0851624"
dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
mode: GRACEFUL
new_leader_uuid: "4a640ba2733143fb9807a7675ed337e3"
 from {username='slave'} at 127.0.0.1:57560
I20260808 01:48:11.318811  8049 raft_consensus.cc:606] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 LEADER]: Received request to transfer leadership to TS 4a640ba2733143fb9807a7675ed337e3
I20260808 01:48:11.321662  8049 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "68c2bac20b264a42adee938c783e6102"
dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
mode: GRACEFUL
new_leader_uuid: "b822036e833c464b81cf37b24ec55c88"
 from {username='slave'} at 127.0.0.1:57560
I20260808 01:48:11.322131  8049 raft_consensus.cc:606] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 LEADER]: Received request to transfer leadership to TS b822036e833c464b81cf37b24ec55c88
I20260808 01:48:11.324481  8049 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4"
dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
mode: GRACEFUL
new_leader_uuid: "b822036e833c464b81cf37b24ec55c88"
 from {username='slave'} at 127.0.0.1:57560
I20260808 01:48:11.324924  8049 raft_consensus.cc:606] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 LEADER]: Received request to transfer leadership to TS b822036e833c464b81cf37b24ec55c88
I20260808 01:48:11.327425  8049 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f441f7e95cba45ed9cb394a38cb4b4a2"
dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
mode: GRACEFUL
new_leader_uuid: "4a640ba2733143fb9807a7675ed337e3"
 from {username='slave'} at 127.0.0.1:57560
I20260808 01:48:11.328030  8049 raft_consensus.cc:606] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 LEADER]: Received request to transfer leadership to TS 4a640ba2733143fb9807a7675ed337e3
I20260808 01:48:11.330706  8049 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f98266402a6c4c82ae25f5fa9faaff51"
dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
mode: GRACEFUL
new_leader_uuid: "b822036e833c464b81cf37b24ec55c88"
 from {username='slave'} at 127.0.0.1:57560
I20260808 01:48:11.331152  8049 raft_consensus.cc:606] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 LEADER]: Received request to transfer leadership to TS b822036e833c464b81cf37b24ec55c88
I20260808 01:48:11.332212  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260808 01:48:11.332690  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:11.357784  8355 raft_consensus.cc:993] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b: : Instructing follower 4a640ba2733143fb9807a7675ed337e3 to start an election
I20260808 01:48:11.358492  8353 raft_consensus.cc:1081] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 LEADER]: Signalling peer 4a640ba2733143fb9807a7675ed337e3 to start an election
I20260808 01:48:11.360503  8128 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f441f7e95cba45ed9cb394a38cb4b4a2"
dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
 from {username='slave'} at 127.0.0.1:45316
I20260808 01:48:11.361092  8128 raft_consensus.cc:493] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:11.361450  8128 raft_consensus.cc:3060] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:11.368012  8128 raft_consensus.cc:515] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:11.369977  8128 leader_election.cc:290] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 3 election: Requested vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507)
I20260808 01:48:11.371209  7971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f441f7e95cba45ed9cb394a38cb4b4a2" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b822036e833c464b81cf37b24ec55c88"
I20260808 01:48:11.371191  8049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f441f7e95cba45ed9cb394a38cb4b4a2" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
I20260808 01:48:11.371961  7971 raft_consensus.cc:3060] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:11.372007  8049 raft_consensus.cc:3055] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:48:11.372390  8049 raft_consensus.cc:740] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Running, Role: LEADER
I20260808 01:48:11.373013  8049 consensus_queue.cc:260] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:11.373929  8049 raft_consensus.cc:3060] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:11.377159  7971 raft_consensus.cc:2468] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 3.
I20260808 01:48:11.378698  8092 leader_election.cc:304] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:48:11.379381  8049 raft_consensus.cc:2468] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 3.
I20260808 01:48:11.379690  8346 raft_consensus.cc:2804] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:48:11.380839  8346 raft_consensus.cc:697] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 3 LEADER]: Becoming Leader. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Running, Role: LEADER
I20260808 01:48:11.381815  8346 consensus_queue.cc:237] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:11.388710  7862 catalog_manager.cc:5697] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 reported cstate change: term changed from 2 to 3, leader changed from 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194) to 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195). New cstate: current_term: 3 leader_uuid: "4a640ba2733143fb9807a7675ed337e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: HEALTHY } } }
I20260808 01:48:11.487931  8355 raft_consensus.cc:993] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b: : Instructing follower b822036e833c464b81cf37b24ec55c88 to start an election
I20260808 01:48:11.488471  8353 raft_consensus.cc:1081] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 LEADER]: Signalling peer b822036e833c464b81cf37b24ec55c88 to start an election
I20260808 01:48:11.490110  7971 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "68c2bac20b264a42adee938c783e6102"
dest_uuid: "b822036e833c464b81cf37b24ec55c88"
 from {username='slave'} at 127.0.0.1:45130
I20260808 01:48:11.490593  7971 raft_consensus.cc:493] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:11.490901  7971 raft_consensus.cc:3060] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:11.495592  7971 raft_consensus.cc:515] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:11.497573  7971 leader_election.cc:290] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 3 election: Requested vote from peers 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:48:11.498792  8049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68c2bac20b264a42adee938c783e6102" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
I20260808 01:48:11.498833  8128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68c2bac20b264a42adee938c783e6102" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
I20260808 01:48:11.499481  8049 raft_consensus.cc:3055] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:48:11.499550  8128 raft_consensus.cc:3060] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:11.499805  8049 raft_consensus.cc:740] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Running, Role: LEADER
I20260808 01:48:11.500643  8049 consensus_queue.cc:260] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:11.501493  8049 raft_consensus.cc:3060] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:11.504549  8128 raft_consensus.cc:2468] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 3.
I20260808 01:48:11.505914  7935 leader_election.cc:304] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:48:11.507210  8049 raft_consensus.cc:2468] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 3.
I20260808 01:48:11.508085  8362 raft_consensus.cc:2804] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:48:11.509559  8362 raft_consensus.cc:697] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [term 3 LEADER]: Becoming Leader. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Running, Role: LEADER
I20260808 01:48:11.510521  8362 consensus_queue.cc:237] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:11.518529  7862 catalog_manager.cc:5697] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 reported cstate change: term changed from 2 to 3, leader changed from 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194) to b822036e833c464b81cf37b24ec55c88 (127.7.165.193). New cstate: current_term: 3 leader_uuid: "b822036e833c464b81cf37b24ec55c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:11.540570  8355 raft_consensus.cc:993] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b: : Instructing follower b822036e833c464b81cf37b24ec55c88 to start an election
I20260808 01:48:11.541040  8353 raft_consensus.cc:1081] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 LEADER]: Signalling peer b822036e833c464b81cf37b24ec55c88 to start an election
I20260808 01:48:11.542774  7971 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f98266402a6c4c82ae25f5fa9faaff51"
dest_uuid: "b822036e833c464b81cf37b24ec55c88"
 from {username='slave'} at 127.0.0.1:45130
I20260808 01:48:11.543314  7971 raft_consensus.cc:493] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:11.543640  7971 raft_consensus.cc:3060] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:48:11.547976  7971 raft_consensus.cc:515] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:11.549615  7971 leader_election.cc:290] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 4 election: Requested vote from peers 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:48:11.550668  8049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f98266402a6c4c82ae25f5fa9faaff51" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
I20260808 01:48:11.550753  8128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f98266402a6c4c82ae25f5fa9faaff51" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
I20260808 01:48:11.551425  8049 raft_consensus.cc:3055] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:48:11.551580  8128 raft_consensus.cc:3060] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:48:11.551730  8049 raft_consensus.cc:740] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Running, Role: LEADER
I20260808 01:48:11.552601  8049 consensus_queue.cc:260] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:11.553539  8049 raft_consensus.cc:3060] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:48:11.557176  8128 raft_consensus.cc:2468] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 4.
I20260808 01:48:11.558389  7935 leader_election.cc:304] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:48:11.558530  8049 raft_consensus.cc:2468] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 4.
I20260808 01:48:11.559185  8362 raft_consensus.cc:2804] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:48:11.559736  8362 raft_consensus.cc:697] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [term 4 LEADER]: Becoming Leader. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Running, Role: LEADER
I20260808 01:48:11.561123  8362 consensus_queue.cc:237] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:11.569262  7862 catalog_manager.cc:5697] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 reported cstate change: term changed from 3 to 4, leader changed from 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194) to b822036e833c464b81cf37b24ec55c88 (127.7.165.193). New cstate: current_term: 4 leader_uuid: "b822036e833c464b81cf37b24ec55c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:11.580593  8355 raft_consensus.cc:993] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b: : Instructing follower 4a640ba2733143fb9807a7675ed337e3 to start an election
I20260808 01:48:11.581063  8353 raft_consensus.cc:1081] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 LEADER]: Signalling peer 4a640ba2733143fb9807a7675ed337e3 to start an election
I20260808 01:48:11.582468  8128 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2d3db31373fa48e7a8e8a5a6e0851624"
dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
 from {username='slave'} at 127.0.0.1:45316
I20260808 01:48:11.583096  8128 raft_consensus.cc:493] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:11.583420  8128 raft_consensus.cc:3060] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:48:11.587968  8128 raft_consensus.cc:515] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:11.589854  8128 leader_election.cc:290] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 4 election: Requested vote from peers 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507), b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863)
I20260808 01:48:11.590883  8049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d3db31373fa48e7a8e8a5a6e0851624" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
I20260808 01:48:11.591146  7971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d3db31373fa48e7a8e8a5a6e0851624" candidate_uuid: "4a640ba2733143fb9807a7675ed337e3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b822036e833c464b81cf37b24ec55c88"
I20260808 01:48:11.591728  8049 raft_consensus.cc:3055] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:48:11.591734  7971 raft_consensus.cc:3060] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:48:11.592128  8049 raft_consensus.cc:740] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Running, Role: LEADER
I20260808 01:48:11.592850  8049 consensus_queue.cc:260] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:11.593783  8049 raft_consensus.cc:3060] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:48:11.596879  7971 raft_consensus.cc:2468] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 4.
I20260808 01:48:11.597985  8092 leader_election.cc:304] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:48:11.598165  8049 raft_consensus.cc:2468] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a640ba2733143fb9807a7675ed337e3 in term 4.
I20260808 01:48:11.598860  8346 raft_consensus.cc:2804] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:48:11.599442  8346 raft_consensus.cc:697] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 4 LEADER]: Becoming Leader. State: Replica: 4a640ba2733143fb9807a7675ed337e3, State: Running, Role: LEADER
I20260808 01:48:11.600368  8346 consensus_queue.cc:237] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:11.608362  7862 catalog_manager.cc:5697] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 reported cstate change: term changed from 3 to 4, leader changed from 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194) to 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195). New cstate: current_term: 4 leader_uuid: "4a640ba2733143fb9807a7675ed337e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: HEALTHY } } }
I20260808 01:48:11.866142  8049 raft_consensus.cc:1275] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Refusing update from remote peer 4a640ba2733143fb9807a7675ed337e3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:11.867947  8346 consensus_queue.cc:1048] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:48:11.879972  7971 raft_consensus.cc:1275] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Refusing update from remote peer 4a640ba2733143fb9807a7675ed337e3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:11.882483  8356 consensus_queue.cc:1048] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:48:11.895237  8355 raft_consensus.cc:993] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b: : Instructing follower b822036e833c464b81cf37b24ec55c88 to start an election
I20260808 01:48:11.895816  8353 raft_consensus.cc:1081] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 LEADER]: Signalling peer b822036e833c464b81cf37b24ec55c88 to start an election
I20260808 01:48:11.897574  7971 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4"
dest_uuid: "b822036e833c464b81cf37b24ec55c88"
 from {username='slave'} at 127.0.0.1:45130
I20260808 01:48:11.898115  7971 raft_consensus.cc:493] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:11.898449  7971 raft_consensus.cc:3060] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:48:11.903077  7971 raft_consensus.cc:515] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:11.904491  7971 leader_election.cc:290] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 4 election: Requested vote from peers 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507), 4a640ba2733143fb9807a7675ed337e3 (127.7.165.195:36653)
I20260808 01:48:11.905491  8049 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "282686b2e6f746d3b2eb16cf77647c1b"
I20260808 01:48:11.906057  8049 raft_consensus.cc:3055] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:48:11.906414  8049 raft_consensus.cc:740] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 282686b2e6f746d3b2eb16cf77647c1b, State: Running, Role: LEADER
I20260808 01:48:11.906301  8128 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e6647c3e5e146d185e0230bd07689e4" candidate_uuid: "b822036e833c464b81cf37b24ec55c88" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4a640ba2733143fb9807a7675ed337e3"
I20260808 01:48:11.907186  8128 raft_consensus.cc:3060] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:48:11.907207  8049 consensus_queue.cc:260] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:11.908422  8127 raft_consensus.cc:1275] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Refusing update from remote peer b822036e833c464b81cf37b24ec55c88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:11.908677  8049 raft_consensus.cc:3060] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:48:11.910223  8362 consensus_queue.cc:1048] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:48:11.913714  8128 raft_consensus.cc:2468] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 4.
I20260808 01:48:11.914695  7935 leader_election.cc:304] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3, b822036e833c464b81cf37b24ec55c88; no voters: 
I20260808 01:48:11.915581  8362 raft_consensus.cc:2804] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:48:11.916132  8362 raft_consensus.cc:697] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 4 LEADER]: Becoming Leader. State: Replica: b822036e833c464b81cf37b24ec55c88, State: Running, Role: LEADER
I20260808 01:48:11.915923  8049 raft_consensus.cc:2468] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b822036e833c464b81cf37b24ec55c88 in term 4.
I20260808 01:48:11.917124  8362 consensus_queue.cc:237] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:11.922595  8049 raft_consensus.cc:1275] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Refusing update from remote peer b822036e833c464b81cf37b24ec55c88: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:11.925323  8363 consensus_queue.cc:1048] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:48:11.931255  7860 catalog_manager.cc:5697] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 reported cstate change: term changed from 3 to 4, leader changed from 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194) to b822036e833c464b81cf37b24ec55c88 (127.7.165.193). New cstate: current_term: 4 leader_uuid: "b822036e833c464b81cf37b24ec55c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:11.946017  8127 raft_consensus.cc:1275] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 4 FOLLOWER]: Refusing update from remote peer b822036e833c464b81cf37b24ec55c88: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:48:11.947839  8362 consensus_queue.cc:1048] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:48:11.960292  8049 raft_consensus.cc:1275] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 4 FOLLOWER]: Refusing update from remote peer b822036e833c464b81cf37b24ec55c88: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:48:11.962212  8362 consensus_queue.cc:1048] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:48:12.015717  8049 raft_consensus.cc:1275] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 4 FOLLOWER]: Refusing update from remote peer 4a640ba2733143fb9807a7675ed337e3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:48:12.017112  8356 consensus_queue.cc:1048] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:48:12.026548  7971 raft_consensus.cc:1275] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 4 FOLLOWER]: Refusing update from remote peer 4a640ba2733143fb9807a7675ed337e3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:48:12.028945  8346 consensus_queue.cc:1048] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:48:12.333966  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:12.339439  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:12.339924  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:12.500959  8049 raft_consensus.cc:1275] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 4 FOLLOWER]: Refusing update from remote peer b822036e833c464b81cf37b24ec55c88: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:48:12.502360  8362 consensus_queue.cc:1048] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:48:12.512446  8127 raft_consensus.cc:1275] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 4 FOLLOWER]: Refusing update from remote peer b822036e833c464b81cf37b24ec55c88: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:48:12.514338  8384 consensus_queue.cc:1048] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:48:13.341133  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:13.346495  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:13.347182  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:14.348356  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:14.353538  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:14.354063  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:15.355342  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:15.360574  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:15.361096  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:16.362324  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:16.367545  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:16.368070  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:17.369222  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:17.374491  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:17.375064  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:18.376341  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:18.382519  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:18.383095  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:19.384354  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:19.389910  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:19.390553  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:20.392131  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:20.398157  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:20.398749  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
W20260808 01:48:21.116789  8092 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:40863, user_credentials={real_user=slave}} blocked reactor thread for 46129.7us
I20260808 01:48:21.516978  7831 auto_leader_rebalancer-test.cc:448] The leader distribution is 
I20260808 01:48:21.517673  7831 tablet_server.cc:179] TabletServer@127.7.165.193:0 shutting down...
I20260808 01:48:21.552979  7831 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260808 01:48:21.554093  7831 tablet_replica.cc:333] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88: stopping tablet replica
I20260808 01:48:21.554996  7831 raft_consensus.cc:2243] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:21.555717  7831 raft_consensus.cc:2272] T 8064f9a1744648c4963af166d4b67cfa P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:21.558465  7831 tablet_replica.cc:333] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88: stopping tablet replica
I20260808 01:48:21.559235  7831 raft_consensus.cc:2243] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:21.559746  7831 raft_consensus.cc:2272] T 31d230e1f50b4c888c85615b35087706 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:21.561677  7831 tablet_replica.cc:333] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88: stopping tablet replica
I20260808 01:48:21.562337  7831 raft_consensus.cc:2243] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [term 4 LEADER]: Raft consensus shutting down.
I20260808 01:48:21.563336  7831 raft_consensus.cc:2272] T f98266402a6c4c82ae25f5fa9faaff51 P b822036e833c464b81cf37b24ec55c88 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:21.565544  7831 tablet_replica.cc:333] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88: stopping tablet replica
I20260808 01:48:21.566329  7831 raft_consensus.cc:2243] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 4 LEADER]: Raft consensus shutting down.
I20260808 01:48:21.567449  7831 raft_consensus.cc:2272] T 6e6647c3e5e146d185e0230bd07689e4 P b822036e833c464b81cf37b24ec55c88 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:21.569892  7831 tablet_replica.cc:333] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88: stopping tablet replica
I20260808 01:48:21.570415  7831 raft_consensus.cc:2243] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:21.570914  7831 raft_consensus.cc:2272] T cc6dc97c4b8e482683facaeb4e261128 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:21.572717  7831 tablet_replica.cc:333] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88: stopping tablet replica
I20260808 01:48:21.573343  7831 raft_consensus.cc:2243] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:21.573879  7831 raft_consensus.cc:2272] T 82541e7deca64da19bc6dd1e54b4a3c8 P b822036e833c464b81cf37b24ec55c88 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:21.575731  7831 tablet_replica.cc:333] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88: stopping tablet replica
I20260808 01:48:21.576177  7831 raft_consensus.cc:2243] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:21.576603  7831 raft_consensus.cc:2272] T 2d3db31373fa48e7a8e8a5a6e0851624 P b822036e833c464b81cf37b24ec55c88 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:21.578375  7831 tablet_replica.cc:333] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88: stopping tablet replica
I20260808 01:48:21.578974  7831 raft_consensus.cc:2243] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [term 3 LEADER]: Raft consensus shutting down.
I20260808 01:48:21.580214  7831 raft_consensus.cc:2272] T 68c2bac20b264a42adee938c783e6102 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260808 01:48:21.582036  8092 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:48:21.583129  7831 tablet_replica.cc:333] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88: stopping tablet replica
I20260808 01:48:21.583856  7831 raft_consensus.cc:2243] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:21.584676  7831 raft_consensus.cc:2272] T f441f7e95cba45ed9cb394a38cb4b4a2 P b822036e833c464b81cf37b24ec55c88 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260808 01:48:21.586428  8092 consensus_peers.cc:597] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 -> Peer b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863): Couldn't send request to peer b822036e833c464b81cf37b24ec55c88. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:21.611253  8014 consensus_peers.cc:597] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b -> Peer b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863): Couldn't send request to peer b822036e833c464b81cf37b24ec55c88. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:21.737015  8014 consensus_peers.cc:597] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b -> Peer b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863): Couldn't send request to peer b822036e833c464b81cf37b24ec55c88. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:40863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:21.737286  7831 tablet_server.cc:196] TabletServer@127.7.165.193:0 shutdown complete.
I20260808 01:48:21.761101  7831 tablet_server.cc:179] TabletServer@127.7.165.194:0 shutting down...
I20260808 01:48:21.795302  7831 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260808 01:48:21.796074  7831 tablet_replica.cc:333] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b: stopping tablet replica
I20260808 01:48:21.796782  7831 raft_consensus.cc:2243] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b [term 2 LEADER]: Raft consensus shutting down.
I20260808 01:48:21.797672  7831 raft_consensus.cc:2272] T 8064f9a1744648c4963af166d4b67cfa P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:21.799606  7831 tablet_replica.cc:333] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b: stopping tablet replica
I20260808 01:48:21.800163  7831 raft_consensus.cc:2243] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 LEADER]: Raft consensus shutting down.
I20260808 01:48:21.800942  7831 raft_consensus.cc:2272] T 31d230e1f50b4c888c85615b35087706 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:21.803079  7831 tablet_replica.cc:333] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b: stopping tablet replica
I20260808 01:48:21.803798  7831 raft_consensus.cc:2243] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:21.804363  7831 raft_consensus.cc:2272] T f98266402a6c4c82ae25f5fa9faaff51 P 282686b2e6f746d3b2eb16cf77647c1b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:21.806561  7831 tablet_replica.cc:333] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b: stopping tablet replica
I20260808 01:48:21.807109  7831 raft_consensus.cc:2243] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:21.807476  7831 raft_consensus.cc:2272] T 2d3db31373fa48e7a8e8a5a6e0851624 P 282686b2e6f746d3b2eb16cf77647c1b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:21.809114  7831 tablet_replica.cc:333] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b: stopping tablet replica
I20260808 01:48:21.809587  7831 raft_consensus.cc:2243] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:21.809988  7831 raft_consensus.cc:2272] T 6e6647c3e5e146d185e0230bd07689e4 P 282686b2e6f746d3b2eb16cf77647c1b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:21.811990  7831 tablet_replica.cc:333] T 82541e7deca64da19bc6dd1e54b4a3c8 P 282686b2e6f746d3b2eb16cf77647c1b: stopping tablet replica
I20260808 01:48:21.812496  7831 raft_consensus.cc:2243] T 82541e7deca64da19bc6dd1e54b4a3c8 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:21.812909  7831 raft_consensus.cc:2272] T 82541e7deca64da19bc6dd1e54b4a3c8 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:21.814639  7831 tablet_replica.cc:333] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b: stopping tablet replica
I20260808 01:48:21.815224  7831 raft_consensus.cc:2243] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 LEADER]: Raft consensus shutting down.
I20260808 01:48:21.815955  7831 raft_consensus.cc:2272] T cc6dc97c4b8e482683facaeb4e261128 P 282686b2e6f746d3b2eb16cf77647c1b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:21.817917  7831 tablet_replica.cc:333] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b: stopping tablet replica
I20260808 01:48:21.818377  7831 raft_consensus.cc:2243] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:21.818831  7831 raft_consensus.cc:2272] T f441f7e95cba45ed9cb394a38cb4b4a2 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:21.821077  7831 tablet_replica.cc:333] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b: stopping tablet replica
I20260808 01:48:21.821707  7831 raft_consensus.cc:2243] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:21.822268  7831 raft_consensus.cc:2272] T 68c2bac20b264a42adee938c783e6102 P 282686b2e6f746d3b2eb16cf77647c1b [term 3 FOLLOWER]: Raft consensus is shut down!
W20260808 01:48:21.823549  8089 consensus_peers.cc:597] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 -> Peer 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507): Couldn't send request to peer 282686b2e6f746d3b2eb16cf77647c1b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:21.842090  8092 consensus_peers.cc:597] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 -> Peer b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863): Couldn't send request to peer b822036e833c464b81cf37b24ec55c88. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:40863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:21.875696  8092 consensus_peers.cc:597] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 -> Peer b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863): Couldn't send request to peer b822036e833c464b81cf37b24ec55c88. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:40863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:22.087319  8070 debug-util.cc:398] Leaking SignalData structure 0x7b0800126700 after lost signal to thread 7834
W20260808 01:48:22.088737  8070 debug-util.cc:398] Leaking SignalData structure 0x7b0800126de0 after lost signal to thread 7899
W20260808 01:48:22.091511  8070 debug-util.cc:398] Leaking SignalData structure 0x7b08000a7280 after lost signal to thread 8154
W20260808 01:48:22.208160  8089 consensus_peers.cc:597] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 -> Peer 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507): Couldn't send request to peer 282686b2e6f746d3b2eb16cf77647c1b. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:34507: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:22.248980  8089 consensus_peers.cc:597] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 -> Peer 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507): Couldn't send request to peer 282686b2e6f746d3b2eb16cf77647c1b. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:34507: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:22.852885  7831 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 8070)
I20260808 01:48:22.908056  8406 raft_consensus.cc:493] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b822036e833c464b81cf37b24ec55c88)
I20260808 01:48:22.908805  8406 raft_consensus.cc:515] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:22.911849  8406 leader_election.cc:290] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507)
W20260808 01:48:22.918128  8092 leader_election.cc:336] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863): Network error: Client connection negotiation failed: client connection to 127.7.165.193:40863: connect: Connection refused (error 111)
W20260808 01:48:22.922483  8089 leader_election.cc:336] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507): Network error: Client connection negotiation failed: client connection to 127.7.165.194:34507: connect: Connection refused (error 111)
I20260808 01:48:22.922955  8089 leader_election.cc:304] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3; no voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88
I20260808 01:48:22.923691  8406 raft_consensus.cc:2749] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:48:22.987289  8406 raft_consensus.cc:493] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b822036e833c464b81cf37b24ec55c88)
I20260808 01:48:22.987952  8406 raft_consensus.cc:515] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:22.990474  8406 leader_election.cc:290] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507)
W20260808 01:48:22.995524  8089 leader_election.cc:336] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507): Network error: Client connection negotiation failed: client connection to 127.7.165.194:34507: connect: Connection refused (error 111)
W20260808 01:48:22.996208  8092 leader_election.cc:336] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863): Network error: Client connection negotiation failed: client connection to 127.7.165.193:40863: connect: Connection refused (error 111)
I20260808 01:48:22.996600  8092 leader_election.cc:304] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3; no voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88
I20260808 01:48:22.997282  8406 raft_consensus.cc:2749] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:48:23.027385  8406 raft_consensus.cc:493] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 282686b2e6f746d3b2eb16cf77647c1b)
I20260808 01:48:23.028051  8406 raft_consensus.cc:515] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:23.030572  8406 leader_election.cc:290] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507)
W20260808 01:48:23.035245  8092 leader_election.cc:336] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863): Network error: Client connection negotiation failed: client connection to 127.7.165.193:40863: connect: Connection refused (error 111)
W20260808 01:48:23.035837  8089 leader_election.cc:336] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507): Network error: Client connection negotiation failed: client connection to 127.7.165.194:34507: connect: Connection refused (error 111)
I20260808 01:48:23.036213  8089 leader_election.cc:304] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3; no voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88
I20260808 01:48:23.036885  8406 raft_consensus.cc:2749] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:48:23.068296  8406 raft_consensus.cc:493] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b822036e833c464b81cf37b24ec55c88)
I20260808 01:48:23.068912  8406 raft_consensus.cc:515] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:23.071273  8406 leader_election.cc:290] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507)
W20260808 01:48:23.075773  8092 leader_election.cc:336] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863): Network error: Client connection negotiation failed: client connection to 127.7.165.193:40863: connect: Connection refused (error 111)
W20260808 01:48:23.076332  8089 leader_election.cc:336] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507): Network error: Client connection negotiation failed: client connection to 127.7.165.194:34507: connect: Connection refused (error 111)
I20260808 01:48:23.076711  8089 leader_election.cc:304] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3; no voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88
I20260808 01:48:23.077351  8406 raft_consensus.cc:2749] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:48:23.108808  8406 raft_consensus.cc:493] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 282686b2e6f746d3b2eb16cf77647c1b)
I20260808 01:48:23.109545  8406 raft_consensus.cc:515] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:23.112131  8406 leader_election.cc:290] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507)
W20260808 01:48:23.117134  8092 leader_election.cc:336] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863): Network error: Client connection negotiation failed: client connection to 127.7.165.193:40863: connect: Connection refused (error 111)
W20260808 01:48:23.117780  8089 leader_election.cc:336] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507): Network error: Client connection negotiation failed: client connection to 127.7.165.194:34507: connect: Connection refused (error 111)
I20260808 01:48:23.118152  8089 leader_election.cc:304] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3; no voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88
I20260808 01:48:23.118829  8406 raft_consensus.cc:2749] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:48:23.181597  8406 raft_consensus.cc:493] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 282686b2e6f746d3b2eb16cf77647c1b)
I20260808 01:48:23.182240  8406 raft_consensus.cc:515] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b822036e833c464b81cf37b24ec55c88" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 40863 } } peers { permanent_uuid: "282686b2e6f746d3b2eb16cf77647c1b" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 34507 } } peers { permanent_uuid: "4a640ba2733143fb9807a7675ed337e3" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 36653 } }
I20260808 01:48:23.184841  8406 leader_election.cc:290] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863), 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507)
W20260808 01:48:23.189870  8092 leader_election.cc:336] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863): Network error: Client connection negotiation failed: client connection to 127.7.165.193:40863: connect: Connection refused (error 111)
W20260808 01:48:23.190500  8089 leader_election.cc:336] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507): Network error: Client connection negotiation failed: client connection to 127.7.165.194:34507: connect: Connection refused (error 111)
I20260808 01:48:23.190922  8089 leader_election.cc:304] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a640ba2733143fb9807a7675ed337e3; no voters: 282686b2e6f746d3b2eb16cf77647c1b, b822036e833c464b81cf37b24ec55c88
I20260808 01:48:23.191568  8406 raft_consensus.cc:2749] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:48:23.853397  7831 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 8070)
W20260808 01:48:24.145466  8092 consensus_peers.cc:597] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 -> Peer b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863): Couldn't send request to peer b822036e833c464b81cf37b24ec55c88. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:40863: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:48:24.174229  8092 consensus_peers.cc:597] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 -> Peer b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863): Couldn't send request to peer b822036e833c464b81cf37b24ec55c88. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:40863: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:48:24.302294  8089 consensus_peers.cc:597] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 -> Peer 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507): Couldn't send request to peer 282686b2e6f746d3b2eb16cf77647c1b. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:34507: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:48:24.335522  8092 consensus_peers.cc:597] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 -> Peer b822036e833c464b81cf37b24ec55c88 (127.7.165.193:40863): Couldn't send request to peer b822036e833c464b81cf37b24ec55c88. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:40863: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:48:24.570608  8089 consensus_peers.cc:597] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 -> Peer 282686b2e6f746d3b2eb16cf77647c1b (127.7.165.194:34507): Couldn't send request to peer 282686b2e6f746d3b2eb16cf77647c1b. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:34507: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:48:24.717669  7831 tablet_server.cc:196] TabletServer@127.7.165.194:0 shutdown complete.
I20260808 01:48:24.742103  7831 tablet_server.cc:179] TabletServer@127.7.165.195:0 shutting down...
I20260808 01:48:24.781693  7831 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260808 01:48:24.783218  7831 tablet_replica.cc:333] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3: stopping tablet replica
I20260808 01:48:24.784382  7831 raft_consensus.cc:2243] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:24.785233  7831 raft_consensus.cc:2272] T 31d230e1f50b4c888c85615b35087706 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:24.787670  7831 tablet_replica.cc:333] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3: stopping tablet replica
I20260808 01:48:24.788304  7831 raft_consensus.cc:2243] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:24.788967  7831 raft_consensus.cc:2272] T f98266402a6c4c82ae25f5fa9faaff51 P 4a640ba2733143fb9807a7675ed337e3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:24.790814  7831 tablet_replica.cc:333] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3: stopping tablet replica
I20260808 01:48:24.791393  7831 raft_consensus.cc:2243] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:24.791978  7831 raft_consensus.cc:2272] T 68c2bac20b264a42adee938c783e6102 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:24.794268  7831 tablet_replica.cc:333] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3: stopping tablet replica
I20260808 01:48:24.794931  7831 raft_consensus.cc:2243] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 3 LEADER]: Raft consensus shutting down.
I20260808 01:48:24.795806  7831 raft_consensus.cc:2272] T f441f7e95cba45ed9cb394a38cb4b4a2 P 4a640ba2733143fb9807a7675ed337e3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:24.797868  7831 tablet_replica.cc:333] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3: stopping tablet replica
I20260808 01:48:24.798455  7831 raft_consensus.cc:2243] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 [term 2 LEADER]: Raft consensus shutting down.
I20260808 01:48:24.799311  7831 raft_consensus.cc:2272] T 82541e7deca64da19bc6dd1e54b4a3c8 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:24.801170  7831 tablet_replica.cc:333] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3: stopping tablet replica
I20260808 01:48:24.801713  7831 raft_consensus.cc:2243] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:24.802379  7831 raft_consensus.cc:2272] T 6e6647c3e5e146d185e0230bd07689e4 P 4a640ba2733143fb9807a7675ed337e3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:24.804639  7831 tablet_replica.cc:333] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3: stopping tablet replica
I20260808 01:48:24.805223  7831 raft_consensus.cc:2243] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 4 LEADER]: Raft consensus shutting down.
I20260808 01:48:24.806293  7831 raft_consensus.cc:2272] T 2d3db31373fa48e7a8e8a5a6e0851624 P 4a640ba2733143fb9807a7675ed337e3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:24.808321  7831 tablet_replica.cc:333] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3: stopping tablet replica
I20260808 01:48:24.808872  7831 raft_consensus.cc:2243] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:24.809362  7831 raft_consensus.cc:2272] T cc6dc97c4b8e482683facaeb4e261128 P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:24.811539  7831 tablet_replica.cc:333] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3: stopping tablet replica
I20260808 01:48:24.812083  7831 raft_consensus.cc:2243] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:24.812656  7831 raft_consensus.cc:2272] T 8064f9a1744648c4963af166d4b67cfa P 4a640ba2733143fb9807a7675ed337e3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:24.844738  7831 tablet_server.cc:196] TabletServer@127.7.165.195:0 shutdown complete.
I20260808 01:48:24.867441  7831 master.cc:562] Master@127.7.165.254:36421 shutting down...
I20260808 01:48:25.109445  7831 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [term 1 LEADER]: Raft consensus shutting down.
I20260808 01:48:25.110232  7831 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:25.110790  7831 tablet_replica.cc:333] T 00000000000000000000000000000000 P 45f1bafd07d4419b8e68b4d0d13a6759: stopping tablet replica
I20260808 01:48:25.134459  7831 master.cc:584] Master@127.7.165.254:36421 shutdown complete.
282686b2e6f746d3b2eb16cf77647c1b  4
4a640ba2733143fb9807a7675ed337e3  1
b822036e833c464b81cf37b24ec55c88  4
282686b2e6f746d3b2eb16cf77647c1b  3
4a640ba2733143fb9807a7675ed337e3  3
b822036e833c464b81cf37b24ec55c88  3
282686b2e6f746d3b2eb16cf77647c1b  8
4a640ba2733143fb9807a7675ed337e3  1
b822036e833c464b81cf37b24ec55c88  0
282686b2e6f746d3b2eb16cf77647c1b  3
4a640ba2733143fb9807a7675ed337e3  3
b822036e833c464b81cf37b24ec55c88  3
[       OK ] LeaderRebalancerTest.FunctionalTestForDivided (35325 ms)
[ RUN      ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260808 01:48:25.189816  7831 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.7.165.254:45413
I20260808 01:48:25.191428  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:25.198268  8411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:25.198586  8412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:25.199343  8414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:25.199740  7831 server_base.cc:1061] running on GCE node
I20260808 01:48:25.201247  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:48:25.201449  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:48:25.201572  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786153705201560 us; error 0 us; skew 500 ppm
I20260808 01:48:25.202149  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:25.204818  7831 webserver.cc:533] Webserver started at http://127.7.165.254:42207/ using document root <none> and password file <none>
I20260808 01:48:25.205315  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:25.205498  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:25.205735  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:25.207111  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/master-0-root/instance:
uuid: "289bf78aad494e20a2a7447e705689ec"
format_stamp: "Formatted at 2026-08-08 01:48:25 on dist-test-slave-8tdl"
I20260808 01:48:25.212839  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.001s
I20260808 01:48:25.218020  8419 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:25.219084  7831 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260808 01:48:25.219429  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/master-0-root
uuid: "289bf78aad494e20a2a7447e705689ec"
format_stamp: "Formatted at 2026-08-08 01:48:25 on dist-test-slave-8tdl"
I20260808 01:48:25.219782  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:25.249688  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:48:25.251283  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:25.296420  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.254:45413
I20260808 01:48:25.296542  8474 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.254:45413 every 8 connection(s)
I20260808 01:48:25.301091  8475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:25.311609  8475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec: Bootstrap starting.
I20260808 01:48:25.316749  8475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:25.322357  8475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec: No bootstrap required, opened a new log
I20260808 01:48:25.324939  8475 raft_consensus.cc:359] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289bf78aad494e20a2a7447e705689ec" member_type: VOTER }
I20260808 01:48:25.325626  8475 raft_consensus.cc:385] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:25.325896  8475 raft_consensus.cc:740] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289bf78aad494e20a2a7447e705689ec, State: Initialized, Role: FOLLOWER
I20260808 01:48:25.326560  8475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289bf78aad494e20a2a7447e705689ec" member_type: VOTER }
I20260808 01:48:25.327095  8475 raft_consensus.cc:399] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 01:48:25.327346  8475 raft_consensus.cc:493] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 01:48:25.327644  8475 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:25.332618  8475 raft_consensus.cc:515] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289bf78aad494e20a2a7447e705689ec" member_type: VOTER }
I20260808 01:48:25.333319  8475 leader_election.cc:304] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 289bf78aad494e20a2a7447e705689ec; no voters: 
I20260808 01:48:25.335352  8475 leader_election.cc:290] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260808 01:48:25.335784  8478 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:25.337666  8478 raft_consensus.cc:697] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [term 1 LEADER]: Becoming Leader. State: Replica: 289bf78aad494e20a2a7447e705689ec, State: Running, Role: LEADER
I20260808 01:48:25.338626  8478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289bf78aad494e20a2a7447e705689ec" member_type: VOTER }
I20260808 01:48:25.339565  8475 sys_catalog.cc:565] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [sys.catalog]: configured and running, proceeding with master startup.
I20260808 01:48:25.344831  8481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 289bf78aad494e20a2a7447e705689ec. Latest consensus state: current_term: 1 leader_uuid: "289bf78aad494e20a2a7447e705689ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289bf78aad494e20a2a7447e705689ec" member_type: VOTER } }
I20260808 01:48:25.344831  8479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "289bf78aad494e20a2a7447e705689ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289bf78aad494e20a2a7447e705689ec" member_type: VOTER } }
I20260808 01:48:25.345709  8481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [sys.catalog]: This master's current role is: LEADER
I20260808 01:48:25.345734  8479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [sys.catalog]: This master's current role is: LEADER
I20260808 01:48:25.361158  7831 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
W20260808 01:48:25.366325  8495 catalog_manager.cc:1594] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 01:48:25.366801  8495 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 01:48:25.367209  8487 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260808 01:48:25.375099  8487 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260808 01:48:25.389377  8487 catalog_manager.cc:1383] Generated new cluster ID: b2618789e8e34153a6850e67341f65e9
I20260808 01:48:25.389772  8487 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260808 01:48:25.413070  8487 catalog_manager.cc:1406] Generated new certificate authority record
I20260808 01:48:25.414528  8487 catalog_manager.cc:1540] Loading token signing keys...
I20260808 01:48:25.435340  8487 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec: Generated new TSK 0
I20260808 01:48:25.436051  8487 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260808 01:48:25.492911  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:25.499811  8497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:25.500830  8498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:25.503849  8500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:25.504102  7831 server_base.cc:1061] running on GCE node
I20260808 01:48:25.505311  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:48:25.505513  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:48:25.505707  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786153705505693 us; error 0 us; skew 500 ppm
I20260808 01:48:25.506289  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:25.508852  7831 webserver.cc:533] Webserver started at http://127.7.165.193:34807/ using document root <none> and password file <none>
I20260808 01:48:25.509433  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:25.509630  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:25.509925  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:25.511217  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/ts-0-root/instance:
uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
format_stamp: "Formatted at 2026-08-08 01:48:25 on dist-test-slave-8tdl"
I20260808 01:48:25.516407  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260808 01:48:25.520990  8505 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:25.521986  7831 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260808 01:48:25.522406  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/ts-0-root
uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
format_stamp: "Formatted at 2026-08-08 01:48:25 on dist-test-slave-8tdl"
I20260808 01:48:25.522765  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:25.537972  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:48:25.539686  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:25.541431  7831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:25.544087  7831 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:25.544291  7831 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:25.544603  7831 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:25.544792  7831 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:25.593744  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.193:37659
I20260808 01:48:25.593840  8569 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.193:37659 every 8 connection(s)
I20260808 01:48:25.599704  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:25.608794  8574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:25.609648  8575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:25.613122  8579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:25.614427  7831 server_base.cc:1061] running on GCE node
I20260808 01:48:25.616712  8570 heartbeater.cc:344] Connected to a master server at 127.7.165.254:45413
I20260808 01:48:25.617033  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260808 01:48:25.617182  8570 heartbeater.cc:461] Registering TS with master...
W20260808 01:48:25.617323  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:48:25.617547  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786153705617527 us; error 0 us; skew 500 ppm
I20260808 01:48:25.618142  8570 heartbeater.cc:507] Master 127.7.165.254:45413 requested a full tablet report, sending...
I20260808 01:48:25.618347  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:25.620955  8437 ts_manager.cc:194] Registered new tserver with Master: d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659)
I20260808 01:48:25.621729  7831 webserver.cc:533] Webserver started at http://127.7.165.194:32881/ using document root <none> and password file <none>
I20260808 01:48:25.622309  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:25.622581  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:25.622936  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:25.623271  8437 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58396
I20260808 01:48:25.624565  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/ts-1-root/instance:
uuid: "593c8fd64bf14a9996d36e2145636220"
format_stamp: "Formatted at 2026-08-08 01:48:25 on dist-test-slave-8tdl"
I20260808 01:48:25.630733  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260808 01:48:25.635157  8584 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:25.636152  7831 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260808 01:48:25.636507  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/ts-1-root
uuid: "593c8fd64bf14a9996d36e2145636220"
format_stamp: "Formatted at 2026-08-08 01:48:25 on dist-test-slave-8tdl"
I20260808 01:48:25.636806  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:25.653556  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:48:25.655150  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:25.656898  7831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:25.659461  7831 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:25.659727  7831 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:25.659989  7831 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:25.660179  7831 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:25.707895  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.194:32883
I20260808 01:48:25.708057  8647 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.194:32883 every 8 connection(s)
I20260808 01:48:25.713375  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:25.723852  8652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:25.726220  8651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:25.728725  8654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:25.731160  7831 server_base.cc:1061] running on GCE node
I20260808 01:48:25.732309  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:48:25.732578  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:48:25.732805  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786153705732768 us; error 0 us; skew 500 ppm
I20260808 01:48:25.733294  8648 heartbeater.cc:344] Connected to a master server at 127.7.165.254:45413
I20260808 01:48:25.733625  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:25.733754  8648 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:25.734807  8648 heartbeater.cc:507] Master 127.7.165.254:45413 requested a full tablet report, sending...
I20260808 01:48:25.736785  7831 webserver.cc:533] Webserver started at http://127.7.165.195:39805/ using document root <none> and password file <none>
I20260808 01:48:25.737391  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:25.737577  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:25.737519  8437 ts_manager.cc:194] Registered new tserver with Master: 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883)
I20260808 01:48:25.738126  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:25.739295  8437 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58404
I20260808 01:48:25.739475  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/ts-2-root/instance:
uuid: "96823ad418f9403cbaaac1ddff4834b2"
format_stamp: "Formatted at 2026-08-08 01:48:25 on dist-test-slave-8tdl"
I20260808 01:48:25.745047  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260808 01:48:25.748905  8659 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:25.750026  7831 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260808 01:48:25.750306  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/ts-2-root
uuid: "96823ad418f9403cbaaac1ddff4834b2"
format_stamp: "Formatted at 2026-08-08 01:48:25 on dist-test-slave-8tdl"
I20260808 01:48:25.750577  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786153669593643-7831-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:25.771095  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:48:25.772488  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:25.774530  7831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:25.777213  7831 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:25.777433  7831 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:25.778005  7831 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:25.778235  7831 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s	user 0.000s	sys 0.000s
I20260808 01:48:25.827525  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.195:39507
I20260808 01:48:25.827665  8725 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.195:39507 every 8 connection(s)
I20260808 01:48:25.845150  8726 heartbeater.cc:344] Connected to a master server at 127.7.165.254:45413
I20260808 01:48:25.845602  8726 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:25.846410  8726 heartbeater.cc:507] Master 127.7.165.254:45413 requested a full tablet report, sending...
I20260808 01:48:25.848803  8437 ts_manager.cc:194] Registered new tserver with Master: 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:25.849287  7831 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017858532s
I20260808 01:48:25.850422  8437 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58408
I20260808 01:48:25.877588  8437 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58412:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260808 01:48:25.881438  8437 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 01:48:25.982906  8613 tablet_service.cc:1511] Processing CreateTablet for tablet 490b233715cf4853916dd7273575b3a0 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260808 01:48:25.984756  8613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 490b233715cf4853916dd7273575b3a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:25.991868  8612 tablet_service.cc:1511] Processing CreateTablet for tablet 537921115d13455d98e32f8ad895e87a (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260808 01:48:25.993496  8612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 537921115d13455d98e32f8ad895e87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:25.995572  8534 tablet_service.cc:1511] Processing CreateTablet for tablet 490b233715cf4853916dd7273575b3a0 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260808 01:48:25.995879  8611 tablet_service.cc:1511] Processing CreateTablet for tablet 20f572798f1646acb92966b083e07eb2 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260808 01:48:25.996701  8533 tablet_service.cc:1511] Processing CreateTablet for tablet 537921115d13455d98e32f8ad895e87a (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260808 01:48:25.997327  8534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 490b233715cf4853916dd7273575b3a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:25.998124  8533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 537921115d13455d98e32f8ad895e87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:25.998657  8610 tablet_service.cc:1511] Processing CreateTablet for tablet 7e38a43fd7704fe8bec9648856983350 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260808 01:48:25.999539  8688 tablet_service.cc:1511] Processing CreateTablet for tablet 490b233715cf4853916dd7273575b3a0 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260808 01:48:26.001124  8688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 490b233715cf4853916dd7273575b3a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.002039  8609 tablet_service.cc:1511] Processing CreateTablet for tablet e61a690621224f02a28c606e3228c838 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260808 01:48:26.004122  8687 tablet_service.cc:1511] Processing CreateTablet for tablet 537921115d13455d98e32f8ad895e87a (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260808 01:48:26.006089  8687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 537921115d13455d98e32f8ad895e87a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.008068  8686 tablet_service.cc:1511] Processing CreateTablet for tablet 20f572798f1646acb92966b083e07eb2 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260808 01:48:26.009856  8686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20f572798f1646acb92966b083e07eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.012212  8608 tablet_service.cc:1511] Processing CreateTablet for tablet cc880f9cc7844dae9f308ef3c2b9bbdb (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260808 01:48:26.015260  8532 tablet_service.cc:1511] Processing CreateTablet for tablet 20f572798f1646acb92966b083e07eb2 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260808 01:48:26.016870  8532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20f572798f1646acb92966b083e07eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.016582  8683 tablet_service.cc:1511] Processing CreateTablet for tablet cc880f9cc7844dae9f308ef3c2b9bbdb (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260808 01:48:26.012212  8685 tablet_service.cc:1511] Processing CreateTablet for tablet 7e38a43fd7704fe8bec9648856983350 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260808 01:48:26.019922  8684 tablet_service.cc:1511] Processing CreateTablet for tablet e61a690621224f02a28c606e3228c838 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260808 01:48:26.022359  8683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc880f9cc7844dae9f308ef3c2b9bbdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.023247  8685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e38a43fd7704fe8bec9648856983350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:25.997359  8611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20f572798f1646acb92966b083e07eb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.027055  8610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e38a43fd7704fe8bec9648856983350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.029487  8531 tablet_service.cc:1511] Processing CreateTablet for tablet 7e38a43fd7704fe8bec9648856983350 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260808 01:48:26.031229  8608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc880f9cc7844dae9f308ef3c2b9bbdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.032081  8530 tablet_service.cc:1511] Processing CreateTablet for tablet e61a690621224f02a28c606e3228c838 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260808 01:48:26.035403  8741 tablet_bootstrap.cc:492] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220: Bootstrap starting.
I20260808 01:48:26.036152  8684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e61a690621224f02a28c606e3228c838. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.035871  8529 tablet_service.cc:1511] Processing CreateTablet for tablet cc880f9cc7844dae9f308ef3c2b9bbdb (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260808 01:48:26.037985  8531 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e38a43fd7704fe8bec9648856983350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.042045  8530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e61a690621224f02a28c606e3228c838. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.042642  8743 tablet_bootstrap.cc:492] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2: Bootstrap starting.
I20260808 01:48:26.051316  8529 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc880f9cc7844dae9f308ef3c2b9bbdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.043810  8741 tablet_bootstrap.cc:654] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.057133  8609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e61a690621224f02a28c606e3228c838. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.057942  8743 tablet_bootstrap.cc:654] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.081112  8533 tablet_service.cc:1511] Processing CreateTablet for tablet d471f57bba4e49f9b0e8c29880e2a864 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260808 01:48:26.083532  8533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d471f57bba4e49f9b0e8c29880e2a864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.088605  8742 tablet_bootstrap.cc:492] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5: Bootstrap starting.
I20260808 01:48:26.103170  8686 tablet_service.cc:1511] Processing CreateTablet for tablet d471f57bba4e49f9b0e8c29880e2a864 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260808 01:48:26.105274  8686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d471f57bba4e49f9b0e8c29880e2a864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.106644  8528 tablet_service.cc:1511] Processing CreateTablet for tablet e2186a1e5b7a4dcb828dc73a73036684 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260808 01:48:26.108268  8528 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2186a1e5b7a4dcb828dc73a73036684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.116330  8607 tablet_service.cc:1511] Processing CreateTablet for tablet e2186a1e5b7a4dcb828dc73a73036684 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260808 01:48:26.119865  8607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2186a1e5b7a4dcb828dc73a73036684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.127658  8613 tablet_service.cc:1511] Processing CreateTablet for tablet d471f57bba4e49f9b0e8c29880e2a864 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260808 01:48:26.129570  8613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d471f57bba4e49f9b0e8c29880e2a864. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.132789  8742 tablet_bootstrap.cc:654] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.143488  8741 tablet_bootstrap.cc:492] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220: No bootstrap required, opened a new log
I20260808 01:48:26.144212  8741 ts_tablet_manager.cc:1403] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220: Time spent bootstrapping tablet: real 0.109s	user 0.025s	sys 0.016s
I20260808 01:48:26.172103  8685 tablet_service.cc:1511] Processing CreateTablet for tablet e2186a1e5b7a4dcb828dc73a73036684 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260808 01:48:26.186115  8685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2186a1e5b7a4dcb828dc73a73036684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.192596  8741 raft_consensus.cc:359] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.193442  8741 raft_consensus.cc:385] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.193781  8741 raft_consensus.cc:740] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.194780  8742 tablet_bootstrap.cc:492] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5: No bootstrap required, opened a new log
I20260808 01:48:26.195333  8742 ts_tablet_manager.cc:1403] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent bootstrapping tablet: real 0.107s	user 0.018s	sys 0.035s
I20260808 01:48:26.197974  8743 tablet_bootstrap.cc:492] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2: No bootstrap required, opened a new log
I20260808 01:48:26.204975  8743 ts_tablet_manager.cc:1403] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2: Time spent bootstrapping tablet: real 0.163s	user 0.017s	sys 0.006s
I20260808 01:48:26.206461  8741 consensus_queue.cc:260] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.211489  8743 raft_consensus.cc:359] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.216485  8743 raft_consensus.cc:385] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.216981  8743 raft_consensus.cc:740] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.218225  8743 consensus_queue.cc:260] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.218434  8610 tablet_service.cc:1511] Processing CreateTablet for tablet f390638b5fbd41b686f3fa99a5898f77 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260808 01:48:26.222554  8532 tablet_service.cc:1511] Processing CreateTablet for tablet f390638b5fbd41b686f3fa99a5898f77 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260808 01:48:26.223752  8688 tablet_service.cc:1511] Processing CreateTablet for tablet f390638b5fbd41b686f3fa99a5898f77 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260808 01:48:26.226862  8529 tablet_service.cc:1511] Processing CreateTablet for tablet 598364b36429453793bb8922ded6a556 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260808 01:48:26.227275  8611 tablet_service.cc:1511] Processing CreateTablet for tablet 598364b36429453793bb8922ded6a556 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260808 01:48:26.230145  8688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f390638b5fbd41b686f3fa99a5898f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.230209  8529 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 598364b36429453793bb8922ded6a556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.232873  8742 raft_consensus.cc:359] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.234375  8742 raft_consensus.cc:385] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.235014  8742 raft_consensus.cc:740] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.236066  8742 consensus_queue.cc:260] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.233040  8610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f390638b5fbd41b686f3fa99a5898f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.241982  8532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f390638b5fbd41b686f3fa99a5898f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.246583  8611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 598364b36429453793bb8922ded6a556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.247216  8726 heartbeater.cc:499] Master 127.7.165.254:45413 was elected leader, sending a full tablet report...
I20260808 01:48:26.254729  8741 ts_tablet_manager.cc:1434] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220: Time spent starting tablet: real 0.110s	user 0.018s	sys 0.001s
I20260808 01:48:26.255993  8648 heartbeater.cc:499] Master 127.7.165.254:45413 was elected leader, sending a full tablet report...
I20260808 01:48:26.260615  8741 tablet_bootstrap.cc:492] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220: Bootstrap starting.
I20260808 01:48:26.267803  8685 tablet_service.cc:1511] Processing CreateTablet for tablet 598364b36429453793bb8922ded6a556 (DEFAULT_TABLE table=test-workload [id=fde7544d4e504faf9a39e8fc80aabf2d]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260808 01:48:26.269094  8685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 598364b36429453793bb8922ded6a556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:26.271347  8742 ts_tablet_manager.cc:1434] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent starting tablet: real 0.076s	user 0.011s	sys 0.000s
I20260808 01:48:26.272332  8570 heartbeater.cc:499] Master 127.7.165.254:45413 was elected leader, sending a full tablet report...
I20260808 01:48:26.272836  8742 tablet_bootstrap.cc:492] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5: Bootstrap starting.
I20260808 01:48:26.283905  8743 ts_tablet_manager.cc:1434] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2: Time spent starting tablet: real 0.078s	user 0.011s	sys 0.000s
I20260808 01:48:26.285269  8743 tablet_bootstrap.cc:492] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2: Bootstrap starting.
I20260808 01:48:26.292232  8743 tablet_bootstrap.cc:654] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.302809  8742 tablet_bootstrap.cc:654] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.305786  8741 tablet_bootstrap.cc:654] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.320658  8742 tablet_bootstrap.cc:492] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5: No bootstrap required, opened a new log
I20260808 01:48:26.321244  8742 ts_tablet_manager.cc:1403] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent bootstrapping tablet: real 0.049s	user 0.008s	sys 0.004s
I20260808 01:48:26.321393  8741 tablet_bootstrap.cc:492] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220: No bootstrap required, opened a new log
I20260808 01:48:26.321965  8741 ts_tablet_manager.cc:1403] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220: Time spent bootstrapping tablet: real 0.062s	user 0.014s	sys 0.004s
I20260808 01:48:26.324575  8742 raft_consensus.cc:359] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.324869  8741 raft_consensus.cc:359] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.325430  8742 raft_consensus.cc:385] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.325575  8741 raft_consensus.cc:385] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.325788  8742 raft_consensus.cc:740] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.325901  8741 raft_consensus.cc:740] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.326649  8741 consensus_queue.cc:260] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.326645  8742 consensus_queue.cc:260] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.329128  8742 ts_tablet_manager.cc:1434] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.002s
I20260808 01:48:26.330199  8742 tablet_bootstrap.cc:492] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5: Bootstrap starting.
I20260808 01:48:26.329484  8743 tablet_bootstrap.cc:492] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2: No bootstrap required, opened a new log
I20260808 01:48:26.331022  8741 ts_tablet_manager.cc:1434] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.001s
I20260808 01:48:26.331260  8743 ts_tablet_manager.cc:1403] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2: Time spent bootstrapping tablet: real 0.046s	user 0.012s	sys 0.004s
I20260808 01:48:26.332073  8741 tablet_bootstrap.cc:492] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220: Bootstrap starting.
W20260808 01:48:26.332954  8727 tablet.cc:2406] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:48:26.336174  8743 raft_consensus.cc:359] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.338572  8742 tablet_bootstrap.cc:654] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.340461  8743 raft_consensus.cc:385] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.339824  8741 tablet_bootstrap.cc:654] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.341060  8743 raft_consensus.cc:740] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.342919  8743 consensus_queue.cc:260] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.347343  8750 raft_consensus.cc:493] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:26.348590  8750 raft_consensus.cc:515] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.355381  8741 tablet_bootstrap.cc:492] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220: No bootstrap required, opened a new log
W20260808 01:48:26.355867  8489 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:48:26.356163  8741 ts_tablet_manager.cc:1403] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220: Time spent bootstrapping tablet: real 0.024s	user 0.013s	sys 0.004s
I20260808 01:48:26.358414  8742 tablet_bootstrap.cc:492] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5: No bootstrap required, opened a new log
I20260808 01:48:26.359156  8742 ts_tablet_manager.cc:1403] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent bootstrapping tablet: real 0.029s	user 0.013s	sys 0.010s
I20260808 01:48:26.346378  8743 ts_tablet_manager.cc:1434] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2: Time spent starting tablet: real 0.015s	user 0.003s	sys 0.003s
I20260808 01:48:26.360611  8743 tablet_bootstrap.cc:492] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2: Bootstrap starting.
I20260808 01:48:26.359074  8750 leader_election.cc:290] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:26.362941  8742 raft_consensus.cc:359] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.363824  8742 raft_consensus.cc:385] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.363886  8749 raft_consensus.cc:493] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:26.364174  8742 raft_consensus.cc:740] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.364881  8749 raft_consensus.cc:515] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.365694  8742 consensus_queue.cc:260] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.369230  8742 ts_tablet_manager.cc:1434] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260808 01:48:26.370478  8742 tablet_bootstrap.cc:492] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5: Bootstrap starting.
I20260808 01:48:26.373278  8743 tablet_bootstrap.cc:654] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.375005  8748 raft_consensus.cc:493] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:26.375769  8748 raft_consensus.cc:515] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.378006  8742 tablet_bootstrap.cc:654] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.379534  8749 leader_election.cc:290] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883)
I20260808 01:48:26.391409  8748 leader_election.cc:290] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:26.358824  8741 raft_consensus.cc:359] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.403079  8741 raft_consensus.cc:385] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.403527  8741 raft_consensus.cc:740] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.406272  8741 consensus_queue.cc:260] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.426107  8741 ts_tablet_manager.cc:1434] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220: Time spent starting tablet: real 0.069s	user 0.007s	sys 0.000s
I20260808 01:48:26.427237  8741 tablet_bootstrap.cc:492] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220: Bootstrap starting.
I20260808 01:48:26.434307  8748 raft_consensus.cc:493] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:26.434953  8748 raft_consensus.cc:515] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.437247  8748 leader_election.cc:290] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:26.468098  8699 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "537921115d13455d98e32f8ad895e87a" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96823ad418f9403cbaaac1ddff4834b2" is_pre_election: true
I20260808 01:48:26.470568  8699 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e61a690621224f02a28c606e3228c838" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96823ad418f9403cbaaac1ddff4834b2" is_pre_election: true
W20260808 01:48:26.470783  8587 leader_election.cc:343] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:48:26.472556  8587 leader_election.cc:343] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:26.473726  8544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "537921115d13455d98e32f8ad895e87a" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" is_pre_election: true
I20260808 01:48:26.474089  8543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e61a690621224f02a28c606e3228c838" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" is_pre_election: true
I20260808 01:48:26.474553  8544 raft_consensus.cc:2468] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 593c8fd64bf14a9996d36e2145636220 in term 0.
I20260808 01:48:26.474889  8543 raft_consensus.cc:2468] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 593c8fd64bf14a9996d36e2145636220 in term 0.
I20260808 01:48:26.476130  8542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e38a43fd7704fe8bec9648856983350" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" is_pre_election: true
I20260808 01:48:26.477555  8742 tablet_bootstrap.cc:492] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5: No bootstrap required, opened a new log
I20260808 01:48:26.477556  8743 tablet_bootstrap.cc:492] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2: No bootstrap required, opened a new log
I20260808 01:48:26.478232  8743 ts_tablet_manager.cc:1403] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2: Time spent bootstrapping tablet: real 0.118s	user 0.042s	sys 0.031s
I20260808 01:48:26.478879  8623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "490b233715cf4853916dd7273575b3a0" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593c8fd64bf14a9996d36e2145636220" is_pre_election: true
I20260808 01:48:26.479672  8623 raft_consensus.cc:2468] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 0.
I20260808 01:48:26.481134  8622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e38a43fd7704fe8bec9648856983350" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593c8fd64bf14a9996d36e2145636220" is_pre_election: true
I20260808 01:48:26.481925  8741 tablet_bootstrap.cc:654] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.481768  8743 raft_consensus.cc:359] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.482558  8743 raft_consensus.cc:385] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.482963  8743 raft_consensus.cc:740] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.483412  8585 leader_election.cc:304] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 96823ad418f9403cbaaac1ddff4834b2
I20260808 01:48:26.483923  8506 leader_election.cc:304] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:26.483841  8743 consensus_queue.cc:260] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.484457  8748 raft_consensus.cc:2804] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:26.484865  8748 raft_consensus.cc:493] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:26.485347  8748 raft_consensus.cc:3060] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:26.484889  8750 raft_consensus.cc:2804] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:26.487145  8743 ts_tablet_manager.cc:1434] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260808 01:48:26.487301  8750 raft_consensus.cc:493] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:26.478232  8742 ts_tablet_manager.cc:1403] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent bootstrapping tablet: real 0.108s	user 0.036s	sys 0.059s
I20260808 01:48:26.487800  8750 raft_consensus.cc:3060] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:26.492173  8742 raft_consensus.cc:359] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.493247  8742 raft_consensus.cc:385] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.493688  8742 raft_consensus.cc:740] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.494843  8742 consensus_queue.cc:260] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
W20260808 01:48:26.498131  8660 leader_election.cc:343] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:26.495630  8585 leader_election.cc:304] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 96823ad418f9403cbaaac1ddff4834b2
W20260808 01:48:26.498958  8660 leader_election.cc:343] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:26.499404  8660 leader_election.cc:304] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96823ad418f9403cbaaac1ddff4834b2; no voters: 593c8fd64bf14a9996d36e2145636220, d731ac9726624d11a9f44e9bdbbf3fa5
I20260808 01:48:26.500224  8750 raft_consensus.cc:515] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.501636  8749 raft_consensus.cc:2749] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:48:26.501750  8748 raft_consensus.cc:515] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.506172  8749 raft_consensus.cc:493] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:26.506717  8749 raft_consensus.cc:515] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.508037  8748 leader_election.cc:290] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 1 election: Requested vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:26.509094  8749 leader_election.cc:290] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883)
I20260808 01:48:26.509094  8748 raft_consensus.cc:2804] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:26.510282  8748 raft_consensus.cc:493] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:26.511154  8542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20f572798f1646acb92966b083e07eb2" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" is_pre_election: true
I20260808 01:48:26.513717  8622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20f572798f1646acb92966b083e07eb2" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593c8fd64bf14a9996d36e2145636220" is_pre_election: true
I20260808 01:48:26.515058  8542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "537921115d13455d98e32f8ad895e87a" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
I20260808 01:48:26.515679  8542 raft_consensus.cc:3060] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:48:26.515802  8660 leader_election.cc:343] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:48:26.516626  8660 leader_election.cc:343] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:26.517050  8660 leader_election.cc:304] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96823ad418f9403cbaaac1ddff4834b2; no voters: 593c8fd64bf14a9996d36e2145636220, d731ac9726624d11a9f44e9bdbbf3fa5
I20260808 01:48:26.517765  8750 leader_election.cc:290] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 election: Requested vote from peers 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:26.523905  8542 raft_consensus.cc:2468] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593c8fd64bf14a9996d36e2145636220 in term 1.
I20260808 01:48:26.525415  8622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "490b233715cf4853916dd7273575b3a0" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593c8fd64bf14a9996d36e2145636220"
I20260808 01:48:26.526127  8622 raft_consensus.cc:3060] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:26.527240  8699 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "490b233715cf4853916dd7273575b3a0" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96823ad418f9403cbaaac1ddff4834b2" is_pre_election: true
I20260808 01:48:26.527719  8697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "490b233715cf4853916dd7273575b3a0" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
W20260808 01:48:26.528908  8508 leader_election.cc:343] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:26.529325  8698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "537921115d13455d98e32f8ad895e87a" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
I20260808 01:48:26.530203  8698 raft_consensus.cc:3060] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:48:26.530381  8508 leader_election.cc:343] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:26.526147  8749 raft_consensus.cc:2749] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:48:26.532078  8743 tablet_bootstrap.cc:492] T cc880f9cc7844dae9f308ef3c2b9bbdb P 96823ad418f9403cbaaac1ddff4834b2: Bootstrap starting.
I20260808 01:48:26.536357  8622 raft_consensus.cc:2468] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 1.
I20260808 01:48:26.538180  8585 leader_election.cc:304] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:26.539844  8743 tablet_bootstrap.cc:654] T cc880f9cc7844dae9f308ef3c2b9bbdb P 96823ad418f9403cbaaac1ddff4834b2: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.540518  8506 leader_election.cc:304] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 96823ad418f9403cbaaac1ddff4834b2
I20260808 01:48:26.541528  8750 raft_consensus.cc:2804] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:26.542066  8750 raft_consensus.cc:697] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Becoming Leader. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Running, Role: LEADER
I20260808 01:48:26.542829  8698 raft_consensus.cc:2468] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593c8fd64bf14a9996d36e2145636220 in term 1.
I20260808 01:48:26.542985  8750 consensus_queue.cc:237] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.553032  8748 raft_consensus.cc:3060] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:26.553516  8741 tablet_bootstrap.cc:492] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220: No bootstrap required, opened a new log
I20260808 01:48:26.555285  8741 ts_tablet_manager.cc:1403] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220: Time spent bootstrapping tablet: real 0.128s	user 0.017s	sys 0.037s
I20260808 01:48:26.557240  8743 tablet_bootstrap.cc:492] T cc880f9cc7844dae9f308ef3c2b9bbdb P 96823ad418f9403cbaaac1ddff4834b2: No bootstrap required, opened a new log
I20260808 01:48:26.557799  8743 ts_tablet_manager.cc:1403] T cc880f9cc7844dae9f308ef3c2b9bbdb P 96823ad418f9403cbaaac1ddff4834b2: Time spent bootstrapping tablet: real 0.026s	user 0.008s	sys 0.009s
I20260808 01:48:26.560201  8743 raft_consensus.cc:359] T cc880f9cc7844dae9f308ef3c2b9bbdb P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.560887  8743 raft_consensus.cc:385] T cc880f9cc7844dae9f308ef3c2b9bbdb P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.561239  8743 raft_consensus.cc:740] T cc880f9cc7844dae9f308ef3c2b9bbdb P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.562069  8743 consensus_queue.cc:260] T cc880f9cc7844dae9f308ef3c2b9bbdb P 96823ad418f9403cbaaac1ddff4834b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.560513  8741 raft_consensus.cc:359] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.563920  8741 raft_consensus.cc:385] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.569669  8741 raft_consensus.cc:740] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.570586  8742 ts_tablet_manager.cc:1434] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent starting tablet: real 0.082s	user 0.048s	sys 0.032s
I20260808 01:48:26.571135  8741 consensus_queue.cc:260] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.571848  8742 tablet_bootstrap.cc:492] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5: Bootstrap starting.
I20260808 01:48:26.573155  8748 raft_consensus.cc:515] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.554073  8769 raft_consensus.cc:2804] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:26.564504  8743 ts_tablet_manager.cc:1434] T cc880f9cc7844dae9f308ef3c2b9bbdb P 96823ad418f9403cbaaac1ddff4834b2: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260808 01:48:26.572268  8436 catalog_manager.cc:5697] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:26.580979  8742 tablet_bootstrap.cc:654] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.588078  8743 tablet_bootstrap.cc:492] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2: Bootstrap starting.
I20260808 01:48:26.592044  8542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e61a690621224f02a28c606e3228c838" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
I20260808 01:48:26.592757  8542 raft_consensus.cc:3060] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:26.598589  8743 tablet_bootstrap.cc:654] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.605146  8542 raft_consensus.cc:2468] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593c8fd64bf14a9996d36e2145636220 in term 1.
I20260808 01:48:26.608616  8698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e61a690621224f02a28c606e3228c838" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
W20260808 01:48:26.620576  8587 leader_election.cc:343] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:26.609338  8748 leader_election.cc:290] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 1 election: Requested vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:26.614748  8743 tablet_bootstrap.cc:492] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2: No bootstrap required, opened a new log
I20260808 01:48:26.622318  8743 ts_tablet_manager.cc:1403] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2: Time spent bootstrapping tablet: real 0.035s	user 0.013s	sys 0.011s
I20260808 01:48:26.623652  8585 leader_election.cc:304] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 96823ad418f9403cbaaac1ddff4834b2
I20260808 01:48:26.626026  8748 raft_consensus.cc:2804] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:26.626914  8748 raft_consensus.cc:697] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 1 LEADER]: Becoming Leader. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Running, Role: LEADER
I20260808 01:48:26.617782  8769 raft_consensus.cc:697] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 1 LEADER]: Becoming Leader. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Running, Role: LEADER
I20260808 01:48:26.627152  8743 raft_consensus.cc:359] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.617122  8773 raft_consensus.cc:493] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:26.629242  8773 raft_consensus.cc:515] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.629403  8769 consensus_queue.cc:237] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.628549  8743 raft_consensus.cc:385] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.634101  8743 raft_consensus.cc:740] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.635272  8743 consensus_queue.cc:260] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.637126  8748 consensus_queue.cc:237] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.639425  8773 leader_election.cc:290] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:26.641134  8698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc880f9cc7844dae9f308ef3c2b9bbdb" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96823ad418f9403cbaaac1ddff4834b2" is_pre_election: true
I20260808 01:48:26.642097  8698 raft_consensus.cc:2468] T cc880f9cc7844dae9f308ef3c2b9bbdb P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 593c8fd64bf14a9996d36e2145636220 in term 0.
I20260808 01:48:26.643345  8743 ts_tablet_manager.cc:1434] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2: Time spent starting tablet: real 0.020s	user 0.010s	sys 0.000s
I20260808 01:48:26.644398  8587 leader_election.cc:304] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, 96823ad418f9403cbaaac1ddff4834b2; no voters: 
I20260808 01:48:26.645898  8743 tablet_bootstrap.cc:492] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2: Bootstrap starting.
I20260808 01:48:26.648471  8542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc880f9cc7844dae9f308ef3c2b9bbdb" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" is_pre_election: true
W20260808 01:48:26.650898  8585 leader_election.cc:343] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:26.654120  8776 raft_consensus.cc:2804] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:26.654949  8776 raft_consensus.cc:493] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:26.655562  8776 raft_consensus.cc:3060] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:26.661113  8743 tablet_bootstrap.cc:654] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.678043  8741 ts_tablet_manager.cc:1434] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220: Time spent starting tablet: real 0.122s	user 0.044s	sys 0.050s
I20260808 01:48:26.679384  8741 tablet_bootstrap.cc:492] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220: Bootstrap starting.
I20260808 01:48:26.679461  8742 tablet_bootstrap.cc:492] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5: No bootstrap required, opened a new log
I20260808 01:48:26.680122  8742 ts_tablet_manager.cc:1403] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent bootstrapping tablet: real 0.109s	user 0.031s	sys 0.051s
I20260808 01:48:26.683609  8742 raft_consensus.cc:359] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.684384  8742 raft_consensus.cc:385] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.684760  8742 raft_consensus.cc:740] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.685307  8743 tablet_bootstrap.cc:492] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2: No bootstrap required, opened a new log
I20260808 01:48:26.685892  8743 ts_tablet_manager.cc:1403] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2: Time spent bootstrapping tablet: real 0.040s	user 0.018s	sys 0.000s
I20260808 01:48:26.685721  8742 consensus_queue.cc:260] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.687486  8741 tablet_bootstrap.cc:654] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.688745  8742 ts_tablet_manager.cc:1434] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260808 01:48:26.689839  8742 tablet_bootstrap.cc:492] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5: Bootstrap starting.
I20260808 01:48:26.689692  8743 raft_consensus.cc:359] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.690529  8743 raft_consensus.cc:385] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.690986  8743 raft_consensus.cc:740] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.692005  8743 consensus_queue.cc:260] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.695188  8743 ts_tablet_manager.cc:1434] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.002s
I20260808 01:48:26.696750  8743 tablet_bootstrap.cc:492] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2: Bootstrap starting.
I20260808 01:48:26.697036  8742 tablet_bootstrap.cc:654] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.701262  8741 tablet_bootstrap.cc:492] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220: No bootstrap required, opened a new log
I20260808 01:48:26.701826  8741 ts_tablet_manager.cc:1403] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220: Time spent bootstrapping tablet: real 0.023s	user 0.014s	sys 0.005s
I20260808 01:48:26.704530  8743 tablet_bootstrap.cc:654] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.704864  8741 raft_consensus.cc:359] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.705657  8741 raft_consensus.cc:385] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.705971  8741 raft_consensus.cc:740] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.707085  8741 consensus_queue.cc:260] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.710708  8741 ts_tablet_manager.cc:1434] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.001s
I20260808 01:48:26.712201  8741 tablet_bootstrap.cc:492] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220: Bootstrap starting.
I20260808 01:48:26.718091  8742 tablet_bootstrap.cc:492] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5: No bootstrap required, opened a new log
I20260808 01:48:26.718727  8742 ts_tablet_manager.cc:1403] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent bootstrapping tablet: real 0.029s	user 0.011s	sys 0.015s
I20260808 01:48:26.723603  8741 tablet_bootstrap.cc:654] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.724988  8743 tablet_bootstrap.cc:492] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2: No bootstrap required, opened a new log
I20260808 01:48:26.725581  8743 ts_tablet_manager.cc:1403] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2: Time spent bootstrapping tablet: real 0.029s	user 0.013s	sys 0.012s
I20260808 01:48:26.727943  8742 raft_consensus.cc:359] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.728780  8742 raft_consensus.cc:385] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.729107  8742 raft_consensus.cc:740] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.729168  8743 raft_consensus.cc:359] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.730290  8743 raft_consensus.cc:385] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.729965  8742 consensus_queue.cc:260] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.730747  8743 raft_consensus.cc:740] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.732734  8742 ts_tablet_manager.cc:1434] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.001s
I20260808 01:48:26.733924  8742 tablet_bootstrap.cc:492] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5: Bootstrap starting.
I20260808 01:48:26.739202  8776 raft_consensus.cc:515] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.732031  8743 consensus_queue.cc:260] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.743122  8743 ts_tablet_manager.cc:1434] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2: Time spent starting tablet: real 0.017s	user 0.011s	sys 0.001s
I20260808 01:48:26.744150  8743 tablet_bootstrap.cc:492] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2: Bootstrap starting.
I20260808 01:48:26.740882  8742 tablet_bootstrap.cc:654] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.750147  8434 catalog_manager.cc:5697] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 reported cstate change: term changed from 0 to 1, leader changed from <none> to 593c8fd64bf14a9996d36e2145636220 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "593c8fd64bf14a9996d36e2145636220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:26.753968  8743 tablet_bootstrap.cc:654] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.759152  8698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc880f9cc7844dae9f308ef3c2b9bbdb" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
I20260808 01:48:26.760089  8698 raft_consensus.cc:3060] T cc880f9cc7844dae9f308ef3c2b9bbdb P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:26.762331  8542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc880f9cc7844dae9f308ef3c2b9bbdb" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
W20260808 01:48:26.765329  8585 leader_election.cc:343] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:26.769740  8776 leader_election.cc:290] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 1 election: Requested vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:26.771199  8743 tablet_bootstrap.cc:492] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2: No bootstrap required, opened a new log
I20260808 01:48:26.771757  8743 ts_tablet_manager.cc:1403] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2: Time spent bootstrapping tablet: real 0.028s	user 0.014s	sys 0.002s
I20260808 01:48:26.772432  8698 raft_consensus.cc:2468] T cc880f9cc7844dae9f308ef3c2b9bbdb P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593c8fd64bf14a9996d36e2145636220 in term 1.
I20260808 01:48:26.774667  8743 raft_consensus.cc:359] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.775084  8587 leader_election.cc:304] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, 96823ad418f9403cbaaac1ddff4834b2; no voters: d731ac9726624d11a9f44e9bdbbf3fa5
I20260808 01:48:26.775496  8743 raft_consensus.cc:385] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.775815  8743 raft_consensus.cc:740] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.776795  8741 tablet_bootstrap.cc:492] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220: No bootstrap required, opened a new log
I20260808 01:48:26.775098  8742 tablet_bootstrap.cc:492] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5: No bootstrap required, opened a new log
I20260808 01:48:26.776513  8743 consensus_queue.cc:260] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.777451  8741 ts_tablet_manager.cc:1403] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220: Time spent bootstrapping tablet: real 0.066s	user 0.022s	sys 0.033s
I20260808 01:48:26.777190  8434 catalog_manager.cc:5697] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 reported cstate change: term changed from 0 to 1, leader changed from <none> to 593c8fd64bf14a9996d36e2145636220 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "593c8fd64bf14a9996d36e2145636220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:26.776719  8776 raft_consensus.cc:2804] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:26.777606  8742 ts_tablet_manager.cc:1403] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent bootstrapping tablet: real 0.044s	user 0.013s	sys 0.006s
I20260808 01:48:26.780828  8741 raft_consensus.cc:359] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.781631  8741 raft_consensus.cc:385] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.781970  8741 raft_consensus.cc:740] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.782788  8741 consensus_queue.cc:260] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.783329  8742 raft_consensus.cc:359] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.784150  8742 raft_consensus.cc:385] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.784524  8742 raft_consensus.cc:740] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.779489  8776 raft_consensus.cc:697] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 1 LEADER]: Becoming Leader. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Running, Role: LEADER
I20260808 01:48:26.785393  8742 consensus_queue.cc:260] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.785914  8776 consensus_queue.cc:237] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.778926  8743 ts_tablet_manager.cc:1434] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260808 01:48:26.788033  8743 tablet_bootstrap.cc:492] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2: Bootstrap starting.
I20260808 01:48:26.794368  8743 tablet_bootstrap.cc:654] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.794785  8768 raft_consensus.cc:493] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:26.788005  8742 ts_tablet_manager.cc:1434] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260808 01:48:26.795337  8768 raft_consensus.cc:515] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.796279  8742 tablet_bootstrap.cc:492] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5: Bootstrap starting.
I20260808 01:48:26.797753  8768 leader_election.cc:290] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:26.797957  8741 ts_tablet_manager.cc:1434] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.000s
I20260808 01:48:26.799105  8741 tablet_bootstrap.cc:492] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220: Bootstrap starting.
I20260808 01:48:26.799705  8698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e38a43fd7704fe8bec9648856983350" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96823ad418f9403cbaaac1ddff4834b2" is_pre_election: true
I20260808 01:48:26.800411  8698 raft_consensus.cc:2468] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 0.
I20260808 01:48:26.802762  8508 leader_election.cc:304] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96823ad418f9403cbaaac1ddff4834b2, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:26.801578  8622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e38a43fd7704fe8bec9648856983350" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593c8fd64bf14a9996d36e2145636220" is_pre_election: true
I20260808 01:48:26.804201  8622 raft_consensus.cc:2468] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 0.
I20260808 01:48:26.806012  8741 tablet_bootstrap.cc:654] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.808341  8742 tablet_bootstrap.cc:654] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.809188  8768 raft_consensus.cc:2804] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:26.809755  8768 raft_consensus.cc:493] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:26.810187  8768 raft_consensus.cc:3060] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:26.818737  8743 tablet_bootstrap.cc:492] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2: No bootstrap required, opened a new log
I20260808 01:48:26.819362  8743 ts_tablet_manager.cc:1403] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2: Time spent bootstrapping tablet: real 0.032s	user 0.012s	sys 0.000s
I20260808 01:48:26.821770  8768 raft_consensus.cc:515] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.821481  8434 catalog_manager.cc:5697] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 reported cstate change: term changed from 0 to 1, leader changed from <none> to 593c8fd64bf14a9996d36e2145636220 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "593c8fd64bf14a9996d36e2145636220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:26.822832  8741 tablet_bootstrap.cc:492] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220: No bootstrap required, opened a new log
I20260808 01:48:26.822379  8743 raft_consensus.cc:359] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.823348  8741 ts_tablet_manager.cc:1403] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220: Time spent bootstrapping tablet: real 0.024s	user 0.017s	sys 0.004s
I20260808 01:48:26.823415  8743 raft_consensus.cc:385] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.823963  8743 raft_consensus.cc:740] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.824925  8743 consensus_queue.cc:260] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.825186  8622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e38a43fd7704fe8bec9648856983350" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593c8fd64bf14a9996d36e2145636220"
I20260808 01:48:26.825927  8622 raft_consensus.cc:3060] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:26.826593  8741 raft_consensus.cc:359] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.827348  8741 raft_consensus.cc:385] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.827433  8698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e38a43fd7704fe8bec9648856983350" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
I20260808 01:48:26.827700  8741 raft_consensus.cc:740] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.828022  8698 raft_consensus.cc:3060] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:26.828541  8741 consensus_queue.cc:260] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.831063  8741 ts_tablet_manager.cc:1434] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260808 01:48:26.832105  8741 tablet_bootstrap.cc:492] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220: Bootstrap starting.
I20260808 01:48:26.833263  8622 raft_consensus.cc:2468] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 1.
I20260808 01:48:26.834921  8506 leader_election.cc:304] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:26.835920  8750 raft_consensus.cc:2804] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:26.838109  8742 tablet_bootstrap.cc:492] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5: No bootstrap required, opened a new log
I20260808 01:48:26.838649  8742 ts_tablet_manager.cc:1403] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent bootstrapping tablet: real 0.043s	user 0.008s	sys 0.003s
I20260808 01:48:26.840185  8741 tablet_bootstrap.cc:654] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.842495  8742 raft_consensus.cc:359] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.843326  8742 raft_consensus.cc:385] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.843595  8742 raft_consensus.cc:740] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.844463  8742 consensus_queue.cc:260] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.850382  8698 raft_consensus.cc:2468] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 1.
I20260808 01:48:26.851554  8749 raft_consensus.cc:493] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:26.851825  8768 leader_election.cc:290] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 election: Requested vote from peers 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:26.852349  8749 raft_consensus.cc:515] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.858980  8542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d471f57bba4e49f9b0e8c29880e2a864" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" is_pre_election: true
I20260808 01:48:26.859871  8542 raft_consensus.cc:2468] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96823ad418f9403cbaaac1ddff4834b2 in term 0.
I20260808 01:48:26.860745  8749 leader_election.cc:290] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883)
I20260808 01:48:26.861545  8660 leader_election.cc:304] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96823ad418f9403cbaaac1ddff4834b2, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:26.861873  8750 raft_consensus.cc:697] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Becoming Leader. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Running, Role: LEADER
I20260808 01:48:26.862823  8749 raft_consensus.cc:2804] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:26.863467  8749 raft_consensus.cc:493] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:26.863881  8749 raft_consensus.cc:3060] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:26.863691  8750 consensus_queue.cc:237] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.866611  8622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d471f57bba4e49f9b0e8c29880e2a864" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593c8fd64bf14a9996d36e2145636220" is_pre_election: true
I20260808 01:48:26.871608  8622 raft_consensus.cc:2468] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96823ad418f9403cbaaac1ddff4834b2 in term 0.
I20260808 01:48:26.872819  8749 raft_consensus.cc:515] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.868875  8743 ts_tablet_manager.cc:1434] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2: Time spent starting tablet: real 0.049s	user 0.006s	sys 0.000s
I20260808 01:48:26.875979  8542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d471f57bba4e49f9b0e8c29880e2a864" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
I20260808 01:48:26.876650  8542 raft_consensus.cc:3060] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:26.876732  8623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d471f57bba4e49f9b0e8c29880e2a864" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593c8fd64bf14a9996d36e2145636220"
I20260808 01:48:26.877312  8623 raft_consensus.cc:3060] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:26.884430  8542 raft_consensus.cc:2468] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96823ad418f9403cbaaac1ddff4834b2 in term 1.
I20260808 01:48:26.885905  8660 leader_election.cc:304] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96823ad418f9403cbaaac1ddff4834b2, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:26.875007  8749 leader_election.cc:290] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 election: Requested vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883)
I20260808 01:48:26.890753  8749 raft_consensus.cc:2804] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:26.891291  8749 raft_consensus.cc:697] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 LEADER]: Becoming Leader. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Running, Role: LEADER
I20260808 01:48:26.892189  8749 consensus_queue.cc:237] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.894081  8623 raft_consensus.cc:2468] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96823ad418f9403cbaaac1ddff4834b2 in term 1.
I20260808 01:48:26.895526  8743 tablet_bootstrap.cc:492] T f390638b5fbd41b686f3fa99a5898f77 P 96823ad418f9403cbaaac1ddff4834b2: Bootstrap starting.
I20260808 01:48:26.910377  8436 catalog_manager.cc:5697] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:26.913578  8742 ts_tablet_manager.cc:1434] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent starting tablet: real 0.074s	user 0.028s	sys 0.042s
I20260808 01:48:26.914012  8799 raft_consensus.cc:493] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:26.917708  8799 raft_consensus.cc:515] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.918892  8741 tablet_bootstrap.cc:492] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220: No bootstrap required, opened a new log
I20260808 01:48:26.919430  8741 ts_tablet_manager.cc:1403] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220: Time spent bootstrapping tablet: real 0.088s	user 0.044s	sys 0.030s
I20260808 01:48:26.914877  8742 tablet_bootstrap.cc:492] T 598364b36429453793bb8922ded6a556 P d731ac9726624d11a9f44e9bdbbf3fa5: Bootstrap starting.
I20260808 01:48:26.917166  8743 tablet_bootstrap.cc:654] T f390638b5fbd41b686f3fa99a5898f77 P 96823ad418f9403cbaaac1ddff4834b2: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.922911  8741 raft_consensus.cc:359] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.923365  8799 leader_election.cc:290] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883)
I20260808 01:48:26.923656  8741 raft_consensus.cc:385] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.923147  8623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e61a690621224f02a28c606e3228c838" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593c8fd64bf14a9996d36e2145636220" is_pre_election: true
I20260808 01:48:26.924072  8741 raft_consensus.cc:740] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.923147  8542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e61a690621224f02a28c606e3228c838" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" is_pre_election: true
I20260808 01:48:26.925346  8542 raft_consensus.cc:2393] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 96823ad418f9403cbaaac1ddff4834b2 in current term 1: Already voted for candidate 593c8fd64bf14a9996d36e2145636220 in this term.
I20260808 01:48:26.924998  8741 consensus_queue.cc:260] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.928213  8660 leader_election.cc:304] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96823ad418f9403cbaaac1ddff4834b2; no voters: 593c8fd64bf14a9996d36e2145636220, d731ac9726624d11a9f44e9bdbbf3fa5
I20260808 01:48:26.930178  8698 raft_consensus.cc:3060] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:26.934110  8742 tablet_bootstrap.cc:654] T 598364b36429453793bb8922ded6a556 P d731ac9726624d11a9f44e9bdbbf3fa5: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.935871  8741 ts_tablet_manager.cc:1434] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.000s
I20260808 01:48:26.939400  8795 consensus_queue.cc:1048] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:48:26.941912  8437 catalog_manager.cc:5697] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "96823ad418f9403cbaaac1ddff4834b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: HEALTHY } } }
I20260808 01:48:26.943413  8749 raft_consensus.cc:3060] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:26.946874  8741 tablet_bootstrap.cc:492] T 598364b36429453793bb8922ded6a556 P 593c8fd64bf14a9996d36e2145636220: Bootstrap starting.
I20260808 01:48:26.949831  8742 tablet_bootstrap.cc:492] T 598364b36429453793bb8922ded6a556 P d731ac9726624d11a9f44e9bdbbf3fa5: No bootstrap required, opened a new log
I20260808 01:48:26.950403  8742 ts_tablet_manager.cc:1403] T 598364b36429453793bb8922ded6a556 P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent bootstrapping tablet: real 0.036s	user 0.011s	sys 0.003s
I20260808 01:48:26.956323  8741 tablet_bootstrap.cc:654] T 598364b36429453793bb8922ded6a556 P 593c8fd64bf14a9996d36e2145636220: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:26.956799  8742 raft_consensus.cc:359] T 598364b36429453793bb8922ded6a556 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:26.957533  8742 raft_consensus.cc:385] T 598364b36429453793bb8922ded6a556 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:26.957816  8742 raft_consensus.cc:740] T 598364b36429453793bb8922ded6a556 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Initialized, Role: FOLLOWER
I20260808 01:48:26.958534  8742 consensus_queue.cc:260] T 598364b36429453793bb8922ded6a556 P d731ac9726624d11a9f44e9bdbbf3fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.006404  8799 raft_consensus.cc:493] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:27.007190  8799 raft_consensus.cc:515] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.017263  8742 ts_tablet_manager.cc:1434] T 598364b36429453793bb8922ded6a556 P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent starting tablet: real 0.067s	user 0.007s	sys 0.000s
I20260808 01:48:27.018855  8742 tablet_bootstrap.cc:492] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5: Bootstrap starting.
I20260808 01:48:27.030166  8749 raft_consensus.cc:2749] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:48:27.032687  8542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2186a1e5b7a4dcb828dc73a73036684" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" is_pre_election: true
I20260808 01:48:27.033432  8542 raft_consensus.cc:2468] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96823ad418f9403cbaaac1ddff4834b2 in term 0.
I20260808 01:48:27.033746  8742 tablet_bootstrap.cc:654] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:27.034070  8799 leader_election.cc:290] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883)
I20260808 01:48:27.035249  8623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2186a1e5b7a4dcb828dc73a73036684" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593c8fd64bf14a9996d36e2145636220" is_pre_election: true
I20260808 01:48:27.035940  8623 raft_consensus.cc:2468] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96823ad418f9403cbaaac1ddff4834b2 in term 0.
I20260808 01:48:27.036044  8660 leader_election.cc:304] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96823ad418f9403cbaaac1ddff4834b2, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:27.039124  8799 raft_consensus.cc:2804] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:27.040472  8799 raft_consensus.cc:493] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:27.041579  8799 raft_consensus.cc:3060] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:27.044288  8743 tablet_bootstrap.cc:492] T f390638b5fbd41b686f3fa99a5898f77 P 96823ad418f9403cbaaac1ddff4834b2: No bootstrap required, opened a new log
I20260808 01:48:27.044315  8741 tablet_bootstrap.cc:492] T 598364b36429453793bb8922ded6a556 P 593c8fd64bf14a9996d36e2145636220: No bootstrap required, opened a new log
I20260808 01:48:27.044924  8743 ts_tablet_manager.cc:1403] T f390638b5fbd41b686f3fa99a5898f77 P 96823ad418f9403cbaaac1ddff4834b2: Time spent bootstrapping tablet: real 0.150s	user 0.042s	sys 0.023s
I20260808 01:48:27.045001  8741 ts_tablet_manager.cc:1403] T 598364b36429453793bb8922ded6a556 P 593c8fd64bf14a9996d36e2145636220: Time spent bootstrapping tablet: real 0.098s	user 0.014s	sys 0.021s
I20260808 01:48:27.043200  8795 consensus_queue.cc:1048] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260808 01:48:27.047816  8743 raft_consensus.cc:359] T f390638b5fbd41b686f3fa99a5898f77 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.048512  8743 raft_consensus.cc:385] T f390638b5fbd41b686f3fa99a5898f77 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:27.048789  8743 raft_consensus.cc:740] T f390638b5fbd41b686f3fa99a5898f77 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Initialized, Role: FOLLOWER
I20260808 01:48:27.049597  8741 raft_consensus.cc:359] T 598364b36429453793bb8922ded6a556 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.049647  8743 consensus_queue.cc:260] T f390638b5fbd41b686f3fa99a5898f77 P 96823ad418f9403cbaaac1ddff4834b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.050777  8741 raft_consensus.cc:385] T 598364b36429453793bb8922ded6a556 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:27.051121  8741 raft_consensus.cc:740] T 598364b36429453793bb8922ded6a556 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Initialized, Role: FOLLOWER
I20260808 01:48:27.052480  8741 consensus_queue.cc:260] T 598364b36429453793bb8922ded6a556 P 593c8fd64bf14a9996d36e2145636220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.054391  8742 tablet_bootstrap.cc:492] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5: No bootstrap required, opened a new log
I20260808 01:48:27.054767  8741 ts_tablet_manager.cc:1434] T 598364b36429453793bb8922ded6a556 P 593c8fd64bf14a9996d36e2145636220: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.001s
I20260808 01:48:27.054926  8742 ts_tablet_manager.cc:1403] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent bootstrapping tablet: real 0.036s	user 0.013s	sys 0.006s
I20260808 01:48:27.055738  8741 tablet_bootstrap.cc:492] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220: Bootstrap starting.
I20260808 01:48:27.056530  8776 consensus_queue.cc:1048] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:27.057574  8742 raft_consensus.cc:359] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.058265  8742 raft_consensus.cc:385] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:27.058547  8742 raft_consensus.cc:740] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Initialized, Role: FOLLOWER
I20260808 01:48:27.059257  8742 consensus_queue.cc:260] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.062041  8741 tablet_bootstrap.cc:654] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:27.065212  8743 ts_tablet_manager.cc:1434] T f390638b5fbd41b686f3fa99a5898f77 P 96823ad418f9403cbaaac1ddff4834b2: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.001s
I20260808 01:48:27.067819  8799 raft_consensus.cc:515] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.071301  8543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2186a1e5b7a4dcb828dc73a73036684" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
I20260808 01:48:27.071830  8742 ts_tablet_manager.cc:1434] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.000s
I20260808 01:48:27.071985  8543 raft_consensus.cc:3060] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:27.074384  8795 raft_consensus.cc:493] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:27.074769  8622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2186a1e5b7a4dcb828dc73a73036684" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593c8fd64bf14a9996d36e2145636220"
I20260808 01:48:27.075419  8622 raft_consensus.cc:3060] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:27.075137  8795 raft_consensus.cc:515] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.078626  8698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc880f9cc7844dae9f308ef3c2b9bbdb" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96823ad418f9403cbaaac1ddff4834b2" is_pre_election: true
I20260808 01:48:27.078953  8621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc880f9cc7844dae9f308ef3c2b9bbdb" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593c8fd64bf14a9996d36e2145636220" is_pre_election: true
I20260808 01:48:27.079582  8698 raft_consensus.cc:2393] T cc880f9cc7844dae9f308ef3c2b9bbdb P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d731ac9726624d11a9f44e9bdbbf3fa5 in current term 1: Already voted for candidate 593c8fd64bf14a9996d36e2145636220 in this term.
I20260808 01:48:27.081349  8795 leader_election.cc:290] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:27.083072  8622 raft_consensus.cc:2468] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96823ad418f9403cbaaac1ddff4834b2 in term 1.
I20260808 01:48:27.081503  8506 leader_election.cc:304] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 593c8fd64bf14a9996d36e2145636220, 96823ad418f9403cbaaac1ddff4834b2
I20260808 01:48:27.084462  8660 leader_election.cc:304] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, 96823ad418f9403cbaaac1ddff4834b2; no voters: 
I20260808 01:48:27.087517  8749 raft_consensus.cc:2804] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:27.088065  8749 raft_consensus.cc:697] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 LEADER]: Becoming Leader. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Running, Role: LEADER
I20260808 01:48:27.088933  8749 consensus_queue.cc:237] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.089545  8776 consensus_queue.cc:1048] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:27.088655  8795 raft_consensus.cc:3060] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:27.084789  8799 leader_election.cc:290] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 election: Requested vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883)
I20260808 01:48:27.100935  8793 raft_consensus.cc:493] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:27.101799  8793 raft_consensus.cc:515] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.104769  8793 leader_election.cc:290] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:27.106493  8697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20f572798f1646acb92966b083e07eb2" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96823ad418f9403cbaaac1ddff4834b2" is_pre_election: true
I20260808 01:48:27.107298  8697 raft_consensus.cc:2468] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 0.
I20260808 01:48:27.107458  8622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20f572798f1646acb92966b083e07eb2" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593c8fd64bf14a9996d36e2145636220" is_pre_election: true
I20260808 01:48:27.108800  8622 raft_consensus.cc:2468] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 0.
I20260808 01:48:27.109033  8508 leader_election.cc:304] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96823ad418f9403cbaaac1ddff4834b2, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:27.109989  8793 raft_consensus.cc:2804] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:27.110347  8793 raft_consensus.cc:493] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:27.110769  8793 raft_consensus.cc:3060] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:27.113595  8543 raft_consensus.cc:2468] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96823ad418f9403cbaaac1ddff4834b2 in term 1.
I20260808 01:48:27.124665  8749 raft_consensus.cc:493] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:27.125366  8749 raft_consensus.cc:515] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.129678  8749 leader_election.cc:290] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883)
I20260808 01:48:27.119395  8793 raft_consensus.cc:515] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.121035  8437 catalog_manager.cc:5697] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "96823ad418f9403cbaaac1ddff4834b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: HEALTHY } } }
I20260808 01:48:27.131057  8543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598364b36429453793bb8922ded6a556" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" is_pre_election: true
I20260808 01:48:27.131771  8543 raft_consensus.cc:2468] T 598364b36429453793bb8922ded6a556 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96823ad418f9403cbaaac1ddff4834b2 in term 0.
I20260808 01:48:27.133168  8660 leader_election.cc:304] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96823ad418f9403cbaaac1ddff4834b2, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:27.134866  8749 raft_consensus.cc:2804] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:27.135411  8749 raft_consensus.cc:493] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:27.135849  8749 raft_consensus.cc:3060] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:27.141958  8793 leader_election.cc:290] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 election: Requested vote from peers 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:27.142021  8698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20f572798f1646acb92966b083e07eb2" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
I20260808 01:48:27.142771  8698 raft_consensus.cc:3060] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:27.142531  8621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20f572798f1646acb92966b083e07eb2" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593c8fd64bf14a9996d36e2145636220"
I20260808 01:48:27.143435  8621 raft_consensus.cc:3060] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:27.148401  8749 raft_consensus.cc:515] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.157512  8543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598364b36429453793bb8922ded6a556" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
I20260808 01:48:27.157677  8772 consensus_queue.cc:1048] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:48:27.158222  8543 raft_consensus.cc:3060] T 598364b36429453793bb8922ded6a556 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:27.158241  8698 raft_consensus.cc:2468] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 1.
I20260808 01:48:27.159955  8508 leader_election.cc:304] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96823ad418f9403cbaaac1ddff4834b2, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:27.160866  8793 raft_consensus.cc:2804] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:27.161368  8793 raft_consensus.cc:697] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Becoming Leader. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Running, Role: LEADER
I20260808 01:48:27.162276  8793 consensus_queue.cc:237] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.164204  8749 leader_election.cc:290] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 election: Requested vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883)
I20260808 01:48:27.165731  8620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598364b36429453793bb8922ded6a556" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593c8fd64bf14a9996d36e2145636220"
I20260808 01:48:27.166437  8620 raft_consensus.cc:3060] T 598364b36429453793bb8922ded6a556 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:27.166304  8622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "598364b36429453793bb8922ded6a556" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593c8fd64bf14a9996d36e2145636220" is_pre_election: true
I20260808 01:48:27.167979  8543 raft_consensus.cc:2468] T 598364b36429453793bb8922ded6a556 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96823ad418f9403cbaaac1ddff4834b2 in term 1.
I20260808 01:48:27.169824  8660 leader_election.cc:304] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96823ad418f9403cbaaac1ddff4834b2, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:27.174588  8620 raft_consensus.cc:2468] T 598364b36429453793bb8922ded6a556 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96823ad418f9403cbaaac1ddff4834b2 in term 1.
I20260808 01:48:27.180212  8749 raft_consensus.cc:2804] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:27.181290  8749 raft_consensus.cc:697] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 LEADER]: Becoming Leader. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Running, Role: LEADER
I20260808 01:48:27.182612  8749 consensus_queue.cc:237] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.200196  8621 raft_consensus.cc:2468] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 1.
I20260808 01:48:27.205901  8741 tablet_bootstrap.cc:492] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220: No bootstrap required, opened a new log
I20260808 01:48:27.206447  8741 ts_tablet_manager.cc:1403] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220: Time spent bootstrapping tablet: real 0.151s	user 0.050s	sys 0.069s
I20260808 01:48:27.207638  8795 raft_consensus.cc:2749] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:48:27.213577  8741 raft_consensus.cc:359] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.214308  8741 raft_consensus.cc:385] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:27.214569  8741 raft_consensus.cc:740] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Initialized, Role: FOLLOWER
I20260808 01:48:27.215378  8741 consensus_queue.cc:260] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.215754  8769 consensus_queue.cc:1048] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:27.218658  8434 catalog_manager.cc:5697] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:27.230260  8741 ts_tablet_manager.cc:1434] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220: Time spent starting tablet: real 0.023s	user 0.012s	sys 0.007s
I20260808 01:48:27.238775  8437 catalog_manager.cc:5697] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "96823ad418f9403cbaaac1ddff4834b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: HEALTHY } } }
I20260808 01:48:27.238852  8793 raft_consensus.cc:493] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:27.240816  8793 raft_consensus.cc:515] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.243188  8793 leader_election.cc:290] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:27.244131  8621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f390638b5fbd41b686f3fa99a5898f77" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593c8fd64bf14a9996d36e2145636220" is_pre_election: true
I20260808 01:48:27.244828  8621 raft_consensus.cc:2468] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 0.
I20260808 01:48:27.244939  8697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f390638b5fbd41b686f3fa99a5898f77" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96823ad418f9403cbaaac1ddff4834b2" is_pre_election: true
I20260808 01:48:27.245726  8697 raft_consensus.cc:2468] T f390638b5fbd41b686f3fa99a5898f77 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 0.
I20260808 01:48:27.246320  8506 leader_election.cc:304] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:27.247659  8793 raft_consensus.cc:2804] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:27.248081  8793 raft_consensus.cc:493] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:27.248390  8793 raft_consensus.cc:3060] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:27.256596  8793 raft_consensus.cc:515] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.258515  8793 leader_election.cc:290] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 election: Requested vote from peers 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:27.259611  8621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f390638b5fbd41b686f3fa99a5898f77" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593c8fd64bf14a9996d36e2145636220"
I20260808 01:48:27.260181  8621 raft_consensus.cc:3060] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:27.259928  8698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f390638b5fbd41b686f3fa99a5898f77" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
I20260808 01:48:27.260718  8698 raft_consensus.cc:3060] T f390638b5fbd41b686f3fa99a5898f77 P 96823ad418f9403cbaaac1ddff4834b2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:27.267673  8621 raft_consensus.cc:2468] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 1.
I20260808 01:48:27.267673  8698 raft_consensus.cc:2468] T f390638b5fbd41b686f3fa99a5898f77 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 1.
I20260808 01:48:27.269279  8508 leader_election.cc:304] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96823ad418f9403cbaaac1ddff4834b2, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:27.270068  8793 raft_consensus.cc:2804] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:27.270577  8793 raft_consensus.cc:697] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Becoming Leader. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Running, Role: LEADER
I20260808 01:48:27.271445  8793 consensus_queue.cc:237] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.279100  8436 catalog_manager.cc:5697] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:27.282902  8769 consensus_queue.cc:1048] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:27.283674  8818 consensus_queue.cc:1048] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:27.298355  8793 consensus_queue.cc:1048] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:48:27.306123  8799 consensus_queue.cc:1048] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:27.324265  8793 consensus_queue.cc:1048] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:27.326782  8799 consensus_queue.cc:1048] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:27.490773  8621 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "537921115d13455d98e32f8ad895e87a"
dest_uuid: "593c8fd64bf14a9996d36e2145636220"
mode: GRACEFUL
new_leader_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
 from {username='slave'} at 127.0.0.1:44304
I20260808 01:48:27.491410  8621 raft_consensus.cc:606] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 1 LEADER]: Received request to transfer leadership to TS d731ac9726624d11a9f44e9bdbbf3fa5
I20260808 01:48:27.504864  8621 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e61a690621224f02a28c606e3228c838"
dest_uuid: "593c8fd64bf14a9996d36e2145636220"
mode: GRACEFUL
new_leader_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
 from {username='slave'} at 127.0.0.1:44304
I20260808 01:48:27.505409  8621 raft_consensus.cc:606] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 1 LEADER]: Received request to transfer leadership to TS d731ac9726624d11a9f44e9bdbbf3fa5
I20260808 01:48:27.523664  8698 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d471f57bba4e49f9b0e8c29880e2a864"
dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
mode: GRACEFUL
new_leader_uuid: "593c8fd64bf14a9996d36e2145636220"
 from {username='slave'} at 127.0.0.1:41274
I20260808 01:48:27.524338  8698 raft_consensus.cc:606] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 LEADER]: Received request to transfer leadership to TS 593c8fd64bf14a9996d36e2145636220
I20260808 01:48:27.530740  8698 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e2186a1e5b7a4dcb828dc73a73036684"
dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
mode: GRACEFUL
new_leader_uuid: "593c8fd64bf14a9996d36e2145636220"
 from {username='slave'} at 127.0.0.1:41274
I20260808 01:48:27.531339  8698 raft_consensus.cc:606] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 LEADER]: Received request to transfer leadership to TS 593c8fd64bf14a9996d36e2145636220
I20260808 01:48:27.546645  8543 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f390638b5fbd41b686f3fa99a5898f77"
dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
mode: GRACEFUL
new_leader_uuid: "593c8fd64bf14a9996d36e2145636220"
 from {username='slave'} at 127.0.0.1:34862
I20260808 01:48:27.547319  8543 raft_consensus.cc:606] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Received request to transfer leadership to TS 593c8fd64bf14a9996d36e2145636220
I20260808 01:48:27.620409  8793 consensus_queue.cc:1048] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:27.631062  8749 consensus_queue.cc:1048] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:27.632377  8793 consensus_queue.cc:1048] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:27.639946  8749 consensus_queue.cc:1048] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:27.653847  8799 raft_consensus.cc:993] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2: : Instructing follower 593c8fd64bf14a9996d36e2145636220 to start an election
I20260808 01:48:27.654376  8799 raft_consensus.cc:1081] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 LEADER]: Signalling peer 593c8fd64bf14a9996d36e2145636220 to start an election
I20260808 01:48:27.656833  8620 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e2186a1e5b7a4dcb828dc73a73036684"
dest_uuid: "593c8fd64bf14a9996d36e2145636220"
 from {username='slave'} at 127.0.0.1:44296
I20260808 01:48:27.659729  8620 raft_consensus.cc:493] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:27.660159  8620 raft_consensus.cc:3060] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:27.661942  8749 consensus_queue.cc:1048] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:27.667985  8620 raft_consensus.cc:515] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.676033  8620 leader_election.cc:290] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 2 election: Requested vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:27.678634  8543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2186a1e5b7a4dcb828dc73a73036684" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
I20260808 01:48:27.679165  8698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2186a1e5b7a4dcb828dc73a73036684" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
I20260808 01:48:27.679358  8543 raft_consensus.cc:3060] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:27.680567  8698 raft_consensus.cc:3055] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:48:27.680943  8698 raft_consensus.cc:740] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Running, Role: LEADER
I20260808 01:48:27.681943  8698 consensus_queue.cc:260] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.683784  8698 raft_consensus.cc:3060] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:27.686133  8799 consensus_queue.cc:1048] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:48:27.687454  8543 raft_consensus.cc:2468] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593c8fd64bf14a9996d36e2145636220 in term 2.
I20260808 01:48:27.689055  8585 leader_election.cc:304] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:27.690032  8772 raft_consensus.cc:2804] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:48:27.690526  8772 raft_consensus.cc:697] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 2 LEADER]: Becoming Leader. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Running, Role: LEADER
I20260808 01:48:27.691434  8772 consensus_queue.cc:237] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.693303  8698 raft_consensus.cc:2468] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593c8fd64bf14a9996d36e2145636220 in term 2.
I20260808 01:48:27.704450  8435 catalog_manager.cc:5697] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 reported cstate change: term changed from 1 to 2, leader changed from 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195) to 593c8fd64bf14a9996d36e2145636220 (127.7.165.194). New cstate: current_term: 2 leader_uuid: "593c8fd64bf14a9996d36e2145636220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:27.715350  8818 raft_consensus.cc:993] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220: : Instructing follower d731ac9726624d11a9f44e9bdbbf3fa5 to start an election
I20260808 01:48:27.715912  8772 raft_consensus.cc:1081] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 1 LEADER]: Signalling peer d731ac9726624d11a9f44e9bdbbf3fa5 to start an election
I20260808 01:48:27.717854  8543 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "537921115d13455d98e32f8ad895e87a"
dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
 from {username='slave'} at 127.0.0.1:34860
I20260808 01:48:27.718480  8543 raft_consensus.cc:493] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:27.718950  8543 raft_consensus.cc:3060] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:27.728116  8543 raft_consensus.cc:515] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.730497  8543 leader_election.cc:290] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 2 election: Requested vote from peers 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:27.732005  8698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "537921115d13455d98e32f8ad895e87a" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
I20260808 01:48:27.732107  8621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "537921115d13455d98e32f8ad895e87a" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "593c8fd64bf14a9996d36e2145636220"
I20260808 01:48:27.732793  8698 raft_consensus.cc:3060] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:27.732869  8621 raft_consensus.cc:3055] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:48:27.733275  8621 raft_consensus.cc:740] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Running, Role: LEADER
I20260808 01:48:27.733984  8621 consensus_queue.cc:260] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.735088  8621 raft_consensus.cc:3060] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:27.739265  8698 raft_consensus.cc:2468] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 2.
I20260808 01:48:27.740772  8508 leader_election.cc:304] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96823ad418f9403cbaaac1ddff4834b2, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:27.741714  8793 raft_consensus.cc:2804] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:48:27.742182  8793 raft_consensus.cc:697] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 LEADER]: Becoming Leader. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Running, Role: LEADER
I20260808 01:48:27.742875  8621 raft_consensus.cc:2468] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 2.
I20260808 01:48:27.743043  8793 consensus_queue.cc:237] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.751168  8818 raft_consensus.cc:993] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220: : Instructing follower d731ac9726624d11a9f44e9bdbbf3fa5 to start an election
I20260808 01:48:27.751643  8818 raft_consensus.cc:1081] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 1 LEADER]: Signalling peer d731ac9726624d11a9f44e9bdbbf3fa5 to start an election
I20260808 01:48:27.751485  8435 catalog_manager.cc:5697] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 reported cstate change: term changed from 1 to 2, leader changed from 593c8fd64bf14a9996d36e2145636220 (127.7.165.194) to d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193). New cstate: current_term: 2 leader_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:27.753643  8543 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e61a690621224f02a28c606e3228c838"
dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
 from {username='slave'} at 127.0.0.1:34860
I20260808 01:48:27.754235  8543 raft_consensus.cc:493] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:27.754616  8543 raft_consensus.cc:3060] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:27.762382  8543 raft_consensus.cc:515] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.764556  8543 leader_election.cc:290] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 2 election: Requested vote from peers 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:27.765625  8621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e61a690621224f02a28c606e3228c838" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "593c8fd64bf14a9996d36e2145636220"
I20260808 01:48:27.765785  8698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e61a690621224f02a28c606e3228c838" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
I20260808 01:48:27.766268  8621 raft_consensus.cc:3055] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:48:27.766376  8698 raft_consensus.cc:3060] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:27.766635  8621 raft_consensus.cc:740] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Running, Role: LEADER
I20260808 01:48:27.767558  8621 consensus_queue.cc:260] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.768643  8621 raft_consensus.cc:3060] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:27.772593  8698 raft_consensus.cc:2468] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 2.
I20260808 01:48:27.773969  8508 leader_election.cc:304] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96823ad418f9403cbaaac1ddff4834b2, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:27.774147  8621 raft_consensus.cc:2468] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 2.
I20260808 01:48:27.774921  8793 raft_consensus.cc:2804] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:48:27.775444  8793 raft_consensus.cc:697] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 LEADER]: Becoming Leader. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Running, Role: LEADER
I20260808 01:48:27.776381  8793 consensus_queue.cc:237] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.784273  8435 catalog_manager.cc:5697] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 reported cstate change: term changed from 1 to 2, leader changed from 593c8fd64bf14a9996d36e2145636220 (127.7.165.194) to d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193). New cstate: current_term: 2 leader_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:27.786669  8793 consensus_queue.cc:1048] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:27.801633  8793 consensus_queue.cc:1048] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:27.818419  8750 raft_consensus.cc:993] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5: : Instructing follower 593c8fd64bf14a9996d36e2145636220 to start an election
I20260808 01:48:27.818943  8839 raft_consensus.cc:1081] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Signalling peer 593c8fd64bf14a9996d36e2145636220 to start an election
I20260808 01:48:27.820658  8621 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f390638b5fbd41b686f3fa99a5898f77"
dest_uuid: "593c8fd64bf14a9996d36e2145636220"
 from {username='slave'} at 127.0.0.1:44294
I20260808 01:48:27.821383  8621 raft_consensus.cc:493] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:27.821822  8621 raft_consensus.cc:3060] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:27.829305  8621 raft_consensus.cc:515] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.831589  8621 leader_election.cc:290] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 2 election: Requested vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:27.833037  8698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f390638b5fbd41b686f3fa99a5898f77" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
I20260808 01:48:27.833693  8698 raft_consensus.cc:3060] T f390638b5fbd41b686f3fa99a5898f77 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:27.834527  8543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f390638b5fbd41b686f3fa99a5898f77" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
I20260808 01:48:27.835225  8543 raft_consensus.cc:3055] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:48:27.835548  8543 raft_consensus.cc:740] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Running, Role: LEADER
I20260808 01:48:27.836408  8543 consensus_queue.cc:260] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.837777  8543 raft_consensus.cc:3060] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:27.841102  8698 raft_consensus.cc:2468] T f390638b5fbd41b686f3fa99a5898f77 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593c8fd64bf14a9996d36e2145636220 in term 2.
I20260808 01:48:27.842489  8587 leader_election.cc:304] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, 96823ad418f9403cbaaac1ddff4834b2; no voters: 
I20260808 01:48:27.843433  8772 raft_consensus.cc:2804] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:48:27.843824  8543 raft_consensus.cc:2468] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593c8fd64bf14a9996d36e2145636220 in term 2.
I20260808 01:48:27.843992  8772 raft_consensus.cc:697] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 [term 2 LEADER]: Becoming Leader. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Running, Role: LEADER
I20260808 01:48:27.845569  8772 consensus_queue.cc:237] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.853263  8434 catalog_manager.cc:5697] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 reported cstate change: term changed from 1 to 2, leader changed from d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193) to 593c8fd64bf14a9996d36e2145636220 (127.7.165.194). New cstate: current_term: 2 leader_uuid: "593c8fd64bf14a9996d36e2145636220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:27.880964  8799 raft_consensus.cc:993] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2: : Instructing follower 593c8fd64bf14a9996d36e2145636220 to start an election
I20260808 01:48:27.881495  8797 raft_consensus.cc:1081] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 LEADER]: Signalling peer 593c8fd64bf14a9996d36e2145636220 to start an election
I20260808 01:48:27.883219  8621 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d471f57bba4e49f9b0e8c29880e2a864"
dest_uuid: "593c8fd64bf14a9996d36e2145636220"
 from {username='slave'} at 127.0.0.1:44296
I20260808 01:48:27.883785  8621 raft_consensus.cc:493] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:27.884289  8621 raft_consensus.cc:3060] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:27.889506  8621 raft_consensus.cc:515] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.891304  8621 leader_election.cc:290] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 2 election: Requested vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:27.892277  8543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d471f57bba4e49f9b0e8c29880e2a864" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
I20260808 01:48:27.892517  8698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d471f57bba4e49f9b0e8c29880e2a864" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
I20260808 01:48:27.893059  8543 raft_consensus.cc:3060] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:27.893252  8698 raft_consensus.cc:3055] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:48:27.893656  8698 raft_consensus.cc:740] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Running, Role: LEADER
I20260808 01:48:27.894399  8698 consensus_queue.cc:260] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.895462  8698 raft_consensus.cc:3060] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:27.898831  8543 raft_consensus.cc:2468] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593c8fd64bf14a9996d36e2145636220 in term 2.
I20260808 01:48:27.900343  8585 leader_election.cc:304] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:27.901279  8772 raft_consensus.cc:2804] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:48:27.901522  8698 raft_consensus.cc:2468] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593c8fd64bf14a9996d36e2145636220 in term 2.
I20260808 01:48:27.901808  8772 raft_consensus.cc:697] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 2 LEADER]: Becoming Leader. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Running, Role: LEADER
I20260808 01:48:27.902833  8772 consensus_queue.cc:237] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:27.910130  8434 catalog_manager.cc:5697] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 reported cstate change: term changed from 1 to 2, leader changed from 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195) to 593c8fd64bf14a9996d36e2145636220 (127.7.165.194). New cstate: current_term: 2 leader_uuid: "593c8fd64bf14a9996d36e2145636220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:28.171332  8698 raft_consensus.cc:1275] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Refusing update from remote peer d731ac9726624d11a9f44e9bdbbf3fa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:28.172817  8839 consensus_queue.cc:1048] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:48:28.182739  8621 raft_consensus.cc:1275] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Refusing update from remote peer d731ac9726624d11a9f44e9bdbbf3fa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:28.185118  8839 consensus_queue.cc:1048] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:48:28.261094  8543 raft_consensus.cc:1275] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Refusing update from remote peer 593c8fd64bf14a9996d36e2145636220: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:28.262627  8772 consensus_queue.cc:1048] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:48:28.269275  8697 raft_consensus.cc:1275] T f390638b5fbd41b686f3fa99a5898f77 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Refusing update from remote peer 593c8fd64bf14a9996d36e2145636220: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:28.272044  8697 raft_consensus.cc:1275] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Refusing update from remote peer 593c8fd64bf14a9996d36e2145636220: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:28.272698  8772 consensus_queue.cc:1048] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:48:28.275401  8818 consensus_queue.cc:1048] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:48:28.297593  8621 raft_consensus.cc:1275] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Refusing update from remote peer d731ac9726624d11a9f44e9bdbbf3fa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:28.299265  8543 raft_consensus.cc:1275] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Refusing update from remote peer 593c8fd64bf14a9996d36e2145636220: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:28.299387  8839 consensus_queue.cc:1048] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:48:28.301191  8772 consensus_queue.cc:1048] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:48:28.311276  8698 raft_consensus.cc:1275] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Refusing update from remote peer 593c8fd64bf14a9996d36e2145636220: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:28.313844  8698 raft_consensus.cc:1275] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Refusing update from remote peer d731ac9726624d11a9f44e9bdbbf3fa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:28.315110  8845 consensus_queue.cc:1048] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:48:28.315623  8839 consensus_queue.cc:1048] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:48:28.328840  8543 raft_consensus.cc:1275] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Refusing update from remote peer 593c8fd64bf14a9996d36e2145636220: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:28.331358  8818 consensus_queue.cc:1048] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:48:30.592717  7831 auto_leader_rebalancer-test.cc:473] The leader distribution is 
I20260808 01:48:30.593717  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:30.619494  8543 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "490b233715cf4853916dd7273575b3a0"
dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
mode: GRACEFUL
new_leader_uuid: "96823ad418f9403cbaaac1ddff4834b2"
 from {username='slave'} at 127.0.0.1:34878
I20260808 01:48:30.620339  8543 raft_consensus.cc:606] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Received request to transfer leadership to TS 96823ad418f9403cbaaac1ddff4834b2
I20260808 01:48:30.623732  8543 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7e38a43fd7704fe8bec9648856983350"
dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
mode: GRACEFUL
new_leader_uuid: "96823ad418f9403cbaaac1ddff4834b2"
 from {username='slave'} at 127.0.0.1:34878
I20260808 01:48:30.624440  8543 raft_consensus.cc:606] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Received request to transfer leadership to TS 96823ad418f9403cbaaac1ddff4834b2
I20260808 01:48:30.625887  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 01:48:30.626407  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:30.792546  8850 raft_consensus.cc:993] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5: : Instructing follower 96823ad418f9403cbaaac1ddff4834b2 to start an election
I20260808 01:48:30.793247  8795 raft_consensus.cc:1081] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Signalling peer 96823ad418f9403cbaaac1ddff4834b2 to start an election
I20260808 01:48:30.795125  8697 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7e38a43fd7704fe8bec9648856983350"
dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
 from {username='slave'} at 127.0.0.1:41254
I20260808 01:48:30.795661  8697 raft_consensus.cc:493] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:30.796064  8697 raft_consensus.cc:3060] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:30.805217  8697 raft_consensus.cc:515] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:30.809230  8697 leader_election.cc:290] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 2 election: Requested vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883)
I20260808 01:48:30.811112  8543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e38a43fd7704fe8bec9648856983350" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
I20260808 01:48:30.811602  8621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e38a43fd7704fe8bec9648856983350" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "593c8fd64bf14a9996d36e2145636220"
I20260808 01:48:30.812001  8543 raft_consensus.cc:3055] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:48:30.812193  8621 raft_consensus.cc:3060] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:30.812398  8543 raft_consensus.cc:740] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Running, Role: LEADER
I20260808 01:48:30.813357  8543 consensus_queue.cc:260] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:30.814741  8543 raft_consensus.cc:3060] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:30.819105  8621 raft_consensus.cc:2468] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96823ad418f9403cbaaac1ddff4834b2 in term 2.
I20260808 01:48:30.821043  8660 leader_election.cc:304] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, 96823ad418f9403cbaaac1ddff4834b2; no voters: 
I20260808 01:48:30.822181  8799 raft_consensus.cc:2804] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:48:30.822105  8543 raft_consensus.cc:2468] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96823ad418f9403cbaaac1ddff4834b2 in term 2.
I20260808 01:48:30.824157  8799 raft_consensus.cc:697] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 LEADER]: Becoming Leader. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Running, Role: LEADER
I20260808 01:48:30.825372  8799 consensus_queue.cc:237] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:30.836576  8436 catalog_manager.cc:5697] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 reported cstate change: term changed from 1 to 2, leader changed from d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193) to 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195). New cstate: current_term: 2 leader_uuid: "96823ad418f9403cbaaac1ddff4834b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: HEALTHY } } }
I20260808 01:48:31.122318  8795 raft_consensus.cc:993] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5: : Instructing follower 96823ad418f9403cbaaac1ddff4834b2 to start an election
I20260808 01:48:31.122934  8850 raft_consensus.cc:1081] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Signalling peer 96823ad418f9403cbaaac1ddff4834b2 to start an election
I20260808 01:48:31.124742  8698 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "490b233715cf4853916dd7273575b3a0"
dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
 from {username='slave'} at 127.0.0.1:41254
I20260808 01:48:31.125181  8698 raft_consensus.cc:493] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:31.125536  8698 raft_consensus.cc:3060] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:31.131474  8698 raft_consensus.cc:515] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:31.133057  8698 leader_election.cc:290] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 2 election: Requested vote from peers 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883), d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659)
I20260808 01:48:31.134243  8543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "490b233715cf4853916dd7273575b3a0" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
I20260808 01:48:31.134182  8620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "490b233715cf4853916dd7273575b3a0" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "593c8fd64bf14a9996d36e2145636220"
I20260808 01:48:31.135140  8543 raft_consensus.cc:3055] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:48:31.135140  8620 raft_consensus.cc:3060] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:31.135789  8543 raft_consensus.cc:740] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Running, Role: LEADER
I20260808 01:48:31.136828  8543 consensus_queue.cc:260] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:31.138461  8543 raft_consensus.cc:3060] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:31.144426  8620 raft_consensus.cc:2468] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96823ad418f9403cbaaac1ddff4834b2 in term 2.
I20260808 01:48:31.146165  8660 leader_election.cc:304] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, 96823ad418f9403cbaaac1ddff4834b2; no voters: 
I20260808 01:48:31.147044  8799 raft_consensus.cc:2804] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:48:31.147006  8543 raft_consensus.cc:2468] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96823ad418f9403cbaaac1ddff4834b2 in term 2.
I20260808 01:48:31.147745  8799 raft_consensus.cc:697] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 LEADER]: Becoming Leader. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Running, Role: LEADER
I20260808 01:48:31.148722  8799 consensus_queue.cc:237] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:31.161844  8436 catalog_manager.cc:5697] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 reported cstate change: term changed from 1 to 2, leader changed from d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193) to 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195). New cstate: current_term: 2 leader_uuid: "96823ad418f9403cbaaac1ddff4834b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: HEALTHY } } }
I20260808 01:48:31.298372  8620 raft_consensus.cc:1275] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Refusing update from remote peer 96823ad418f9403cbaaac1ddff4834b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:31.300113  8799 consensus_queue.cc:1048] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:48:31.315651  8543 raft_consensus.cc:1275] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Refusing update from remote peer 96823ad418f9403cbaaac1ddff4834b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:31.317521  8854 consensus_queue.cc:1048] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:48:31.594255  8543 raft_consensus.cc:1275] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Refusing update from remote peer 96823ad418f9403cbaaac1ddff4834b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:31.595781  8854 consensus_queue.cc:1048] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:48:31.605711  8620 raft_consensus.cc:1275] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Refusing update from remote peer 96823ad418f9403cbaaac1ddff4834b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:31.607719  8863 consensus_queue.cc:1048] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:48:31.627525  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:31.632923  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:31.633396  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:32.634413  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:32.640398  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:32.640864  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:33.642371  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:33.648001  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:33.648494  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:34.649683  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:34.656230  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:34.656695  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:35.657720  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:35.663908  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:35.664379  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:36.665505  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:36.671223  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:36.671697  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:37.672647  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:37.679617  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:37.680258  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:38.714421  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:38.720180  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:38.720777  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
W20260808 01:48:39.424038  8471 debug-util.cc:398] Leaking SignalData structure 0x7b08000522e0 after lost signal to thread 7834
W20260808 01:48:39.424907  8471 debug-util.cc:398] Leaking SignalData structure 0x7b08000529c0 after lost signal to thread 8474
W20260808 01:48:39.425741  8471 debug-util.cc:398] Leaking SignalData structure 0x7b08001b3b40 after lost signal to thread 8569
W20260808 01:48:39.426478  8471 debug-util.cc:398] Leaking SignalData structure 0x7b08000d8380 after lost signal to thread 8647
W20260808 01:48:39.427515  8471 debug-util.cc:398] Leaking SignalData structure 0x7b0800094c40 after lost signal to thread 8725
I20260808 01:48:39.722230  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:39.732798  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:39.733678  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:40.781042  7831 auto_leader_rebalancer-test.cc:492] The leader distribution is 
I20260808 01:48:40.787006  8697 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "490b233715cf4853916dd7273575b3a0"
dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
mode: GRACEFUL
new_leader_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
 from {username='slave'} at 127.0.0.1:41274
I20260808 01:48:40.787909  8697 raft_consensus.cc:606] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 LEADER]: Received request to transfer leadership to TS d731ac9726624d11a9f44e9bdbbf3fa5
I20260808 01:48:40.802850  8697 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7e38a43fd7704fe8bec9648856983350"
dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
mode: GRACEFUL
new_leader_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
 from {username='slave'} at 127.0.0.1:41274
I20260808 01:48:40.803526  8697 raft_consensus.cc:606] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 LEADER]: Received request to transfer leadership to TS d731ac9726624d11a9f44e9bdbbf3fa5
I20260808 01:48:40.814206  8621 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cc880f9cc7844dae9f308ef3c2b9bbdb"
dest_uuid: "593c8fd64bf14a9996d36e2145636220"
mode: GRACEFUL
new_leader_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
 from {username='slave'} at 127.0.0.1:44304
I20260808 01:48:40.815294  8621 raft_consensus.cc:606] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 1 LEADER]: Received request to transfer leadership to TS d731ac9726624d11a9f44e9bdbbf3fa5
I20260808 01:48:40.822731  8621 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d471f57bba4e49f9b0e8c29880e2a864"
dest_uuid: "593c8fd64bf14a9996d36e2145636220"
mode: GRACEFUL
new_leader_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
 from {username='slave'} at 127.0.0.1:44304
I20260808 01:48:40.823314  8621 raft_consensus.cc:606] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 2 LEADER]: Received request to transfer leadership to TS d731ac9726624d11a9f44e9bdbbf3fa5
I20260808 01:48:40.829876  8621 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e2186a1e5b7a4dcb828dc73a73036684"
dest_uuid: "593c8fd64bf14a9996d36e2145636220"
mode: GRACEFUL
new_leader_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
 from {username='slave'} at 127.0.0.1:44304
I20260808 01:48:40.830497  8621 raft_consensus.cc:606] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 2 LEADER]: Received request to transfer leadership to TS d731ac9726624d11a9f44e9bdbbf3fa5
I20260808 01:48:40.954423  8876 raft_consensus.cc:993] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2: : Instructing follower d731ac9726624d11a9f44e9bdbbf3fa5 to start an election
I20260808 01:48:40.955195  8882 raft_consensus.cc:1081] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 LEADER]: Signalling peer d731ac9726624d11a9f44e9bdbbf3fa5 to start an election
I20260808 01:48:40.958582  8543 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7e38a43fd7704fe8bec9648856983350"
dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
 from {username='slave'} at 127.0.0.1:34846
I20260808 01:48:40.959339  8543 raft_consensus.cc:493] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:40.960142  8543 raft_consensus.cc:3060] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:40.970854  8543 raft_consensus.cc:515] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:40.973623  8543 leader_election.cc:290] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 3 election: Requested vote from peers 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:40.975023  8621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e38a43fd7704fe8bec9648856983350" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "593c8fd64bf14a9996d36e2145636220"
I20260808 01:48:40.975095  8697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e38a43fd7704fe8bec9648856983350" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
I20260808 01:48:40.976110  8621 raft_consensus.cc:3060] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:40.976223  8697 raft_consensus.cc:3055] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:48:40.976902  8697 raft_consensus.cc:740] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Running, Role: LEADER
I20260808 01:48:40.978160  8697 consensus_queue.cc:260] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:40.980545  8697 raft_consensus.cc:3060] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:40.989081  8621 raft_consensus.cc:2468] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 3.
I20260808 01:48:40.991051  8506 leader_election.cc:304] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:40.992259  8879 raft_consensus.cc:2804] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:48:40.992584  8697 raft_consensus.cc:2468] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 3.
I20260808 01:48:40.993752  8879 raft_consensus.cc:697] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 LEADER]: Becoming Leader. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Running, Role: LEADER
I20260808 01:48:40.995328  8879 consensus_queue.cc:237] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:41.017271  8435 catalog_manager.cc:5697] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 reported cstate change: term changed from 2 to 3, leader changed from 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195) to d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193). New cstate: current_term: 3 leader_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:41.018478  8886 raft_consensus.cc:993] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220: : Instructing follower d731ac9726624d11a9f44e9bdbbf3fa5 to start an election
I20260808 01:48:41.019294  8881 raft_consensus.cc:1081] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 2 LEADER]: Signalling peer d731ac9726624d11a9f44e9bdbbf3fa5 to start an election
I20260808 01:48:41.022317  8543 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e2186a1e5b7a4dcb828dc73a73036684"
dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
 from {username='slave'} at 127.0.0.1:34860
I20260808 01:48:41.023748  8543 raft_consensus.cc:493] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:41.024163  8543 raft_consensus.cc:3060] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:41.024778  8886 raft_consensus.cc:993] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220: : Instructing follower d731ac9726624d11a9f44e9bdbbf3fa5 to start an election
I20260808 01:48:41.025369  8881 raft_consensus.cc:1081] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 1 LEADER]: Signalling peer d731ac9726624d11a9f44e9bdbbf3fa5 to start an election
I20260808 01:48:41.028770  8542 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cc880f9cc7844dae9f308ef3c2b9bbdb"
dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
 from {username='slave'} at 127.0.0.1:34860
I20260808 01:48:41.029556  8542 raft_consensus.cc:493] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:41.029964  8542 raft_consensus.cc:3060] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:41.031049  8543 raft_consensus.cc:515] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:41.033306  8543 leader_election.cc:290] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 3 election: Requested vote from peers 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:41.034480  8697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2186a1e5b7a4dcb828dc73a73036684" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
I20260808 01:48:41.035108  8697 raft_consensus.cc:3060] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:41.043610  8621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2186a1e5b7a4dcb828dc73a73036684" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "593c8fd64bf14a9996d36e2145636220"
I20260808 01:48:41.044476  8621 raft_consensus.cc:3055] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:48:41.044857  8621 raft_consensus.cc:740] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Running, Role: LEADER
I20260808 01:48:41.047408  8542 raft_consensus.cc:515] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:41.051378  8542 leader_election.cc:290] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 2 election: Requested vote from peers 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:41.052706  8699 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc880f9cc7844dae9f308ef3c2b9bbdb" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
I20260808 01:48:41.053519  8699 raft_consensus.cc:3060] T cc880f9cc7844dae9f308ef3c2b9bbdb P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:41.055541  8623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc880f9cc7844dae9f308ef3c2b9bbdb" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "593c8fd64bf14a9996d36e2145636220"
I20260808 01:48:41.056288  8623 raft_consensus.cc:3055] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:48:41.056631  8623 raft_consensus.cc:740] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Running, Role: LEADER
I20260808 01:48:41.057637  8623 consensus_queue.cc:260] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:41.059716  8623 raft_consensus.cc:3060] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:41.063022  8699 raft_consensus.cc:2468] T cc880f9cc7844dae9f308ef3c2b9bbdb P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 2.
I20260808 01:48:41.065236  8508 leader_election.cc:304] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96823ad418f9403cbaaac1ddff4834b2, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:41.066960  8879 raft_consensus.cc:2804] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:48:41.046231  8621 consensus_queue.cc:260] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:41.067976  8621 raft_consensus.cc:3060] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:41.068706  8881 raft_consensus.cc:993] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220: : Instructing follower d731ac9726624d11a9f44e9bdbbf3fa5 to start an election
I20260808 01:48:41.069176  8881 raft_consensus.cc:1081] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 2 LEADER]: Signalling peer d731ac9726624d11a9f44e9bdbbf3fa5 to start an election
I20260808 01:48:41.076018  8697 raft_consensus.cc:2468] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 3.
I20260808 01:48:41.077461  8542 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d471f57bba4e49f9b0e8c29880e2a864"
dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
 from {username='slave'} at 127.0.0.1:34860
I20260808 01:48:41.078094  8542 raft_consensus.cc:493] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:41.079005  8698 raft_consensus.cc:1240] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 3 FOLLOWER]: Rejecting Update request from peer 593c8fd64bf14a9996d36e2145636220 for earlier term 2. Current term is 3. Ops: []
I20260808 01:48:41.078495  8542 raft_consensus.cc:3060] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:41.085688  8621 raft_consensus.cc:2468] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 3.
I20260808 01:48:41.087916  8506 leader_election.cc:304] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:41.089833  8883 raft_consensus.cc:2804] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:48:41.090603  8883 raft_consensus.cc:697] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 LEADER]: Becoming Leader. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Running, Role: LEADER
I20260808 01:48:41.092072  8883 consensus_queue.cc:237] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:41.092644  8542 raft_consensus.cc:515] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:41.081970  8623 raft_consensus.cc:2468] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 2.
I20260808 01:48:41.102828  8879 raft_consensus.cc:697] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 LEADER]: Becoming Leader. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Running, Role: LEADER
I20260808 01:48:41.110394  8698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d471f57bba4e49f9b0e8c29880e2a864" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
I20260808 01:48:41.111317  8698 raft_consensus.cc:3060] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:41.109768  8879 consensus_queue.cc:237] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:41.110778  8542 leader_election.cc:290] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 3 election: Requested vote from peers 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:41.119261  8621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d471f57bba4e49f9b0e8c29880e2a864" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "593c8fd64bf14a9996d36e2145636220"
I20260808 01:48:41.120499  8621 raft_consensus.cc:3055] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:48:41.121073  8621 raft_consensus.cc:740] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Running, Role: LEADER
I20260808 01:48:41.122934  8621 consensus_queue.cc:260] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:41.125018  8621 raft_consensus.cc:3060] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:41.126225  8698 raft_consensus.cc:2468] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 3.
I20260808 01:48:41.127741  8508 leader_election.cc:304] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96823ad418f9403cbaaac1ddff4834b2, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:41.130451  8891 raft_consensus.cc:2804] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:48:41.131702  8891 raft_consensus.cc:697] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 LEADER]: Becoming Leader. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Running, Role: LEADER
I20260808 01:48:41.134109  8891 consensus_queue.cc:237] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:41.138731  8621 raft_consensus.cc:2468] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 3.
I20260808 01:48:41.139390  8882 raft_consensus.cc:993] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2: : Instructing follower d731ac9726624d11a9f44e9bdbbf3fa5 to start an election
I20260808 01:48:41.142792  8876 raft_consensus.cc:1081] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 LEADER]: Signalling peer d731ac9726624d11a9f44e9bdbbf3fa5 to start an election
I20260808 01:48:41.146715  8542 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "490b233715cf4853916dd7273575b3a0"
dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
 from {username='slave'} at 127.0.0.1:34846
I20260808 01:48:41.148257  8542 raft_consensus.cc:493] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:41.149402  8542 raft_consensus.cc:3060] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:41.166579  8542 raft_consensus.cc:515] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:41.176379  8621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "490b233715cf4853916dd7273575b3a0" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "593c8fd64bf14a9996d36e2145636220"
I20260808 01:48:41.176662  8542 leader_election.cc:290] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 3 election: Requested vote from peers 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:41.177196  8435 catalog_manager.cc:5697] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 reported cstate change: term changed from 1 to 2, leader changed from 593c8fd64bf14a9996d36e2145636220 (127.7.165.194) to d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193). New cstate: current_term: 2 leader_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:41.177181  8621 raft_consensus.cc:3060] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:41.178253  8697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "490b233715cf4853916dd7273575b3a0" candidate_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
I20260808 01:48:41.179049  8697 raft_consensus.cc:3055] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:48:41.179380  8697 raft_consensus.cc:740] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Running, Role: LEADER
I20260808 01:48:41.180193  8697 consensus_queue.cc:260] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:41.181617  8697 raft_consensus.cc:3060] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:41.181447  8435 catalog_manager.cc:5697] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 reported cstate change: term changed from 2 to 3, leader changed from 593c8fd64bf14a9996d36e2145636220 (127.7.165.194) to d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193). New cstate: current_term: 3 leader_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:41.183400  8435 catalog_manager.cc:5697] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 reported cstate change: term changed from 2 to 3, leader changed from 593c8fd64bf14a9996d36e2145636220 (127.7.165.194) to d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193). New cstate: current_term: 3 leader_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:41.186131  8621 raft_consensus.cc:2468] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 3.
I20260808 01:48:41.187587  8506 leader_election.cc:304] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, d731ac9726624d11a9f44e9bdbbf3fa5; no voters: 
I20260808 01:48:41.188934  8891 raft_consensus.cc:2804] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:48:41.190011  8697 raft_consensus.cc:2468] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731ac9726624d11a9f44e9bdbbf3fa5 in term 3.
I20260808 01:48:41.189868  8891 raft_consensus.cc:697] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 LEADER]: Becoming Leader. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Running, Role: LEADER
I20260808 01:48:41.191543  8891 consensus_queue.cc:237] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:41.213732  8435 catalog_manager.cc:5697] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 reported cstate change: term changed from 2 to 3, leader changed from 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195) to d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193). New cstate: current_term: 3 leader_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:41.389497  8621 raft_consensus.cc:1275] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220 [term 3 FOLLOWER]: Refusing update from remote peer d731ac9726624d11a9f44e9bdbbf3fa5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:41.391511  8891 consensus_queue.cc:1048] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:48:41.421446  8697 raft_consensus.cc:1275] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 3 FOLLOWER]: Refusing update from remote peer d731ac9726624d11a9f44e9bdbbf3fa5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:41.427480  8879 consensus_queue.cc:1048] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:48:41.492341  8697 raft_consensus.cc:1275] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 3 FOLLOWER]: Refusing update from remote peer d731ac9726624d11a9f44e9bdbbf3fa5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:41.494442  8879 consensus_queue.cc:1048] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:48:41.518620  8623 raft_consensus.cc:1275] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 3 FOLLOWER]: Refusing update from remote peer d731ac9726624d11a9f44e9bdbbf3fa5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:41.520180  8698 raft_consensus.cc:1275] T cc880f9cc7844dae9f308ef3c2b9bbdb P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Refusing update from remote peer d731ac9726624d11a9f44e9bdbbf3fa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:41.521013  8879 consensus_queue.cc:1048] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:48:41.528317  8621 raft_consensus.cc:1275] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Refusing update from remote peer d731ac9726624d11a9f44e9bdbbf3fa5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:41.528707  8893 consensus_queue.cc:1048] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260808 01:48:41.531756  8883 consensus_queue.cc:1048] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:48:41.560969  8623 raft_consensus.cc:1275] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 3 FOLLOWER]: Refusing update from remote peer d731ac9726624d11a9f44e9bdbbf3fa5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:41.564195  8893 consensus_queue.cc:1048] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:48:41.609331  8698 raft_consensus.cc:1275] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 3 FOLLOWER]: Refusing update from remote peer d731ac9726624d11a9f44e9bdbbf3fa5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:41.611541  8879 consensus_queue.cc:1048] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:48:41.662020  8698 raft_consensus.cc:1275] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [term 3 FOLLOWER]: Refusing update from remote peer d731ac9726624d11a9f44e9bdbbf3fa5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:41.663558  8893 consensus_queue.cc:1048] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:48:41.676257  8623 raft_consensus.cc:1275] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220 [term 3 FOLLOWER]: Refusing update from remote peer d731ac9726624d11a9f44e9bdbbf3fa5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:41.677991  8891 consensus_queue.cc:1048] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:48:43.878010  7831 auto_leader_rebalancer-test.cc:507] The leader distribution is 
I20260808 01:48:43.878867  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:43.886945  8542 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "20f572798f1646acb92966b083e07eb2"
dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
mode: GRACEFUL
new_leader_uuid: "593c8fd64bf14a9996d36e2145636220"
 from {username='slave'} at 127.0.0.1:34878
I20260808 01:48:43.887486  8542 raft_consensus.cc:606] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Received request to transfer leadership to TS 593c8fd64bf14a9996d36e2145636220
I20260808 01:48:43.891577  8542 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "537921115d13455d98e32f8ad895e87a"
dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
mode: GRACEFUL
new_leader_uuid: "96823ad418f9403cbaaac1ddff4834b2"
 from {username='slave'} at 127.0.0.1:34878
I20260808 01:48:43.892581  8542 raft_consensus.cc:606] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 LEADER]: Received request to transfer leadership to TS 96823ad418f9403cbaaac1ddff4834b2
I20260808 01:48:43.895869  8542 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7e38a43fd7704fe8bec9648856983350"
dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
mode: GRACEFUL
new_leader_uuid: "96823ad418f9403cbaaac1ddff4834b2"
 from {username='slave'} at 127.0.0.1:34878
I20260808 01:48:43.896534  8542 raft_consensus.cc:606] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 LEADER]: Received request to transfer leadership to TS 96823ad418f9403cbaaac1ddff4834b2
I20260808 01:48:43.899514  8542 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d471f57bba4e49f9b0e8c29880e2a864"
dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
mode: GRACEFUL
new_leader_uuid: "593c8fd64bf14a9996d36e2145636220"
 from {username='slave'} at 127.0.0.1:34878
I20260808 01:48:43.900293  8542 raft_consensus.cc:606] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 LEADER]: Received request to transfer leadership to TS 593c8fd64bf14a9996d36e2145636220
I20260808 01:48:43.902101  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260808 01:48:43.902578  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:44.038367  8891 raft_consensus.cc:993] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5: : Instructing follower 96823ad418f9403cbaaac1ddff4834b2 to start an election
I20260808 01:48:44.038987  8891 raft_consensus.cc:1081] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 LEADER]: Signalling peer 96823ad418f9403cbaaac1ddff4834b2 to start an election
I20260808 01:48:44.040529  8697 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7e38a43fd7704fe8bec9648856983350"
dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
 from {username='slave'} at 127.0.0.1:41254
I20260808 01:48:44.041085  8697 raft_consensus.cc:493] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:44.041342  8697 raft_consensus.cc:3060] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:48:44.046170  8697 raft_consensus.cc:515] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:44.048621  8697 leader_election.cc:290] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 4 election: Requested vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883)
I20260808 01:48:44.049634  8542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e38a43fd7704fe8bec9648856983350" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
I20260808 01:48:44.049793  8623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7e38a43fd7704fe8bec9648856983350" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "593c8fd64bf14a9996d36e2145636220"
I20260808 01:48:44.050379  8542 raft_consensus.cc:3055] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:48:44.050776  8542 raft_consensus.cc:740] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Running, Role: LEADER
I20260808 01:48:44.050810  8623 raft_consensus.cc:3060] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:48:44.051509  8542 consensus_queue.cc:260] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:44.052588  8542 raft_consensus.cc:3060] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:48:44.056635  8623 raft_consensus.cc:2468] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96823ad418f9403cbaaac1ddff4834b2 in term 4.
I20260808 01:48:44.057492  8542 raft_consensus.cc:2468] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96823ad418f9403cbaaac1ddff4834b2 in term 4.
I20260808 01:48:44.057994  8660 leader_election.cc:304] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, 96823ad418f9403cbaaac1ddff4834b2; no voters: 
I20260808 01:48:44.058815  8915 raft_consensus.cc:2804] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:48:44.060041  8915 raft_consensus.cc:697] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 4 LEADER]: Becoming Leader. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Running, Role: LEADER
I20260808 01:48:44.060865  8915 consensus_queue.cc:237] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:44.067529  8436 catalog_manager.cc:5697] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 reported cstate change: term changed from 3 to 4, leader changed from d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193) to 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195). New cstate: current_term: 4 leader_uuid: "96823ad418f9403cbaaac1ddff4834b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: HEALTHY } } }
I20260808 01:48:44.095039  8891 raft_consensus.cc:993] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5: : Instructing follower 593c8fd64bf14a9996d36e2145636220 to start an election
I20260808 01:48:44.095592  8916 raft_consensus.cc:1081] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 LEADER]: Signalling peer 593c8fd64bf14a9996d36e2145636220 to start an election
I20260808 01:48:44.097247  8623 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d471f57bba4e49f9b0e8c29880e2a864"
dest_uuid: "593c8fd64bf14a9996d36e2145636220"
 from {username='slave'} at 127.0.0.1:44294
I20260808 01:48:44.097661  8623 raft_consensus.cc:493] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:44.097883  8623 raft_consensus.cc:3060] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:48:44.103026  8623 raft_consensus.cc:515] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:44.104854  8623 leader_election.cc:290] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 4 election: Requested vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:44.106042  8697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d471f57bba4e49f9b0e8c29880e2a864" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
I20260808 01:48:44.106042  8542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d471f57bba4e49f9b0e8c29880e2a864" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
I20260808 01:48:44.106851  8697 raft_consensus.cc:3060] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:48:44.106873  8542 raft_consensus.cc:3055] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:48:44.107409  8542 raft_consensus.cc:740] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Running, Role: LEADER
I20260808 01:48:44.108180  8542 consensus_queue.cc:260] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:44.109395  8542 raft_consensus.cc:3060] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:48:44.114181  8697 raft_consensus.cc:2468] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593c8fd64bf14a9996d36e2145636220 in term 4.
I20260808 01:48:44.115665  8587 leader_election.cc:304] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, 96823ad418f9403cbaaac1ddff4834b2; no voters: 
I20260808 01:48:44.116005  8542 raft_consensus.cc:2468] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593c8fd64bf14a9996d36e2145636220 in term 4.
I20260808 01:48:44.116436  8914 raft_consensus.cc:2804] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:48:44.117770  8914 raft_consensus.cc:697] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 4 LEADER]: Becoming Leader. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Running, Role: LEADER
I20260808 01:48:44.118541  8914 consensus_queue.cc:237] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:44.126514  8436 catalog_manager.cc:5697] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 reported cstate change: term changed from 3 to 4, leader changed from d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193) to 593c8fd64bf14a9996d36e2145636220 (127.7.165.194). New cstate: current_term: 4 leader_uuid: "593c8fd64bf14a9996d36e2145636220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:44.130334  8891 raft_consensus.cc:993] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5: : Instructing follower 593c8fd64bf14a9996d36e2145636220 to start an election
I20260808 01:48:44.131057  8916 raft_consensus.cc:1081] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Signalling peer 593c8fd64bf14a9996d36e2145636220 to start an election
I20260808 01:48:44.133039  8623 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "20f572798f1646acb92966b083e07eb2"
dest_uuid: "593c8fd64bf14a9996d36e2145636220"
 from {username='slave'} at 127.0.0.1:44294
I20260808 01:48:44.133728  8623 raft_consensus.cc:493] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:44.134094  8623 raft_consensus.cc:3060] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:44.139292  8623 raft_consensus.cc:515] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:44.141409  8623 leader_election.cc:290] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 2 election: Requested vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195:39507)
I20260808 01:48:44.141867  8542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20f572798f1646acb92966b083e07eb2" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
I20260808 01:48:44.142069  8697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20f572798f1646acb92966b083e07eb2" candidate_uuid: "593c8fd64bf14a9996d36e2145636220" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
I20260808 01:48:44.142571  8697 raft_consensus.cc:3060] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:44.142530  8542 raft_consensus.cc:3055] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:48:44.143160  8542 raft_consensus.cc:740] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Running, Role: LEADER
I20260808 01:48:44.144047  8542 consensus_queue.cc:260] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:44.145424  8542 raft_consensus.cc:3060] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:48:44.150377  8697 raft_consensus.cc:2468] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593c8fd64bf14a9996d36e2145636220 in term 2.
I20260808 01:48:44.151945  8587 leader_election.cc:304] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, 96823ad418f9403cbaaac1ddff4834b2; no voters: 
I20260808 01:48:44.152752  8914 raft_consensus.cc:2804] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:48:44.152788  8542 raft_consensus.cc:2468] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 593c8fd64bf14a9996d36e2145636220 in term 2.
I20260808 01:48:44.153266  8914 raft_consensus.cc:697] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 [term 2 LEADER]: Becoming Leader. State: Replica: 593c8fd64bf14a9996d36e2145636220, State: Running, Role: LEADER
I20260808 01:48:44.154294  8914 consensus_queue.cc:237] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:44.162247  8436 catalog_manager.cc:5697] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 reported cstate change: term changed from 1 to 2, leader changed from d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193) to 593c8fd64bf14a9996d36e2145636220 (127.7.165.194). New cstate: current_term: 2 leader_uuid: "593c8fd64bf14a9996d36e2145636220" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:44.255318  8916 raft_consensus.cc:993] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5: : Instructing follower 96823ad418f9403cbaaac1ddff4834b2 to start an election
I20260808 01:48:44.255936  8891 raft_consensus.cc:1081] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 LEADER]: Signalling peer 96823ad418f9403cbaaac1ddff4834b2 to start an election
I20260808 01:48:44.257707  8697 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "537921115d13455d98e32f8ad895e87a"
dest_uuid: "96823ad418f9403cbaaac1ddff4834b2"
 from {username='slave'} at 127.0.0.1:41254
I20260808 01:48:44.258293  8697 raft_consensus.cc:493] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:48:44.258554  8697 raft_consensus.cc:3060] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:44.264249  8697 raft_consensus.cc:515] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:44.266757  8697 leader_election.cc:290] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 3 election: Requested vote from peers d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193:37659), 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883)
I20260808 01:48:44.267735  8542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "537921115d13455d98e32f8ad895e87a" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5"
I20260808 01:48:44.268049  8623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "537921115d13455d98e32f8ad895e87a" candidate_uuid: "96823ad418f9403cbaaac1ddff4834b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "593c8fd64bf14a9996d36e2145636220"
I20260808 01:48:44.268350  8542 raft_consensus.cc:3055] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:48:44.268555  8623 raft_consensus.cc:3060] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:44.268694  8542 raft_consensus.cc:740] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d731ac9726624d11a9f44e9bdbbf3fa5, State: Running, Role: LEADER
I20260808 01:48:44.269541  8542 consensus_queue.cc:260] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:44.270777  8542 raft_consensus.cc:3060] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:48:44.275558  8623 raft_consensus.cc:2468] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96823ad418f9403cbaaac1ddff4834b2 in term 3.
I20260808 01:48:44.277144  8660 leader_election.cc:304] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 593c8fd64bf14a9996d36e2145636220, 96823ad418f9403cbaaac1ddff4834b2; no voters: 
I20260808 01:48:44.277971  8915 raft_consensus.cc:2804] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:48:44.278534  8915 raft_consensus.cc:697] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [term 3 LEADER]: Becoming Leader. State: Replica: 96823ad418f9403cbaaac1ddff4834b2, State: Running, Role: LEADER
I20260808 01:48:44.279481  8915 consensus_queue.cc:237] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } }
I20260808 01:48:44.280373  8542 raft_consensus.cc:2468] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96823ad418f9403cbaaac1ddff4834b2 in term 3.
I20260808 01:48:44.287220  8436 catalog_manager.cc:5697] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 reported cstate change: term changed from 2 to 3, leader changed from d731ac9726624d11a9f44e9bdbbf3fa5 (127.7.165.193) to 96823ad418f9403cbaaac1ddff4834b2 (127.7.165.195). New cstate: current_term: 3 leader_uuid: "96823ad418f9403cbaaac1ddff4834b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 } health_report { overall_health: HEALTHY } } }
I20260808 01:48:44.467087  8542 raft_consensus.cc:1275] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 4 FOLLOWER]: Refusing update from remote peer 96823ad418f9403cbaaac1ddff4834b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:48:44.468542  8915 consensus_queue.cc:1048] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:48:44.484877  8623 raft_consensus.cc:1275] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220 [term 4 FOLLOWER]: Refusing update from remote peer 96823ad418f9403cbaaac1ddff4834b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:48:44.487732  8917 consensus_queue.cc:1048] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:48:44.674038  8623 raft_consensus.cc:1275] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 3 FOLLOWER]: Refusing update from remote peer 96823ad418f9403cbaaac1ddff4834b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:44.675496  8917 consensus_queue.cc:1048] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593c8fd64bf14a9996d36e2145636220" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:48:44.687703  8542 raft_consensus.cc:1275] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 FOLLOWER]: Refusing update from remote peer 96823ad418f9403cbaaac1ddff4834b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:48:44.689580  8915 consensus_queue.cc:1048] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:48:44.701491  8543 raft_consensus.cc:1275] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Refusing update from remote peer 593c8fd64bf14a9996d36e2145636220: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:44.704262  8697 raft_consensus.cc:1275] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 4 FOLLOWER]: Refusing update from remote peer 593c8fd64bf14a9996d36e2145636220: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:48:44.704185  8914 consensus_queue.cc:1048] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:48:44.710008  8914 consensus_queue.cc:1048] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:48:44.717892  8542 raft_consensus.cc:1275] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 4 FOLLOWER]: Refusing update from remote peer 593c8fd64bf14a9996d36e2145636220: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:48:44.731698  8698 raft_consensus.cc:1275] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Refusing update from remote peer 593c8fd64bf14a9996d36e2145636220: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:48:44.736172  8938 consensus_queue.cc:1048] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d731ac9726624d11a9f44e9bdbbf3fa5" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 37659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:48:44.738193  8942 consensus_queue.cc:1048] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96823ad418f9403cbaaac1ddff4834b2" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260808 01:48:44.903846  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:44.910861  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:44.911379  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:45.912477  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:45.919407  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:45.919896  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:46.921067  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:46.927429  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:46.927963  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:47.929139  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:47.935400  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:47.936025  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:48.937188  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:48.944795  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:48.945330  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:49.946437  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:49.955211  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:49.955770  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:50.957130  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:50.963097  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:50.963706  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:51.965065  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:51.970878  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:51.971439  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:52.972592  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:48:52.978542  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:52.979121  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:54.018425  7831 auto_leader_rebalancer-test.cc:518] The leader distribution is 
I20260808 01:48:54.019109  7831 tablet_server.cc:179] TabletServer@127.7.165.193:0 shutting down...
I20260808 01:48:54.059441  7831 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260808 01:48:54.060462  7831 tablet_replica.cc:333] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5: stopping tablet replica
I20260808 01:48:54.061667  7831 raft_consensus.cc:2243] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.062762  7831 raft_consensus.cc:2272] T f390638b5fbd41b686f3fa99a5898f77 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.064975  7831 tablet_replica.cc:333] T 598364b36429453793bb8922ded6a556 P d731ac9726624d11a9f44e9bdbbf3fa5: stopping tablet replica
I20260808 01:48:54.065516  7831 raft_consensus.cc:2243] T 598364b36429453793bb8922ded6a556 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.065917  7831 raft_consensus.cc:2272] T 598364b36429453793bb8922ded6a556 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.067921  7831 tablet_replica.cc:333] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5: stopping tablet replica
I20260808 01:48:54.068584  7831 raft_consensus.cc:2243] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.069094  7831 raft_consensus.cc:2272] T 20f572798f1646acb92966b083e07eb2 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.071242  7831 tablet_replica.cc:333] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5: stopping tablet replica
I20260808 01:48:54.071789  7831 raft_consensus.cc:2243] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 LEADER]: Raft consensus shutting down.
I20260808 01:48:54.072963  7831 raft_consensus.cc:2272] T cc880f9cc7844dae9f308ef3c2b9bbdb P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.075237  7831 tablet_replica.cc:333] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5: stopping tablet replica
I20260808 01:48:54.075834  7831 raft_consensus.cc:2243] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.076337  7831 raft_consensus.cc:2272] T 537921115d13455d98e32f8ad895e87a P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.079178  7831 tablet_replica.cc:333] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5: stopping tablet replica
I20260808 01:48:54.079759  7831 raft_consensus.cc:2243] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 LEADER]: Raft consensus shutting down.
I20260808 01:48:54.080680  7831 raft_consensus.cc:2272] T e2186a1e5b7a4dcb828dc73a73036684 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.082647  7831 tablet_replica.cc:333] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5: stopping tablet replica
I20260808 01:48:54.083290  7831 raft_consensus.cc:2243] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.083873  7831 raft_consensus.cc:2272] T 7e38a43fd7704fe8bec9648856983350 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.085981  7831 tablet_replica.cc:333] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5: stopping tablet replica
I20260808 01:48:54.086529  7831 raft_consensus.cc:2243] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.087122  7831 raft_consensus.cc:2272] T d471f57bba4e49f9b0e8c29880e2a864 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.088969  7831 tablet_replica.cc:333] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5: stopping tablet replica
I20260808 01:48:54.089473  7831 raft_consensus.cc:2243] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 LEADER]: Raft consensus shutting down.
I20260808 01:48:54.090301  7831 raft_consensus.cc:2272] T e61a690621224f02a28c606e3228c838 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.092242  7831 tablet_replica.cc:333] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5: stopping tablet replica
I20260808 01:48:54.092732  7831 raft_consensus.cc:2243] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 LEADER]: Raft consensus shutting down.
I20260808 01:48:54.093636  7831 raft_consensus.cc:2272] T 490b233715cf4853916dd7273575b3a0 P d731ac9726624d11a9f44e9bdbbf3fa5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.117579  7831 tablet_server.cc:196] TabletServer@127.7.165.193:0 shutdown complete.
I20260808 01:48:54.141930  7831 tablet_server.cc:179] TabletServer@127.7.165.194:0 shutting down...
I20260808 01:48:54.177307  7831 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260808 01:48:54.178380  7831 tablet_replica.cc:333] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220: stopping tablet replica
I20260808 01:48:54.179458  7831 raft_consensus.cc:2243] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 [term 2 LEADER]: Raft consensus shutting down.
I20260808 01:48:54.180595  7831 raft_consensus.cc:2272] T f390638b5fbd41b686f3fa99a5898f77 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.183058  7831 tablet_replica.cc:333] T 598364b36429453793bb8922ded6a556 P 593c8fd64bf14a9996d36e2145636220: stopping tablet replica
I20260808 01:48:54.183745  7831 raft_consensus.cc:2243] T 598364b36429453793bb8922ded6a556 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.184307  7831 raft_consensus.cc:2272] T 598364b36429453793bb8922ded6a556 P 593c8fd64bf14a9996d36e2145636220 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.186434  7831 tablet_replica.cc:333] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220: stopping tablet replica
I20260808 01:48:54.187199  7831 raft_consensus.cc:2243] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 [term 2 LEADER]: Raft consensus shutting down.
I20260808 01:48:54.188110  7831 raft_consensus.cc:2272] T 20f572798f1646acb92966b083e07eb2 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.190515  7831 tablet_replica.cc:333] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220: stopping tablet replica
I20260808 01:48:54.191236  7831 raft_consensus.cc:2243] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.191736  7831 raft_consensus.cc:2272] T 7e38a43fd7704fe8bec9648856983350 P 593c8fd64bf14a9996d36e2145636220 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.193869  7831 tablet_replica.cc:333] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220: stopping tablet replica
I20260808 01:48:54.194530  7831 raft_consensus.cc:2243] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.195160  7831 raft_consensus.cc:2272] T e2186a1e5b7a4dcb828dc73a73036684 P 593c8fd64bf14a9996d36e2145636220 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.197302  7831 tablet_replica.cc:333] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220: stopping tablet replica
I20260808 01:48:54.197912  7831 raft_consensus.cc:2243] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.198427  7831 raft_consensus.cc:2272] T cc880f9cc7844dae9f308ef3c2b9bbdb P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.200827  7831 tablet_replica.cc:333] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220: stopping tablet replica
I20260808 01:48:54.201705  7831 raft_consensus.cc:2243] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.202404  7831 raft_consensus.cc:2272] T 537921115d13455d98e32f8ad895e87a P 593c8fd64bf14a9996d36e2145636220 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260808 01:48:54.202828  8660 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 50 similar messages]
I20260808 01:48:54.205186  7831 tablet_replica.cc:333] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220: stopping tablet replica
I20260808 01:48:54.206094  7831 raft_consensus.cc:2243] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 4 LEADER]: Raft consensus shutting down.
W20260808 01:48:54.206537  8660 consensus_peers.cc:597] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 -> Peer 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883): Couldn't send request to peer 593c8fd64bf14a9996d36e2145636220. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:54.207629  7831 raft_consensus.cc:2272] T d471f57bba4e49f9b0e8c29880e2a864 P 593c8fd64bf14a9996d36e2145636220 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.209975  7831 tablet_replica.cc:333] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220: stopping tablet replica
I20260808 01:48:54.210510  7831 raft_consensus.cc:2243] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.211052  7831 raft_consensus.cc:2272] T e61a690621224f02a28c606e3228c838 P 593c8fd64bf14a9996d36e2145636220 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.213011  7831 tablet_replica.cc:333] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220: stopping tablet replica
I20260808 01:48:54.213529  7831 raft_consensus.cc:2243] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.214035  7831 raft_consensus.cc:2272] T 490b233715cf4853916dd7273575b3a0 P 593c8fd64bf14a9996d36e2145636220 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.247987  7831 tablet_server.cc:196] TabletServer@127.7.165.194:0 shutdown complete.
I20260808 01:48:54.270241  7831 tablet_server.cc:179] TabletServer@127.7.165.195:0 shutting down...
I20260808 01:48:54.305750  7831 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260808 01:48:54.306605  7831 tablet_replica.cc:333] T f390638b5fbd41b686f3fa99a5898f77 P 96823ad418f9403cbaaac1ddff4834b2: stopping tablet replica
I20260808 01:48:54.307736  7831 raft_consensus.cc:2243] T f390638b5fbd41b686f3fa99a5898f77 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.308502  7831 raft_consensus.cc:2272] T f390638b5fbd41b686f3fa99a5898f77 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.310851  7831 tablet_replica.cc:333] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2: stopping tablet replica
I20260808 01:48:54.311421  7831 raft_consensus.cc:2243] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.311899  7831 raft_consensus.cc:2272] T d471f57bba4e49f9b0e8c29880e2a864 P 96823ad418f9403cbaaac1ddff4834b2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.313791  7831 tablet_replica.cc:333] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2: stopping tablet replica
I20260808 01:48:54.314314  7831 raft_consensus.cc:2243] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.314785  7831 raft_consensus.cc:2272] T e61a690621224f02a28c606e3228c838 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.316583  7831 tablet_replica.cc:333] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2: stopping tablet replica
I20260808 01:48:54.317085  7831 raft_consensus.cc:2243] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.317489  7831 raft_consensus.cc:2272] T 490b233715cf4853916dd7273575b3a0 P 96823ad418f9403cbaaac1ddff4834b2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.319310  7831 tablet_replica.cc:333] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2: stopping tablet replica
I20260808 01:48:54.321182  7831 raft_consensus.cc:2243] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.321784  7831 raft_consensus.cc:2272] T e2186a1e5b7a4dcb828dc73a73036684 P 96823ad418f9403cbaaac1ddff4834b2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.324307  7831 tablet_replica.cc:333] T cc880f9cc7844dae9f308ef3c2b9bbdb P 96823ad418f9403cbaaac1ddff4834b2: stopping tablet replica
I20260808 01:48:54.325024  7831 raft_consensus.cc:2243] T cc880f9cc7844dae9f308ef3c2b9bbdb P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.325552  7831 raft_consensus.cc:2272] T cc880f9cc7844dae9f308ef3c2b9bbdb P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260808 01:48:54.327210  8660 consensus_peers.cc:597] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 -> Peer 593c8fd64bf14a9996d36e2145636220 (127.7.165.194:32883): Couldn't send request to peer 593c8fd64bf14a9996d36e2145636220. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:32883: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:54.327960  7831 tablet_replica.cc:333] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2: stopping tablet replica
I20260808 01:48:54.328563  7831 raft_consensus.cc:2243] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [term 3 LEADER]: Raft consensus shutting down.
I20260808 01:48:54.329411  7831 raft_consensus.cc:2272] T 537921115d13455d98e32f8ad895e87a P 96823ad418f9403cbaaac1ddff4834b2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.331528  7831 tablet_replica.cc:333] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2: stopping tablet replica
I20260808 01:48:54.332105  7831 raft_consensus.cc:2243] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 4 LEADER]: Raft consensus shutting down.
I20260808 01:48:54.332924  7831 raft_consensus.cc:2272] T 7e38a43fd7704fe8bec9648856983350 P 96823ad418f9403cbaaac1ddff4834b2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.334973  7831 tablet_replica.cc:333] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2: stopping tablet replica
I20260808 01:48:54.335526  7831 raft_consensus.cc:2243] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 01:48:54.336334  7831 raft_consensus.cc:2272] T 598364b36429453793bb8922ded6a556 P 96823ad418f9403cbaaac1ddff4834b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.338150  7831 tablet_replica.cc:333] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2: stopping tablet replica
I20260808 01:48:54.338714  7831 raft_consensus.cc:2243] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:54.339166  7831 raft_consensus.cc:2272] T 20f572798f1646acb92966b083e07eb2 P 96823ad418f9403cbaaac1ddff4834b2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.370514  7831 tablet_server.cc:196] TabletServer@127.7.165.195:0 shutdown complete.
I20260808 01:48:54.395352  7831 master.cc:562] Master@127.7.165.254:45413 shutting down...
I20260808 01:48:54.423374  7831 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [term 1 LEADER]: Raft consensus shutting down.
I20260808 01:48:54.424038  7831 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:54.424396  7831 tablet_replica.cc:333] T 00000000000000000000000000000000 P 289bf78aad494e20a2a7447e705689ec: stopping tablet replica
I20260808 01:48:54.464661  7831 master.cc:584] Master@127.7.165.254:45413 shutdown complete.
593c8fd64bf14a9996d36e2145636220  4
96823ad418f9403cbaaac1ddff4834b2  1
d731ac9726624d11a9f44e9bdbbf3fa5  5
593c8fd64bf14a9996d36e2145636220  4
96823ad418f9403cbaaac1ddff4834b2  3
d731ac9726624d11a9f44e9bdbbf3fa5  3
593c8fd64bf14a9996d36e2145636220  1
96823ad418f9403cbaaac1ddff4834b2  1
d731ac9726624d11a9f44e9bdbbf3fa5  8
593c8fd64bf14a9996d36e2145636220  3
96823ad418f9403cbaaac1ddff4834b2  3
d731ac9726624d11a9f44e9bdbbf3fa5  4
[       OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (29329 ms)
[ RUN      ] LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass
I20260808 01:48:54.518635  7831 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.7.165.254:43379
I20260808 01:48:54.520159  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:54.526808  8960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:54.527416  8959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:54.528093  8962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:54.529752  7831 server_base.cc:1061] running on GCE node
I20260808 01:48:54.531178  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:48:54.531387  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:48:54.531584  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786153734531567 us; error 0 us; skew 500 ppm
I20260808 01:48:54.532205  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:54.535059  7831 webserver.cc:533] Webserver started at http://127.7.165.254:42355/ using document root <none> and password file <none>
I20260808 01:48:54.535640  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:54.535852  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:54.536164  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:54.537482  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/master-0-root/instance:
uuid: "7fd9d1d2beb9480fb21203a4e1aa0d8e"
format_stamp: "Formatted at 2026-08-08 01:48:54 on dist-test-slave-8tdl"
I20260808 01:48:54.542842  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.003s
I20260808 01:48:54.546934  8968 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:54.547991  7831 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260808 01:48:54.548262  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/master-0-root
uuid: "7fd9d1d2beb9480fb21203a4e1aa0d8e"
format_stamp: "Formatted at 2026-08-08 01:48:54 on dist-test-slave-8tdl"
I20260808 01:48:54.548547  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:54.588330  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:48:54.589833  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:54.636447  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.254:43379
I20260808 01:48:54.636551  9023 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.254:43379 every 8 connection(s)
I20260808 01:48:54.641000  9024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:54.652112  9024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e: Bootstrap starting.
I20260808 01:48:54.657430  9024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:54.662545  9024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e: No bootstrap required, opened a new log
I20260808 01:48:54.665138  9024 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd9d1d2beb9480fb21203a4e1aa0d8e" member_type: VOTER }
I20260808 01:48:54.665616  9024 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:54.665921  9024 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd9d1d2beb9480fb21203a4e1aa0d8e, State: Initialized, Role: FOLLOWER
I20260808 01:48:54.666807  9024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd9d1d2beb9480fb21203a4e1aa0d8e" member_type: VOTER }
I20260808 01:48:54.667580  9024 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 01:48:54.668052  9024 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 01:48:54.668417  9024 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:54.673348  9024 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd9d1d2beb9480fb21203a4e1aa0d8e" member_type: VOTER }
I20260808 01:48:54.673985  9024 leader_election.cc:304] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fd9d1d2beb9480fb21203a4e1aa0d8e; no voters: 
I20260808 01:48:54.675469  9024 leader_election.cc:290] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260808 01:48:54.676009  9027 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:54.677487  9027 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [term 1 LEADER]: Becoming Leader. State: Replica: 7fd9d1d2beb9480fb21203a4e1aa0d8e, State: Running, Role: LEADER
I20260808 01:48:54.678334  9027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd9d1d2beb9480fb21203a4e1aa0d8e" member_type: VOTER }
I20260808 01:48:54.678584  9024 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 01:48:54.683739  9028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fd9d1d2beb9480fb21203a4e1aa0d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd9d1d2beb9480fb21203a4e1aa0d8e" member_type: VOTER } }
I20260808 01:48:54.683908  9029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fd9d1d2beb9480fb21203a4e1aa0d8e. Latest consensus state: current_term: 1 leader_uuid: "7fd9d1d2beb9480fb21203a4e1aa0d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd9d1d2beb9480fb21203a4e1aa0d8e" member_type: VOTER } }
I20260808 01:48:54.684665  9029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [sys.catalog]: This master's current role is: LEADER
I20260808 01:48:54.686539  9028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [sys.catalog]: This master's current role is: LEADER
I20260808 01:48:54.687044  9035 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260808 01:48:54.693243  9035 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260808 01:48:54.697307  7831 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260808 01:48:54.704883  9035 catalog_manager.cc:1383] Generated new cluster ID: 1e92541ddf9f44cd9de054c34f542612
I20260808 01:48:54.705260  9035 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260808 01:48:54.719520  9035 catalog_manager.cc:1406] Generated new certificate authority record
I20260808 01:48:54.720968  9035 catalog_manager.cc:1540] Loading token signing keys...
I20260808 01:48:54.740031  9035 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e: Generated new TSK 0
I20260808 01:48:54.740731  9035 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260808 01:48:54.765151  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:54.772528  9047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:54.773859  9048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:54.776752  9050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:54.777060  7831 server_base.cc:1061] running on GCE node
I20260808 01:48:54.778236  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:48:54.778523  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:48:54.778740  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786153734778688 us; error 0 us; skew 500 ppm
I20260808 01:48:54.779389  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:54.782757  7831 webserver.cc:533] Webserver started at http://127.7.165.193:45929/ using document root <none> and password file <none>
I20260808 01:48:54.783639  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:54.783931  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:54.784272  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:54.785543  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/ts-0-root/instance:
uuid: "e91d6edcb50845178ee3c8d60fb4ac4c"
format_stamp: "Formatted at 2026-08-08 01:48:54 on dist-test-slave-8tdl"
I20260808 01:48:54.791491  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.002s
I20260808 01:48:54.795959  9055 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:54.797034  7831 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260808 01:48:54.797367  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/ts-0-root
uuid: "e91d6edcb50845178ee3c8d60fb4ac4c"
format_stamp: "Formatted at 2026-08-08 01:48:54 on dist-test-slave-8tdl"
I20260808 01:48:54.797652  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:54.811379  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:48:54.812812  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:54.814427  7831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:54.817001  7831 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:54.817200  7831 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:54.817513  7831 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:54.817674  7831 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:54.864359  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.193:43869
I20260808 01:48:54.864480  9121 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.193:43869 every 8 connection(s)
I20260808 01:48:54.869863  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:54.878542  9126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:54.879729  9127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:54.888623  9122 heartbeater.cc:344] Connected to a master server at 127.7.165.254:43379
I20260808 01:48:54.889130  9122 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:54.890172  9122 heartbeater.cc:507] Master 127.7.165.254:43379 requested a full tablet report, sending...
W20260808 01:48:54.890540  9130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:54.890585  7831 server_base.cc:1061] running on GCE node
I20260808 01:48:54.893204  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:48:54.893471  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:48:54.893682  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786153734893660 us; error 0 us; skew 500 ppm
I20260808 01:48:54.894109  8986 ts_manager.cc:194] Registered new tserver with Master: e91d6edcb50845178ee3c8d60fb4ac4c (127.7.165.193:43869)
I20260808 01:48:54.894297  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:54.896499  8986 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47824
I20260808 01:48:54.897396  7831 webserver.cc:533] Webserver started at http://127.7.165.194:39651/ using document root <none> and password file <none>
I20260808 01:48:54.897964  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:54.898192  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:54.898500  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:54.900074  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/ts-1-root/instance:
uuid: "3aaa5fb00ae84069a009f4dee2917bc2"
format_stamp: "Formatted at 2026-08-08 01:48:54 on dist-test-slave-8tdl"
I20260808 01:48:54.905741  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20260808 01:48:54.909933  9135 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:54.910997  7831 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260808 01:48:54.911345  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/ts-1-root
uuid: "3aaa5fb00ae84069a009f4dee2917bc2"
format_stamp: "Formatted at 2026-08-08 01:48:54 on dist-test-slave-8tdl"
I20260808 01:48:54.911669  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:54.931604  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:48:54.933265  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:54.935117  7831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:54.937696  7831 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:54.937961  7831 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:54.938407  7831 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:54.938668  7831 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:54.991955  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.194:42547
I20260808 01:48:54.992125  9200 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.194:42547 every 8 connection(s)
I20260808 01:48:54.997011  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:55.008337  9206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:55.009843  9205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:55.012162  7831 server_base.cc:1061] running on GCE node
W20260808 01:48:55.012710  9208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:55.012879  9201 heartbeater.cc:344] Connected to a master server at 127.7.165.254:43379
I20260808 01:48:55.013432  9201 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:55.013898  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:48:55.014179  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:48:55.014317  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786153735014305 us; error 0 us; skew 500 ppm
I20260808 01:48:55.014321  9201 heartbeater.cc:507] Master 127.7.165.254:43379 requested a full tablet report, sending...
I20260808 01:48:55.015009  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:55.016970  8986 ts_manager.cc:194] Registered new tserver with Master: 3aaa5fb00ae84069a009f4dee2917bc2 (127.7.165.194:42547)
I20260808 01:48:55.017995  7831 webserver.cc:533] Webserver started at http://127.7.165.195:46023/ using document root <none> and password file <none>
I20260808 01:48:55.018719  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:55.018994  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:55.018996  8986 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47838
I20260808 01:48:55.019452  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:55.021111  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/ts-2-root/instance:
uuid: "dfff33c1d94b49e38d105070579e7d6c"
format_stamp: "Formatted at 2026-08-08 01:48:55 on dist-test-slave-8tdl"
I20260808 01:48:55.026398  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.007s	sys 0.000s
I20260808 01:48:55.030290  9213 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:55.031376  7831 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260808 01:48:55.031687  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/ts-2-root
uuid: "dfff33c1d94b49e38d105070579e7d6c"
format_stamp: "Formatted at 2026-08-08 01:48:55 on dist-test-slave-8tdl"
I20260808 01:48:55.032049  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass.1786153669593643-7831-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:55.062222  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:48:55.063633  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:55.065002  7831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:55.067649  7831 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:55.067899  7831 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:55.068169  7831 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:55.068360  7831 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:55.116408  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.195:40603
I20260808 01:48:55.116573  9278 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.195:40603 every 8 connection(s)
I20260808 01:48:55.132479  9279 heartbeater.cc:344] Connected to a master server at 127.7.165.254:43379
I20260808 01:48:55.132931  9279 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:55.133865  9279 heartbeater.cc:507] Master 127.7.165.254:43379 requested a full tablet report, sending...
I20260808 01:48:55.136399  8986 ts_manager.cc:194] Registered new tserver with Master: dfff33c1d94b49e38d105070579e7d6c (127.7.165.195:40603)
I20260808 01:48:55.137595  7831 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017661459s
I20260808 01:48:55.138126  8986 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47844
I20260808 01:48:55.173207  8986 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47858:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000UUU\025\004\001\000\252\252\252*\004\001\000\377\377\377?\004\001\000TUUU\004\001\000\251\252\252j""\004\001\000UUU\025\004\001\000\252\252\252*\004\001\000\377\377\377?\004\001\000TUUU\004\001\000\251\252\252j"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260808 01:48:55.176601  8986 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 01:48:55.245179  9164 tablet_service.cc:1511] Processing CreateTablet for tablet af8385293960449aba32c949e652aea6 (DEFAULT_TABLE table=test-workload [id=048111c724ca42e5861351d18f7b61e5]), partition=RANGE (key) PARTITION VALUES < 357913941
I20260808 01:48:55.247118  9164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af8385293960449aba32c949e652aea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:55.246654  9085 tablet_service.cc:1511] Processing CreateTablet for tablet 68eef7a69e004073b251ab0e3e6445a1 (DEFAULT_TABLE table=test-workload [id=048111c724ca42e5861351d18f7b61e5]), partition=RANGE (key) PARTITION 357913941 <= VALUES < 715827882
I20260808 01:48:55.248662  9085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68eef7a69e004073b251ab0e3e6445a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:55.251993  9163 tablet_service.cc:1511] Processing CreateTablet for tablet 68eef7a69e004073b251ab0e3e6445a1 (DEFAULT_TABLE table=test-workload [id=048111c724ca42e5861351d18f7b61e5]), partition=RANGE (key) PARTITION 357913941 <= VALUES < 715827882
I20260808 01:48:55.253558  9163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68eef7a69e004073b251ab0e3e6445a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:55.255861  9086 tablet_service.cc:1511] Processing CreateTablet for tablet af8385293960449aba32c949e652aea6 (DEFAULT_TABLE table=test-workload [id=048111c724ca42e5861351d18f7b61e5]), partition=RANGE (key) PARTITION VALUES < 357913941
I20260808 01:48:55.257350  9086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af8385293960449aba32c949e652aea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:55.263187  9162 tablet_service.cc:1511] Processing CreateTablet for tablet 9cc5901f3ab34ee7a109a146c958de2e (DEFAULT_TABLE table=test-workload [id=048111c724ca42e5861351d18f7b61e5]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1073741823
I20260808 01:48:55.265302  9162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cc5901f3ab34ee7a109a146c958de2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:55.268014  9161 tablet_service.cc:1511] Processing CreateTablet for tablet 33f9ea1ccfd24813b8afcb138e17565a (DEFAULT_TABLE table=test-workload [id=048111c724ca42e5861351d18f7b61e5]), partition=RANGE (key) PARTITION 1073741823 <= VALUES < 1431655764
I20260808 01:48:55.269915  9161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f9ea1ccfd24813b8afcb138e17565a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:55.270416  9159 tablet_service.cc:1511] Processing CreateTablet for tablet 5f5125e2eade46d4bd25cf04eeb19170 (DEFAULT_TABLE table=test-workload [id=048111c724ca42e5861351d18f7b61e5]), partition=RANGE (key) PARTITION 1789569705 <= VALUES
I20260808 01:48:55.271977  9159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f5125e2eade46d4bd25cf04eeb19170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:55.269923  9160 tablet_service.cc:1511] Processing CreateTablet for tablet 90faceeb397646dc9c3d19c907d4acdc (DEFAULT_TABLE table=test-workload [id=048111c724ca42e5861351d18f7b61e5]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1789569705
I20260808 01:48:55.276587  9160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90faceeb397646dc9c3d19c907d4acdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:55.296972  9084 tablet_service.cc:1511] Processing CreateTablet for tablet 9cc5901f3ab34ee7a109a146c958de2e (DEFAULT_TABLE table=test-workload [id=048111c724ca42e5861351d18f7b61e5]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1073741823
I20260808 01:48:55.298483  9084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cc5901f3ab34ee7a109a146c958de2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:55.309381  9083 tablet_service.cc:1511] Processing CreateTablet for tablet 33f9ea1ccfd24813b8afcb138e17565a (DEFAULT_TABLE table=test-workload [id=048111c724ca42e5861351d18f7b61e5]), partition=RANGE (key) PARTITION 1073741823 <= VALUES < 1431655764
I20260808 01:48:55.311117  9083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f9ea1ccfd24813b8afcb138e17565a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:55.322060  9082 tablet_service.cc:1511] Processing CreateTablet for tablet 90faceeb397646dc9c3d19c907d4acdc (DEFAULT_TABLE table=test-workload [id=048111c724ca42e5861351d18f7b61e5]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1789569705
I20260808 01:48:55.323961  9082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90faceeb397646dc9c3d19c907d4acdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:55.325871  9237 tablet_service.cc:1511] Processing CreateTablet for tablet 5f5125e2eade46d4bd25cf04eeb19170 (DEFAULT_TABLE table=test-workload [id=048111c724ca42e5861351d18f7b61e5]), partition=RANGE (key) PARTITION 1789569705 <= VALUES
I20260808 01:48:55.327548  9237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f5125e2eade46d4bd25cf04eeb19170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:55.328676  9240 tablet_service.cc:1511] Processing CreateTablet for tablet 9cc5901f3ab34ee7a109a146c958de2e (DEFAULT_TABLE table=test-workload [id=048111c724ca42e5861351d18f7b61e5]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1073741823
I20260808 01:48:55.332319  9242 tablet_service.cc:1511] Processing CreateTablet for tablet 68eef7a69e004073b251ab0e3e6445a1 (DEFAULT_TABLE table=test-workload [id=048111c724ca42e5861351d18f7b61e5]), partition=RANGE (key) PARTITION 357913941 <= VALUES < 715827882
I20260808 01:48:55.334381  9238 tablet_service.cc:1511] Processing CreateTablet for tablet 90faceeb397646dc9c3d19c907d4acdc (DEFAULT_TABLE table=test-workload [id=048111c724ca42e5861351d18f7b61e5]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1789569705
I20260808 01:48:55.336057  9080 tablet_service.cc:1511] Processing CreateTablet for tablet 5f5125e2eade46d4bd25cf04eeb19170 (DEFAULT_TABLE table=test-workload [id=048111c724ca42e5861351d18f7b61e5]), partition=RANGE (key) PARTITION 1789569705 <= VALUES
I20260808 01:48:55.326375  9243 tablet_service.cc:1511] Processing CreateTablet for tablet af8385293960449aba32c949e652aea6 (DEFAULT_TABLE table=test-workload [id=048111c724ca42e5861351d18f7b61e5]), partition=RANGE (key) PARTITION VALUES < 357913941
I20260808 01:48:55.332412  9239 tablet_service.cc:1511] Processing CreateTablet for tablet 33f9ea1ccfd24813b8afcb138e17565a (DEFAULT_TABLE table=test-workload [id=048111c724ca42e5861351d18f7b61e5]), partition=RANGE (key) PARTITION 1073741823 <= VALUES < 1431655764
I20260808 01:48:55.338196  9240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cc5901f3ab34ee7a109a146c958de2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:55.339242  9239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f9ea1ccfd24813b8afcb138e17565a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:55.342958  9243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af8385293960449aba32c949e652aea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:55.343750  9080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f5125e2eade46d4bd25cf04eeb19170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:55.344393  9242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68eef7a69e004073b251ab0e3e6445a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:55.351276  9238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90faceeb397646dc9c3d19c907d4acdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:55.373230  9294 tablet_bootstrap.cc:492] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2: Bootstrap starting.
I20260808 01:48:55.393517  9295 tablet_bootstrap.cc:492] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c: Bootstrap starting.
I20260808 01:48:55.402186  9294 tablet_bootstrap.cc:654] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:55.410902  9295 tablet_bootstrap.cc:654] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:55.425276  9296 tablet_bootstrap.cc:492] T 5f5125e2eade46d4bd25cf04eeb19170 P dfff33c1d94b49e38d105070579e7d6c: Bootstrap starting.
I20260808 01:48:55.425310  9294 tablet_bootstrap.cc:492] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2: No bootstrap required, opened a new log
I20260808 01:48:55.426133  9294 ts_tablet_manager.cc:1403] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2: Time spent bootstrapping tablet: real 0.053s	user 0.017s	sys 0.004s
I20260808 01:48:55.426776  9295 tablet_bootstrap.cc:492] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c: No bootstrap required, opened a new log
I20260808 01:48:55.427351  9295 ts_tablet_manager.cc:1403] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c: Time spent bootstrapping tablet: real 0.038s	user 0.005s	sys 0.015s
I20260808 01:48:55.428848  9294 raft_consensus.cc:359] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.429513  9294 raft_consensus.cc:385] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:55.429797  9294 raft_consensus.cc:740] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aaa5fb00ae84069a009f4dee2917bc2, State: Initialized, Role: FOLLOWER
I20260808 01:48:55.430240  9295 raft_consensus.cc:359] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.430962  9295 raft_consensus.cc:385] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:55.431509  9295 raft_consensus.cc:740] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91d6edcb50845178ee3c8d60fb4ac4c, State: Initialized, Role: FOLLOWER
I20260808 01:48:55.432256  9295 consensus_queue.cc:260] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.430555  9294 consensus_queue.cc:260] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.434947  9122 heartbeater.cc:499] Master 127.7.165.254:43379 was elected leader, sending a full tablet report...
I20260808 01:48:55.435554  9295 ts_tablet_manager.cc:1434] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20260808 01:48:55.436533  9295 tablet_bootstrap.cc:492] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c: Bootstrap starting.
I20260808 01:48:55.439272  9296 tablet_bootstrap.cc:654] T 5f5125e2eade46d4bd25cf04eeb19170 P dfff33c1d94b49e38d105070579e7d6c: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:55.440043  9201 heartbeater.cc:499] Master 127.7.165.254:43379 was elected leader, sending a full tablet report...
I20260808 01:48:55.440742  9294 ts_tablet_manager.cc:1434] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.003s
I20260808 01:48:55.441882  9294 tablet_bootstrap.cc:492] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2: Bootstrap starting.
I20260808 01:48:55.449431  9294 tablet_bootstrap.cc:654] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:55.452656  9296 tablet_bootstrap.cc:492] T 5f5125e2eade46d4bd25cf04eeb19170 P dfff33c1d94b49e38d105070579e7d6c: No bootstrap required, opened a new log
I20260808 01:48:55.453243  9296 ts_tablet_manager.cc:1403] T 5f5125e2eade46d4bd25cf04eeb19170 P dfff33c1d94b49e38d105070579e7d6c: Time spent bootstrapping tablet: real 0.028s	user 0.010s	sys 0.011s
I20260808 01:48:55.456776  9296 raft_consensus.cc:359] T 5f5125e2eade46d4bd25cf04eeb19170 P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.457675  9296 raft_consensus.cc:385] T 5f5125e2eade46d4bd25cf04eeb19170 P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:55.458537  9295 tablet_bootstrap.cc:654] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:55.458001  9296 raft_consensus.cc:740] T 5f5125e2eade46d4bd25cf04eeb19170 P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfff33c1d94b49e38d105070579e7d6c, State: Initialized, Role: FOLLOWER
I20260808 01:48:55.459570  9296 consensus_queue.cc:260] T 5f5125e2eade46d4bd25cf04eeb19170 P dfff33c1d94b49e38d105070579e7d6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.470402  9296 ts_tablet_manager.cc:1434] T 5f5125e2eade46d4bd25cf04eeb19170 P dfff33c1d94b49e38d105070579e7d6c: Time spent starting tablet: real 0.017s	user 0.009s	sys 0.006s
I20260808 01:48:55.472358  9279 heartbeater.cc:499] Master 127.7.165.254:43379 was elected leader, sending a full tablet report...
I20260808 01:48:55.473699  9296 tablet_bootstrap.cc:492] T 90faceeb397646dc9c3d19c907d4acdc P dfff33c1d94b49e38d105070579e7d6c: Bootstrap starting.
I20260808 01:48:55.479574  9295 tablet_bootstrap.cc:492] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c: No bootstrap required, opened a new log
I20260808 01:48:55.480201  9295 ts_tablet_manager.cc:1403] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c: Time spent bootstrapping tablet: real 0.044s	user 0.011s	sys 0.012s
I20260808 01:48:55.481415  9296 tablet_bootstrap.cc:654] T 90faceeb397646dc9c3d19c907d4acdc P dfff33c1d94b49e38d105070579e7d6c: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:55.484369  9295 raft_consensus.cc:359] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.485574  9295 raft_consensus.cc:385] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:55.486097  9295 raft_consensus.cc:740] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91d6edcb50845178ee3c8d60fb4ac4c, State: Initialized, Role: FOLLOWER
I20260808 01:48:55.487480  9295 consensus_queue.cc:260] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.491721  9295 ts_tablet_manager.cc:1434] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c: Time spent starting tablet: real 0.011s	user 0.011s	sys 0.000s
I20260808 01:48:55.493055  9295 tablet_bootstrap.cc:492] T 9cc5901f3ab34ee7a109a146c958de2e P e91d6edcb50845178ee3c8d60fb4ac4c: Bootstrap starting.
I20260808 01:48:55.494513  9296 tablet_bootstrap.cc:492] T 90faceeb397646dc9c3d19c907d4acdc P dfff33c1d94b49e38d105070579e7d6c: No bootstrap required, opened a new log
I20260808 01:48:55.495143  9296 ts_tablet_manager.cc:1403] T 90faceeb397646dc9c3d19c907d4acdc P dfff33c1d94b49e38d105070579e7d6c: Time spent bootstrapping tablet: real 0.022s	user 0.012s	sys 0.008s
I20260808 01:48:55.498903  9296 raft_consensus.cc:359] T 90faceeb397646dc9c3d19c907d4acdc P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.499873  9296 raft_consensus.cc:385] T 90faceeb397646dc9c3d19c907d4acdc P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:55.500262  9296 raft_consensus.cc:740] T 90faceeb397646dc9c3d19c907d4acdc P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfff33c1d94b49e38d105070579e7d6c, State: Initialized, Role: FOLLOWER
I20260808 01:48:55.501695  9296 consensus_queue.cc:260] T 90faceeb397646dc9c3d19c907d4acdc P dfff33c1d94b49e38d105070579e7d6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.500599  9295 tablet_bootstrap.cc:654] T 9cc5901f3ab34ee7a109a146c958de2e P e91d6edcb50845178ee3c8d60fb4ac4c: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:55.504674  9296 ts_tablet_manager.cc:1434] T 90faceeb397646dc9c3d19c907d4acdc P dfff33c1d94b49e38d105070579e7d6c: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260808 01:48:55.505822  9296 tablet_bootstrap.cc:492] T 68eef7a69e004073b251ab0e3e6445a1 P dfff33c1d94b49e38d105070579e7d6c: Bootstrap starting.
I20260808 01:48:55.507085  9294 tablet_bootstrap.cc:492] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2: No bootstrap required, opened a new log
I20260808 01:48:55.507668  9294 ts_tablet_manager.cc:1403] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2: Time spent bootstrapping tablet: real 0.066s	user 0.023s	sys 0.037s
I20260808 01:48:55.511823  9294 raft_consensus.cc:359] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.513440  9294 raft_consensus.cc:385] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:55.513932  9294 raft_consensus.cc:740] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aaa5fb00ae84069a009f4dee2917bc2, State: Initialized, Role: FOLLOWER
I20260808 01:48:55.515242  9294 consensus_queue.cc:260] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.515965  9296 tablet_bootstrap.cc:654] T 68eef7a69e004073b251ab0e3e6445a1 P dfff33c1d94b49e38d105070579e7d6c: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:55.517822  9294 ts_tablet_manager.cc:1434] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260808 01:48:55.519290  9294 tablet_bootstrap.cc:492] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2: Bootstrap starting.
I20260808 01:48:55.527689  9294 tablet_bootstrap.cc:654] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:55.559092  9299 raft_consensus.cc:493] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:55.559805  9299 raft_consensus.cc:515] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.578892  9299 leader_election.cc:290] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aaa5fb00ae84069a009f4dee2917bc2 (127.7.165.194:42547), dfff33c1d94b49e38d105070579e7d6c (127.7.165.195:40603)
I20260808 01:48:55.620268  9299 raft_consensus.cc:493] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:55.622825  9299 raft_consensus.cc:515] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.626363  9299 leader_election.cc:290] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aaa5fb00ae84069a009f4dee2917bc2 (127.7.165.194:42547), dfff33c1d94b49e38d105070579e7d6c (127.7.165.195:40603)
I20260808 01:48:55.646632  9295 tablet_bootstrap.cc:492] T 9cc5901f3ab34ee7a109a146c958de2e P e91d6edcb50845178ee3c8d60fb4ac4c: No bootstrap required, opened a new log
I20260808 01:48:55.647480  9295 ts_tablet_manager.cc:1403] T 9cc5901f3ab34ee7a109a146c958de2e P e91d6edcb50845178ee3c8d60fb4ac4c: Time spent bootstrapping tablet: real 0.155s	user 0.053s	sys 0.069s
I20260808 01:48:55.652839  9295 raft_consensus.cc:359] T 9cc5901f3ab34ee7a109a146c958de2e P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.654016  9295 raft_consensus.cc:385] T 9cc5901f3ab34ee7a109a146c958de2e P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:55.654485  9295 raft_consensus.cc:740] T 9cc5901f3ab34ee7a109a146c958de2e P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91d6edcb50845178ee3c8d60fb4ac4c, State: Initialized, Role: FOLLOWER
I20260808 01:48:55.656147  9295 consensus_queue.cc:260] T 9cc5901f3ab34ee7a109a146c958de2e P e91d6edcb50845178ee3c8d60fb4ac4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.658754  9295 ts_tablet_manager.cc:1434] T 9cc5901f3ab34ee7a109a146c958de2e P e91d6edcb50845178ee3c8d60fb4ac4c: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.002s
I20260808 01:48:55.659842  9295 tablet_bootstrap.cc:492] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c: Bootstrap starting.
I20260808 01:48:55.667121  9295 tablet_bootstrap.cc:654] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:55.667214  9175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f5125e2eade46d4bd25cf04eeb19170" candidate_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" is_pre_election: true
I20260808 01:48:55.667646  9174 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68eef7a69e004073b251ab0e3e6445a1" candidate_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" is_pre_election: true
W20260808 01:48:55.669894  9060 leader_election.cc:343] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aaa5fb00ae84069a009f4dee2917bc2 (127.7.165.194:42547): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:48:55.670816  9060 leader_election.cc:343] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aaa5fb00ae84069a009f4dee2917bc2 (127.7.165.194:42547): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:55.672549  9294 tablet_bootstrap.cc:492] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2: No bootstrap required, opened a new log
I20260808 01:48:55.672967  9295 tablet_bootstrap.cc:492] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c: No bootstrap required, opened a new log
I20260808 01:48:55.673117  9294 ts_tablet_manager.cc:1403] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2: Time spent bootstrapping tablet: real 0.154s	user 0.045s	sys 0.076s
I20260808 01:48:55.673484  9295 ts_tablet_manager.cc:1403] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c: Time spent bootstrapping tablet: real 0.014s	user 0.005s	sys 0.005s
I20260808 01:48:55.676337  9294 raft_consensus.cc:359] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.676708  9295 raft_consensus.cc:359] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.677343  9294 raft_consensus.cc:385] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:55.677496  9295 raft_consensus.cc:385] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:55.677786  9294 raft_consensus.cc:740] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aaa5fb00ae84069a009f4dee2917bc2, State: Initialized, Role: FOLLOWER
I20260808 01:48:55.677980  9295 raft_consensus.cc:740] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91d6edcb50845178ee3c8d60fb4ac4c, State: Initialized, Role: FOLLOWER
I20260808 01:48:55.678663  9296 tablet_bootstrap.cc:492] T 68eef7a69e004073b251ab0e3e6445a1 P dfff33c1d94b49e38d105070579e7d6c: No bootstrap required, opened a new log
I20260808 01:48:55.679214  9296 ts_tablet_manager.cc:1403] T 68eef7a69e004073b251ab0e3e6445a1 P dfff33c1d94b49e38d105070579e7d6c: Time spent bootstrapping tablet: real 0.174s	user 0.071s	sys 0.069s
I20260808 01:48:55.678916  9295 consensus_queue.cc:260] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.681362  9253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f5125e2eade46d4bd25cf04eeb19170" candidate_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfff33c1d94b49e38d105070579e7d6c" is_pre_election: true
I20260808 01:48:55.681659  9295 ts_tablet_manager.cc:1434] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260808 01:48:55.682156  9253 raft_consensus.cc:2468] T 5f5125e2eade46d4bd25cf04eeb19170 P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e91d6edcb50845178ee3c8d60fb4ac4c in term 0.
I20260808 01:48:55.682928  9295 tablet_bootstrap.cc:492] T 33f9ea1ccfd24813b8afcb138e17565a P e91d6edcb50845178ee3c8d60fb4ac4c: Bootstrap starting.
I20260808 01:48:55.682521  9296 raft_consensus.cc:359] T 68eef7a69e004073b251ab0e3e6445a1 P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.683342  9296 raft_consensus.cc:385] T 68eef7a69e004073b251ab0e3e6445a1 P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:55.683661  9296 raft_consensus.cc:740] T 68eef7a69e004073b251ab0e3e6445a1 P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfff33c1d94b49e38d105070579e7d6c, State: Initialized, Role: FOLLOWER
I20260808 01:48:55.683676  9252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68eef7a69e004073b251ab0e3e6445a1" candidate_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfff33c1d94b49e38d105070579e7d6c" is_pre_election: true
I20260808 01:48:55.684304  9057 leader_election.cc:304] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dfff33c1d94b49e38d105070579e7d6c, e91d6edcb50845178ee3c8d60fb4ac4c; no voters: 3aaa5fb00ae84069a009f4dee2917bc2
I20260808 01:48:55.684532  9296 consensus_queue.cc:260] T 68eef7a69e004073b251ab0e3e6445a1 P dfff33c1d94b49e38d105070579e7d6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.685240  9299 raft_consensus.cc:2804] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:55.685591  9252 raft_consensus.cc:2468] T 68eef7a69e004073b251ab0e3e6445a1 P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e91d6edcb50845178ee3c8d60fb4ac4c in term 0.
I20260808 01:48:55.688185  9296 ts_tablet_manager.cc:1434] T 68eef7a69e004073b251ab0e3e6445a1 P dfff33c1d94b49e38d105070579e7d6c: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.004s
I20260808 01:48:55.689594  9296 tablet_bootstrap.cc:492] T 33f9ea1ccfd24813b8afcb138e17565a P dfff33c1d94b49e38d105070579e7d6c: Bootstrap starting.
I20260808 01:48:55.685654  9299 raft_consensus.cc:493] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:55.690784  9295 tablet_bootstrap.cc:654] T 33f9ea1ccfd24813b8afcb138e17565a P e91d6edcb50845178ee3c8d60fb4ac4c: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:55.690930  9299 raft_consensus.cc:3060] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:48:55.693749  9039 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:48:55.690840  9057 leader_election.cc:304] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dfff33c1d94b49e38d105070579e7d6c, e91d6edcb50845178ee3c8d60fb4ac4c; no voters: 3aaa5fb00ae84069a009f4dee2917bc2
I20260808 01:48:55.698750  9296 tablet_bootstrap.cc:654] T 33f9ea1ccfd24813b8afcb138e17565a P dfff33c1d94b49e38d105070579e7d6c: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:55.679230  9294 consensus_queue.cc:260] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.701016  9300 raft_consensus.cc:493] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:55.702062  9300 raft_consensus.cc:515] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.708815  9295 tablet_bootstrap.cc:492] T 33f9ea1ccfd24813b8afcb138e17565a P e91d6edcb50845178ee3c8d60fb4ac4c: No bootstrap required, opened a new log
I20260808 01:48:55.709522  9295 ts_tablet_manager.cc:1403] T 33f9ea1ccfd24813b8afcb138e17565a P e91d6edcb50845178ee3c8d60fb4ac4c: Time spent bootstrapping tablet: real 0.027s	user 0.011s	sys 0.011s
I20260808 01:48:55.709843  9296 tablet_bootstrap.cc:492] T 33f9ea1ccfd24813b8afcb138e17565a P dfff33c1d94b49e38d105070579e7d6c: No bootstrap required, opened a new log
I20260808 01:48:55.710391  9296 ts_tablet_manager.cc:1403] T 33f9ea1ccfd24813b8afcb138e17565a P dfff33c1d94b49e38d105070579e7d6c: Time spent bootstrapping tablet: real 0.021s	user 0.007s	sys 0.010s
I20260808 01:48:55.713831  9295 raft_consensus.cc:359] T 33f9ea1ccfd24813b8afcb138e17565a P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.714313  9296 raft_consensus.cc:359] T 33f9ea1ccfd24813b8afcb138e17565a P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.714977  9300 leader_election.cc:290] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e91d6edcb50845178ee3c8d60fb4ac4c (127.7.165.193:43869), dfff33c1d94b49e38d105070579e7d6c (127.7.165.195:40603)
I20260808 01:48:55.715219  9296 raft_consensus.cc:385] T 33f9ea1ccfd24813b8afcb138e17565a P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:55.715143  9295 raft_consensus.cc:385] T 33f9ea1ccfd24813b8afcb138e17565a P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:55.715723  9296 raft_consensus.cc:740] T 33f9ea1ccfd24813b8afcb138e17565a P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfff33c1d94b49e38d105070579e7d6c, State: Initialized, Role: FOLLOWER
I20260808 01:48:55.715900  9295 raft_consensus.cc:740] T 33f9ea1ccfd24813b8afcb138e17565a P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91d6edcb50845178ee3c8d60fb4ac4c, State: Initialized, Role: FOLLOWER
I20260808 01:48:55.716846  9296 consensus_queue.cc:260] T 33f9ea1ccfd24813b8afcb138e17565a P dfff33c1d94b49e38d105070579e7d6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.717561  9295 consensus_queue.cc:260] T 33f9ea1ccfd24813b8afcb138e17565a P e91d6edcb50845178ee3c8d60fb4ac4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.718216  9300 raft_consensus.cc:493] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:55.718786  9300 raft_consensus.cc:515] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.719766  9296 ts_tablet_manager.cc:1434] T 33f9ea1ccfd24813b8afcb138e17565a P dfff33c1d94b49e38d105070579e7d6c: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260808 01:48:55.722620  9296 tablet_bootstrap.cc:492] T af8385293960449aba32c949e652aea6 P dfff33c1d94b49e38d105070579e7d6c: Bootstrap starting.
I20260808 01:48:55.728452  9299 raft_consensus.cc:515] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.733757  9299 leader_election.cc:290] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [CANDIDATE]: Term 1 election: Requested vote from peers 3aaa5fb00ae84069a009f4dee2917bc2 (127.7.165.194:42547), dfff33c1d94b49e38d105070579e7d6c (127.7.165.195:40603)
I20260808 01:48:55.737560  9299 raft_consensus.cc:2804] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:55.740903  9299 raft_consensus.cc:493] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:55.736418  9300 leader_election.cc:290] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e91d6edcb50845178ee3c8d60fb4ac4c (127.7.165.193:43869), dfff33c1d94b49e38d105070579e7d6c (127.7.165.195:40603)
I20260808 01:48:55.734280  9296 tablet_bootstrap.cc:654] T af8385293960449aba32c949e652aea6 P dfff33c1d94b49e38d105070579e7d6c: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:55.725498  9294 ts_tablet_manager.cc:1434] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2: Time spent starting tablet: real 0.052s	user 0.006s	sys 0.003s
I20260808 01:48:55.744972  9294 tablet_bootstrap.cc:492] T 90faceeb397646dc9c3d19c907d4acdc P 3aaa5fb00ae84069a009f4dee2917bc2: Bootstrap starting.
I20260808 01:48:55.742551  9175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f5125e2eade46d4bd25cf04eeb19170" candidate_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aaa5fb00ae84069a009f4dee2917bc2"
I20260808 01:48:55.741637  9299 raft_consensus.cc:3060] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:48:55.752024  9060 leader_election.cc:343] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aaa5fb00ae84069a009f4dee2917bc2 (127.7.165.194:42547): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:48:55.739935  9057 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:40603, user_credentials={real_user=slave}} blocked reactor thread for 49320.3us
I20260808 01:48:55.758910  9252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f5125e2eade46d4bd25cf04eeb19170" candidate_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfff33c1d94b49e38d105070579e7d6c"
I20260808 01:48:55.759582  9252 raft_consensus.cc:3060] T 5f5125e2eade46d4bd25cf04eeb19170 P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:55.762164  9296 tablet_bootstrap.cc:492] T af8385293960449aba32c949e652aea6 P dfff33c1d94b49e38d105070579e7d6c: No bootstrap required, opened a new log
I20260808 01:48:55.762796  9296 ts_tablet_manager.cc:1403] T af8385293960449aba32c949e652aea6 P dfff33c1d94b49e38d105070579e7d6c: Time spent bootstrapping tablet: real 0.040s	user 0.014s	sys 0.001s
I20260808 01:48:55.766063  9296 raft_consensus.cc:359] T af8385293960449aba32c949e652aea6 P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.767014  9296 raft_consensus.cc:385] T af8385293960449aba32c949e652aea6 P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:55.767379  9296 raft_consensus.cc:740] T af8385293960449aba32c949e652aea6 P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfff33c1d94b49e38d105070579e7d6c, State: Initialized, Role: FOLLOWER
I20260808 01:48:55.767019  9299 raft_consensus.cc:515] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.767997  9252 raft_consensus.cc:2468] T 5f5125e2eade46d4bd25cf04eeb19170 P dfff33c1d94b49e38d105070579e7d6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e91d6edcb50845178ee3c8d60fb4ac4c in term 1.
I20260808 01:48:55.768312  9296 consensus_queue.cc:260] T af8385293960449aba32c949e652aea6 P dfff33c1d94b49e38d105070579e7d6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.770830  9057 leader_election.cc:304] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dfff33c1d94b49e38d105070579e7d6c, e91d6edcb50845178ee3c8d60fb4ac4c; no voters: 3aaa5fb00ae84069a009f4dee2917bc2
I20260808 01:48:55.769358  9295 ts_tablet_manager.cc:1434] T 33f9ea1ccfd24813b8afcb138e17565a P e91d6edcb50845178ee3c8d60fb4ac4c: Time spent starting tablet: real 0.059s	user 0.017s	sys 0.022s
I20260808 01:48:55.772156  9296 ts_tablet_manager.cc:1434] T af8385293960449aba32c949e652aea6 P dfff33c1d94b49e38d105070579e7d6c: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.003s
I20260808 01:48:55.772157  9299 leader_election.cc:290] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [CANDIDATE]: Term 1 election: Requested vote from peers 3aaa5fb00ae84069a009f4dee2917bc2 (127.7.165.194:42547), dfff33c1d94b49e38d105070579e7d6c (127.7.165.195:40603)
I20260808 01:48:55.773519  9296 tablet_bootstrap.cc:492] T 9cc5901f3ab34ee7a109a146c958de2e P dfff33c1d94b49e38d105070579e7d6c: Bootstrap starting.
I20260808 01:48:55.772197  9318 raft_consensus.cc:2804] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:55.774214  9318 raft_consensus.cc:697] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 LEADER]: Becoming Leader. State: Replica: e91d6edcb50845178ee3c8d60fb4ac4c, State: Running, Role: LEADER
I20260808 01:48:55.775293  9175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68eef7a69e004073b251ab0e3e6445a1" candidate_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aaa5fb00ae84069a009f4dee2917bc2"
I20260808 01:48:55.775202  9318 consensus_queue.cc:237] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
W20260808 01:48:55.776933  9060 leader_election.cc:343] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3aaa5fb00ae84069a009f4dee2917bc2 (127.7.165.194:42547): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:55.781279  9296 tablet_bootstrap.cc:654] T 9cc5901f3ab34ee7a109a146c958de2e P dfff33c1d94b49e38d105070579e7d6c: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:55.777056  9295 tablet_bootstrap.cc:492] T af8385293960449aba32c949e652aea6 P e91d6edcb50845178ee3c8d60fb4ac4c: Bootstrap starting.
I20260808 01:48:55.786101  9252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68eef7a69e004073b251ab0e3e6445a1" candidate_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfff33c1d94b49e38d105070579e7d6c"
I20260808 01:48:55.786840  9252 raft_consensus.cc:3060] T 68eef7a69e004073b251ab0e3e6445a1 P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:55.756403  9294 tablet_bootstrap.cc:654] T 90faceeb397646dc9c3d19c907d4acdc P 3aaa5fb00ae84069a009f4dee2917bc2: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:55.789718  9253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33f9ea1ccfd24813b8afcb138e17565a" candidate_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfff33c1d94b49e38d105070579e7d6c" is_pre_election: true
I20260808 01:48:55.790748  9253 raft_consensus.cc:2468] T 33f9ea1ccfd24813b8afcb138e17565a P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aaa5fb00ae84069a009f4dee2917bc2 in term 0.
I20260808 01:48:55.792128  9251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cc5901f3ab34ee7a109a146c958de2e" candidate_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfff33c1d94b49e38d105070579e7d6c" is_pre_election: true
I20260808 01:48:55.792637  9096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33f9ea1ccfd24813b8afcb138e17565a" candidate_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" is_pre_election: true
I20260808 01:48:55.793366  9096 raft_consensus.cc:2468] T 33f9ea1ccfd24813b8afcb138e17565a P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aaa5fb00ae84069a009f4dee2917bc2 in term 0.
I20260808 01:48:55.795632  9096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cc5901f3ab34ee7a109a146c958de2e" candidate_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" is_pre_election: true
I20260808 01:48:55.795362  8986 catalog_manager.cc:5697] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c reported cstate change: term changed from 0 to 1, leader changed from <none> to e91d6edcb50845178ee3c8d60fb4ac4c (127.7.165.193). New cstate: current_term: 1 leader_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:55.796336  9096 raft_consensus.cc:2468] T 9cc5901f3ab34ee7a109a146c958de2e P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aaa5fb00ae84069a009f4dee2917bc2 in term 0.
I20260808 01:48:55.798076  9138 leader_election.cc:304] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aaa5fb00ae84069a009f4dee2917bc2, dfff33c1d94b49e38d105070579e7d6c; no voters: 
I20260808 01:48:55.799038  9300 raft_consensus.cc:2804] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:55.799466  9300 raft_consensus.cc:493] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:55.799875  9300 raft_consensus.cc:3060] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:55.800168  9295 tablet_bootstrap.cc:654] T af8385293960449aba32c949e652aea6 P e91d6edcb50845178ee3c8d60fb4ac4c: Neither blocks nor log segments found. Creating new log.
W20260808 01:48:55.811158  9138 leader_election.cc:343] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dfff33c1d94b49e38d105070579e7d6c (127.7.165.195:40603): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:55.812812  9138 leader_election.cc:304] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aaa5fb00ae84069a009f4dee2917bc2, e91d6edcb50845178ee3c8d60fb4ac4c; no voters: dfff33c1d94b49e38d105070579e7d6c
I20260808 01:48:55.813875  9316 raft_consensus.cc:2804] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:55.814306  9316 raft_consensus.cc:493] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:55.814749  9316 raft_consensus.cc:3060] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:55.810160  9300 raft_consensus.cc:515] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.816814  9295 tablet_bootstrap.cc:492] T af8385293960449aba32c949e652aea6 P e91d6edcb50845178ee3c8d60fb4ac4c: No bootstrap required, opened a new log
I20260808 01:48:55.820350  9296 tablet_bootstrap.cc:492] T 9cc5901f3ab34ee7a109a146c958de2e P dfff33c1d94b49e38d105070579e7d6c: No bootstrap required, opened a new log
I20260808 01:48:55.820942  9299 raft_consensus.cc:493] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:55.822157  9296 ts_tablet_manager.cc:1403] T 9cc5901f3ab34ee7a109a146c958de2e P dfff33c1d94b49e38d105070579e7d6c: Time spent bootstrapping tablet: real 0.049s	user 0.019s	sys 0.021s
I20260808 01:48:55.822149  9299 raft_consensus.cc:515] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.824698  9299 leader_election.cc:290] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aaa5fb00ae84069a009f4dee2917bc2 (127.7.165.194:42547), dfff33c1d94b49e38d105070579e7d6c (127.7.165.195:40603)
I20260808 01:48:55.825297  9295 ts_tablet_manager.cc:1403] T af8385293960449aba32c949e652aea6 P e91d6edcb50845178ee3c8d60fb4ac4c: Time spent bootstrapping tablet: real 0.048s	user 0.011s	sys 0.015s
I20260808 01:48:55.825492  9296 raft_consensus.cc:359] T 9cc5901f3ab34ee7a109a146c958de2e P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.826282  9296 raft_consensus.cc:385] T 9cc5901f3ab34ee7a109a146c958de2e P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:55.826299  9175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90faceeb397646dc9c3d19c907d4acdc" candidate_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" is_pre_election: true
I20260808 01:48:55.826817  9296 raft_consensus.cc:740] T 9cc5901f3ab34ee7a109a146c958de2e P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfff33c1d94b49e38d105070579e7d6c, State: Initialized, Role: FOLLOWER
I20260808 01:48:55.827785  9296 consensus_queue.cc:260] T 9cc5901f3ab34ee7a109a146c958de2e P dfff33c1d94b49e38d105070579e7d6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
W20260808 01:48:55.828194  9060 leader_election.cc:343] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3aaa5fb00ae84069a009f4dee2917bc2 (127.7.165.194:42547): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:55.826399  9251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90faceeb397646dc9c3d19c907d4acdc" candidate_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfff33c1d94b49e38d105070579e7d6c" is_pre_election: true
I20260808 01:48:55.829035  9251 raft_consensus.cc:2468] T 90faceeb397646dc9c3d19c907d4acdc P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e91d6edcb50845178ee3c8d60fb4ac4c in term 0.
I20260808 01:48:55.830327  9296 ts_tablet_manager.cc:1434] T 9cc5901f3ab34ee7a109a146c958de2e P dfff33c1d94b49e38d105070579e7d6c: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260808 01:48:55.830116  9316 raft_consensus.cc:515] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.830356  9057 leader_election.cc:304] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dfff33c1d94b49e38d105070579e7d6c, e91d6edcb50845178ee3c8d60fb4ac4c; no voters: 3aaa5fb00ae84069a009f4dee2917bc2
I20260808 01:48:55.831480  9252 raft_consensus.cc:2468] T 68eef7a69e004073b251ab0e3e6445a1 P dfff33c1d94b49e38d105070579e7d6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e91d6edcb50845178ee3c8d60fb4ac4c in term 1.
I20260808 01:48:55.816215  9294 tablet_bootstrap.cc:492] T 90faceeb397646dc9c3d19c907d4acdc P 3aaa5fb00ae84069a009f4dee2917bc2: No bootstrap required, opened a new log
I20260808 01:48:55.831923  9295 raft_consensus.cc:359] T af8385293960449aba32c949e652aea6 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.832698  9294 ts_tablet_manager.cc:1403] T 90faceeb397646dc9c3d19c907d4acdc P 3aaa5fb00ae84069a009f4dee2917bc2: Time spent bootstrapping tablet: real 0.088s	user 0.012s	sys 0.004s
I20260808 01:48:55.833565  9096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cc5901f3ab34ee7a109a146c958de2e" candidate_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c"
I20260808 01:48:55.834271  9096 raft_consensus.cc:3060] T 9cc5901f3ab34ee7a109a146c958de2e P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:55.834091  9057 leader_election.cc:304] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dfff33c1d94b49e38d105070579e7d6c, e91d6edcb50845178ee3c8d60fb4ac4c; no voters: 3aaa5fb00ae84069a009f4dee2917bc2
I20260808 01:48:55.832875  9295 raft_consensus.cc:385] T af8385293960449aba32c949e652aea6 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:55.835853  9300 leader_election.cc:290] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [CANDIDATE]: Term 1 election: Requested vote from peers e91d6edcb50845178ee3c8d60fb4ac4c (127.7.165.193:43869), dfff33c1d94b49e38d105070579e7d6c (127.7.165.195:40603)
I20260808 01:48:55.836158  9295 raft_consensus.cc:740] T af8385293960449aba32c949e652aea6 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91d6edcb50845178ee3c8d60fb4ac4c, State: Initialized, Role: FOLLOWER
I20260808 01:48:55.835192  9318 raft_consensus.cc:2804] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:55.836997  9318 raft_consensus.cc:697] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 LEADER]: Becoming Leader. State: Replica: e91d6edcb50845178ee3c8d60fb4ac4c, State: Running, Role: LEADER
I20260808 01:48:55.837349  9295 consensus_queue.cc:260] T af8385293960449aba32c949e652aea6 P e91d6edcb50845178ee3c8d60fb4ac4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.834282  9299 raft_consensus.cc:2804] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:55.839124  9299 raft_consensus.cc:493] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:55.838853  9294 raft_consensus.cc:359] T 90faceeb397646dc9c3d19c907d4acdc P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.839540  9299 raft_consensus.cc:3060] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:55.839725  9294 raft_consensus.cc:385] T 90faceeb397646dc9c3d19c907d4acdc P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:55.840098  9294 raft_consensus.cc:740] T 90faceeb397646dc9c3d19c907d4acdc P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aaa5fb00ae84069a009f4dee2917bc2, State: Initialized, Role: FOLLOWER
I20260808 01:48:55.841104  9295 ts_tablet_manager.cc:1434] T af8385293960449aba32c949e652aea6 P e91d6edcb50845178ee3c8d60fb4ac4c: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.001s
I20260808 01:48:55.841107  9294 consensus_queue.cc:260] T 90faceeb397646dc9c3d19c907d4acdc P 3aaa5fb00ae84069a009f4dee2917bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.842805  9252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33f9ea1ccfd24813b8afcb138e17565a" candidate_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfff33c1d94b49e38d105070579e7d6c"
I20260808 01:48:55.843606  9252 raft_consensus.cc:3060] T 33f9ea1ccfd24813b8afcb138e17565a P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:55.842851  9095 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33f9ea1ccfd24813b8afcb138e17565a" candidate_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c"
I20260808 01:48:55.845093  9294 ts_tablet_manager.cc:1434] T 90faceeb397646dc9c3d19c907d4acdc P 3aaa5fb00ae84069a009f4dee2917bc2: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.004s
I20260808 01:48:55.845463  9095 raft_consensus.cc:3060] T 33f9ea1ccfd24813b8afcb138e17565a P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:55.837885  9318 consensus_queue.cc:237] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.846602  9294 tablet_bootstrap.cc:492] T 68eef7a69e004073b251ab0e3e6445a1 P 3aaa5fb00ae84069a009f4dee2917bc2: Bootstrap starting.
I20260808 01:48:55.856560  9316 leader_election.cc:290] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [CANDIDATE]: Term 1 election: Requested vote from peers e91d6edcb50845178ee3c8d60fb4ac4c (127.7.165.193:43869), dfff33c1d94b49e38d105070579e7d6c (127.7.165.195:40603)
I20260808 01:48:55.858465  9095 raft_consensus.cc:2468] T 33f9ea1ccfd24813b8afcb138e17565a P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aaa5fb00ae84069a009f4dee2917bc2 in term 1.
I20260808 01:48:55.859751  9096 raft_consensus.cc:2468] T 9cc5901f3ab34ee7a109a146c958de2e P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aaa5fb00ae84069a009f4dee2917bc2 in term 1.
I20260808 01:48:55.860762  9251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cc5901f3ab34ee7a109a146c958de2e" candidate_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfff33c1d94b49e38d105070579e7d6c"
I20260808 01:48:55.861645  9251 raft_consensus.cc:3060] T 9cc5901f3ab34ee7a109a146c958de2e P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:55.861747  9138 leader_election.cc:304] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aaa5fb00ae84069a009f4dee2917bc2, e91d6edcb50845178ee3c8d60fb4ac4c; no voters: 
I20260808 01:48:55.862902  9138 leader_election.cc:304] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aaa5fb00ae84069a009f4dee2917bc2, e91d6edcb50845178ee3c8d60fb4ac4c; no voters: 
I20260808 01:48:55.863749  9300 raft_consensus.cc:2804] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:55.863843  9316 raft_consensus.cc:2804] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:55.872646  9252 raft_consensus.cc:2468] T 33f9ea1ccfd24813b8afcb138e17565a P dfff33c1d94b49e38d105070579e7d6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aaa5fb00ae84069a009f4dee2917bc2 in term 1.
I20260808 01:48:55.873119  9299 raft_consensus.cc:515] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.880645  8983 catalog_manager.cc:5697] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c reported cstate change: term changed from 0 to 1, leader changed from <none> to e91d6edcb50845178ee3c8d60fb4ac4c (127.7.165.193). New cstate: current_term: 1 leader_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:55.882237  9175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90faceeb397646dc9c3d19c907d4acdc" candidate_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aaa5fb00ae84069a009f4dee2917bc2"
I20260808 01:48:55.883185  9175 raft_consensus.cc:3060] T 90faceeb397646dc9c3d19c907d4acdc P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:55.898926  9175 raft_consensus.cc:2468] T 90faceeb397646dc9c3d19c907d4acdc P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e91d6edcb50845178ee3c8d60fb4ac4c in term 1.
I20260808 01:48:55.899880  9331 raft_consensus.cc:493] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:55.901010  9060 leader_election.cc:304] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aaa5fb00ae84069a009f4dee2917bc2, e91d6edcb50845178ee3c8d60fb4ac4c; no voters: 
I20260808 01:48:55.901573  9331 raft_consensus.cc:515] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.902781  9318 raft_consensus.cc:2804] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:55.903290  9318 raft_consensus.cc:697] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 LEADER]: Becoming Leader. State: Replica: e91d6edcb50845178ee3c8d60fb4ac4c, State: Running, Role: LEADER
I20260808 01:48:55.904666  9331 leader_election.cc:290] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e91d6edcb50845178ee3c8d60fb4ac4c (127.7.165.193:43869), dfff33c1d94b49e38d105070579e7d6c (127.7.165.195:40603)
I20260808 01:48:55.904536  9318 consensus_queue.cc:237] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.906661  9300 raft_consensus.cc:697] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 LEADER]: Becoming Leader. State: Replica: 3aaa5fb00ae84069a009f4dee2917bc2, State: Running, Role: LEADER
I20260808 01:48:55.911769  9294 tablet_bootstrap.cc:654] T 68eef7a69e004073b251ab0e3e6445a1 P 3aaa5fb00ae84069a009f4dee2917bc2: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:55.911691  9096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af8385293960449aba32c949e652aea6" candidate_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" is_pre_election: true
I20260808 01:48:55.912034  9300 consensus_queue.cc:237] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.912426  9251 raft_consensus.cc:2468] T 9cc5901f3ab34ee7a109a146c958de2e P dfff33c1d94b49e38d105070579e7d6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aaa5fb00ae84069a009f4dee2917bc2 in term 1.
I20260808 01:48:55.910429  9316 raft_consensus.cc:697] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 LEADER]: Becoming Leader. State: Replica: 3aaa5fb00ae84069a009f4dee2917bc2, State: Running, Role: LEADER
I20260808 01:48:55.913350  9252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af8385293960449aba32c949e652aea6" candidate_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfff33c1d94b49e38d105070579e7d6c" is_pre_election: true
I20260808 01:48:55.914098  9252 raft_consensus.cc:2468] T af8385293960449aba32c949e652aea6 P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aaa5fb00ae84069a009f4dee2917bc2 in term 0.
I20260808 01:48:55.913908  9316 consensus_queue.cc:237] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.912508  9096 raft_consensus.cc:2468] T af8385293960449aba32c949e652aea6 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aaa5fb00ae84069a009f4dee2917bc2 in term 0.
I20260808 01:48:55.918169  9138 leader_election.cc:304] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aaa5fb00ae84069a009f4dee2917bc2, dfff33c1d94b49e38d105070579e7d6c; no voters: 
I20260808 01:48:55.921835  9252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90faceeb397646dc9c3d19c907d4acdc" candidate_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfff33c1d94b49e38d105070579e7d6c"
I20260808 01:48:55.910471  9299 leader_election.cc:290] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [CANDIDATE]: Term 1 election: Requested vote from peers 3aaa5fb00ae84069a009f4dee2917bc2 (127.7.165.194:42547), dfff33c1d94b49e38d105070579e7d6c (127.7.165.195:40603)
I20260808 01:48:55.922860  9252 raft_consensus.cc:3060] T 90faceeb397646dc9c3d19c907d4acdc P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:55.920745  9330 raft_consensus.cc:2804] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:55.930821  9330 raft_consensus.cc:493] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:55.931227  9330 raft_consensus.cc:3060] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:55.930223  9294 tablet_bootstrap.cc:492] T 68eef7a69e004073b251ab0e3e6445a1 P 3aaa5fb00ae84069a009f4dee2917bc2: No bootstrap required, opened a new log
I20260808 01:48:55.934139  9294 ts_tablet_manager.cc:1403] T 68eef7a69e004073b251ab0e3e6445a1 P 3aaa5fb00ae84069a009f4dee2917bc2: Time spent bootstrapping tablet: real 0.088s	user 0.011s	sys 0.007s
I20260808 01:48:55.937587  9294 raft_consensus.cc:359] T 68eef7a69e004073b251ab0e3e6445a1 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.938401  9294 raft_consensus.cc:385] T 68eef7a69e004073b251ab0e3e6445a1 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:55.937826  8983 catalog_manager.cc:5697] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c reported cstate change: term changed from 0 to 1, leader changed from <none> to e91d6edcb50845178ee3c8d60fb4ac4c (127.7.165.193). New cstate: current_term: 1 leader_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:55.942793  9294 raft_consensus.cc:740] T 68eef7a69e004073b251ab0e3e6445a1 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aaa5fb00ae84069a009f4dee2917bc2, State: Initialized, Role: FOLLOWER
I20260808 01:48:55.947499  9294 consensus_queue.cc:260] T 68eef7a69e004073b251ab0e3e6445a1 P 3aaa5fb00ae84069a009f4dee2917bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.950064  9294 ts_tablet_manager.cc:1434] T 68eef7a69e004073b251ab0e3e6445a1 P 3aaa5fb00ae84069a009f4dee2917bc2: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.000s
I20260808 01:48:55.946344  9252 raft_consensus.cc:2468] T 90faceeb397646dc9c3d19c907d4acdc P dfff33c1d94b49e38d105070579e7d6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e91d6edcb50845178ee3c8d60fb4ac4c in term 1.
I20260808 01:48:55.952888  9294 tablet_bootstrap.cc:492] T 5f5125e2eade46d4bd25cf04eeb19170 P 3aaa5fb00ae84069a009f4dee2917bc2: Bootstrap starting.
I20260808 01:48:55.952718  9330 raft_consensus.cc:515] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.955487  8986 catalog_manager.cc:5697] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aaa5fb00ae84069a009f4dee2917bc2 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:55.956740  9096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af8385293960449aba32c949e652aea6" candidate_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c"
I20260808 01:48:55.957511  9096 raft_consensus.cc:3060] T af8385293960449aba32c949e652aea6 P e91d6edcb50845178ee3c8d60fb4ac4c [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:55.957516  9252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af8385293960449aba32c949e652aea6" candidate_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfff33c1d94b49e38d105070579e7d6c"
I20260808 01:48:55.958385  9252 raft_consensus.cc:3060] T af8385293960449aba32c949e652aea6 P dfff33c1d94b49e38d105070579e7d6c [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:55.960319  9294 tablet_bootstrap.cc:654] T 5f5125e2eade46d4bd25cf04eeb19170 P 3aaa5fb00ae84069a009f4dee2917bc2: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:55.962893  9330 leader_election.cc:290] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [CANDIDATE]: Term 1 election: Requested vote from peers e91d6edcb50845178ee3c8d60fb4ac4c (127.7.165.193:43869), dfff33c1d94b49e38d105070579e7d6c (127.7.165.195:40603)
I20260808 01:48:55.965440  9252 raft_consensus.cc:2468] T af8385293960449aba32c949e652aea6 P dfff33c1d94b49e38d105070579e7d6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aaa5fb00ae84069a009f4dee2917bc2 in term 1.
I20260808 01:48:55.967351  9138 leader_election.cc:304] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aaa5fb00ae84069a009f4dee2917bc2, dfff33c1d94b49e38d105070579e7d6c; no voters: 
I20260808 01:48:55.968478  9330 raft_consensus.cc:2804] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:55.969007  9330 raft_consensus.cc:697] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 LEADER]: Becoming Leader. State: Replica: 3aaa5fb00ae84069a009f4dee2917bc2, State: Running, Role: LEADER
I20260808 01:48:55.969560  9096 raft_consensus.cc:2468] T af8385293960449aba32c949e652aea6 P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aaa5fb00ae84069a009f4dee2917bc2 in term 1.
I20260808 01:48:55.969849  9330 consensus_queue.cc:237] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.971531  9294 tablet_bootstrap.cc:492] T 5f5125e2eade46d4bd25cf04eeb19170 P 3aaa5fb00ae84069a009f4dee2917bc2: No bootstrap required, opened a new log
I20260808 01:48:55.972218  9294 ts_tablet_manager.cc:1403] T 5f5125e2eade46d4bd25cf04eeb19170 P 3aaa5fb00ae84069a009f4dee2917bc2: Time spent bootstrapping tablet: real 0.020s	user 0.018s	sys 0.000s
I20260808 01:48:55.976009  9294 raft_consensus.cc:359] T 5f5125e2eade46d4bd25cf04eeb19170 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.976814  9294 raft_consensus.cc:385] T 5f5125e2eade46d4bd25cf04eeb19170 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:55.977151  9294 raft_consensus.cc:740] T 5f5125e2eade46d4bd25cf04eeb19170 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aaa5fb00ae84069a009f4dee2917bc2, State: Initialized, Role: FOLLOWER
I20260808 01:48:55.977941  9294 consensus_queue.cc:260] T 5f5125e2eade46d4bd25cf04eeb19170 P 3aaa5fb00ae84069a009f4dee2917bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.980422  9294 ts_tablet_manager.cc:1434] T 5f5125e2eade46d4bd25cf04eeb19170 P 3aaa5fb00ae84069a009f4dee2917bc2: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260808 01:48:55.980924  9330 raft_consensus.cc:493] T 68eef7a69e004073b251ab0e3e6445a1 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:55.981452  9330 raft_consensus.cc:515] T 68eef7a69e004073b251ab0e3e6445a1 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } }
I20260808 01:48:55.981139  8986 catalog_manager.cc:5697] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aaa5fb00ae84069a009f4dee2917bc2 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:55.982822  8986 catalog_manager.cc:5697] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aaa5fb00ae84069a009f4dee2917bc2 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 43869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:55.983462  9330 leader_election.cc:290] T 68eef7a69e004073b251ab0e3e6445a1 P 3aaa5fb00ae84069a009f4dee2917bc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e91d6edcb50845178ee3c8d60fb4ac4c (127.7.165.193:43869), dfff33c1d94b49e38d105070579e7d6c (127.7.165.195:40603)
I20260808 01:48:55.984352  9096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68eef7a69e004073b251ab0e3e6445a1" candidate_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c" is_pre_election: true
I20260808 01:48:55.984772  9252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "68eef7a69e004073b251ab0e3e6445a1" candidate_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfff33c1d94b49e38d105070579e7d6c" is_pre_election: true
I20260808 01:48:55.985574  9252 raft_consensus.cc:2393] T 68eef7a69e004073b251ab0e3e6445a1 P dfff33c1d94b49e38d105070579e7d6c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3aaa5fb00ae84069a009f4dee2917bc2 in current term 1: Already voted for candidate e91d6edcb50845178ee3c8d60fb4ac4c in this term.
I20260808 01:48:55.986595  9138 leader_election.cc:304] T 68eef7a69e004073b251ab0e3e6445a1 P 3aaa5fb00ae84069a009f4dee2917bc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3aaa5fb00ae84069a009f4dee2917bc2; no voters: dfff33c1d94b49e38d105070579e7d6c, e91d6edcb50845178ee3c8d60fb4ac4c
I20260808 01:48:55.987499  9330 raft_consensus.cc:3060] T 68eef7a69e004073b251ab0e3e6445a1 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:55.994799  9330 raft_consensus.cc:2749] T 68eef7a69e004073b251ab0e3e6445a1 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:48:56.180761  9175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "af8385293960449aba32c949e652aea6"
dest_uuid: "3aaa5fb00ae84069a009f4dee2917bc2"
mode: GRACEFUL
new_leader_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c"
 from {username='slave'} at 127.0.0.1:36488
I20260808 01:48:56.181452  9175 raft_consensus.cc:606] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 LEADER]: Received request to transfer leadership to TS e91d6edcb50845178ee3c8d60fb4ac4c
I20260808 01:48:56.186806  9318 consensus_queue.cc:1048] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:56.195189  9175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9cc5901f3ab34ee7a109a146c958de2e"
dest_uuid: "3aaa5fb00ae84069a009f4dee2917bc2"
mode: GRACEFUL
new_leader_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c"
 from {username='slave'} at 127.0.0.1:36488
I20260808 01:48:56.195873  9175 raft_consensus.cc:606] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 LEADER]: Received request to transfer leadership to TS e91d6edcb50845178ee3c8d60fb4ac4c
I20260808 01:48:56.204842  9175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "33f9ea1ccfd24813b8afcb138e17565a"
dest_uuid: "3aaa5fb00ae84069a009f4dee2917bc2"
mode: GRACEFUL
new_leader_uuid: "e91d6edcb50845178ee3c8d60fb4ac4c"
 from {username='slave'} at 127.0.0.1:36488
I20260808 01:48:56.209403  9175 raft_consensus.cc:606] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 LEADER]: Received request to transfer leadership to TS e91d6edcb50845178ee3c8d60fb4ac4c
I20260808 01:48:56.215607  9175 raft_consensus.cc:3060] T 5f5125e2eade46d4bd25cf04eeb19170 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:56.225514  9299 consensus_queue.cc:1048] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aaa5fb00ae84069a009f4dee2917bc2" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 42547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:48:56.258222  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
W20260808 01:48:56.263280  7831 auto_leader_rebalancer.cc:456] leader transfer for tablet 90faceeb397646dc9c3d19c907d4acdc from e91d6edcb50845178ee3c8d60fb4ac4c to dfff33c1d94b49e38d105070579e7d6c failed: Service unavailable: TEST: forced leader step-down failure
W20260808 01:48:56.264050  7831 auto_leader_rebalancer.cc:456] leader transfer for tablet af8385293960449aba32c949e652aea6 from 3aaa5fb00ae84069a009f4dee2917bc2 to dfff33c1d94b49e38d105070579e7d6c failed: Service unavailable: TEST: forced leader step-down failure
I20260808 01:48:56.264350  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:48:56.264799  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:48:56.265332  7831 tablet_server.cc:179] TabletServer@127.7.165.193:0 shutting down...
I20260808 01:48:56.290313  7831 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260808 01:48:56.291543  7831 tablet_replica.cc:333] T af8385293960449aba32c949e652aea6 P e91d6edcb50845178ee3c8d60fb4ac4c: stopping tablet replica
I20260808 01:48:56.292368  7831 raft_consensus.cc:2243] T af8385293960449aba32c949e652aea6 P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:56.292937  7831 raft_consensus.cc:2272] T af8385293960449aba32c949e652aea6 P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:56.295468  7831 tablet_replica.cc:333] T 33f9ea1ccfd24813b8afcb138e17565a P e91d6edcb50845178ee3c8d60fb4ac4c: stopping tablet replica
I20260808 01:48:56.296068  7831 raft_consensus.cc:2243] T 33f9ea1ccfd24813b8afcb138e17565a P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:56.296463  7831 raft_consensus.cc:2272] T 33f9ea1ccfd24813b8afcb138e17565a P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:56.298518  7831 tablet_replica.cc:333] T 9cc5901f3ab34ee7a109a146c958de2e P e91d6edcb50845178ee3c8d60fb4ac4c: stopping tablet replica
I20260808 01:48:56.299338  7831 raft_consensus.cc:2243] T 9cc5901f3ab34ee7a109a146c958de2e P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:56.299834  7831 raft_consensus.cc:2272] T 9cc5901f3ab34ee7a109a146c958de2e P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:56.301721  7831 tablet_replica.cc:333] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c: stopping tablet replica
I20260808 01:48:56.302325  7831 raft_consensus.cc:2243] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 LEADER]: Raft consensus shutting down.
I20260808 01:48:56.303095  7831 pending_rounds.cc:70] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c: Trying to abort 1 pending ops.
I20260808 01:48:56.303453  7831 pending_rounds.cc:77] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316085702034132992 op_type: NO_OP noop_request { }
I20260808 01:48:56.303844  7831 raft_consensus.cc:2889] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260808 01:48:56.304184  7831 raft_consensus.cc:2272] T 68eef7a69e004073b251ab0e3e6445a1 P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:56.306491  7831 tablet_replica.cc:333] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c: stopping tablet replica
I20260808 01:48:56.307138  7831 raft_consensus.cc:2243] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 LEADER]: Raft consensus shutting down.
I20260808 01:48:56.307812  7831 pending_rounds.cc:70] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c: Trying to abort 1 pending ops.
I20260808 01:48:56.308060  7831 pending_rounds.cc:77] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316085702283726848 op_type: NO_OP noop_request { }
I20260808 01:48:56.308375  7831 raft_consensus.cc:2889] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260808 01:48:56.308699  7831 raft_consensus.cc:2272] T 90faceeb397646dc9c3d19c907d4acdc P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:56.311319  7831 tablet_replica.cc:333] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c: stopping tablet replica
I20260808 01:48:56.311973  7831 raft_consensus.cc:2243] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 LEADER]: Raft consensus shutting down.
I20260808 01:48:56.312804  7831 raft_consensus.cc:2272] T 5f5125e2eade46d4bd25cf04eeb19170 P e91d6edcb50845178ee3c8d60fb4ac4c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260808 01:48:56.316668  9138 consensus_peers.cc:597] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 -> Peer e91d6edcb50845178ee3c8d60fb4ac4c (127.7.165.193:43869): Couldn't send request to peer e91d6edcb50845178ee3c8d60fb4ac4c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:56.343740  7831 tablet_server.cc:196] TabletServer@127.7.165.193:0 shutdown complete.
I20260808 01:48:56.357982  7831 tablet_server.cc:179] TabletServer@127.7.165.194:0 shutting down...
I20260808 01:48:56.365991  9330 consensus_queue.cc:1048] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 01:48:56.381031  9138 consensus_peers.cc:597] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 -> Peer e91d6edcb50845178ee3c8d60fb4ac4c (127.7.165.193:43869): Couldn't send request to peer e91d6edcb50845178ee3c8d60fb4ac4c. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:43869: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:48:56.404693  9138 consensus_peers.cc:597] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 -> Peer e91d6edcb50845178ee3c8d60fb4ac4c (127.7.165.193:43869): Couldn't send request to peer e91d6edcb50845178ee3c8d60fb4ac4c. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:43869: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:48:56.406850  7831 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260808 01:48:56.408560  7831 tablet_replica.cc:333] T 5f5125e2eade46d4bd25cf04eeb19170 P 3aaa5fb00ae84069a009f4dee2917bc2: stopping tablet replica
I20260808 01:48:56.409229  7831 raft_consensus.cc:2243] T 5f5125e2eade46d4bd25cf04eeb19170 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:56.409734  7831 raft_consensus.cc:2272] T 5f5125e2eade46d4bd25cf04eeb19170 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:56.412897  7831 tablet_replica.cc:333] T 90faceeb397646dc9c3d19c907d4acdc P 3aaa5fb00ae84069a009f4dee2917bc2: stopping tablet replica
I20260808 01:48:56.413552  7831 raft_consensus.cc:2243] T 90faceeb397646dc9c3d19c907d4acdc P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:56.414000  7831 raft_consensus.cc:2272] T 90faceeb397646dc9c3d19c907d4acdc P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:56.414355  9316 consensus_queue.cc:1048] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfff33c1d94b49e38d105070579e7d6c" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 40603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:56.415910  7831 tablet_replica.cc:333] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2: stopping tablet replica
I20260808 01:48:56.416534  7831 raft_consensus.cc:2243] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 01:48:56.417358  7831 raft_consensus.cc:2272] T 33f9ea1ccfd24813b8afcb138e17565a P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:56.420207  7831 tablet_replica.cc:333] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2: stopping tablet replica
I20260808 01:48:56.421029  7831 raft_consensus.cc:2243] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 01:48:56.421931  7831 pending_rounds.cc:70] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2: Trying to abort 1 pending ops.
I20260808 01:48:56.422194  7831 pending_rounds.cc:77] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316085702347915264 op_type: NO_OP noop_request { }
I20260808 01:48:56.422538  7831 raft_consensus.cc:2889] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260808 01:48:56.422966  7831 raft_consensus.cc:2272] T 9cc5901f3ab34ee7a109a146c958de2e P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:56.426524  7831 tablet_replica.cc:333] T 68eef7a69e004073b251ab0e3e6445a1 P 3aaa5fb00ae84069a009f4dee2917bc2: stopping tablet replica
I20260808 01:48:56.427343  7831 raft_consensus.cc:2243] T 68eef7a69e004073b251ab0e3e6445a1 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:56.428021  7831 raft_consensus.cc:2272] T 68eef7a69e004073b251ab0e3e6445a1 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:56.430470  7831 tablet_replica.cc:333] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2: stopping tablet replica
I20260808 01:48:56.431109  7831 raft_consensus.cc:2243] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 LEADER]: Raft consensus shutting down.
I20260808 01:48:56.431934  7831 raft_consensus.cc:2272] T af8385293960449aba32c949e652aea6 P 3aaa5fb00ae84069a009f4dee2917bc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:56.458552  7831 tablet_server.cc:196] TabletServer@127.7.165.194:0 shutdown complete.
I20260808 01:48:56.472512  7831 tablet_server.cc:179] TabletServer@127.7.165.195:0 shutting down...
I20260808 01:48:56.494089  7831 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260808 01:48:56.495047  7831 tablet_replica.cc:333] T 9cc5901f3ab34ee7a109a146c958de2e P dfff33c1d94b49e38d105070579e7d6c: stopping tablet replica
I20260808 01:48:56.495765  7831 raft_consensus.cc:2243] T 9cc5901f3ab34ee7a109a146c958de2e P dfff33c1d94b49e38d105070579e7d6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:56.496269  7831 raft_consensus.cc:2272] T 9cc5901f3ab34ee7a109a146c958de2e P dfff33c1d94b49e38d105070579e7d6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:56.498296  7831 tablet_replica.cc:333] T 33f9ea1ccfd24813b8afcb138e17565a P dfff33c1d94b49e38d105070579e7d6c: stopping tablet replica
I20260808 01:48:56.499032  7831 raft_consensus.cc:2243] T 33f9ea1ccfd24813b8afcb138e17565a P dfff33c1d94b49e38d105070579e7d6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:56.499610  7831 raft_consensus.cc:2272] T 33f9ea1ccfd24813b8afcb138e17565a P dfff33c1d94b49e38d105070579e7d6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:56.502190  7831 tablet_replica.cc:333] T af8385293960449aba32c949e652aea6 P dfff33c1d94b49e38d105070579e7d6c: stopping tablet replica
I20260808 01:48:56.502983  7831 raft_consensus.cc:2243] T af8385293960449aba32c949e652aea6 P dfff33c1d94b49e38d105070579e7d6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:56.503484  7831 raft_consensus.cc:2272] T af8385293960449aba32c949e652aea6 P dfff33c1d94b49e38d105070579e7d6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:56.505810  7831 tablet_replica.cc:333] T 68eef7a69e004073b251ab0e3e6445a1 P dfff33c1d94b49e38d105070579e7d6c: stopping tablet replica
I20260808 01:48:56.506422  7831 raft_consensus.cc:2243] T 68eef7a69e004073b251ab0e3e6445a1 P dfff33c1d94b49e38d105070579e7d6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:56.506910  7831 raft_consensus.cc:2272] T 68eef7a69e004073b251ab0e3e6445a1 P dfff33c1d94b49e38d105070579e7d6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:56.508960  7831 tablet_replica.cc:333] T 90faceeb397646dc9c3d19c907d4acdc P dfff33c1d94b49e38d105070579e7d6c: stopping tablet replica
I20260808 01:48:56.509550  7831 raft_consensus.cc:2243] T 90faceeb397646dc9c3d19c907d4acdc P dfff33c1d94b49e38d105070579e7d6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:56.509976  7831 raft_consensus.cc:2272] T 90faceeb397646dc9c3d19c907d4acdc P dfff33c1d94b49e38d105070579e7d6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:56.512104  7831 tablet_replica.cc:333] T 5f5125e2eade46d4bd25cf04eeb19170 P dfff33c1d94b49e38d105070579e7d6c: stopping tablet replica
I20260808 01:48:56.512683  7831 raft_consensus.cc:2243] T 5f5125e2eade46d4bd25cf04eeb19170 P dfff33c1d94b49e38d105070579e7d6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:48:56.513358  7831 raft_consensus.cc:2272] T 5f5125e2eade46d4bd25cf04eeb19170 P dfff33c1d94b49e38d105070579e7d6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:56.540021  7831 tablet_server.cc:196] TabletServer@127.7.165.195:0 shutdown complete.
I20260808 01:48:56.556201  7831 master.cc:562] Master@127.7.165.254:43379 shutting down...
I20260808 01:48:56.576292  7831 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [term 1 LEADER]: Raft consensus shutting down.
I20260808 01:48:56.576975  7831 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:48:56.577332  7831 tablet_replica.cc:333] T 00000000000000000000000000000000 P 7fd9d1d2beb9480fb21203a4e1aa0d8e: stopping tablet replica
I20260808 01:48:56.599752  7831 master.cc:584] Master@127.7.165.254:43379 shutdown complete.
[       OK ] LeaderRebalancerTest.TestStepDownFailureDoesNotAbortPass (2120 ms)
[ RUN      ] LeaderRebalancerTest.AddTserver
I20260808 01:48:56.639487  7831 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.7.165.254:43565
I20260808 01:48:56.640884  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:56.647089  9350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:56.647109  9349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:56.650332  7831 server_base.cc:1061] running on GCE node
W20260808 01:48:56.650588  9352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:56.651873  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:48:56.652122  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:48:56.652274  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786153736652259 us; error 0 us; skew 500 ppm
I20260808 01:48:56.652920  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:56.656498  7831 webserver.cc:533] Webserver started at http://127.7.165.254:46593/ using document root <none> and password file <none>
I20260808 01:48:56.657128  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:56.657351  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:56.657676  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:56.659057  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/master-0-root/instance:
uuid: "3cd8e8730323483db1c5f22e90bb080c"
format_stamp: "Formatted at 2026-08-08 01:48:56 on dist-test-slave-8tdl"
I20260808 01:48:56.664808  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260808 01:48:56.670302  9358 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:56.671520  7831 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260808 01:48:56.671839  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/master-0-root
uuid: "3cd8e8730323483db1c5f22e90bb080c"
format_stamp: "Formatted at 2026-08-08 01:48:56 on dist-test-slave-8tdl"
I20260808 01:48:56.672123  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:56.693370  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:48:56.695528  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:56.743235  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.254:43565
I20260808 01:48:56.743636  9412 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.254:43565 every 8 connection(s)
I20260808 01:48:56.749325  9413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:56.763222  9413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Bootstrap starting.
I20260808 01:48:56.767985  9413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:56.773011  9413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: No bootstrap required, opened a new log
I20260808 01:48:56.776152  9413 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd8e8730323483db1c5f22e90bb080c" member_type: VOTER }
I20260808 01:48:56.776957  9413 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:56.777212  9413 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd8e8730323483db1c5f22e90bb080c, State: Initialized, Role: FOLLOWER
I20260808 01:48:56.777927  9413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd8e8730323483db1c5f22e90bb080c" member_type: VOTER }
I20260808 01:48:56.778442  9413 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 01:48:56.778662  9413 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 01:48:56.779024  9413 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:56.784219  9413 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd8e8730323483db1c5f22e90bb080c" member_type: VOTER }
I20260808 01:48:56.784813  9413 leader_election.cc:304] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cd8e8730323483db1c5f22e90bb080c; no voters: 
I20260808 01:48:56.786375  9413 leader_election.cc:290] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260808 01:48:56.786849  9416 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:56.788661  9416 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [term 1 LEADER]: Becoming Leader. State: Replica: 3cd8e8730323483db1c5f22e90bb080c, State: Running, Role: LEADER
I20260808 01:48:56.789340  9416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd8e8730323483db1c5f22e90bb080c" member_type: VOTER }
I20260808 01:48:56.790241  9413 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 01:48:56.794025  9417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cd8e8730323483db1c5f22e90bb080c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd8e8730323483db1c5f22e90bb080c" member_type: VOTER } }
I20260808 01:48:56.794025  9418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cd8e8730323483db1c5f22e90bb080c. Latest consensus state: current_term: 1 leader_uuid: "3cd8e8730323483db1c5f22e90bb080c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd8e8730323483db1c5f22e90bb080c" member_type: VOTER } }
I20260808 01:48:56.794875  9417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [sys.catalog]: This master's current role is: LEADER
I20260808 01:48:56.795027  9418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [sys.catalog]: This master's current role is: LEADER
I20260808 01:48:56.803792  9421 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260808 01:48:56.809146  9421 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260808 01:48:56.813485  7831 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260808 01:48:56.819386  9421 catalog_manager.cc:1383] Generated new cluster ID: a7eb82ab67824fd18499b5a7b7da6adb
I20260808 01:48:56.819802  9421 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260808 01:48:56.850059  9421 catalog_manager.cc:1406] Generated new certificate authority record
I20260808 01:48:56.851679  9421 catalog_manager.cc:1540] Loading token signing keys...
I20260808 01:48:56.872481  9421 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Generated new TSK 0
I20260808 01:48:56.873210  9421 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260808 01:48:56.881839  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:56.889623  9434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:56.891036  9435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:56.892459  9437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:56.892745  7831 server_base.cc:1061] running on GCE node
I20260808 01:48:56.893743  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:48:56.893990  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:48:56.894177  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786153736894159 us; error 0 us; skew 500 ppm
I20260808 01:48:56.895157  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:56.898152  7831 webserver.cc:533] Webserver started at http://127.7.165.193:38849/ using document root <none> and password file <none>
I20260808 01:48:56.898830  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:56.899083  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:56.899395  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:56.900803  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-0-root/instance:
uuid: "6e7856bed1be43acaa821290d506ee97"
format_stamp: "Formatted at 2026-08-08 01:48:56 on dist-test-slave-8tdl"
I20260808 01:48:56.906257  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260808 01:48:56.911096  9442 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:56.912175  7831 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260808 01:48:56.912596  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-0-root
uuid: "6e7856bed1be43acaa821290d506ee97"
format_stamp: "Formatted at 2026-08-08 01:48:56 on dist-test-slave-8tdl"
I20260808 01:48:56.912979  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:56.926982  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:48:56.928354  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:56.930066  7831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:56.933060  7831 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:56.933357  7831 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:56.933674  7831 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:56.933876  7831 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:56.986123  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.193:39359
I20260808 01:48:56.986282  9506 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.193:39359 every 8 connection(s)
I20260808 01:48:56.992401  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:57.001089  9511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:57.002614  9512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:57.010823  9507 heartbeater.cc:344] Connected to a master server at 127.7.165.254:43565
W20260808 01:48:57.010928  9514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:57.011684  7831 server_base.cc:1061] running on GCE node
I20260808 01:48:57.011773  9507 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:57.013242  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260808 01:48:57.013208  9507 heartbeater.cc:507] Master 127.7.165.254:43565 requested a full tablet report, sending...
W20260808 01:48:57.013633  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:48:57.014093  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786153737014065 us; error 0 us; skew 500 ppm
I20260808 01:48:57.015105  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:57.016853  9377 ts_manager.cc:194] Registered new tserver with Master: 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359)
I20260808 01:48:57.018596  7831 webserver.cc:533] Webserver started at http://127.7.165.194:46123/ using document root <none> and password file <none>
I20260808 01:48:57.019335  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:57.019290  9377 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50064
I20260808 01:48:57.019716  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:57.020077  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:57.021533  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-1-root/instance:
uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
format_stamp: "Formatted at 2026-08-08 01:48:57 on dist-test-slave-8tdl"
I20260808 01:48:57.027081  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.004s
I20260808 01:48:57.031738  9519 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:57.032867  7831 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260808 01:48:57.033267  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-1-root
uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
format_stamp: "Formatted at 2026-08-08 01:48:57 on dist-test-slave-8tdl"
I20260808 01:48:57.033631  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:57.050848  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:48:57.052191  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:57.053825  7831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:57.056602  7831 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:57.056859  7831 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:57.057204  7831 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:57.057381  7831 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:57.118408  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.194:36095
I20260808 01:48:57.118532  9585 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.194:36095 every 8 connection(s)
I20260808 01:48:57.124897  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:48:57.135418  9589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:48:57.137641  9590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:57.144420  9586 heartbeater.cc:344] Connected to a master server at 127.7.165.254:43565
W20260808 01:48:57.144795  9593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:48:57.145146  9586 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:57.145510  7831 server_base.cc:1061] running on GCE node
I20260808 01:48:57.146274  9586 heartbeater.cc:507] Master 127.7.165.254:43565 requested a full tablet report, sending...
I20260808 01:48:57.146858  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:48:57.147272  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:48:57.147588  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786153737147561 us; error 0 us; skew 500 ppm
I20260808 01:48:57.148558  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:48:57.150419  9377 ts_manager.cc:194] Registered new tserver with Master: 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:48:57.153574  9377 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50068
I20260808 01:48:57.154066  7831 webserver.cc:533] Webserver started at http://127.7.165.195:34945/ using document root <none> and password file <none>
I20260808 01:48:57.155320  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:48:57.155637  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:48:57.156006  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:48:57.157167  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-2-root/instance:
uuid: "259bcb3f0d254e8b9effac3277e654e0"
format_stamp: "Formatted at 2026-08-08 01:48:57 on dist-test-slave-8tdl"
I20260808 01:48:57.163913  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.008s	sys 0.000s
I20260808 01:48:57.168769  9597 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:57.170221  7831 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.002s
I20260808 01:48:57.170657  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-2-root
uuid: "259bcb3f0d254e8b9effac3277e654e0"
format_stamp: "Formatted at 2026-08-08 01:48:57 on dist-test-slave-8tdl"
I20260808 01:48:57.171077  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:48:57.187211  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:48:57.189294  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:48:57.191726  7831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:48:57.195082  7831 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:48:57.195381  7831 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:57.195676  7831 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:48:57.195878  7831 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:48:57.261355  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.195:34565
I20260808 01:48:57.261413  9663 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.195:34565 every 8 connection(s)
I20260808 01:48:57.285224  9664 heartbeater.cc:344] Connected to a master server at 127.7.165.254:43565
I20260808 01:48:57.285941  9664 heartbeater.cc:461] Registering TS with master...
I20260808 01:48:57.287676  9664 heartbeater.cc:507] Master 127.7.165.254:43565 requested a full tablet report, sending...
I20260808 01:48:57.291697  9377 ts_manager.cc:194] Registered new tserver with Master: 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:48:57.292168  7831 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.025226089s
I20260808 01:48:57.294477  9377 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50084
I20260808 01:48:57.338238  9377 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50086:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260808 01:48:57.345844  9377 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 01:48:57.601301  9549 tablet_service.cc:1511] Processing CreateTablet for tablet 4f04078e3beb42eda250da45f7485537 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260808 01:48:57.602792  9549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f04078e3beb42eda250da45f7485537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.602974  9545 tablet_service.cc:1511] Processing CreateTablet for tablet 8c49fd321f8b4f89868246239530e78b (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260808 01:48:57.604467  9545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c49fd321f8b4f89868246239530e78b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.606312  9548 tablet_service.cc:1511] Processing CreateTablet for tablet cacb9622c47c4737beff104125e98b77 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260808 01:48:57.607860  9548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cacb9622c47c4737beff104125e98b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.607735  9544 tablet_service.cc:1511] Processing CreateTablet for tablet 145c5a9982a24c8fa4e86d0d99cbbdfb (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260808 01:48:57.609182  9544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 145c5a9982a24c8fa4e86d0d99cbbdfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.609090  9471 tablet_service.cc:1511] Processing CreateTablet for tablet 4f04078e3beb42eda250da45f7485537 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260808 01:48:57.610085  9547 tablet_service.cc:1511] Processing CreateTablet for tablet 4c485e8f4266462e8413079adcf8faef (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260808 01:48:57.610625  9471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f04078e3beb42eda250da45f7485537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.611567  9547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c485e8f4266462e8413079adcf8faef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.612695  9546 tablet_service.cc:1511] Processing CreateTablet for tablet 626249dbd6eb4d66bfa8cce2e716cbe2 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260808 01:48:57.624169  9546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 626249dbd6eb4d66bfa8cce2e716cbe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.607904  9543 tablet_service.cc:1511] Processing CreateTablet for tablet fca874793205463dbb922993f57bf7f5 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260808 01:48:57.628443  9543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fca874793205463dbb922993f57bf7f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.612914  9469 tablet_service.cc:1511] Processing CreateTablet for tablet 4c485e8f4266462e8413079adcf8faef (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260808 01:48:57.632576  9469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c485e8f4266462e8413079adcf8faef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.614102  9468 tablet_service.cc:1511] Processing CreateTablet for tablet 626249dbd6eb4d66bfa8cce2e716cbe2 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260808 01:48:57.636924  9468 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 626249dbd6eb4d66bfa8cce2e716cbe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.615633  9467 tablet_service.cc:1511] Processing CreateTablet for tablet 8c49fd321f8b4f89868246239530e78b (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260808 01:48:57.640540  9467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c49fd321f8b4f89868246239530e78b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.616837  9466 tablet_service.cc:1511] Processing CreateTablet for tablet 145c5a9982a24c8fa4e86d0d99cbbdfb (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260808 01:48:57.644492  9466 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 145c5a9982a24c8fa4e86d0d99cbbdfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.646364  9682 tablet_bootstrap.cc:492] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:48:57.618112  9465 tablet_service.cc:1511] Processing CreateTablet for tablet fca874793205463dbb922993f57bf7f5 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260808 01:48:57.647903  9465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fca874793205463dbb922993f57bf7f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.611860  9470 tablet_service.cc:1511] Processing CreateTablet for tablet cacb9622c47c4737beff104125e98b77 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260808 01:48:57.651252  9470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cacb9622c47c4737beff104125e98b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.653126  9682 tablet_bootstrap.cc:654] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:57.679968  9682 tablet_bootstrap.cc:492] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:48:57.680521  9682 ts_tablet_manager.cc:1403] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.034s	user 0.020s	sys 0.008s
I20260808 01:48:57.683837  9682 raft_consensus.cc:359] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:57.684697  9682 raft_consensus.cc:385] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:57.685029  9682 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:48:57.685860  9682 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:57.706285  9586 heartbeater.cc:499] Master 127.7.165.254:43565 was elected leader, sending a full tablet report...
I20260808 01:48:57.707306  9469 tablet_service.cc:1511] Processing CreateTablet for tablet 0ea307d9adc442539683547d0176a05b (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260808 01:48:57.708788  9469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ea307d9adc442539683547d0176a05b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.716028  9628 tablet_service.cc:1511] Processing CreateTablet for tablet 4f04078e3beb42eda250da45f7485537 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260808 01:48:57.718057  9628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f04078e3beb42eda250da45f7485537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.719702  9682 ts_tablet_manager.cc:1434] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.039s	user 0.009s	sys 0.006s
I20260808 01:48:57.721272  9682 tablet_bootstrap.cc:492] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:48:57.720449  9625 tablet_service.cc:1511] Processing CreateTablet for tablet 626249dbd6eb4d66bfa8cce2e716cbe2 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260808 01:48:57.718087  9626 tablet_service.cc:1511] Processing CreateTablet for tablet 4c485e8f4266462e8413079adcf8faef (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260808 01:48:57.719753  9627 tablet_service.cc:1511] Processing CreateTablet for tablet cacb9622c47c4737beff104125e98b77 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260808 01:48:57.730545  9625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 626249dbd6eb4d66bfa8cce2e716cbe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.735713  9627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cacb9622c47c4737beff104125e98b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.736212  9622 tablet_service.cc:1511] Processing CreateTablet for tablet 145c5a9982a24c8fa4e86d0d99cbbdfb (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260808 01:48:57.737717  9622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 145c5a9982a24c8fa4e86d0d99cbbdfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.734266  9621 tablet_service.cc:1511] Processing CreateTablet for tablet fca874793205463dbb922993f57bf7f5 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260808 01:48:57.739104  9626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c485e8f4266462e8413079adcf8faef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.739987  9624 tablet_service.cc:1511] Processing CreateTablet for tablet 8c49fd321f8b4f89868246239530e78b (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260808 01:48:57.741771  9682 tablet_bootstrap.cc:654] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:57.742527  9624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c49fd321f8b4f89868246239530e78b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.746861  9684 tablet_bootstrap.cc:492] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:48:57.747586  9621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fca874793205463dbb922993f57bf7f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.755445  9684 tablet_bootstrap.cc:654] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:57.756431  9546 tablet_service.cc:1511] Processing CreateTablet for tablet 0ea307d9adc442539683547d0176a05b (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260808 01:48:57.768568  9620 tablet_service.cc:1511] Processing CreateTablet for tablet 0ea307d9adc442539683547d0176a05b (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260808 01:48:57.770164  9620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ea307d9adc442539683547d0176a05b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.783207  9546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ea307d9adc442539683547d0176a05b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 01:48:57.809057  9426 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:48:57.816563  9468 tablet_service.cc:1511] Processing CreateTablet for tablet b335bc0170854e26a77dbec13dbe9eac (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260808 01:48:57.818148  9468 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b335bc0170854e26a77dbec13dbe9eac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.821132  9547 tablet_service.cc:1511] Processing CreateTablet for tablet b335bc0170854e26a77dbec13dbe9eac (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260808 01:48:57.825922  9547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b335bc0170854e26a77dbec13dbe9eac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.828728  9628 tablet_service.cc:1511] Processing CreateTablet for tablet b335bc0170854e26a77dbec13dbe9eac (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260808 01:48:57.844962  9628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b335bc0170854e26a77dbec13dbe9eac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.846213  9543 tablet_service.cc:1511] Processing CreateTablet for tablet 439040d168824496acc076b1c280bde3 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260808 01:48:57.847651  9543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 439040d168824496acc076b1c280bde3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.849918  9682 tablet_bootstrap.cc:492] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:48:57.858511  9682 ts_tablet_manager.cc:1403] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.137s	user 0.017s	sys 0.038s
I20260808 01:48:57.861752  9682 raft_consensus.cc:359] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:57.855628  9688 tablet_bootstrap.cc:492] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:48:57.863421  9465 tablet_service.cc:1511] Processing CreateTablet for tablet 439040d168824496acc076b1c280bde3 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260808 01:48:57.865041  9465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 439040d168824496acc076b1c280bde3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.867370  9684 tablet_bootstrap.cc:492] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:48:57.867944  9684 ts_tablet_manager.cc:1403] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.122s	user 0.009s	sys 0.035s
W20260808 01:48:57.873775  9587 tablet.cc:2406] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:48:57.862604  9682 raft_consensus.cc:385] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:57.875326  9688 tablet_bootstrap.cc:654] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:57.878953  9682 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:48:57.878211  9684 raft_consensus.cc:359] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:57.879531  9684 raft_consensus.cc:385] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:57.884168  9684 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:48:57.885355  9684 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:57.879968  9682 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:57.882069  9467 tablet_service.cc:1511] Processing CreateTablet for tablet 54a2156aecd649d391b0acd6dc223843 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260808 01:48:57.891312  9548 tablet_service.cc:1511] Processing CreateTablet for tablet 8cbead77343f44dd82199bebcfb2def7 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260808 01:48:57.892829  9548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cbead77343f44dd82199bebcfb2def7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.892903  9620 tablet_service.cc:1511] Processing CreateTablet for tablet 54a2156aecd649d391b0acd6dc223843 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260808 01:48:57.893333  9622 tablet_service.cc:1511] Processing CreateTablet for tablet 8cbead77343f44dd82199bebcfb2def7 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260808 01:48:57.883283  9546 tablet_service.cc:1511] Processing CreateTablet for tablet 54a2156aecd649d391b0acd6dc223843 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260808 01:48:57.882220  9628 tablet_service.cc:1511] Processing CreateTablet for tablet 439040d168824496acc076b1c280bde3 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260808 01:48:57.891808  9467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54a2156aecd649d391b0acd6dc223843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.895433  9620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54a2156aecd649d391b0acd6dc223843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.895959  9546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54a2156aecd649d391b0acd6dc223843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.898038  9628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 439040d168824496acc076b1c280bde3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.899037  9622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cbead77343f44dd82199bebcfb2def7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.919801  9682 ts_tablet_manager.cc:1434] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.061s	user 0.004s	sys 0.003s
I20260808 01:48:57.921123  9682 tablet_bootstrap.cc:492] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:48:57.893673  9469 tablet_service.cc:1511] Processing CreateTablet for tablet 8cbead77343f44dd82199bebcfb2def7 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260808 01:48:57.924675  9469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cbead77343f44dd82199bebcfb2def7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.925971  9468 tablet_service.cc:1511] Processing CreateTablet for tablet e57f5706d264426eb123cb26c6f3b89c (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260808 01:48:57.927505  9468 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e57f5706d264426eb123cb26c6f3b89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.937984  9543 tablet_service.cc:1511] Processing CreateTablet for tablet e57f5706d264426eb123cb26c6f3b89c (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260808 01:48:57.939678  9543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e57f5706d264426eb123cb26c6f3b89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.940338  9685 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:57.940892  9685 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:57.942752  9620 tablet_service.cc:1511] Processing CreateTablet for tablet e57f5706d264426eb123cb26c6f3b89c (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260808 01:48:57.944235  9620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e57f5706d264426eb123cb26c6f3b89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.944033  9466 tablet_service.cc:1511] Processing CreateTablet for tablet 7b5ea129398a42fb9981fca652027cd6 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260808 01:48:57.945428  9466 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b5ea129398a42fb9981fca652027cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.946403  9685 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:48:57.951771  9549 tablet_service.cc:1511] Processing CreateTablet for tablet 7b5ea129398a42fb9981fca652027cd6 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260808 01:48:57.953399  9549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b5ea129398a42fb9981fca652027cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.979691  9684 ts_tablet_manager.cc:1434] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.111s	user 0.010s	sys 0.010s
I20260808 01:48:57.980347  9626 tablet_service.cc:1511] Processing CreateTablet for tablet 7b5ea129398a42fb9981fca652027cd6 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260808 01:48:57.980890  9684 tablet_bootstrap.cc:492] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:48:57.981828  9626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b5ea129398a42fb9981fca652027cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.982736  9507 heartbeater.cc:499] Master 127.7.165.254:43565 was elected leader, sending a full tablet report...
I20260808 01:48:57.981925  9470 tablet_service.cc:1511] Processing CreateTablet for tablet c99678e59ec845b28293f811d76f24e8 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260808 01:48:57.984463  9470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c99678e59ec845b28293f811d76f24e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:57.999332  9684 tablet_bootstrap.cc:654] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:58.000423  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:48:58.001222  9481 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:48:58.002871  9524 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:48:58.004156  9685 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:58.004621  9685 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:58.005036  9685 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:58.006004  9628 tablet_service.cc:1511] Processing CreateTablet for tablet c99678e59ec845b28293f811d76f24e8 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260808 01:48:58.007845  9628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c99678e59ec845b28293f811d76f24e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.008176  9682 tablet_bootstrap.cc:654] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:58.007977  9547 tablet_service.cc:1511] Processing CreateTablet for tablet c99678e59ec845b28293f811d76f24e8 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260808 01:48:58.009601  9547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c99678e59ec845b28293f811d76f24e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.010495  9467 tablet_service.cc:1511] Processing CreateTablet for tablet 9949664c54584858ae8fc74c9dc07a4a (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260808 01:48:58.012158  9467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9949664c54584858ae8fc74c9dc07a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.019095  9545 tablet_service.cc:1511] Processing CreateTablet for tablet 9949664c54584858ae8fc74c9dc07a4a (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260808 01:48:58.020725  9545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9949664c54584858ae8fc74c9dc07a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.023986  9621 tablet_service.cc:1511] Processing CreateTablet for tablet 9949664c54584858ae8fc74c9dc07a4a (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260808 01:48:58.025604  9621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9949664c54584858ae8fc74c9dc07a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.037578  9465 tablet_service.cc:1511] Processing CreateTablet for tablet 7c508218361c446898fede7c5682fa9e (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260808 01:48:58.039291  9465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c508218361c446898fede7c5682fa9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.046663  9685 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:58.065491  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:48:58.066586  9481 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:58.074618  9548 tablet_service.cc:1511] Processing CreateTablet for tablet 7c508218361c446898fede7c5682fa9e (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260808 01:48:58.076380  9548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c508218361c446898fede7c5682fa9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.078481  9481 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:48:58.080992  9524 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:48:58.085595  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:48:58.095599  9695 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election won for term 1
W20260808 01:48:58.100224  9521 leader_election.cc:343] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:58.103885  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:48:58.110713  9620 tablet_service.cc:1511] Processing CreateTablet for tablet 7c508218361c446898fede7c5682fa9e (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260808 01:48:58.112733  9620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c508218361c446898fede7c5682fa9e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 01:48:58.112890  9521 leader_election.cc:343] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:58.118350  9685 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:48:58.122659  9688 tablet_bootstrap.cc:492] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:48:58.136302  9688 ts_tablet_manager.cc:1403] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.281s	user 0.019s	sys 0.012s
I20260808 01:48:58.140894  9682 tablet_bootstrap.cc:492] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:48:58.155210  9543 tablet_service.cc:1511] Processing CreateTablet for tablet 924b85a2ee7e46ecbcfd57257902abef (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260808 01:48:58.156760  9543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 924b85a2ee7e46ecbcfd57257902abef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.156708  9622 tablet_service.cc:1511] Processing CreateTablet for tablet 924b85a2ee7e46ecbcfd57257902abef (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260808 01:48:58.158317  9622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 924b85a2ee7e46ecbcfd57257902abef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.150122  9685 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:58.159238  9685 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:58.161511  9685 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:48:58.162942  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:48:58.155627  9467 tablet_service.cc:1511] Processing CreateTablet for tablet 924b85a2ee7e46ecbcfd57257902abef (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
W20260808 01:48:58.165010  9521 leader_election.cc:343] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:58.166477  9684 tablet_bootstrap.cc:492] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:48:58.167155  9684 ts_tablet_manager.cc:1403] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.186s	user 0.020s	sys 0.014s
I20260808 01:48:58.170472  9546 tablet_service.cc:1511] Processing CreateTablet for tablet d2b2a88358244f93970835ea3120c380 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260808 01:48:58.171763  9628 tablet_service.cc:1511] Processing CreateTablet for tablet d2b2a88358244f93970835ea3120c380 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260808 01:48:58.171856  9466 tablet_service.cc:1511] Processing CreateTablet for tablet d2b2a88358244f93970835ea3120c380 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260808 01:48:58.173449  9628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b2a88358244f93970835ea3120c380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.164606  9467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 924b85a2ee7e46ecbcfd57257902abef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.146710  9695 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:48:58.175765  9546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b2a88358244f93970835ea3120c380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.153354  9682 ts_tablet_manager.cc:1403] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.232s	user 0.023s	sys 0.000s
I20260808 01:48:58.183475  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:48:58.184684  9544 tablet_service.cc:1511] Processing CreateTablet for tablet 2c617cb141a34007b6c1b8af0d911923 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260808 01:48:58.184885  9684 raft_consensus.cc:359] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } }
I20260808 01:48:58.185894  9625 tablet_service.cc:1511] Processing CreateTablet for tablet 2c617cb141a34007b6c1b8af0d911923 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260808 01:48:58.188161  9684 raft_consensus.cc:385] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:58.188524  9684 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:48:58.189383  9684 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } }
I20260808 01:48:58.190318  9468 tablet_service.cc:1511] Processing CreateTablet for tablet 4bcbe89d624b472f8be5abc6b34c9026 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260808 01:48:58.191244  9542 tablet_service.cc:1511] Processing CreateTablet for tablet 4bcbe89d624b472f8be5abc6b34c9026 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260808 01:48:58.192723  9542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bcbe89d624b472f8be5abc6b34c9026. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.193452  9544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c617cb141a34007b6c1b8af0d911923. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.193969  9684 ts_tablet_manager.cc:1434] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.027s	user 0.002s	sys 0.005s
I20260808 01:48:58.195096  9684 tablet_bootstrap.cc:492] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:48:58.194417  9470 tablet_service.cc:1511] Processing CreateTablet for tablet 50fd024764ea49c7a1f16e992e7599f3 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260808 01:48:58.199214  9627 tablet_service.cc:1511] Processing CreateTablet for tablet 4bcbe89d624b472f8be5abc6b34c9026 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260808 01:48:58.202250  9624 tablet_service.cc:1511] Processing CreateTablet for tablet 50fd024764ea49c7a1f16e992e7599f3 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260808 01:48:58.202575  9465 tablet_service.cc:1511] Processing CreateTablet for tablet 2c617cb141a34007b6c1b8af0d911923 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260808 01:48:58.203739  9471 tablet_service.cc:1511] Processing CreateTablet for tablet c85ff498df484fc6ac0913aa57aa868b (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260808 01:48:58.205130  9625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c617cb141a34007b6c1b8af0d911923. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.205853  9540 tablet_service.cc:1511] Processing CreateTablet for tablet 50fd024764ea49c7a1f16e992e7599f3 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260808 01:48:58.207695  9624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50fd024764ea49c7a1f16e992e7599f3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 01:48:58.211045  9524 leader_election.cc:343] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:58.211616  9524 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:48:58.213166  9698 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:48:58.215018  9540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50fd024764ea49c7a1f16e992e7599f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.153162  9688 raft_consensus.cc:359] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:58.219192  9688 raft_consensus.cc:385] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:58.219597  9688 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:48:58.220475  9688 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:58.227710  9627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bcbe89d624b472f8be5abc6b34c9026. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.233364  9539 tablet_service.cc:1511] Processing CreateTablet for tablet c85ff498df484fc6ac0913aa57aa868b (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260808 01:48:58.233522  9682 raft_consensus.cc:359] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } }
I20260808 01:48:58.234378  9682 raft_consensus.cc:385] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:58.235412  9470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50fd024764ea49c7a1f16e992e7599f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.236433  9471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c85ff498df484fc6ac0913aa57aa868b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.237169  9468 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bcbe89d624b472f8be5abc6b34c9026. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.235674  9539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c85ff498df484fc6ac0913aa57aa868b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.238119  9695 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:58.254616  9466 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b2a88358244f93970835ea3120c380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.259131  9465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c617cb141a34007b6c1b8af0d911923. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.270797  9625 tablet_service.cc:1511] Processing CreateTablet for tablet c85ff498df484fc6ac0913aa57aa868b (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260808 01:48:58.272382  9625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c85ff498df484fc6ac0913aa57aa868b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.273898  9549 tablet_service.cc:1511] Processing CreateTablet for tablet 11c99c174e234329801ee9b1ceae68d9 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260808 01:48:58.274366  9469 tablet_service.cc:1511] Processing CreateTablet for tablet 11c99c174e234329801ee9b1ceae68d9 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260808 01:48:58.275913  9469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11c99c174e234329801ee9b1ceae68d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.276851  9549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11c99c174e234329801ee9b1ceae68d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.290171  9621 tablet_service.cc:1511] Processing CreateTablet for tablet 11c99c174e234329801ee9b1ceae68d9 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260808 01:48:58.293045  9463 tablet_service.cc:1511] Processing CreateTablet for tablet 7b1e1cf1fbe745feb69372fed0557ce0 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260808 01:48:58.294809  9463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b1e1cf1fbe745feb69372fed0557ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.298489  9619 tablet_service.cc:1511] Processing CreateTablet for tablet 7b1e1cf1fbe745feb69372fed0557ce0 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260808 01:48:58.263063  9682 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:48:58.300446  9682 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } }
I20260808 01:48:58.302351  9547 tablet_service.cc:1511] Processing CreateTablet for tablet 7b1e1cf1fbe745feb69372fed0557ce0 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260808 01:48:58.303582  9664 heartbeater.cc:499] Master 127.7.165.254:43565 was elected leader, sending a full tablet report...
I20260808 01:48:58.306196  9462 tablet_service.cc:1511] Processing CreateTablet for tablet 030eacff61b14455978d2f0de262a089 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260808 01:48:58.309700  9682 ts_tablet_manager.cc:1434] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.122s	user 0.005s	sys 0.004s
I20260808 01:48:58.314221  9684 tablet_bootstrap.cc:654] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:58.314790  9462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030eacff61b14455978d2f0de262a089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.318959  9621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11c99c174e234329801ee9b1ceae68d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.320886  9619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b1e1cf1fbe745feb69372fed0557ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.323817  9543 tablet_service.cc:1511] Processing CreateTablet for tablet 030eacff61b14455978d2f0de262a089 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260808 01:48:58.325318  9543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030eacff61b14455978d2f0de262a089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.326133  9374 catalog_manager.cc:5697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:58.343842  9701 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:58.355310  9695 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:58.355932  9695 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } }
I20260808 01:48:58.358219  9695 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359)
I20260808 01:48:58.361207  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:48:58.362005  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:48:58.362759  9481 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:48:58.364058  9463 tablet_service.cc:1511] Processing CreateTablet for tablet f358d8403bf440059a6753dc4ed0be38 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
W20260808 01:48:58.364778  9521 leader_election.cc:343] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:58.365633  9463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f358d8403bf440059a6753dc4ed0be38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.366127  9524 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:48:58.369899  9682 tablet_bootstrap.cc:492] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:48:58.369679  9545 tablet_service.cc:1511] Processing CreateTablet for tablet f358d8403bf440059a6753dc4ed0be38 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260808 01:48:58.371260  9695 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:58.371732  9695 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:58.372124  9695 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:58.378006  9547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b1e1cf1fbe745feb69372fed0557ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.387153  9682 tablet_bootstrap.cc:654] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:58.398991  9545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f358d8403bf440059a6753dc4ed0be38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.404942  9688 ts_tablet_manager.cc:1434] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.268s	user 0.020s	sys 0.007s
I20260808 01:48:58.417608  9688 tablet_bootstrap.cc:492] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:48:58.421695  9470 tablet_service.cc:1511] Processing CreateTablet for tablet 2973a134fcef4b8da54331bcd12f1173 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260808 01:48:58.423197  9624 tablet_service.cc:1511] Processing CreateTablet for tablet 030eacff61b14455978d2f0de262a089 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260808 01:48:58.424352  9624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030eacff61b14455978d2f0de262a089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.346616  9701 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:58.429198  9684 tablet_bootstrap.cc:492] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:48:58.429797  9684 ts_tablet_manager.cc:1403] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.235s	user 0.040s	sys 0.041s
I20260808 01:48:58.439917  9684 raft_consensus.cc:359] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:58.442224  9470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2973a134fcef4b8da54331bcd12f1173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.444330  9461 tablet_service.cc:1511] Processing CreateTablet for tablet 244e080726394210b5467bf688813cf0 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260808 01:48:58.446848  9548 tablet_service.cc:1511] Processing CreateTablet for tablet 2973a134fcef4b8da54331bcd12f1173 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260808 01:48:58.460955  9548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2973a134fcef4b8da54331bcd12f1173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.448539  9471 tablet_service.cc:1511] Processing CreateTablet for tablet 17e220e6a25747939bdf18f26b732407 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260808 01:48:58.451346  9466 tablet_service.cc:1511] Processing CreateTablet for tablet 487114a18943467b8d3a6db94b17a84d (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260808 01:48:58.463034  9625 tablet_service.cc:1511] Processing CreateTablet for tablet f358d8403bf440059a6753dc4ed0be38 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260808 01:48:58.462545  9468 tablet_service.cc:1511] Processing CreateTablet for tablet ed168d59c43d4bfaa07d104032ad8977 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260808 01:48:58.464401  9625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f358d8403bf440059a6753dc4ed0be38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.430524  9701 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:48:58.453974  9684 raft_consensus.cc:385] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:58.473204  9684 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:48:58.475225  9628 tablet_service.cc:1511] Processing CreateTablet for tablet 244e080726394210b5467bf688813cf0 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260808 01:48:58.475589  9627 tablet_service.cc:1511] Processing CreateTablet for tablet 17e220e6a25747939bdf18f26b732407 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260808 01:48:58.476763  9628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 244e080726394210b5467bf688813cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.476818  9684 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:58.477584  9627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17e220e6a25747939bdf18f26b732407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.465206  9620 tablet_service.cc:1511] Processing CreateTablet for tablet 2973a134fcef4b8da54331bcd12f1173 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260808 01:48:58.467069  9461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 244e080726394210b5467bf688813cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.468856  9544 tablet_service.cc:1511] Processing CreateTablet for tablet 244e080726394210b5467bf688813cf0 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260808 01:48:58.448010  9695 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } }
I20260808 01:48:58.489457  9544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 244e080726394210b5467bf688813cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.490351  9622 tablet_service.cc:1511] Processing CreateTablet for tablet 487114a18943467b8d3a6db94b17a84d (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260808 01:48:58.503407  9620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2973a134fcef4b8da54331bcd12f1173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.505781  9471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17e220e6a25747939bdf18f26b732407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.517184  9682 tablet_bootstrap.cc:492] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:48:58.517782  9682 ts_tablet_manager.cc:1403] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.148s	user 0.029s	sys 0.007s
I20260808 01:48:58.518407  9545 tablet_service.cc:1511] Processing CreateTablet for tablet 487114a18943467b8d3a6db94b17a84d (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260808 01:48:58.519913  9545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 487114a18943467b8d3a6db94b17a84d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.521376  9682 raft_consensus.cc:359] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:58.523181  9682 raft_consensus.cc:385] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:58.523567  9682 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:48:58.524981  9682 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:58.528053  9682 ts_tablet_manager.cc:1434] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.001s
I20260808 01:48:58.529126  9698 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:58.535046  9698 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:58.536844  9688 tablet_bootstrap.cc:654] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:58.538658  9698 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:48:58.540699  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:48:58.541215  9539 tablet_service.cc:1511] Processing CreateTablet for tablet ed168d59c43d4bfaa07d104032ad8977 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
W20260808 01:48:58.543258  9524 leader_election.cc:343] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:58.532346  9684 ts_tablet_manager.cc:1434] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.102s	user 0.007s	sys 0.003s
I20260808 01:48:58.470237  9546 tablet_service.cc:1511] Processing CreateTablet for tablet 17e220e6a25747939bdf18f26b732407 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260808 01:48:58.547291  9682 tablet_bootstrap.cc:492] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:48:58.547610  9546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17e220e6a25747939bdf18f26b732407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.553658  9549 tablet_service.cc:1511] Processing CreateTablet for tablet 90465c2f160641a7901ab0815e723ce1 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260808 01:48:58.554837  9626 tablet_service.cc:1511] Processing CreateTablet for tablet ed168d59c43d4bfaa07d104032ad8977 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260808 01:48:58.556597  9626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed168d59c43d4bfaa07d104032ad8977. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.561756  9618 tablet_service.cc:1511] Processing CreateTablet for tablet 90465c2f160641a7901ab0815e723ce1 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260808 01:48:58.563731  9618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90465c2f160641a7901ab0815e723ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.578914  9622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 487114a18943467b8d3a6db94b17a84d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.617672  9468 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed168d59c43d4bfaa07d104032ad8977. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.617821  9689 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:58.618661  9689 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:58.619329  9470 tablet_service.cc:1511] Processing CreateTablet for tablet 90465c2f160641a7901ab0815e723ce1 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260808 01:48:58.620766  9470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90465c2f160641a7901ab0815e723ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.622195  9469 tablet_service.cc:1511] Processing CreateTablet for tablet affdd24b03ae4ee795a90703613305f7 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260808 01:48:58.622853  9684 tablet_bootstrap.cc:492] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:48:58.640051  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:48:58.642443  9521 leader_election.cc:343] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:58.662782  9540 tablet_service.cc:1511] Processing CreateTablet for tablet affdd24b03ae4ee795a90703613305f7 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260808 01:48:58.705374  9637 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:58.706663  9549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90465c2f160641a7901ab0815e723ce1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.715204  9469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet affdd24b03ae4ee795a90703613305f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.717084  9545 tablet_service.cc:1511] Processing CreateTablet for tablet 407cb6b113ce432587849bc840ba6872 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260808 01:48:58.545387  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:48:58.718647  9466 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 487114a18943467b8d3a6db94b17a84d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 01:48:58.719758  9521 leader_election.cc:343] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:58.720407  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:48:58.721905  9698 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:48:58.718614  9545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 407cb6b113ce432587849bc840ba6872. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.727932  9695 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359)
W20260808 01:48:58.730882  9360 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 178016us
I20260808 01:48:58.731802  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:48:58.732435  9481 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:58.737736  9621 tablet_service.cc:1511] Processing CreateTablet for tablet affdd24b03ae4ee795a90703613305f7 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260808 01:48:58.747817  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:48:58.748791  9480 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 1: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:48:58.758042  9684 tablet_bootstrap.cc:654] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:58.762547  9695 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:48:58.766870  9539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed168d59c43d4bfaa07d104032ad8977. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.777379  9471 tablet_service.cc:1511] Processing CreateTablet for tablet 407cb6b113ce432587849bc840ba6872 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260808 01:48:58.779168  9465 tablet_service.cc:1511] Processing CreateTablet for tablet df6982ec010444a8854140f7b64e985d (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260808 01:48:58.780686  9465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df6982ec010444a8854140f7b64e985d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.783169  9627 tablet_service.cc:1511] Processing CreateTablet for tablet 407cb6b113ce432587849bc840ba6872 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260808 01:48:58.784974  9621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet affdd24b03ae4ee795a90703613305f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.784552  9625 tablet_service.cc:1511] Processing CreateTablet for tablet df6982ec010444a8854140f7b64e985d (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260808 01:48:58.787175  9689 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:48:58.790050  9481 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:48:58.782398  9462 tablet_service.cc:1511] Processing CreateTablet for tablet 4bce6ea2a8244811bf007638181a417a (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
W20260808 01:48:58.804504  9362 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 145854us
I20260808 01:48:58.812660  9627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 407cb6b113ce432587849bc840ba6872. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.817133  9540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet affdd24b03ae4ee795a90703613305f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.837461  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:48:58.840514  9682 tablet_bootstrap.cc:654] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:58.840660  9463 tablet_service.cc:1511] Processing CreateTablet for tablet d0db7b84c3b040c788e0a3f66657d912 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260808 01:48:58.844803  9599 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:48:58.848354  9524 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:48:58.851491  9695 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:58.852070  9695 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:48:58.852921  9695 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } }
I20260808 01:48:58.861717  9471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 407cb6b113ce432587849bc840ba6872. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.871184  9699 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:48:58.864835  9698 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260808 01:48:58.868432  9625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df6982ec010444a8854140f7b64e985d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.884541  9624 tablet_service.cc:1511] Processing CreateTablet for tablet 4bce6ea2a8244811bf007638181a417a (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260808 01:48:58.885766  9462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bce6ea2a8244811bf007638181a417a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.892279  9624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bce6ea2a8244811bf007638181a417a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.855115  9467 tablet_service.cc:1511] Processing CreateTablet for tablet 82043e5428784a1eb53a98811ecbbbb4 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260808 01:48:58.894013  9467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82043e5428784a1eb53a98811ecbbbb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.875306  9684 tablet_bootstrap.cc:492] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:48:58.908914  9684 ts_tablet_manager.cc:1403] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.286s	user 0.012s	sys 0.016s
I20260808 01:48:58.912111  9684 raft_consensus.cc:359] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:58.912868  9684 raft_consensus.cc:385] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:58.913177  9684 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
W20260808 01:48:58.902542  9426 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:48:58.925414  9688 tablet_bootstrap.cc:492] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:48:58.913957  9628 tablet_service.cc:1511] Processing CreateTablet for tablet d0db7b84c3b040c788e0a3f66657d912 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260808 01:48:58.926013  9688 ts_tablet_manager.cc:1403] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.509s	user 0.015s	sys 0.007s
I20260808 01:48:58.928565  9684 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:58.942579  9684 ts_tablet_manager.cc:1434] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.033s	user 0.006s	sys 0.001s
I20260808 01:48:58.944768  9684 tablet_bootstrap.cc:492] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:48:58.947450  9688 raft_consensus.cc:359] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } }
I20260808 01:48:58.948340  9688 raft_consensus.cc:385] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:58.948663  9688 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:48:58.958209  9545 tablet_service.cc:1511] Processing CreateTablet for tablet df6982ec010444a8854140f7b64e985d (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260808 01:48:58.958514  9549 tablet_service.cc:1511] Processing CreateTablet for tablet 4bce6ea2a8244811bf007638181a417a (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260808 01:48:58.953591  9463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0db7b84c3b040c788e0a3f66657d912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.971478  9547 tablet_service.cc:1511] Processing CreateTablet for tablet d0db7b84c3b040c788e0a3f66657d912 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260808 01:48:58.982648  9622 tablet_service.cc:1511] Processing CreateTablet for tablet c2069a355aad47b48173a35b4efc6f9d (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260808 01:48:58.983206  9545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df6982ec010444a8854140f7b64e985d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.988158  9549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bce6ea2a8244811bf007638181a417a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.992724  9543 tablet_service.cc:1511] Processing CreateTablet for tablet 82043e5428784a1eb53a98811ecbbbb4 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260808 01:48:58.993083  9548 tablet_service.cc:1511] Processing CreateTablet for tablet c094b8e92d5e449aa2641c359d46d622 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260808 01:48:58.995915  9542 tablet_service.cc:1511] Processing CreateTablet for tablet c2069a355aad47b48173a35b4efc6f9d (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260808 01:48:58.961395  9688 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } }
I20260808 01:48:59.002570  9688 ts_tablet_manager.cc:1434] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.071s	user 0.008s	sys 0.001s
I20260808 01:48:59.007392  9688 tablet_bootstrap.cc:492] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:48:59.011194  9628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0db7b84c3b040c788e0a3f66657d912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:58.963394  9619 tablet_service.cc:1511] Processing CreateTablet for tablet 82043e5428784a1eb53a98811ecbbbb4 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260808 01:48:59.053306  9622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2069a355aad47b48173a35b4efc6f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.054054  9468 tablet_service.cc:1511] Processing CreateTablet for tablet c094b8e92d5e449aa2641c359d46d622 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260808 01:48:59.051692  9620 tablet_service.cc:1511] Processing CreateTablet for tablet fa45ab4daad84ee78514f98cb3738f23 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260808 01:48:59.042410  9682 tablet_bootstrap.cc:492] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:48:59.056604  9543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82043e5428784a1eb53a98811ecbbbb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.072747  9619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82043e5428784a1eb53a98811ecbbbb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.073009  9618 tablet_service.cc:1511] Processing CreateTablet for tablet c094b8e92d5e449aa2641c359d46d622 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260808 01:48:59.075438  9547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0db7b84c3b040c788e0a3f66657d912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.076783  9620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa45ab4daad84ee78514f98cb3738f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.079710  9468 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c094b8e92d5e449aa2641c359d46d622. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.084082  9682 ts_tablet_manager.cc:1403] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.538s	user 0.044s	sys 0.017s
I20260808 01:48:59.084519  9548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c094b8e92d5e449aa2641c359d46d622. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.085140  9699 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:59.086272  9542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2069a355aad47b48173a35b4efc6f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.086000  9699 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } }
I20260808 01:48:59.092121  9618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c094b8e92d5e449aa2641c359d46d622. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.099720  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:48:59.099737  9469 tablet_service.cc:1511] Processing CreateTablet for tablet c2069a355aad47b48173a35b4efc6f9d (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260808 01:48:59.107290  9469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2069a355aad47b48173a35b4efc6f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.116460  9682 raft_consensus.cc:359] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:59.117403  9682 raft_consensus.cc:385] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:59.117767  9682 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:48:59.123317  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:48:59.127545  9444 leader_election.cc:343] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:59.129243  9544 tablet_service.cc:1511] Processing CreateTablet for tablet fa45ab4daad84ee78514f98cb3738f23 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260808 01:48:59.118630  9682 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:59.131062  9544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa45ab4daad84ee78514f98cb3738f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.132184  9682 ts_tablet_manager.cc:1434] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.048s	user 0.005s	sys 0.002s
I20260808 01:48:59.135473  9689 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:59.136132  9682 tablet_bootstrap.cc:492] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:48:59.136065  9689 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:59.140632  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:48:59.138352  9689 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:48:59.143152  9444 leader_election.cc:343] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:59.151640  9684 tablet_bootstrap.cc:654] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:59.166137  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:48:59.183630  9545 tablet_service.cc:1511] Processing CreateTablet for tablet 20a4f92418934341a6ac4927db61dc87 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260808 01:48:59.185787  9545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20a4f92418934341a6ac4927db61dc87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.188480  9470 tablet_service.cc:1511] Processing CreateTablet for tablet 3354328ed023401e83a2361925b02e81 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260808 01:48:59.207037  9462 tablet_service.cc:1511] Processing CreateTablet for tablet 20a4f92418934341a6ac4927db61dc87 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
W20260808 01:48:59.164788  9444 leader_election.cc:343] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:59.213658  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:48:59.214735  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 53930.9us
I20260808 01:48:59.215461  9689 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:48:59.216986  9699 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359)
I20260808 01:48:59.218007  9539 tablet_service.cc:1511] Processing CreateTablet for tablet 3354328ed023401e83a2361925b02e81 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260808 01:48:59.218387  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:48:59.219338  9480 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 1: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:48:59.219482  9539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3354328ed023401e83a2361925b02e81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.221403  9601 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:48:59.223878  9624 tablet_service.cc:1511] Processing CreateTablet for tablet 3354328ed023401e83a2361925b02e81 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260808 01:48:59.224237  9695 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:59.223568  9461 tablet_service.cc:1511] Processing CreateTablet for tablet 103b913d6c014692b6dea1780af2aa20 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260808 01:48:59.224934  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:48:59.190548  9470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3354328ed023401e83a2361925b02e81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.232892  9560 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:48:59.233404  9695 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:59.233407  9540 tablet_service.cc:1511] Processing CreateTablet for tablet 103b913d6c014692b6dea1780af2aa20 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260808 01:48:59.235968  9682 tablet_bootstrap.cc:654] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:59.245893  9695 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:48:59.247792  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:48:59.247539  9625 tablet_service.cc:1511] Processing CreateTablet for tablet 20a4f92418934341a6ac4927db61dc87 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260808 01:48:59.246497  9540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 103b913d6c014692b6dea1780af2aa20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.231145  9699 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:48:59.260269  9524 leader_election.cc:343] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:59.261987  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:48:59.262001  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:48:59.263160  9689 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:59.263573  9689 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:48:59.274921  9521 leader_election.cc:343] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:59.278327  9521 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:48:59.264026  9689 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:59.270725  9471 tablet_service.cc:1511] Processing CreateTablet for tablet fa45ab4daad84ee78514f98cb3738f23 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260808 01:48:59.287067  9722 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:48:59.273663  9462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20a4f92418934341a6ac4927db61dc87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.289446  9461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 103b913d6c014692b6dea1780af2aa20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.281509  9688 tablet_bootstrap.cc:654] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:59.290388  9471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa45ab4daad84ee78514f98cb3738f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.278877  9466 tablet_service.cc:1511] Processing CreateTablet for tablet 2f740cd3515641a8b2c4add8f809dd82 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260808 01:48:59.277439  9625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20a4f92418934341a6ac4927db61dc87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.299644  9466 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f740cd3515641a8b2c4add8f809dd82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.318501  9689 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:59.314121  9627 tablet_service.cc:1511] Processing CreateTablet for tablet 103b913d6c014692b6dea1780af2aa20 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260808 01:48:59.323009  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:48:59.323655  9560 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:59.326433  9546 tablet_service.cc:1511] Processing CreateTablet for tablet 5cf7e7eadc2347aa84c98413849ebfc9 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260808 01:48:59.326563  9469 tablet_service.cc:1511] Processing CreateTablet for tablet e3814b146ccd472f9304a0be09ad3738 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260808 01:48:59.328037  9546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf7e7eadc2347aa84c98413849ebfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.328321  9469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3814b146ccd472f9304a0be09ad3738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.335428  9624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3354328ed023401e83a2361925b02e81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.347095  9621 tablet_service.cc:1511] Processing CreateTablet for tablet 5cf7e7eadc2347aa84c98413849ebfc9 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260808 01:48:59.355640  9465 tablet_service.cc:1511] Processing CreateTablet for tablet 5cf7e7eadc2347aa84c98413849ebfc9 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260808 01:48:59.361483  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:48:59.362942  9465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf7e7eadc2347aa84c98413849ebfc9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 01:48:59.363073  9444 leader_election.cc:343] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:59.369279  9621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf7e7eadc2347aa84c98413849ebfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.368402  9626 tablet_service.cc:1511] Processing CreateTablet for tablet 2f740cd3515641a8b2c4add8f809dd82 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260808 01:48:59.373561  9627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 103b913d6c014692b6dea1780af2aa20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.374432  9626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f740cd3515641a8b2c4add8f809dd82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.356806  9689 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:48:59.401755  9468 tablet_service.cc:1511] Processing CreateTablet for tablet 3759209338e14437aff5f92a2fff36da (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260808 01:48:59.401034  9374 catalog_manager.cc:5697] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:59.403306  9468 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3759209338e14437aff5f92a2fff36da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.403853  9622 tablet_service.cc:1511] Processing CreateTablet for tablet e3814b146ccd472f9304a0be09ad3738 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260808 01:48:59.405238  9622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3814b146ccd472f9304a0be09ad3738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.409101  9684 tablet_bootstrap.cc:492] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:48:59.412133  9684 ts_tablet_manager.cc:1403] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.468s	user 0.019s	sys 0.018s
I20260808 01:48:59.399259  9543 tablet_service.cc:1511] Processing CreateTablet for tablet e3814b146ccd472f9304a0be09ad3738 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260808 01:48:59.444415  9688 tablet_bootstrap.cc:492] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:48:59.457208  9543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3814b146ccd472f9304a0be09ad3738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.445739  9699 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:48:59.433857  9560 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:48:59.445479  9628 tablet_service.cc:1511] Processing CreateTablet for tablet 360b0cf59e7749058ca972bb487cbd10 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260808 01:48:59.462472  9688 ts_tablet_manager.cc:1403] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.456s	user 0.017s	sys 0.010s
I20260808 01:48:59.469142  9628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 360b0cf59e7749058ca972bb487cbd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.440554  9463 tablet_service.cc:1511] Processing CreateTablet for tablet ea7d6b03a61944ea8a8201984886f30a (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260808 01:48:59.440554  9542 tablet_service.cc:1511] Processing CreateTablet for tablet 360b0cf59e7749058ca972bb487cbd10 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260808 01:48:59.474766  9542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 360b0cf59e7749058ca972bb487cbd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.479471  9463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea7d6b03a61944ea8a8201984886f30a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.471076  9684 raft_consensus.cc:359] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:59.480661  9684 raft_consensus.cc:385] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:59.481290  9684 raft_consensus.cc:740] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:48:59.484211  9684 consensus_queue.cc:260] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:59.425848  9549 tablet_service.cc:1511] Processing CreateTablet for tablet 2f740cd3515641a8b2c4add8f809dd82 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260808 01:48:59.494967  9688 raft_consensus.cc:359] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:59.495833  9688 raft_consensus.cc:385] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:59.496146  9688 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:48:59.496836  9722 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:48:59.496977  9688 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:59.500305  9682 tablet_bootstrap.cc:492] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:48:59.500674  9688 ts_tablet_manager.cc:1434] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.032s	user 0.003s	sys 0.004s
I20260808 01:48:59.500979  9682 ts_tablet_manager.cc:1403] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.368s	user 0.013s	sys 0.015s
I20260808 01:48:59.501729  9688 tablet_bootstrap.cc:492] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:48:59.455061  9467 tablet_service.cc:1511] Processing CreateTablet for tablet 360b0cf59e7749058ca972bb487cbd10 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260808 01:48:59.507218  9467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 360b0cf59e7749058ca972bb487cbd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.507395  9549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f740cd3515641a8b2c4add8f809dd82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.510114  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:48:59.511538  9689 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:48:59.506080  9682 raft_consensus.cc:359] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:59.515053  9682 raft_consensus.cc:385] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:59.514389  9684 ts_tablet_manager.cc:1434] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.100s	user 0.009s	sys 0.000s
I20260808 01:48:59.516790  9682 raft_consensus.cc:740] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:48:59.522718  9682 consensus_queue.cc:260] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:59.533825  9684 tablet_bootstrap.cc:492] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:48:59.545894  9722 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:48:59.549256  9682 ts_tablet_manager.cc:1434] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.048s	user 0.006s	sys 0.001s
I20260808 01:48:59.586587  9682 tablet_bootstrap.cc:492] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:48:59.587796  9471 tablet_service.cc:1511] Processing CreateTablet for tablet 0b4331dfdd5b43448a400f2eb7b1be70 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260808 01:48:59.589289  9471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b4331dfdd5b43448a400f2eb7b1be70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.551753  9684 tablet_bootstrap.cc:654] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:59.557969  9466 tablet_service.cc:1511] Processing CreateTablet for tablet 9bf65b9272854b479e73f2cea21e93f2 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
W20260808 01:48:59.558979  9360 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 46483.4us
I20260808 01:48:59.567703  9688 tablet_bootstrap.cc:654] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:48:59.572378  9470 tablet_service.cc:1511] Processing CreateTablet for tablet 7d3a904863b545ffa227b964ee3a3f18 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260808 01:48:59.596256  9620 tablet_service.cc:1511] Processing CreateTablet for tablet 3759209338e14437aff5f92a2fff36da (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260808 01:48:59.596508  9547 tablet_service.cc:1511] Processing CreateTablet for tablet 3759209338e14437aff5f92a2fff36da (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260808 01:48:59.597762  9620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3759209338e14437aff5f92a2fff36da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.597891  9547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3759209338e14437aff5f92a2fff36da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.600376  9621 tablet_service.cc:1511] Processing CreateTablet for tablet 9bf65b9272854b479e73f2cea21e93f2 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260808 01:48:59.601915  9621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bf65b9272854b479e73f2cea21e93f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.601817  9548 tablet_service.cc:1511] Processing CreateTablet for tablet ea7d6b03a61944ea8a8201984886f30a (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260808 01:48:59.603760  9619 tablet_service.cc:1511] Processing CreateTablet for tablet 7d3a904863b545ffa227b964ee3a3f18 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260808 01:48:59.603860  9545 tablet_service.cc:1511] Processing CreateTablet for tablet 9bf65b9272854b479e73f2cea21e93f2 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260808 01:48:59.604552  9539 tablet_service.cc:1511] Processing CreateTablet for tablet 7d3a904863b545ffa227b964ee3a3f18 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260808 01:48:59.605201  9619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d3a904863b545ffa227b964ee3a3f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.607609  9618 tablet_service.cc:1511] Processing CreateTablet for tablet cab93716156d46d3b0bda8d98d1fc76c (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260808 01:48:59.609001  9618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cab93716156d46d3b0bda8d98d1fc76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.608757  9544 tablet_service.cc:1511] Processing CreateTablet for tablet cab93716156d46d3b0bda8d98d1fc76c (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260808 01:48:59.610913  9544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cab93716156d46d3b0bda8d98d1fc76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.615463  9548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea7d6b03a61944ea8a8201984886f30a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.617116  9539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d3a904863b545ffa227b964ee3a3f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.619277  9545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bf65b9272854b479e73f2cea21e93f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.620054  9624 tablet_service.cc:1511] Processing CreateTablet for tablet 0b4331dfdd5b43448a400f2eb7b1be70 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260808 01:48:59.627512  9628 tablet_service.cc:1511] Processing CreateTablet for tablet ea7d6b03a61944ea8a8201984886f30a (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260808 01:48:59.628921  9628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea7d6b03a61944ea8a8201984886f30a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.584758  9468 tablet_service.cc:1511] Processing CreateTablet for tablet cab93716156d46d3b0bda8d98d1fc76c (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260808 01:48:59.636246  9466 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bf65b9272854b479e73f2cea21e93f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.638854  9540 tablet_service.cc:1511] Processing CreateTablet for tablet 0b4331dfdd5b43448a400f2eb7b1be70 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260808 01:48:59.640259  9540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b4331dfdd5b43448a400f2eb7b1be70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.661092  9735 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:59.662369  9735 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:59.680243  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:48:59.694892  9689 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:48:59.702193  9689 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:59.704685  9560 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:48:59.708774  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:48:59.712435  9467 tablet_service.cc:1511] Processing CreateTablet for tablet 03b17f21031f4ff4828c85b7332fcc2d (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260808 01:48:59.716960  9625 tablet_service.cc:1511] Processing CreateTablet for tablet 03b17f21031f4ff4828c85b7332fcc2d (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260808 01:48:59.711467  9462 tablet_service.cc:1511] Processing CreateTablet for tablet 32724e13cbcb49e991b11d18699e899a (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260808 01:48:59.726032  9628 tablet_service.cc:1511] Processing CreateTablet for tablet 3b8e0e8fda6744ad877514b11658b9d9 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260808 01:48:59.730191  9545 tablet_service.cc:1511] Processing CreateTablet for tablet 3b8e0e8fda6744ad877514b11658b9d9 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260808 01:48:59.731750  9545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8e0e8fda6744ad877514b11658b9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.742985  9624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b4331dfdd5b43448a400f2eb7b1be70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.763367  9735 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:48:59.764498  9735 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:59.765208  9735 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:59.765848  9735 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:59.769279  9625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03b17f21031f4ff4828c85b7332fcc2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.770922  9628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8e0e8fda6744ad877514b11658b9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.696661  9470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d3a904863b545ffa227b964ee3a3f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.786542  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:48:59.811127  9444 leader_election.cc:343] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:59.813887  9545 tablet_service.cc:1511] Processing CreateTablet for tablet 03b17f21031f4ff4828c85b7332fcc2d (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260808 01:48:59.815655  9545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03b17f21031f4ff4828c85b7332fcc2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.818250  9542 tablet_service.cc:1511] Processing CreateTablet for tablet 32724e13cbcb49e991b11d18699e899a (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260808 01:48:59.820793  9542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32724e13cbcb49e991b11d18699e899a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.826540  9621 tablet_service.cc:1511] Processing CreateTablet for tablet 32724e13cbcb49e991b11d18699e899a (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260808 01:48:59.829622  9621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32724e13cbcb49e991b11d18699e899a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.831547  9546 tablet_service.cc:1511] Processing CreateTablet for tablet 460e1059f5504707b891c4b0e0ac6a09 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260808 01:48:59.820825  9465 tablet_service.cc:1511] Processing CreateTablet for tablet 460e1059f5504707b891c4b0e0ac6a09 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260808 01:48:59.841650  9469 tablet_service.cc:1511] Processing CreateTablet for tablet 3b8e0e8fda6744ad877514b11658b9d9 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260808 01:48:59.843291  9468 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cab93716156d46d3b0bda8d98d1fc76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.853708  9546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 460e1059f5504707b891c4b0e0ac6a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.854600  9688 tablet_bootstrap.cc:492] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:48:59.855520  9688 ts_tablet_manager.cc:1403] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.354s	user 0.022s	sys 0.004s
I20260808 01:48:59.856568  9462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32724e13cbcb49e991b11d18699e899a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.859794  9465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 460e1059f5504707b891c4b0e0ac6a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.863413  9467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03b17f21031f4ff4828c85b7332fcc2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.874267  9699 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:48:59.874791  9684 tablet_bootstrap.cc:492] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:48:59.875739  9684 ts_tablet_manager.cc:1403] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.358s	user 0.037s	sys 0.030s
I20260808 01:48:59.875412  9699 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:59.887061  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:48:59.890832  9601 leader_election.cc:343] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:48:59.889324  9374 catalog_manager.cc:5697] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:48:59.906185  9699 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:48:59.907783  9469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8e0e8fda6744ad877514b11658b9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:48:59.913528  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:48:59.914947  9560 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:48:59.935343  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:48:59.950891  9699 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:48:59.951550  9699 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:48:59.952224  9699 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:48:59.949721  9628 tablet_service.cc:1511] Processing CreateTablet for tablet 460e1059f5504707b891c4b0e0ac6a09 (DEFAULT_TABLE table=test-workload [id=5ed6eaf552d849608150254f71c99d76]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260808 01:48:59.956460  9628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 460e1059f5504707b891c4b0e0ac6a09. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 01:48:59.959661  9426 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:48:59.963966  9688 raft_consensus.cc:359] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:59.972685  9688 raft_consensus.cc:385] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:48:59.973598  9688 raft_consensus.cc:740] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:48:59.981848  9688 consensus_queue.cc:260] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:48:59.925653  9684 raft_consensus.cc:359] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.002610  9682 tablet_bootstrap.cc:654] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.006978  9684 raft_consensus.cc:385] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:00.057675  9684 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:00.067274  9684 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.078239  9684 ts_tablet_manager.cc:1434] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.202s	user 0.011s	sys 0.005s
I20260808 01:49:00.080063  9684 tablet_bootstrap.cc:492] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:00.039762  9735 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.092825  9684 tablet_bootstrap.cc:654] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.095391  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:00.096701  9560 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:00.105261  9688 ts_tablet_manager.cc:1434] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.249s	user 0.017s	sys 0.026s
I20260808 01:49:00.113611  9689 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:00.135174  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:00.136231  9637 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:00.129532  9735 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:00.172066  9688 tablet_bootstrap.cc:492] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:00.185751  9699 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.205590  9684 tablet_bootstrap.cc:492] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:00.216652  9560 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:00.206354  9684 ts_tablet_manager.cc:1403] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.127s	user 0.033s	sys 0.074s
I20260808 01:49:00.226509  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:00.228168  9558 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:00.227689  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:00.221412  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:00.233527  9735 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:00.239383  9480 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:00.240274  9735 raft_consensus.cc:697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:00.242311  9735 consensus_queue.cc:237] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.254432  9682 tablet_bootstrap.cc:492] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
W20260808 01:49:00.259420  9444 consensus_peers.cc:597] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:00.231040  9699 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:00.268085  9682 ts_tablet_manager.cc:1403] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.682s	user 0.014s	sys 0.034s
I20260808 01:49:00.302196  9688 tablet_bootstrap.cc:654] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.301436  9684 raft_consensus.cc:359] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.304276  9684 raft_consensus.cc:385] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:00.304893  9684 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:00.308292  9637 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:00.307824  9682 raft_consensus.cc:359] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.311115  9682 raft_consensus.cc:385] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:00.311915  9682 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:00.313433  9682 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.307387  9684 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.370487  9480 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:00.383785  9558 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:00.387328  9682 ts_tablet_manager.cc:1434] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.112s	user 0.005s	sys 0.016s
I20260808 01:49:00.388177  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:00.390163  9684 ts_tablet_manager.cc:1434] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.160s	user 0.013s	sys 0.004s
I20260808 01:49:00.393461  9684 tablet_bootstrap.cc:492] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:00.390198  9682 tablet_bootstrap.cc:492] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:00.394901  9373 catalog_manager.cc:5697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:00.403467  9699 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:00.405223  9699 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:00.407742  9688 tablet_bootstrap.cc:492] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:00.407157  9699 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.408725  9688 ts_tablet_manager.cc:1403] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.237s	user 0.022s	sys 0.011s
I20260808 01:49:00.410560  9682 tablet_bootstrap.cc:654] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.411135  9684 tablet_bootstrap.cc:654] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.421303  9688 raft_consensus.cc:359] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.422379  9688 raft_consensus.cc:385] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:00.422864  9688 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:00.424031  9688 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.427376  9688 ts_tablet_manager.cc:1434] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.018s	user 0.000s	sys 0.010s
I20260808 01:49:00.429267  9688 tablet_bootstrap.cc:492] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:00.443513  9684 tablet_bootstrap.cc:492] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:00.444087  9684 ts_tablet_manager.cc:1403] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.051s	user 0.005s	sys 0.019s
I20260808 01:49:00.444314  9682 tablet_bootstrap.cc:492] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:00.444939  9688 tablet_bootstrap.cc:654] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.444983  9682 ts_tablet_manager.cc:1403] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.055s	user 0.018s	sys 0.007s
I20260808 01:49:00.448415  9684 raft_consensus.cc:359] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.449215  9684 raft_consensus.cc:385] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:00.449541  9684 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:00.449666  9682 raft_consensus.cc:359] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.450551  9684 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.451036  9682 raft_consensus.cc:385] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:00.451718  9682 raft_consensus.cc:740] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:00.452983  9682 consensus_queue.cc:260] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.455185  9372 catalog_manager.cc:5697] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:00.457249  9682 ts_tablet_manager.cc:1434] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.000s
I20260808 01:49:00.459553  9682 tablet_bootstrap.cc:492] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:00.464747  9684 ts_tablet_manager.cc:1434] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.020s	user 0.004s	sys 0.006s
I20260808 01:49:00.465968  9688 tablet_bootstrap.cc:492] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:00.466460  9684 tablet_bootstrap.cc:492] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:00.466619  9688 ts_tablet_manager.cc:1403] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.038s	user 0.007s	sys 0.021s
I20260808 01:49:00.467163  9682 tablet_bootstrap.cc:654] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.475368  9684 tablet_bootstrap.cc:654] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.484107  9688 raft_consensus.cc:359] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.485076  9688 raft_consensus.cc:385] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:00.485232  9684 tablet_bootstrap.cc:492] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:00.485461  9688 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:00.485972  9684 ts_tablet_manager.cc:1403] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.020s	user 0.012s	sys 0.002s
I20260808 01:49:00.486406  9688 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.489593  9688 ts_tablet_manager.cc:1434] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.000s
I20260808 01:49:00.491221  9688 tablet_bootstrap.cc:492] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:00.491569  9682 tablet_bootstrap.cc:492] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:00.492182  9682 ts_tablet_manager.cc:1403] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.033s	user 0.014s	sys 0.012s
I20260808 01:49:00.495534  9682 raft_consensus.cc:359] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.496501  9682 raft_consensus.cc:385] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:00.496842  9682 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:00.496889  9684 raft_consensus.cc:359] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.497897  9684 raft_consensus.cc:385] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:00.498212  9684 raft_consensus.cc:740] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:00.498375  9682 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.499055  9688 tablet_bootstrap.cc:654] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.499017  9684 consensus_queue.cc:260] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.501433  9682 ts_tablet_manager.cc:1434] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260808 01:49:00.501844  9684 ts_tablet_manager.cc:1434] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.015s	user 0.010s	sys 0.000s
I20260808 01:49:00.503013  9682 tablet_bootstrap.cc:492] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:00.503015  9684 tablet_bootstrap.cc:492] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:00.506753  9688 tablet_bootstrap.cc:492] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:00.507217  9735 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:00.507447  9688 ts_tablet_manager.cc:1403] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.017s	user 0.011s	sys 0.004s
I20260808 01:49:00.507783  9735 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.510769  9682 tablet_bootstrap.cc:654] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.511855  9684 tablet_bootstrap.cc:654] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.511304  9688 raft_consensus.cc:359] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.511950  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:00.512514  9688 raft_consensus.cc:385] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:00.512858  9688 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:00.513849  9688 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.512849  9558 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:00.515128  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:00.517397  9688 ts_tablet_manager.cc:1434] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260808 01:49:00.519245  9688 tablet_bootstrap.cc:492] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:00.519460  9735 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:00.519220  9444 leader_election.cc:343] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:00.522850  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:00.524408  9735 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:00.524837  9735 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:00.525210  9735 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:00.529871  9682 tablet_bootstrap.cc:492] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:00.529937  9684 tablet_bootstrap.cc:492] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:00.530519  9682 ts_tablet_manager.cc:1403] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.028s	user 0.012s	sys 0.008s
I20260808 01:49:00.530714  9684 ts_tablet_manager.cc:1403] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.028s	user 0.014s	sys 0.005s
I20260808 01:49:00.531872  9688 tablet_bootstrap.cc:654] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.533674  9682 raft_consensus.cc:359] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.534425  9682 raft_consensus.cc:385] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:00.534720  9682 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:00.535591  9682 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.537901  9682 ts_tablet_manager.cc:1434] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260808 01:49:00.535889  9684 raft_consensus.cc:359] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.539203  9682 tablet_bootstrap.cc:492] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:00.539192  9684 raft_consensus.cc:385] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:00.539851  9684 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:00.540742  9684 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.544847  9688 tablet_bootstrap.cc:492] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:00.545447  9688 ts_tablet_manager.cc:1403] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.027s	user 0.010s	sys 0.008s
I20260808 01:49:00.548055  9722 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:00.548540  9688 raft_consensus.cc:359] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.548640  9722 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.549487  9688 raft_consensus.cc:385] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:00.549822  9688 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:00.551020  9682 tablet_bootstrap.cc:654] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.550650  9688 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.553298  9688 ts_tablet_manager.cc:1434] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260808 01:49:00.554620  9688 tablet_bootstrap.cc:492] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:00.554177  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:00.555449  9637 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:00.557693  9521 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:00.560318  9684 ts_tablet_manager.cc:1434] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.000s
I20260808 01:49:00.560115  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:00.561308  9480 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:00.562386  9722 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:00.563033  9722 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:00.563395  9722 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:00.563802  9722 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:00.571954  9688 tablet_bootstrap.cc:654] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.578327  9722 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.582257  9722 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:00.582935  9722 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:00.583166  9684 tablet_bootstrap.cc:492] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:00.583351  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:00.583427  9722 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.584112  9480 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:00.587069  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:00.587785  9637 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:00.590757  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:00.592152  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:00.592928  9481 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
W20260808 01:49:00.593760  9521 leader_election.cc:343] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:00.594131  9722 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:00.597797  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:00.582492  9735 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.600037  9722 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:00.601771  9722 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:00.601933  9682 tablet_bootstrap.cc:492] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:00.602161  9722 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:00.602509  9699 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:00.602555  9682 ts_tablet_manager.cc:1403] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.064s	user 0.025s	sys 0.013s
I20260808 01:49:00.603149  9699 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.608239  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:00.611213  9684 tablet_bootstrap.cc:654] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.611387  9722 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.611997  9688 tablet_bootstrap.cc:492] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:00.608768  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:00.612694  9688 ts_tablet_manager.cc:1403] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.058s	user 0.007s	sys 0.019s
I20260808 01:49:00.611478  9559 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:00.606482  9682 raft_consensus.cc:359] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.610409  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:00.626894  9480 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:00.608274  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:00.608901  9699 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:00.610553  9735 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:00.627415  9722 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:00.635190  9558 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:00.628620  9688 raft_consensus.cc:359] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.632201  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:00.637298  9682 raft_consensus.cc:385] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:00.630656  9637 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:00.637219  9599 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:00.637423  9688 raft_consensus.cc:385] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:00.613682  9481 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:00.637792  9682 raft_consensus.cc:740] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:00.638604  9699 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260808 01:49:00.639078  9521 leader_election.cc:343] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:00.639889  9699 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:00.639729  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:00.640563  9699 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:00.641127  9722 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:00.633713  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:00.641860  9722 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:00.642441  9479 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:00.643170  9722 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.639771  9682 consensus_queue.cc:260] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.649573  9741 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:00.650236  9741 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.650799  9699 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.653296  9741 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:00.655139  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:00.656158  9480 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:00.656422  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:00.657161  9560 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:00.638595  9688 raft_consensus.cc:740] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:00.655254  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:00.659123  9688 consensus_queue.cc:260] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.662034  9688 ts_tablet_manager.cc:1434] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.049s	user 0.010s	sys 0.001s
I20260808 01:49:00.660398  9638 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:00.669203  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:00.672642  9682 ts_tablet_manager.cc:1434] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.069s	user 0.011s	sys 0.002s
I20260808 01:49:00.674017  9688 tablet_bootstrap.cc:492] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:00.674342  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:00.675619  9741 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:00.675774  9560 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:00.659722  9699 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:00.678613  9599 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:00.679104  9735 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:00.680644  9682 tablet_bootstrap.cc:492] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:00.679553  9741 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:00.670603  9479 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:00.683130  9741 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:00.684149  9741 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.683140  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:00.686991  9480 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:00.688109  9763 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:00.688381  9559 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:00.698589  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:00.699232  9763 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:00.699658  9735 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:00.700212  9735 raft_consensus.cc:697] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:00.703859  9638 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:00.706077  9372 catalog_manager.cc:5697] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:00.700270  9763 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.702644  9735 consensus_queue.cc:237] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.720487  9689 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:00.736949  9684 tablet_bootstrap.cc:492] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:00.737596  9684 ts_tablet_manager.cc:1403] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.155s	user 0.027s	sys 0.017s
I20260808 01:49:00.755610  9682 tablet_bootstrap.cc:654] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.755824  9688 tablet_bootstrap.cc:654] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.765014  9684 raft_consensus.cc:359] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.750972  9372 catalog_manager.cc:5697] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:00.765870  9684 raft_consensus.cc:385] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:00.767279  9684 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:00.775038  9377 catalog_manager.cc:5697] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:00.776743  9741 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:00.779454  9741 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.781893  9684 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.785189  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:00.786055  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:00.786713  9558 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:00.787037  9480 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:00.788463  9599 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:00.789919  9741 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:00.790865  9699 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:00.791293  9699 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:00.791632  9699 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:00.790992  9373 catalog_manager.cc:5697] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:00.784602  9684 ts_tablet_manager.cc:1434] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.047s	user 0.009s	sys 0.001s
I20260808 01:49:00.802783  9637 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:00.809546  9684 tablet_bootstrap.cc:492] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:00.817214  9772 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260808 01:49:00.819831  9688 tablet_bootstrap.cc:492] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:00.820641  9688 ts_tablet_manager.cc:1403] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.154s	user 0.021s	sys 0.001s
I20260808 01:49:00.853147  9688 raft_consensus.cc:359] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.853926  9688 raft_consensus.cc:385] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:00.854211  9688 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:00.855921  9688 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.858558  9688 ts_tablet_manager.cc:1434] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.037s	user 0.011s	sys 0.000s
I20260808 01:49:00.864858  9682 tablet_bootstrap.cc:492] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:00.865733  9682 ts_tablet_manager.cc:1403] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.185s	user 0.019s	sys 0.029s
I20260808 01:49:00.866721  9688 tablet_bootstrap.cc:492] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:00.869036  9682 raft_consensus.cc:359] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.869931  9682 raft_consensus.cc:385] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:00.870237  9682 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:00.871363  9682 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.874933  9682 ts_tablet_manager.cc:1434] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.009s	user 0.010s	sys 0.001s
I20260808 01:49:00.876729  9699 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.878291  9682 tablet_bootstrap.cc:492] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:00.881145  9699 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:00.881533  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:00.881810  9684 tablet_bootstrap.cc:654] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.881853  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:00.882283  9480 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:00.882844  9558 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:00.889376  9763 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:00.890005  9763 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.891746  9480 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:00.894431  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:00.895533  9763 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:00.897735  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:00.897749  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:00.900262  9524 leader_election.cc:343] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:00.903517  9699 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
W20260808 01:49:00.903743  9521 leader_election.cc:343] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:00.904346  9521 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:00.912003  9763 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:00.913269  9699 raft_consensus.cc:697] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:00.914666  9699 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.919466  9772 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:00.928789  9688 tablet_bootstrap.cc:654] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.933341  9558 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:00.979314  9689 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:00.980671  9689 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:00.986732  9682 tablet_bootstrap.cc:654] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:00.994536  9689 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:01.025499  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:01.026955  9558 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:01.031169  9688 tablet_bootstrap.cc:492] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:01.031694  9688 ts_tablet_manager.cc:1403] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.172s	user 0.020s	sys 0.000s
I20260808 01:49:01.032907  9684 tablet_bootstrap.cc:492] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:01.033337  9684 ts_tablet_manager.cc:1403] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.224s	user 0.022s	sys 0.022s
W20260808 01:49:01.034854  9426 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:49:01.032215  9372 catalog_manager.cc:5697] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:01.037474  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:01.039858  9699 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:01.040539  9684 raft_consensus.cc:359] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.041400  9684 raft_consensus.cc:385] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:01.040807  9699 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.042128  9684 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:01.043069  9638 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:01.042892  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:01.047303  9689 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:01.048751  9689 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:01.046298  9699 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:01.049721  9689 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:01.052001  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:01.053637  9684 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.055891  9734 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:01.055982  9688 raft_consensus.cc:359] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.057246  9688 raft_consensus.cc:385] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:01.057771  9688 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:01.056947  9734 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.063476  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:01.064873  9559 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:01.068477  9688 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.071062  9599 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:01.072114  9688 ts_tablet_manager.cc:1434] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.040s	user 0.011s	sys 0.000s
I20260808 01:49:01.072659  9699 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:01.073750  9688 tablet_bootstrap.cc:492] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:01.073552  9699 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:01.074393  9699 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:01.079242  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:01.082566  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:01.084780  9734 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:01.087492  9444 leader_election.cc:343] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:49:01.092088  9444 leader_election.cc:343] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:01.093091  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:01.094805  9682 tablet_bootstrap.cc:492] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:01.095338  9682 ts_tablet_manager.cc:1403] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.217s	user 0.017s	sys 0.005s
I20260808 01:49:01.095284  9734 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:01.101270  9684 ts_tablet_manager.cc:1434] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.068s	user 0.009s	sys 0.005s
I20260808 01:49:01.107782  9682 raft_consensus.cc:359] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.108821  9682 raft_consensus.cc:385] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:01.109200  9682 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:01.109786  9684 tablet_bootstrap.cc:492] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:01.110314  9682 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.103622  9763 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:01.108125  9688 tablet_bootstrap.cc:654] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:01.140635  9638 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:01.169693  9682 ts_tablet_manager.cc:1434] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.074s	user 0.012s	sys 0.003s
I20260808 01:49:01.176580  9688 tablet_bootstrap.cc:492] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:01.174654  9699 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.177696  9688 ts_tablet_manager.cc:1403] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.104s	user 0.011s	sys 0.018s
I20260808 01:49:01.167454  9689 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.181229  9682 tablet_bootstrap.cc:492] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:01.185462  9684 tablet_bootstrap.cc:654] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:01.190357  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:01.198553  9689 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:01.199980  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:01.200965  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:01.201424  9637 raft_consensus.cc:2393] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 1: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:49:01.201663  9559 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:01.206457  9481 raft_consensus.cc:2393] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 1: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:01.214072  9722 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:01.188580  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:01.215296  9688 raft_consensus.cc:359] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.184115  9699 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:01.217525  9688 raft_consensus.cc:385] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:01.216619  9760 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:01.223614  9688 raft_consensus.cc:740] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:01.224563  9688 consensus_queue.cc:260] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.225522  9559 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:01.220590  9722 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:01.222129  9599 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:01.227181  9688 ts_tablet_manager.cc:1434] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.031s	user 0.005s	sys 0.003s
I20260808 01:49:01.227270  9741 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:49:01.228582  9688 tablet_bootstrap.cc:492] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:01.228956  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:01.230365  9689 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:01.231194  9689 raft_consensus.cc:697] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:01.232080  9689 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.248891  9682 tablet_bootstrap.cc:654] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:01.250152  9688 tablet_bootstrap.cc:654] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:01.256219  9699 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:01.264632  9684 tablet_bootstrap.cc:492] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:01.265349  9684 ts_tablet_manager.cc:1403] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.156s	user 0.028s	sys 0.040s
I20260808 01:49:01.274564  9684 raft_consensus.cc:359] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.276120  9684 raft_consensus.cc:385] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:01.276602  9684 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:01.277853  9684 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.291213  9734 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:01.298790  9741 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:01.315107  9684 ts_tablet_manager.cc:1434] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.049s	user 0.010s	sys 0.002s
I20260808 01:49:01.321130  9684 tablet_bootstrap.cc:492] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:01.333256  9688 tablet_bootstrap.cc:492] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:01.333843  9688 ts_tablet_manager.cc:1403] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.106s	user 0.031s	sys 0.001s
I20260808 01:49:01.352739  9688 raft_consensus.cc:359] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.353761  9688 raft_consensus.cc:385] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:01.354117  9688 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:01.360208  9688 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.364158  9688 ts_tablet_manager.cc:1434] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.030s	user 0.008s	sys 0.003s
I20260808 01:49:01.365646  9688 tablet_bootstrap.cc:492] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:01.382521  9684 tablet_bootstrap.cc:654] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:01.395277  9689 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:01.394134  9682 tablet_bootstrap.cc:492] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:01.402356  9682 ts_tablet_manager.cc:1403] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.222s	user 0.011s	sys 0.016s
I20260808 01:49:01.411590  9688 tablet_bootstrap.cc:654] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:01.412437  9377 catalog_manager.cc:5697] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:01.436388  9682 raft_consensus.cc:359] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.437678  9682 raft_consensus.cc:385] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:01.437948  9682 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:01.440611  9682 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.449972  9682 ts_tablet_manager.cc:1434] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.047s	user 0.005s	sys 0.005s
I20260808 01:49:01.456079  9682 tablet_bootstrap.cc:492] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:01.463189  9741 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:01.471572  9722 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:01.472718  9722 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.503533  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:01.504967  9479 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:01.506376  9722 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:01.510921  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:01.512344  9638 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:01.517650  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:01.521938  9684 tablet_bootstrap.cc:492] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:01.520712  9741 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:01.529153  9684 ts_tablet_manager.cc:1403] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.208s	user 0.037s	sys 0.013s
I20260808 01:49:01.528730  9722 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:01.527774  9682 tablet_bootstrap.cc:654] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:01.530405  9722 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:01.535583  9722 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:01.548729  9684 raft_consensus.cc:359] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.549419  9684 raft_consensus.cc:385] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:01.549674  9684 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:01.551600  9684 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.556463  9688 tablet_bootstrap.cc:492] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:01.557248  9688 ts_tablet_manager.cc:1403] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.192s	user 0.023s	sys 0.026s
I20260808 01:49:01.561378  9684 ts_tablet_manager.cc:1434] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.003s
I20260808 01:49:01.562811  9688 raft_consensus.cc:359] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.563745  9688 raft_consensus.cc:385] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:01.564131  9688 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:01.565006  9688 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.567435  9684 tablet_bootstrap.cc:492] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:01.567824  9688 ts_tablet_manager.cc:1434] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.004s
I20260808 01:49:01.578845  9688 tablet_bootstrap.cc:492] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:01.584357  9722 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.598891  9741 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:01.599642  9741 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.603502  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:01.605181  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:01.605050  9481 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:01.606364  9637 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:01.610771  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:01.612004  9479 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:01.614895  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:01.616950  9777 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:01.617712  9777 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:01.618341  9777 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:01.602912  9722 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:01.623951  9688 tablet_bootstrap.cc:654] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:01.624007  9682 tablet_bootstrap.cc:492] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:01.626142  9637 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:01.626958  9682 ts_tablet_manager.cc:1403] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.171s	user 0.019s	sys 0.006s
I20260808 01:49:01.628434  9521 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:01.630108  9722 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:01.632336  9741 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:01.630968  9722 raft_consensus.cc:697] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:01.635499  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:01.636351  9722 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.638940  9682 raft_consensus.cc:359] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.639818  9682 raft_consensus.cc:385] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:01.640164  9682 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
W20260808 01:49:01.640434  9599 leader_election.cc:343] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:01.641027  9682 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.646545  9682 ts_tablet_manager.cc:1434] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.004s
I20260808 01:49:01.650727  9684 tablet_bootstrap.cc:654] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:01.651315  9682 tablet_bootstrap.cc:492] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:01.649758  9802 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260808 01:49:01.659405  9741 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:01.660935  9741 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.666265  9688 tablet_bootstrap.cc:492] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:01.674284  9741 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:01.713070  9791 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:01.714311  9791 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.718225  9688 ts_tablet_manager.cc:1403] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.140s	user 0.031s	sys 0.042s
I20260808 01:49:01.718511  9777 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.721769  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:01.720453  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:01.723835  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:01.724221  9558 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:01.724746  9777 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:01.725361  9480 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:01.725742  9791 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:01.733624  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:01.735687  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:01.738175  9638 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 1: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:49:01.728070  9524 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:01.746285  9688 raft_consensus.cc:359] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.747354  9688 raft_consensus.cc:385] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:01.747751  9688 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:01.748729  9688 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.752494  9688 ts_tablet_manager.cc:1434] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.034s	user 0.009s	sys 0.000s
I20260808 01:49:01.733665  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:01.754608  9688 tablet_bootstrap.cc:492] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:01.735008  9558 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:01.755712  9777 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:01.756551  9777 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:01.757398  9777 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:01.764705  9481 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
W20260808 01:49:01.776530  9601 leader_election.cc:343] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:01.779955  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:01.781390  9481 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:01.784160  9802 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:01.793102  9682 tablet_bootstrap.cc:654] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:01.780551  9373 catalog_manager.cc:5697] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:01.804893  9684 tablet_bootstrap.cc:492] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:01.805588  9684 ts_tablet_manager.cc:1403] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.238s	user 0.026s	sys 0.023s
I20260808 01:49:01.812453  9481 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:01.818297  9684 raft_consensus.cc:359] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.830928  9684 raft_consensus.cc:385] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:01.831390  9684 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:01.832635  9684 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.870307  9802 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:01.871876  9802 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.867972  9760 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:01.875690  9760 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.879832  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:01.869103  9684 ts_tablet_manager.cc:1434] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.063s	user 0.012s	sys 0.005s
I20260808 01:49:01.832257  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:01.880940  9559 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:01.885871  9684 tablet_bootstrap.cc:492] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:01.888839  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:01.893424  9688 tablet_bootstrap.cc:654] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:01.899052  9797 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:01.899953  9797 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:01.900671  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:01.902486  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:01.895282  9741 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:01.900748  9797 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:01.904073  9741 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:49:01.892338  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 60321.1us
I20260808 01:49:01.905956  9741 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.912796  9682 tablet_bootstrap.cc:492] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:01.913313  9682 ts_tablet_manager.cc:1403] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.262s	user 0.016s	sys 0.041s
W20260808 01:49:01.910897  9444 leader_election.cc:343] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:01.891460  9760 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:01.928062  9777 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.935436  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:01.914278  9802 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:01.920674  9558 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
W20260808 01:49:01.945279  9521 leader_election.cc:343] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:01.950641  9722 raft_consensus.cc:2764] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260808 01:49:01.959347  9802 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:01.964351  9777 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:01.975930  9682 raft_consensus.cc:359] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.976858  9682 raft_consensus.cc:385] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:01.977190  9682 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:01.979259  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:01.979974  9559 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:01.969420  9802 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.982159  9802 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:01.978057  9682 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.988848  9682 ts_tablet_manager.cc:1434] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.075s	user 0.006s	sys 0.003s
I20260808 01:49:01.990285  9682 tablet_bootstrap.cc:492] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:01.990715  9688 tablet_bootstrap.cc:492] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:01.991308  9688 ts_tablet_manager.cc:1403] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.237s	user 0.024s	sys 0.031s
I20260808 01:49:01.990134  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:01.993952  9636 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:01.996467  9797 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.997802  9688 raft_consensus.cc:359] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:01.998711  9688 raft_consensus.cc:385] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:01.999135  9688 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:01.992923  9684 tablet_bootstrap.cc:654] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:01.997188  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:01.995110  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:02.005659  9480 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 1: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:02.016774  9689 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:02.017279  9689 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:02.010010  9524 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:02.018049  9689 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:02.018124  9797 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:02.018891  9722 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:02.026343  9559 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
W20260808 01:49:02.026628  9444 leader_election.cc:343] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:02.016378  9688 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.030078  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:02.031282  9688 ts_tablet_manager.cc:1434] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.040s	user 0.008s	sys 0.008s
I20260808 01:49:02.031885  9777 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:02.008046  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:02.032980  9688 tablet_bootstrap.cc:492] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:02.032498  9777 raft_consensus.cc:697] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:02.035311  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:02.035876  9777 consensus_queue.cc:237] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.040139  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:02.056948  9722 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:02.065640  9373 catalog_manager.cc:5697] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
W20260808 01:49:02.069330  9601 leader_election.cc:343] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:02.093315  9689 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.098148  9689 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:02.101765  9684 tablet_bootstrap.cc:492] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:02.102437  9684 ts_tablet_manager.cc:1403] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.218s	user 0.020s	sys 0.040s
I20260808 01:49:02.103739  9682 tablet_bootstrap.cc:654] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:02.103161  9814 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20260808 01:49:02.107743  9444 leader_election.cc:343] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:02.112313  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:02.112341  9684 raft_consensus.cc:359] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.113853  9684 raft_consensus.cc:385] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:02.114161  9684 raft_consensus.cc:740] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:02.115607  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:02.115934  9684 consensus_queue.cc:260] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.116776  9636 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:02.120168  9684 ts_tablet_manager.cc:1434] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.017s	user 0.003s	sys 0.007s
I20260808 01:49:02.122001  9684 tablet_bootstrap.cc:492] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
W20260808 01:49:02.133761  9444 leader_election.cc:343] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:49:02.141100  9426 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:49:02.147161  9688 tablet_bootstrap.cc:654] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:02.153650  9741 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:02.154553  9741 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.172657  9373 catalog_manager.cc:5697] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:02.185876  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:02.189065  9601 leader_election.cc:343] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:02.197750  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:02.197906  9636 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
W20260808 01:49:02.199950  9599 leader_election.cc:343] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:02.201026  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:02.201543  9814 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:02.201112  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:02.202486  9814 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.203748  9558 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:02.204591  9819 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:02.205598  9819 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:02.206069  9684 tablet_bootstrap.cc:654] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:02.203898  9777 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:02.208242  9819 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.213047  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:02.213619  9814 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:02.215588  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:02.216634  9524 leader_election.cc:343] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:49:02.218078  9521 leader_election.cc:343] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:02.218858  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:02.194156  9741 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:02.220921  9814 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:02.221411  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:02.232502  9817 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:02.233767  9817 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:02.236146  9817 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.270785  9814 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260808 01:49:02.287631  9682 tablet_bootstrap.cc:492] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:02.288354  9682 ts_tablet_manager.cc:1403] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.298s	user 0.034s	sys 0.035s
I20260808 01:49:02.299146  9684 tablet_bootstrap.cc:492] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:02.300057  9684 ts_tablet_manager.cc:1403] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.178s	user 0.018s	sys 0.013s
I20260808 01:49:02.320870  9688 tablet_bootstrap.cc:492] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:02.321450  9688 ts_tablet_manager.cc:1403] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.289s	user 0.021s	sys 0.017s
I20260808 01:49:02.326783  9684 raft_consensus.cc:359] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.327879  9684 raft_consensus.cc:385] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:02.328267  9684 raft_consensus.cc:740] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:02.329267  9684 consensus_queue.cc:260] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.332449  9684 ts_tablet_manager.cc:1434] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.032s	user 0.012s	sys 0.000s
I20260808 01:49:02.334026  9684 tablet_bootstrap.cc:492] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:02.330543  9688 raft_consensus.cc:359] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.338088  9688 raft_consensus.cc:385] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:02.338433  9688 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:02.339524  9689 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:02.341233  9688 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.340574  9689 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.344118  9688 ts_tablet_manager.cc:1434] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.022s	user 0.010s	sys 0.003s
I20260808 01:49:02.345709  9688 tablet_bootstrap.cc:492] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:02.347954  9689 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:02.350636  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:02.351697  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:02.360252  9444 leader_election.cc:343] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:02.359001  9372 catalog_manager.cc:5697] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
W20260808 01:49:02.366302  9444 leader_election.cc:343] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:02.367316  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:02.370479  9689 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:02.371484  9688 tablet_bootstrap.cc:654] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:02.363533  9372 catalog_manager.cc:5697] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:02.373842  9682 raft_consensus.cc:359] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.374939  9682 raft_consensus.cc:385] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:02.375340  9682 raft_consensus.cc:740] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:02.376294  9682 consensus_queue.cc:260] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.379220  9682 ts_tablet_manager.cc:1434] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.091s	user 0.009s	sys 0.001s
I20260808 01:49:02.380447  9682 tablet_bootstrap.cc:492] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:02.380617  9684 tablet_bootstrap.cc:654] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:02.380524  9777 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:02.396839  9682 tablet_bootstrap.cc:654] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:02.409016  9688 tablet_bootstrap.cc:492] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:02.409766  9688 ts_tablet_manager.cc:1403] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.064s	user 0.013s	sys 0.014s
I20260808 01:49:02.419081  9689 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:02.420212  9689 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.427024  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:02.428535  9689 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:02.440938  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:02.446400  9688 raft_consensus.cc:359] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.428601  9559 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 1: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:49:02.457231  9688 raft_consensus.cc:385] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:02.458400  9688 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:02.457154  9444 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:02.466231  9688 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.473954  9688 ts_tablet_manager.cc:1434] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.064s	user 0.013s	sys 0.003s
I20260808 01:49:02.476940  9688 tablet_bootstrap.cc:492] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:02.483008  9689 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:02.487613  9777 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:02.500224  9684 tablet_bootstrap.cc:492] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:02.501391  9684 ts_tablet_manager.cc:1403] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.168s	user 0.033s	sys 0.040s
I20260808 01:49:02.510277  9682 tablet_bootstrap.cc:492] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:02.511770  9682 ts_tablet_manager.cc:1403] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.132s	user 0.009s	sys 0.016s
I20260808 01:49:02.507654  9684 raft_consensus.cc:359] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.517050  9684 raft_consensus.cc:385] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:02.517326  9684 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:02.518798  9682 raft_consensus.cc:359] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.519797  9682 raft_consensus.cc:385] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:02.520043  9682 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:02.520669  9682 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.525853  9689 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:02.529422  9682 ts_tablet_manager.cc:1434] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.017s	user 0.009s	sys 0.000s
I20260808 01:49:02.530575  9684 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.533833  9684 ts_tablet_manager.cc:1434] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.032s	user 0.011s	sys 0.000s
I20260808 01:49:02.537487  9684 tablet_bootstrap.cc:492] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:02.542040  9682 tablet_bootstrap.cc:492] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:02.555227  9688 tablet_bootstrap.cc:654] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:02.569361  9682 tablet_bootstrap.cc:654] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:02.571167  9684 tablet_bootstrap.cc:654] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:02.588522  9684 tablet_bootstrap.cc:492] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:02.588636  9682 tablet_bootstrap.cc:492] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:02.588636  9688 tablet_bootstrap.cc:492] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:02.589012  9684 ts_tablet_manager.cc:1403] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.052s	user 0.014s	sys 0.015s
I20260808 01:49:02.589273  9682 ts_tablet_manager.cc:1403] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.047s	user 0.012s	sys 0.020s
I20260808 01:49:02.589335  9688 ts_tablet_manager.cc:1403] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.113s	user 0.014s	sys 0.020s
I20260808 01:49:02.593458  9684 raft_consensus.cc:359] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.594087  9684 raft_consensus.cc:385] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:02.594290  9682 raft_consensus.cc:359] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.594409  9684 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:02.603080  9682 raft_consensus.cc:385] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:02.594450  9688 raft_consensus.cc:359] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.603713  9682 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:02.595821  9763 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:02.604115  9688 raft_consensus.cc:385] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:02.604763  9688 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:02.605326  9682 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.603760  9684 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.604791  9763 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.618779  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:02.620110  9480 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:02.623584  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:02.625911  9760 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:02.626927  9760 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:02.627755  9760 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:02.641652  9688 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.643654  9684 ts_tablet_manager.cc:1434] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.054s	user 0.016s	sys 0.029s
I20260808 01:49:02.652871  9684 tablet_bootstrap.cc:492] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:02.659163  9688 ts_tablet_manager.cc:1434] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.069s	user 0.008s	sys 0.005s
I20260808 01:49:02.660439  9763 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:02.663154  9688 tablet_bootstrap.cc:492] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:02.667074  9684 tablet_bootstrap.cc:654] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:02.669467  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:02.672125  9638 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:49:02.679350  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 59567.8us
I20260808 01:49:02.682731  9835 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:02.690538  9760 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.696184  9682 ts_tablet_manager.cc:1434] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.106s	user 0.009s	sys 0.017s
I20260808 01:49:02.698035  9819 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260808 01:49:02.701510  9521 leader_election.cc:343] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:02.706723  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:02.707208  9819 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:02.707855  9480 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:02.708108  9819 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.713160  9817 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:02.716200  9682 tablet_bootstrap.cc:492] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:02.718901  9760 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:02.731500  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:02.733387  9835 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:02.740932  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:02.744784  9763 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:02.748944  9763 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.780339  9777 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:02.781445  9777 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.782370  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:02.796475  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:02.774113  9819 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:02.798278  9444 leader_election.cc:343] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:49:02.812373  9521 leader_election.cc:343] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:02.813537  9835 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:02.818373  9835 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.821610  9777 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:02.814011  9763 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:02.796669  9480 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:02.822382  9835 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:02.837172  9688 tablet_bootstrap.cc:654] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:02.829501  9817 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260808 01:49:02.849830  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:02.863837  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:02.864420  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:02.864749  9763 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election won for term 1
W20260808 01:49:02.842365  9444 leader_election.cc:343] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:02.865463  9763 raft_consensus.cc:697] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:02.865701  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:02.859081  9819 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:02.867193  9819 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.869643  9819 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:02.870245  9819 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:02.871680  9560 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:02.875939  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:02.894250  9444 leader_election.cc:343] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:02.874209  9763 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.873306  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:49:02.897305  9444 leader_election.cc:343] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:02.897974  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:02.899116  9819 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:02.911204  9760 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:02.911813  9760 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.913386  9684 tablet_bootstrap.cc:492] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:02.913885  9684 ts_tablet_manager.cc:1403] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.261s	user 0.021s	sys 0.034s
I20260808 01:49:02.914369  9760 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:02.917075  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:02.917179  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:02.923332  9599 leader_election.cc:343] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:49:02.922363  9524 leader_election.cc:343] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:02.938318  9684 raft_consensus.cc:359] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.939589  9684 raft_consensus.cc:385] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:02.940022  9684 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:02.942147  9684 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:02.948767  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:02.949977  9479 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:02.956573  9682 tablet_bootstrap.cc:654] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:02.952325  9689 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260808 01:49:02.958276  9684 ts_tablet_manager.cc:1434] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.044s	user 0.009s	sys 0.005s
I20260808 01:49:02.960660  9684 tablet_bootstrap.cc:492] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:02.963747  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:02.964124  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:02.964934  9637 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:02.970196  9688 tablet_bootstrap.cc:492] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:02.971107  9688 ts_tablet_manager.cc:1403] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.308s	user 0.034s	sys 0.023s
I20260808 01:49:02.989993  9521 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:02.992216  9521 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:02.994592  9722 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:02.999591  9373 catalog_manager.cc:5697] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:02.996268  9760 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:03.004534  9684 tablet_bootstrap.cc:654] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:03.004755  9760 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:03.005257  9760 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:02.995442  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:03.024839  9741 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:03.026343  9601 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:03.034611  9741 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:03.035519  9741 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:03.036105  9741 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:03.029286  9836 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:03.043723  9836 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.055269  9688 raft_consensus.cc:359] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.056209  9688 raft_consensus.cc:385] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:03.056517  9688 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:03.057507  9688 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.061012  9836 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:03.066336  9688 ts_tablet_manager.cc:1434] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.095s	user 0.009s	sys 0.000s
I20260808 01:49:03.063356  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:03.067719  9688 tablet_bootstrap.cc:492] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
W20260808 01:49:03.071269  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 45007.7us
I20260808 01:49:03.067476  9557 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:03.076027  9599 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:03.081681  9836 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:03.087190  9836 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:03.106441  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:03.099599  9836 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:03.115686  9760 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.119575  9682 tablet_bootstrap.cc:492] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:03.120081  9684 tablet_bootstrap.cc:492] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:03.120149  9682 ts_tablet_manager.cc:1403] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.416s	user 0.018s	sys 0.020s
I20260808 01:49:03.120823  9684 ts_tablet_manager.cc:1403] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.160s	user 0.021s	sys 0.018s
I20260808 01:49:03.113435  9741 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.124966  9760 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:03.125779  9682 raft_consensus.cc:359] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.128087  9682 raft_consensus.cc:385] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:03.128564  9682 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:03.128837  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:03.126494  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:03.129688  9682 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:03.132488  9524 leader_election.cc:343] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:03.138590  9682 ts_tablet_manager.cc:1434] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.018s	user 0.010s	sys 0.000s
W20260808 01:49:03.139775  9599 leader_election.cc:343] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:03.152678  9684 raft_consensus.cc:359] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.156440  9684 raft_consensus.cc:385] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:03.156828  9684 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:03.157918  9684 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:03.159205  9601 leader_election.cc:343] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:03.163345  9688 tablet_bootstrap.cc:654] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:03.139361  9741 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:03.163504  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:03.164892  9480 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:03.163668  9684 ts_tablet_manager.cc:1434] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.042s	user 0.003s	sys 0.006s
I20260808 01:49:03.159811  9682 tablet_bootstrap.cc:492] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:03.178525  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:03.179751  9638 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:03.181761  9684 tablet_bootstrap.cc:492] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:03.198714  9836 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.208793  9682 tablet_bootstrap.cc:654] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:03.210623  9836 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:03.213178  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:03.215878  9559 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:03.222820  9480 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:03.236181  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:03.239537  9684 tablet_bootstrap.cc:654] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:03.243474  9601 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:03.234225  9638 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:03.247704  9741 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:03.248311  9741 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:03.249442  9741 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.236562  9559 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:03.255344  9688 tablet_bootstrap.cc:492] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:03.256865  9688 ts_tablet_manager.cc:1403] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.189s	user 0.014s	sys 0.031s
I20260808 01:49:03.260245  9682 tablet_bootstrap.cc:492] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:03.260819  9682 ts_tablet_manager.cc:1403] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.101s	user 0.012s	sys 0.016s
W20260808 01:49:03.247695  9601 leader_election.cc:343] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:03.266520  9521 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:03.268788  9763 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:03.269603  9763 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:03.269026  9599 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:03.270720  9763 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.272068  9836 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:03.276114  9682 raft_consensus.cc:359] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.277004  9682 raft_consensus.cc:385] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:03.277369  9682 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:03.283879  9836 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:03.287359  9688 raft_consensus.cc:359] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.290359  9688 raft_consensus.cc:385] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:03.290827  9688 raft_consensus.cc:740] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:03.278497  9682 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:03.295210  9426 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:49:03.300225  9688 consensus_queue.cc:260] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.286563  9836 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.306367  9688 ts_tablet_manager.cc:1434] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.049s	user 0.006s	sys 0.003s
I20260808 01:49:03.313583  9688 tablet_bootstrap.cc:492] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:03.314139  9374 catalog_manager.cc:5697] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:03.307956  9682 ts_tablet_manager.cc:1434] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.047s	user 0.011s	sys 0.000s
I20260808 01:49:03.318426  9763 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:03.319077  9682 tablet_bootstrap.cc:492] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:03.329869  9372 catalog_manager.cc:5697] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:03.363003  9688 tablet_bootstrap.cc:654] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:03.364073  9682 tablet_bootstrap.cc:654] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:03.365182  9684 tablet_bootstrap.cc:492] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:03.365689  9684 ts_tablet_manager.cc:1403] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.184s	user 0.032s	sys 0.034s
I20260808 01:49:03.362587  9372 catalog_manager.cc:5697] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:03.376365  9684 raft_consensus.cc:359] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.377111  9684 raft_consensus.cc:385] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:03.377374  9684 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:03.378012  9682 tablet_bootstrap.cc:492] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:03.378028  9684 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.375788  9638 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:03.380615  9684 ts_tablet_manager.cc:1434] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.002s
I20260808 01:49:03.384827  9682 ts_tablet_manager.cc:1403] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.066s	user 0.010s	sys 0.013s
I20260808 01:49:03.389669  9684 tablet_bootstrap.cc:492] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:03.391784  9682 raft_consensus.cc:359] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.392985  9682 raft_consensus.cc:385] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:03.393340  9682 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:03.395058  9682 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.397812  9682 ts_tablet_manager.cc:1434] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260808 01:49:03.399000  9684 tablet_bootstrap.cc:654] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:03.399226  9682 tablet_bootstrap.cc:492] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:03.426693  9722 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:03.439530  9684 tablet_bootstrap.cc:492] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:03.440138  9684 ts_tablet_manager.cc:1403] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.051s	user 0.013s	sys 0.004s
I20260808 01:49:03.440591  9688 tablet_bootstrap.cc:492] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:03.441215  9688 ts_tablet_manager.cc:1403] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.128s	user 0.018s	sys 0.010s
I20260808 01:49:03.444164  9684 raft_consensus.cc:359] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.445125  9684 raft_consensus.cc:385] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:03.445485  9684 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:03.447402  9682 tablet_bootstrap.cc:654] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:03.446431  9684 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.449591  9684 ts_tablet_manager.cc:1434] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260808 01:49:03.449666  9689 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:03.450495  9689 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.444593  9688 raft_consensus.cc:359] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.453888  9722 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:03.454463  9722 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.455240  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:49:03.457294  9444 leader_election.cc:343] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:03.452553  9688 raft_consensus.cc:385] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:03.458752  9689 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:03.451849  9684 tablet_bootstrap.cc:492] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:03.458997  9688 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:03.461238  9741 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:03.461030  9688 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.461935  9741 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.464735  9741 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:03.466646  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:03.468146  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:03.468403  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:03.469090  9559 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:03.470558  9599 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:03.472244  9836 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:03.472764  9836 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:03.472913  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:03.467567  9480 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:03.473228  9836 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:03.475914  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:03.459136  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:03.477497  9763 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:03.478436  9763 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:03.479051  9763 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:03.483788  9722 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:03.485879  9444 leader_election.cc:343] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:03.486505  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:03.487653  9689 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:03.504554  9688 ts_tablet_manager.cc:1434] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.063s	user 0.008s	sys 0.000s
I20260808 01:49:03.505635  9688 tablet_bootstrap.cc:492] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:03.504091  9684 tablet_bootstrap.cc:654] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:03.517604  9682 tablet_bootstrap.cc:492] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:03.518239  9682 ts_tablet_manager.cc:1403] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.119s	user 0.018s	sys 0.028s
I20260808 01:49:03.513839  9836 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.515834  9763 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.521636  9637 raft_consensus.cc:2393] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 1: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:49:03.523190  9688 tablet_bootstrap.cc:654] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:03.523540  9682 raft_consensus.cc:359] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.524286  9682 raft_consensus.cc:385] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:03.524358  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:03.524636  9682 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:03.525113  9480 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:03.525394  9682 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.527953  9682 ts_tablet_manager.cc:1434] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.007s
I20260808 01:49:03.530150  9682 tablet_bootstrap.cc:492] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:03.536556  9836 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:03.538499  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:03.539537  9682 tablet_bootstrap.cc:654] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:03.541347  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:03.542622  9559 raft_consensus.cc:2393] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 1: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:49:03.551221  9599 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:03.552194  9836 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:49:03.552675  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:03.553555  9637 raft_consensus.cc:2393] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 1: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:49:03.554457  9763 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:03.562978  9688 tablet_bootstrap.cc:492] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:03.563593  9688 ts_tablet_manager.cc:1403] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.058s	user 0.015s	sys 0.006s
I20260808 01:49:03.569370  9688 raft_consensus.cc:359] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.570436  9688 raft_consensus.cc:385] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:03.570880  9688 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:03.572029  9688 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.572911  9480 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:03.575589  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:03.578339  9763 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:03.580816  9688 ts_tablet_manager.cc:1434] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.017s	user 0.009s	sys 0.000s
I20260808 01:49:03.580432  9763 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:03.582705  9722 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:03.582540  9763 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.583498  9722 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.591207  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:03.594277  9521 leader_election.cc:343] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:03.595069  9688 tablet_bootstrap.cc:492] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:03.595508  9722 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:03.596949  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:03.603552  9524 leader_election.cc:343] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:03.599795  9684 tablet_bootstrap.cc:492] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:03.604998  9684 ts_tablet_manager.cc:1403] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.153s	user 0.028s	sys 0.037s
I20260808 01:49:03.604287  9524 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:03.607595  9722 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:03.616626  9688 tablet_bootstrap.cc:654] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:03.618053  9682 tablet_bootstrap.cc:492] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:03.618664  9682 ts_tablet_manager.cc:1403] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.089s	user 0.034s	sys 0.034s
I20260808 01:49:03.617362  9684 raft_consensus.cc:359] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.619125  9684 raft_consensus.cc:385] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:03.619578  9684 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:03.620468  9684 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.622534  9372 catalog_manager.cc:5697] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:03.624135  9684 ts_tablet_manager.cc:1434] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.003s
I20260808 01:49:03.625828  9684 tablet_bootstrap.cc:492] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:03.625629  9682 raft_consensus.cc:359] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.626403  9682 raft_consensus.cc:385] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:03.626722  9682 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:03.628721  9682 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.635980  9682 ts_tablet_manager.cc:1434] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.001s
I20260808 01:49:03.636525  9722 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:03.637228  9682 tablet_bootstrap.cc:492] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:03.637094  9722 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.637909  9688 tablet_bootstrap.cc:492] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
W20260808 01:49:03.638429  9599 consensus_peers.cc:597] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:03.641630  9684 tablet_bootstrap.cc:654] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:03.638438  9688 ts_tablet_manager.cc:1403] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.056s	user 0.012s	sys 0.006s
I20260808 01:49:03.651912  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:03.652799  9480 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:03.655378  9722 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:03.657111  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:03.660466  9688 raft_consensus.cc:359] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.661283  9688 raft_consensus.cc:385] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:03.661550  9688 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:03.662001  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:03.665546  9722 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:03.666224  9722 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:03.666474  9688 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.668188  9638 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:03.672747  9688 ts_tablet_manager.cc:1434] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.028s	user 0.008s	sys 0.000s
I20260808 01:49:03.674365  9688 tablet_bootstrap.cc:492] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:03.667598  9722 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:03.688892  9777 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:03.698844  9682 tablet_bootstrap.cc:654] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
W20260808 01:49:03.709085  9524 consensus_peers.cc:597] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:03.725592  9684 tablet_bootstrap.cc:492] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:03.728241  9684 ts_tablet_manager.cc:1403] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.103s	user 0.026s	sys 0.034s
I20260808 01:49:03.731312  9722 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.739148  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:03.741593  9480 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:03.738502  9741 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260808 01:49:03.733943  9684 raft_consensus.cc:359] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.750451  9684 raft_consensus.cc:385] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:03.750918  9684 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:03.751960  9684 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.753882  9684 ts_tablet_manager.cc:1434] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.025s	user 0.008s	sys 0.000s
I20260808 01:49:03.755277  9684 tablet_bootstrap.cc:492] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:03.767367  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:03.768481  9638 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:03.770730  9688 tablet_bootstrap.cc:654] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:03.772377  9722 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:03.790179  9601 consensus_peers.cc:597] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:03.795778  9682 tablet_bootstrap.cc:492] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:03.796581  9682 ts_tablet_manager.cc:1403] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.160s	user 0.021s	sys 0.016s
I20260808 01:49:03.802022  9684 tablet_bootstrap.cc:654] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:03.803464  9682 raft_consensus.cc:359] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.804293  9682 raft_consensus.cc:385] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:03.804572  9682 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:03.809746  9638 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:03.805384  9682 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.816280  9760 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:03.816916  9760 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.816514  9682 ts_tablet_manager.cc:1434] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.020s	user 0.004s	sys 0.003s
I20260808 01:49:03.820215  9682 tablet_bootstrap.cc:492] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:03.823429  9688 tablet_bootstrap.cc:492] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:03.824184  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:03.824184  9688 ts_tablet_manager.cc:1403] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.150s	user 0.020s	sys 0.003s
I20260808 01:49:03.824920  9481 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:03.825017  9760 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:03.826661  9521 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:03.827818  9684 tablet_bootstrap.cc:492] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:03.828291  9524 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:03.829576  9835 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:03.830370  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:03.830078  9835 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.831007  9682 tablet_bootstrap.cc:654] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:03.831338  9638 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:03.833840  9763 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:03.828289  9684 ts_tablet_manager.cc:1403] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.073s	user 0.008s	sys 0.014s
I20260808 01:49:03.828557  9688 raft_consensus.cc:359] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.836263  9688 raft_consensus.cc:385] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:03.836552  9688 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:03.839339  9760 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:03.847983  9760 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:03.841367  9688 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.854888  9760 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:03.964689  9480 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:03.965243  9688 ts_tablet_manager.cc:1434] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.140s	user 0.009s	sys 0.016s
I20260808 01:49:03.966289  9688 tablet_bootstrap.cc:492] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:03.879104  9763 raft_consensus.cc:697] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:03.963573  9836 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:03.978392  9763 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.981166  9836 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.998360  9684 raft_consensus.cc:359] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.999461  9684 raft_consensus.cc:385] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:03.999811  9684 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:04.000739  9684 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:03.975232  9722 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:04.015130  9722 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.018896  9684 ts_tablet_manager.cc:1434] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.183s	user 0.009s	sys 0.001s
I20260808 01:49:04.013715  9835 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:04.040617  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:04.041764  9559 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:04.052600  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:04.053783  9481 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:04.057651  9836 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:04.064913  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:04.068549  9599 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:04.071292  9835 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:04.033540  9817 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:04.073372  9817 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.072167  9835 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:04.076073  9835 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:04.092998  9682 tablet_bootstrap.cc:492] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:04.094554  9682 ts_tablet_manager.cc:1403] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.275s	user 0.017s	sys 0.027s
I20260808 01:49:04.099473  9684 tablet_bootstrap.cc:492] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:04.088390  9817 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:04.105137  9599 leader_election.cc:343] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:04.110890  9817 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:04.124894  9688 tablet_bootstrap.cc:654] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:04.123108  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:04.127276  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:04.133637  9722 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:04.141400  9638 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 1: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:49:04.126413  9817 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.133772  9760 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.155189  9891 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:04.156353  9891 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.160097  9682 raft_consensus.cc:359] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.160876  9682 raft_consensus.cc:385] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:04.161139  9682 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:04.161900  9682 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.162827  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:04.163834  9480 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:04.165966  9682 ts_tablet_manager.cc:1434] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.070s	user 0.006s	sys 0.001s
I20260808 01:49:04.165982  9373 catalog_manager.cc:5697] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:04.167563  9682 tablet_bootstrap.cc:492] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:04.170001  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:04.173581  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:04.186784  9888 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260808 01:49:04.190945  9889 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:04.197359  9763 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:04.200636  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:04.202519  9480 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:04.204367  9817 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:04.208233  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:49:04.216187  9444 leader_election.cc:343] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:04.218305  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:04.222385  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:04.222467  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:04.226604  9882 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:04.227674  9882 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:04.228250  9882 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:04.229930  9889 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:04.233915  9684 tablet_bootstrap.cc:654] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:04.234490  9688 tablet_bootstrap.cc:492] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:04.235107  9688 ts_tablet_manager.cc:1403] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.269s	user 0.024s	sys 0.015s
I20260808 01:49:04.226004  9760 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:04.235143  9444 leader_election.cc:343] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:04.241698  9559 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:04.242605  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:04.244230  9889 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:04.245977  9682 tablet_bootstrap.cc:654] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:04.246378  9882 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.248051  9835 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.249212  9882 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:04.249614  9636 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 1: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:49:04.251974  9524 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:04.253587  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:04.252964  9891 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:04.254293  9760 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260808 01:49:04.256197  9599 leader_election.cc:343] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:04.256278  9763 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:04.257630  9760 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:04.258271  9760 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:04.261443  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:04.267489  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:04.268327  9479 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:49:04.277184  9524 leader_election.cc:343] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:04.282239  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:04.291549  9835 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:04.300129  9480 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:04.301790  9688 raft_consensus.cc:359] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.303926  9688 raft_consensus.cc:385] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:04.304320  9688 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:04.302986  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:04.357350  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:04.358507  9635 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:04.384199  9479 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:04.411087  9688 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.412094  9682 tablet_bootstrap.cc:492] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:04.412796  9682 ts_tablet_manager.cc:1403] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.246s	user 0.013s	sys 0.005s
I20260808 01:49:04.418668  9682 raft_consensus.cc:359] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.302995  9760 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.420532  9682 raft_consensus.cc:385] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:04.422964  9688 ts_tablet_manager.cc:1434] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.187s	user 0.006s	sys 0.020s
I20260808 01:49:04.425427  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:04.426126  9478 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:04.426424  9480 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
W20260808 01:49:04.427073  9670 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.IsCreateTableDone -> {remote=127.7.165.254:43565, user_credentials={real_user=slave}} blocked reactor thread for 58433.2us
I20260808 01:49:04.424613  9682 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:04.426092  9688 tablet_bootstrap.cc:492] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:04.436297  9558 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 1: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:49:04.446583  9682 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.466773  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:04.483666  9684 tablet_bootstrap.cc:492] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:04.484344  9684 ts_tablet_manager.cc:1403] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.448s	user 0.023s	sys 0.019s
I20260808 01:49:04.484918  9682 ts_tablet_manager.cc:1434] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.072s	user 0.013s	sys 0.006s
I20260808 01:49:04.486581  9682 tablet_bootstrap.cc:492] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:04.495674  9777 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:04.498160  9777 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.491979  9882 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:04.512732  9882 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:04.516542  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:04.519743  9741 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:04.520385  9741 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:04.494987  9760 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:04.521938  9741 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.502326  9777 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:04.514591  9882 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.535048  9891 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:04.537222  9891 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.548671  9891 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:04.553637  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:04.561306  9684 raft_consensus.cc:359] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.561913  9558 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:04.562773  9684 raft_consensus.cc:385] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:04.564899  9684 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:04.566603  9684 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:04.556417  9521 leader_election.cc:343] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:04.568159  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:04.569804  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:04.545825  9635 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:04.576587  9521 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:04.581365  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:04.582291  9891 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:04.585278  9684 ts_tablet_manager.cc:1434] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.101s	user 0.012s	sys 0.000s
I20260808 01:49:04.585947  9688 tablet_bootstrap.cc:654] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:04.595244  9477 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 1: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:49:04.589746  9836 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:04.597929  9836 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:04.598556  9836 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:04.601558  9478 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:04.604719  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:04.608635  9521 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:04.610174  9891 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:04.611021  9891 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:04.612294  9891 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.612712  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:04.604669  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:04.614552  9476 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:04.618197  9760 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:04.618217  9682 tablet_bootstrap.cc:654] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:04.626816  9521 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:04.628412  9760 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260808 01:49:04.632102  9903 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260808 01:49:04.641165  9684 tablet_bootstrap.cc:492] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
W20260808 01:49:04.656478  9601 leader_election.cc:343] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:04.649500  9760 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
W20260808 01:49:04.664677  9902 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260808 01:49:04.674436  9836 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.686205  9836 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:04.688999  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:04.689452  9524 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:04.689750  9558 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:04.690909  9760 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:04.691375  9760 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:04.691840  9760 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:04.695308  9688 tablet_bootstrap.cc:492] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:04.696753  9688 ts_tablet_manager.cc:1403] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.271s	user 0.022s	sys 0.014s
I20260808 01:49:04.712513  9688 raft_consensus.cc:359] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.713797  9688 raft_consensus.cc:385] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:04.725286  9760 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.714253  9688 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:04.740404  9688 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.765504  9688 ts_tablet_manager.cc:1434] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.068s	user 0.005s	sys 0.004s
W20260808 01:49:04.744977  9521 leader_election.cc:343] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:04.769935  9372 catalog_manager.cc:5697] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:04.798246  9688 tablet_bootstrap.cc:492] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:04.803885  9373 catalog_manager.cc:5697] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:04.806068  9373 catalog_manager.cc:5697] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:04.806923  9817 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:49:04.790243  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48312us
I20260808 01:49:04.808892  9817 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.816233  9817 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:04.817507  9682 tablet_bootstrap.cc:492] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:04.817915  9682 ts_tablet_manager.cc:1403] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.332s	user 0.030s	sys 0.021s
I20260808 01:49:04.827097  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:04.843595  9558 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:04.844976  9684 tablet_bootstrap.cc:654] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:04.848905  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:04.839087  9760 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:04.849743  9637 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:04.850135  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:04.851169  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:04.851426  9741 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:04.852098  9741 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:04.853245  9741 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.854439  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:04.857225  9682 raft_consensus.cc:359] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.858311  9682 raft_consensus.cc:385] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:04.866892  9682 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
W20260808 01:49:04.877405  9601 leader_election.cc:343] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:04.884434  9817 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:04.885099  9817 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:04.885599  9817 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:49:04.893896  9426 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:49:04.904950  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:04.903692  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:49:04.910091  9521 leader_election.cc:343] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:04.911538  9682 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.910044  9479 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:49:04.919720  9444 leader_election.cc:343] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:04.935642  9682 ts_tablet_manager.cc:1434] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.118s	user 0.011s	sys 0.000s
I20260808 01:49:04.947782  9682 tablet_bootstrap.cc:492] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:04.970098  9817 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.973759  9688 tablet_bootstrap.cc:654] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:04.985452  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:04.986378  9558 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:04.990612  9918 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:04.993462  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:04.994220  9635 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:04.997448  9479 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:05.001286  9524 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:05.002899  9817 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:05.004906  9760 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:05.005897  9760 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:05.007274  9760 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:04.991469  9918 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.016707  9918 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:05.020450  9635 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:05.034662  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:05.037245  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:05.038265  9817 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:05.038911  9817 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:05.040514  9817 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.043463  9684 tablet_bootstrap.cc:492] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:05.043985  9684 ts_tablet_manager.cc:1403] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.411s	user 0.016s	sys 0.008s
I20260808 01:49:05.047432  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:05.044008  9372 catalog_manager.cc:5697] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:05.051443  9684 raft_consensus.cc:359] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.053580  9888 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:05.058107  9558 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
W20260808 01:49:05.063017  9444 leader_election.cc:343] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:05.062364  9684 raft_consensus.cc:385] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:05.064221  9684 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:05.067830  9688 tablet_bootstrap.cc:492] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:05.068302  9688 ts_tablet_manager.cc:1403] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.282s	user 0.008s	sys 0.034s
I20260808 01:49:05.074451  9688 raft_consensus.cc:359] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.075382  9688 raft_consensus.cc:385] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:05.075654  9688 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:05.076599  9688 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.070353  9684 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.089584  9688 ts_tablet_manager.cc:1434] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.003s
W20260808 01:49:05.091097  9444 leader_election.cc:343] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:05.101653  9684 ts_tablet_manager.cc:1434] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.057s	user 0.007s	sys 0.003s
I20260808 01:49:05.102125  9688 tablet_bootstrap.cc:492] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:05.104959  9682 tablet_bootstrap.cc:654] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:05.091835  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:05.108377  9817 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:05.114365  9372 catalog_manager.cc:5697] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:05.116765  9777 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260808 01:49:05.091351  9760 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20260808 01:49:05.123173  9374 catalog_manager.cc:5697] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:05.128053  9684 tablet_bootstrap.cc:492] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:05.132527  9777 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:05.179924  9688 tablet_bootstrap.cc:654] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:05.182746  9684 tablet_bootstrap.cc:654] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:05.193413  9777 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260808 01:49:05.214329  9599 consensus_peers.cc:597] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:05.214936  9682 tablet_bootstrap.cc:492] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:05.215556  9682 ts_tablet_manager.cc:1403] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.273s	user 0.020s	sys 0.004s
I20260808 01:49:05.220889  9682 raft_consensus.cc:359] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.221732  9682 raft_consensus.cc:385] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:05.222029  9682 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:05.222775  9682 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.231844  9682 ts_tablet_manager.cc:1434] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.002s
I20260808 01:49:05.234993  9688 tablet_bootstrap.cc:492] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:05.235574  9682 tablet_bootstrap.cc:492] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:05.235677  9688 ts_tablet_manager.cc:1403] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.134s	user 0.014s	sys 0.016s
I20260808 01:49:05.244079  9688 raft_consensus.cc:359] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.244979  9688 raft_consensus.cc:385] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:05.245268  9688 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:05.246134  9688 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.248571  9688 ts_tablet_manager.cc:1434] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.003s
I20260808 01:49:05.256767  9688 tablet_bootstrap.cc:492] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:05.265290  9688 tablet_bootstrap.cc:654] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:05.273362  9684 tablet_bootstrap.cc:492] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:05.273900  9684 ts_tablet_manager.cc:1403] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.168s	user 0.016s	sys 0.007s
I20260808 01:49:05.278937  9684 raft_consensus.cc:359] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.279939  9684 raft_consensus.cc:385] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:05.280299  9684 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:05.281132  9684 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.310527  9684 ts_tablet_manager.cc:1434] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.036s	user 0.005s	sys 0.005s
I20260808 01:49:05.311674  9684 tablet_bootstrap.cc:492] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:05.317716  9688 tablet_bootstrap.cc:492] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:05.318547  9688 ts_tablet_manager.cc:1403] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.063s	user 0.023s	sys 0.021s
I20260808 01:49:05.321972  9777 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:05.322669  9777 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.323484  9688 raft_consensus.cc:359] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.324337  9688 raft_consensus.cc:385] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:05.324659  9688 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:05.325519  9688 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.328212  9688 ts_tablet_manager.cc:1434] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260808 01:49:05.329589  9688 tablet_bootstrap.cc:492] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:05.331163  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:05.333428  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:05.334721  9476 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:05.337284  9919 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:05.338366  9477 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:05.338204  9919 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.338992  9836 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:05.339797  9836 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.340608  9479 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:05.332033  9558 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:05.325979  9777 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:05.343426  9599 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:05.346454  9836 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:05.345161  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:05.349241  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:05.351012  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:05.347803  9836 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:05.351455  9688 tablet_bootstrap.cc:654] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:05.352097  9836 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:05.352836  9836 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:49:05.357897  9601 leader_election.cc:343] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:05.351460  9684 tablet_bootstrap.cc:654] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:05.356751  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:05.362974  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:05.368001  9919 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:05.369305  9599 leader_election.cc:343] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:05.369722  9836 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.369971  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:05.372190  9836 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:05.376765  9836 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:05.381259  9688 tablet_bootstrap.cc:492] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:05.382155  9688 ts_tablet_manager.cc:1403] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.053s	user 0.015s	sys 0.011s
I20260808 01:49:05.388860  9688 raft_consensus.cc:359] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.390012  9688 raft_consensus.cc:385] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:05.390408  9688 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:05.391312  9688 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.393201  9836 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:05.395074  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:05.395848  9481 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:05.402906  9882 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:05.405390  9688 ts_tablet_manager.cc:1434] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.023s	user 0.010s	sys 0.000s
I20260808 01:49:05.407169  9688 tablet_bootstrap.cc:492] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:05.413339  9682 tablet_bootstrap.cc:654] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:05.425765  9777 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:05.426965  9777 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.435873  9777 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:05.441135  9688 tablet_bootstrap.cc:654] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:05.444968  9684 tablet_bootstrap.cc:492] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:05.448093  9817 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:05.452733  9684 ts_tablet_manager.cc:1403] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.141s	user 0.020s	sys 0.018s
I20260808 01:49:05.449965  9481 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:05.456691  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:05.462301  9684 raft_consensus.cc:359] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.463291  9684 raft_consensus.cc:385] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:05.463609  9684 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:05.464737  9817 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:05.465791  9684 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.433533  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:05.480877  9684 ts_tablet_manager.cc:1434] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.028s	user 0.005s	sys 0.005s
I20260808 01:49:05.492303  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:05.478561  9558 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:05.514483  9741 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:05.515647  9741 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:05.513000  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:05.517089  9479 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 1: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:49:05.520354  9601 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:05.529357  9777 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:05.530169  9558 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:05.517465  9741 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.546800  9684 tablet_bootstrap.cc:492] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:05.593053  9682 tablet_bootstrap.cc:492] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:05.593685  9682 ts_tablet_manager.cc:1403] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.359s	user 0.040s	sys 0.086s
I20260808 01:49:05.596736  9682 raft_consensus.cc:359] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.597316  9682 raft_consensus.cc:385] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:05.597592  9682 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:05.598392  9682 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.604607  9688 tablet_bootstrap.cc:492] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:05.605233  9688 ts_tablet_manager.cc:1403] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.198s	user 0.009s	sys 0.018s
I20260808 01:49:05.612747  9777 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:05.614856  9682 ts_tablet_manager.cc:1434] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.021s	user 0.009s	sys 0.000s
I20260808 01:49:05.621465  9682 tablet_bootstrap.cc:492] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:05.622660  9688 raft_consensus.cc:359] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.623435  9688 raft_consensus.cc:385] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:05.623713  9688 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:05.624429  9688 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.626739  9688 ts_tablet_manager.cc:1434] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.003s
I20260808 01:49:05.630028  9688 tablet_bootstrap.cc:492] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:05.635164  9684 tablet_bootstrap.cc:654] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:05.634989  9760 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:05.619071  9918 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:05.637403  9918 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.670635  9682 tablet_bootstrap.cc:654] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:05.672195  9918 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:05.685329  9918 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:05.687748  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:49:05.696542  9444 leader_election.cc:343] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:05.684564  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:05.705011  9688 tablet_bootstrap.cc:654] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
W20260808 01:49:05.711628  9444 leader_election.cc:343] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:05.712285  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:05.713804  9817 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:49:05.755415  9948 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260808 01:49:05.778120  9372 catalog_manager.cc:5697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:05.799762  9684 tablet_bootstrap.cc:492] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:05.800514  9684 ts_tablet_manager.cc:1403] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.259s	user 0.021s	sys 0.004s
I20260808 01:49:05.812072  9682 tablet_bootstrap.cc:492] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:05.813063  9682 ts_tablet_manager.cc:1403] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.192s	user 0.014s	sys 0.011s
I20260808 01:49:05.829715  9882 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:05.830343  9882 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.831938  9684 raft_consensus.cc:359] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.832535  9882 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:05.832713  9684 raft_consensus.cc:385] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:05.832993  9684 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:05.833817  9684 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.838060  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:05.838876  9559 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:05.840065  9684 ts_tablet_manager.cc:1434] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.039s	user 0.008s	sys 0.000s
I20260808 01:49:05.840296  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:05.841157  9958 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:05.844048  9836 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:05.844476  9836 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:05.844834  9836 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:05.851295  9682 raft_consensus.cc:359] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.852391  9682 raft_consensus.cc:385] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:05.852744  9682 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:05.853677  9682 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.891232  9682 ts_tablet_manager.cc:1434] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.078s	user 0.008s	sys 0.000s
I20260808 01:49:05.893501  9682 tablet_bootstrap.cc:492] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:05.894431  9684 tablet_bootstrap.cc:492] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:05.906142  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:05.929780  9836 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.950317  9760 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:05.951053  9760 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.953019  9836 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:05.953956  9760 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:05.955121  9688 tablet_bootstrap.cc:492] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:05.955627  9688 ts_tablet_manager.cc:1403] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.326s	user 0.024s	sys 0.046s
W20260808 01:49:05.956303  9601 leader_election.cc:343] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:05.959589  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:05.954597  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:05.956629  9882 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:05.962539  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:05.968477  9521 leader_election.cc:343] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:05.968952  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:05.963049  9559 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:05.965147  9882 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:05.974747  9601 leader_election.cc:343] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:49:05.976105  9524 leader_election.cc:343] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:05.976603  9524 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:05.977381  9688 raft_consensus.cc:359] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.978168  9688 raft_consensus.cc:385] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:05.978453  9688 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:05.978474  9888 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:05.979274  9688 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:05.984261  9688 ts_tablet_manager.cc:1434] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.028s	user 0.006s	sys 0.001s
I20260808 01:49:05.985160  9882 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:05.985378  9688 tablet_bootstrap.cc:492] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:05.989481  9684 tablet_bootstrap.cc:654] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:05.990447  9559 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:05.990631  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:05.988538  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:05.995384  9560 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:05.999763  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:06.004863  9599 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:06.006415  9882 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
W20260808 01:49:06.011577  9601 leader_election.cc:343] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:06.019549  9777 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:06.020085  9777 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:06.020675  9777 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:06.011432  9882 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:06.044140  9882 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.047703  9682 tablet_bootstrap.cc:654] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:06.082429  9684 tablet_bootstrap.cc:492] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:06.083045  9684 ts_tablet_manager.cc:1403] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.202s	user 0.016s	sys 0.006s
I20260808 01:49:06.088038  9777 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.095129  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:06.098181  9684 raft_consensus.cc:359] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.100310  9684 raft_consensus.cc:385] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:06.100639  9684 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:06.091310  9777 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:06.111200  9684 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:06.111698  9601 leader_election.cc:343] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:06.118520  9945 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:06.118971  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:06.119611  9559 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:06.121093  9684 ts_tablet_manager.cc:1434] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.038s	user 0.011s	sys 0.000s
I20260808 01:49:06.122279  9684 tablet_bootstrap.cc:492] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:06.127420  9559 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:06.134423  9599 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:06.136531  9741 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:06.145896  9688 tablet_bootstrap.cc:654] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:06.141381  9741 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:06.162361  9882 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:06.152717  9741 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.170071  9684 tablet_bootstrap.cc:654] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:06.176911  9682 tablet_bootstrap.cc:492] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:06.177469  9682 ts_tablet_manager.cc:1403] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.284s	user 0.016s	sys 0.014s
I20260808 01:49:06.184268  9952 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:06.185055  9952 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.188556  9919 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:06.191238  9372 catalog_manager.cc:5697] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:06.193341  9919 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.209266  9682 raft_consensus.cc:359] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.210029  9682 raft_consensus.cc:385] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:06.210317  9682 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:06.211359  9682 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.206353  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:06.217289  9682 ts_tablet_manager.cc:1434] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.040s	user 0.007s	sys 0.002s
I20260808 01:49:06.221478  9682 tablet_bootstrap.cc:492] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:06.222080  9688 tablet_bootstrap.cc:492] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:06.222625  9688 ts_tablet_manager.cc:1403] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.237s	user 0.028s	sys 0.022s
W20260808 01:49:06.231501  9970 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260808 01:49:06.225411  9560 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:06.238590  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:49:06.240411  9444 leader_election.cc:343] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:06.244974  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:06.232012  9952 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:06.246243  9638 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:06.261894  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:06.250809  9688 raft_consensus.cc:359] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.265479  9688 raft_consensus.cc:385] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:06.265920  9688 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:06.266891  9688 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.270745  9817 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:06.280287  9688 ts_tablet_manager.cc:1434] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.057s	user 0.011s	sys 0.000s
I20260808 01:49:06.288086  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:06.290467  9684 tablet_bootstrap.cc:492] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:06.291093  9684 ts_tablet_manager.cc:1403] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.169s	user 0.011s	sys 0.036s
I20260808 01:49:06.294193  9684 raft_consensus.cc:359] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.294999  9684 raft_consensus.cc:385] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:06.295352  9684 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
W20260808 01:49:06.295606  9444 leader_election.cc:343] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:06.296206  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:06.296053  9684 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.295075  9682 tablet_bootstrap.cc:654] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:06.277392  9919 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:06.303844  9688 tablet_bootstrap.cc:492] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:06.280484  9817 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:06.305963  9817 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:06.297523  9952 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:06.317770  9374 catalog_manager.cc:5697] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:06.329317  9688 tablet_bootstrap.cc:654] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:06.318708  9741 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:06.330986  9741 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.333735  9741 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:06.346931  9684 ts_tablet_manager.cc:1434] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.056s	user 0.011s	sys 0.012s
I20260808 01:49:06.348405  9684 tablet_bootstrap.cc:492] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:06.359711  9682 tablet_bootstrap.cc:492] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:06.360476  9682 ts_tablet_manager.cc:1403] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.139s	user 0.010s	sys 0.009s
I20260808 01:49:06.361665  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:06.363433  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:06.365063  9688 tablet_bootstrap.cc:492] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:06.371491  9741 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:06.373315  9688 ts_tablet_manager.cc:1403] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.076s	user 0.011s	sys 0.014s
I20260808 01:49:06.372488  9741 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:06.374662  9599 leader_election.cc:343] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:06.377954  9741 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:49:06.379330  9601 leader_election.cc:343] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:06.379995  9601 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:49:06.381812  9426 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:49:06.384707  9777 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:06.385842  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:06.387970  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:06.389775  9599 leader_election.cc:343] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:49:06.390580  9601 leader_election.cc:343] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:06.391101  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:06.388015  9682 raft_consensus.cc:359] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.392462  9684 tablet_bootstrap.cc:654] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:06.392671  9682 raft_consensus.cc:385] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:06.393110  9682 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:06.393956  9777 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:06.396067  9817 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.398176  9688 raft_consensus.cc:359] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.398973  9688 raft_consensus.cc:385] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:06.398649  9682 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.399255  9688 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:06.402629  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:06.404273  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:06.403268  9817 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:06.405372  9638 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:06.405660  9688 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.406394  9560 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:06.409759  9682 ts_tablet_manager.cc:1434] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.049s	user 0.011s	sys 0.008s
I20260808 01:49:06.411844  9682 tablet_bootstrap.cc:492] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:06.414055  9638 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:06.414340  9688 ts_tablet_manager.cc:1434] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.041s	user 0.011s	sys 0.000s
I20260808 01:49:06.415940  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:06.416883  9688 tablet_bootstrap.cc:492] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:06.418274  9684 tablet_bootstrap.cc:492] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:06.416903  9817 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:06.419864  9817 raft_consensus.cc:697] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:06.421027  9684 ts_tablet_manager.cc:1403] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.073s	user 0.011s	sys 0.015s
I20260808 01:49:06.425289  9682 tablet_bootstrap.cc:654] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:06.421409  9817 consensus_queue.cc:237] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.429685  9684 raft_consensus.cc:359] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.442859  9684 raft_consensus.cc:385] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:06.443451  9684 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:06.444631  9684 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.442302  9688 tablet_bootstrap.cc:654] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:06.460084  9684 ts_tablet_manager.cc:1434] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.038s	user 0.008s	sys 0.003s
I20260808 01:49:06.476816  9684 tablet_bootstrap.cc:492] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:06.484968  9682 tablet_bootstrap.cc:492] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:06.485607  9682 ts_tablet_manager.cc:1403] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.074s	user 0.023s	sys 0.031s
I20260808 01:49:06.499609  9560 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:06.497155  9374 catalog_manager.cc:5697] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:06.505173  9684 tablet_bootstrap.cc:654] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:06.505338  9682 raft_consensus.cc:359] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.506120  9682 raft_consensus.cc:385] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:06.506600  9682 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:06.507920  9682 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.510398  9682 ts_tablet_manager.cc:1434] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.024s	user 0.009s	sys 0.001s
I20260808 01:49:06.522204  9682 tablet_bootstrap.cc:492] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:06.548142  9684 tablet_bootstrap.cc:492] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:06.551715  9688 tablet_bootstrap.cc:492] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:06.552486  9688 ts_tablet_manager.cc:1403] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.136s	user 0.022s	sys 0.015s
I20260808 01:49:06.552681  9684 ts_tablet_manager.cc:1403] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.077s	user 0.019s	sys 0.031s
I20260808 01:49:06.557149  9688 raft_consensus.cc:359] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.558055  9688 raft_consensus.cc:385] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:06.558377  9688 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:06.559403  9688 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.562175  9682 tablet_bootstrap.cc:654] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:06.562839  9684 raft_consensus.cc:359] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.563572  9684 raft_consensus.cc:385] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:06.563843  9684 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:06.564786  9688 ts_tablet_manager.cc:1434] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.003s
I20260808 01:49:06.564636  9684 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.566187  9688 tablet_bootstrap.cc:492] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:06.573868  9684 ts_tablet_manager.cc:1434] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.021s	user 0.002s	sys 0.006s
I20260808 01:49:06.585620  9684 tablet_bootstrap.cc:492] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:06.594053  9477 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:06.605223  9684 tablet_bootstrap.cc:654] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:06.613488  9682 tablet_bootstrap.cc:492] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:06.614665  9682 ts_tablet_manager.cc:1403] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.093s	user 0.018s	sys 0.030s
I20260808 01:49:06.616510  9688 tablet_bootstrap.cc:654] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:06.619169  9882 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:06.620590  9682 raft_consensus.cc:359] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.621512  9682 raft_consensus.cc:385] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:06.621939  9682 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:06.623334  9682 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.623641  9741 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:06.627592  9682 ts_tablet_manager.cc:1434] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.004s
I20260808 01:49:06.642616  9682 tablet_bootstrap.cc:492] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:06.675541  9688 tablet_bootstrap.cc:492] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:06.676187  9688 ts_tablet_manager.cc:1403] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.110s	user 0.011s	sys 0.026s
I20260808 01:49:06.679448  9919 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:06.681568  9919 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.682343  9684 tablet_bootstrap.cc:492] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:06.682893  9684 ts_tablet_manager.cc:1403] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.098s	user 0.017s	sys 0.019s
I20260808 01:49:06.685676  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:06.686452  9559 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:06.687305  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:06.689321  9684 raft_consensus.cc:359] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.692178  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:06.692308  9638 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:06.692631  9684 raft_consensus.cc:385] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:06.692924  9684 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:06.693826  9684 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.694262  9979 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:06.694759  9979 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:06.695133  9979 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:06.696645  9684 ts_tablet_manager.cc:1434] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260808 01:49:06.697621  9684 tablet_bootstrap.cc:492] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:06.684598  9919 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:06.705157  9688 raft_consensus.cc:359] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.710727  9688 raft_consensus.cc:385] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:06.711102  9688 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:06.712302  9682 tablet_bootstrap.cc:654] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:06.714035  9688 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:06.723647  9601 consensus_peers.cc:597] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:06.725060  9836 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:06.725598  9836 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.731730  9760 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:06.732337  9760 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.734015  9688 ts_tablet_manager.cc:1434] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.057s	user 0.007s	sys 0.002s
I20260808 01:49:06.736582  9760 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:06.739773  9688 tablet_bootstrap.cc:492] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:06.736042  9979 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.741394  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:06.742255  9638 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:06.744824  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:06.745862  9481 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:06.747139  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:06.748304  9560 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:06.750872  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:06.754408  9521 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:06.755610  9968 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:06.756168  9968 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:06.756757  9968 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:06.753439  9836 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:06.754146  9947 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:06.759531  9947 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:06.760062  9947 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:06.762285  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:06.767176  9479 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:06.765491  9945 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:06.777045  9688 tablet_bootstrap.cc:654] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:06.777539  9684 tablet_bootstrap.cc:654] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:06.787392  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:06.788597  9559 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:06.804679  9979 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:06.808676  9559 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:06.811218  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:06.814154  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:06.814896  9638 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:06.814381  9915 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:06.817903  9979 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:06.829293  9682 tablet_bootstrap.cc:492] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:06.829880  9682 ts_tablet_manager.cc:1403] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.201s	user 0.025s	sys 0.009s
I20260808 01:49:06.826030  9915 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.837574  9915 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:06.839015  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:06.839920  9635 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:06.841089  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:06.841727  9947 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.842200  9477 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:06.843695  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:06.844432  9968 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.831353  9979 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:06.852352  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:06.853711  9968 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:06.850018  9682 raft_consensus.cc:359] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.855036  9682 raft_consensus.cc:385] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:06.855194  9952 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:06.856345  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:06.855995  9688 tablet_bootstrap.cc:492] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:06.858603  9682 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:06.859354  9688 ts_tablet_manager.cc:1403] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.120s	user 0.018s	sys 0.017s
I20260808 01:49:06.861508  9638 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:06.856707  9952 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.865763  9635 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 1: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:49:06.865263  9682 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.857616  9760 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:06.857442  9477 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:06.869936  9760 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:06.874085  9760 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:06.852413  9979 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.875901  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:06.879925  9684 tablet_bootstrap.cc:492] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:06.880622  9684 ts_tablet_manager.cc:1403] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.183s	user 0.013s	sys 0.018s
I20260808 01:49:06.889694  9760 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.890565  9952 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:06.893800  9760 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:06.895464  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:06.896610  9635 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:06.898373  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:06.899380  9952 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:06.900084  9952 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:06.900722  9952 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:06.901074  9682 ts_tablet_manager.cc:1434] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.071s	user 0.007s	sys 0.002s
I20260808 01:49:06.902251  9682 tablet_bootstrap.cc:492] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:06.903113  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:06.904167  9481 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:06.913837  9684 raft_consensus.cc:359] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.916440  9997 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:06.916770  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:06.917333  9997 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.917757  9688 raft_consensus.cc:359] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:06.918593  9444 leader_election.cc:343] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:06.919085  9688 raft_consensus.cc:385] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:06.919488  9688 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:06.915271  9968 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:06.921424  9968 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.920367  9688 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.922224  9997 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:06.923623  9968 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:06.915567  9684 raft_consensus.cc:385] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260808 01:49:06.931697  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 61271.6us
I20260808 01:49:06.932327  9684 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:06.933118  9684 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:06.943915  9684 ts_tablet_manager.cc:1434] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.063s	user 0.005s	sys 0.006s
I20260808 01:49:06.935626  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:06.930539  9688 ts_tablet_manager.cc:1434] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.071s	user 0.004s	sys 0.004s
I20260808 01:49:06.953763  9638 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:06.960709  9688 tablet_bootstrap.cc:492] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:06.964246  9481 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:06.968809  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:06.973564  9636 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:06.976667  9638 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:06.971669  9684 tablet_bootstrap.cc:492] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:06.977420  9521 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:06.979525  9373 catalog_manager.cc:5697] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:06.982023 10001 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:06.982941  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:06.988574  9919 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:06.991192 10001 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:06.992199  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:06.960189  9947 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:06.997931  9558 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 1: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:49:07.008697  9477 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:06.996721 10001 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:07.018843  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:07.021010  9945 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:49:07.015082  9919 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260808 01:49:07.022396  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:07.023514  9481 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:06.979490  9952 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.029127  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:07.043720  9915 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:07.046797  9915 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:07.045797  9919 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:07.059674  9915 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.064237  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:07.060063  9919 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:07.067492  9444 leader_election.cc:343] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:07.066817  9688 tablet_bootstrap.cc:654] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:07.061133  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:07.096170  9952 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:07.107546  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:07.121199  9919 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:07.137866  9760 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:07.138329  9682 tablet_bootstrap.cc:654] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:07.138820  9760 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:07.145532  9836 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:07.147018  9836 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.142069  9760 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.155200  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:07.157874 10001 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:07.158974  9599 leader_election.cc:343] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:49:07.162424  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 55060.2us
I20260808 01:49:07.164368  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:07.165390 10001 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:07.165570  9478 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:07.166846  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:07.167455  9638 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:07.153311  9836 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:49:07.173408  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 83632.4us
I20260808 01:49:07.175352  9601 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:07.175799  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:49:07.180531  9444 leader_election.cc:343] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:07.184523  9638 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:07.185899  9521 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:07.183768  9836 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:07.190827  9997 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:07.191901  9836 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:07.192205  9997 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:07.192409  9836 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:07.195006  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:07.206897  9688 tablet_bootstrap.cc:492] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:07.207252  9684 tablet_bootstrap.cc:654] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:07.207396  9688 ts_tablet_manager.cc:1403] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.249s	user 0.011s	sys 0.025s
I20260808 01:49:07.211735  9559 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:07.216938  9372 catalog_manager.cc:5697] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:07.221696  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:07.222538  9636 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:07.193286  9997 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.219712  9524 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:49:07.261880  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 45784.5us
I20260808 01:49:07.275982  9688 raft_consensus.cc:359] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.276844  9688 raft_consensus.cc:385] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:07.277133  9688 raft_consensus.cc:740] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:07.278533  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:07.279349  9481 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:07.277918  9688 consensus_queue.cc:260] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.292160  9688 ts_tablet_manager.cc:1434] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.084s	user 0.010s	sys 0.000s
I20260808 01:49:07.314783  9636 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:07.319550  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:07.321154  9919 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:07.322129  9919 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:07.330770  9915 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:07.323573  9919 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.337431  9481 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:07.336318  9836 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.341039  9638 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 1: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:49:07.356035  9682 tablet_bootstrap.cc:492] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:07.356622  9682 ts_tablet_manager.cc:1403] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.455s	user 0.011s	sys 0.039s
I20260808 01:49:07.356510  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:07.357522  9479 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:07.358928  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:07.360898  9682 raft_consensus.cc:359] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.361629  9682 raft_consensus.cc:385] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:07.361940  9682 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
W20260808 01:49:07.361641  9599 leader_election.cc:343] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:07.367537  9836 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:07.362042  9444 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:07.372162  9688 tablet_bootstrap.cc:492] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:07.372704  9682 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.375229  9682 ts_tablet_manager.cc:1434] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.003s
I20260808 01:49:07.380053  9684 tablet_bootstrap.cc:492] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:07.380643  9684 ts_tablet_manager.cc:1403] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.409s	user 0.019s	sys 0.007s
I20260808 01:49:07.385885  9684 raft_consensus.cc:359] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.386662 10005 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:07.395700  9684 raft_consensus.cc:385] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:07.398525  9684 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:07.399422  9684 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.405330  9682 tablet_bootstrap.cc:492] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:07.406167  9741 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:07.406946  9741 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.412700  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:07.402159  9374 catalog_manager.cc:5697] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
W20260808 01:49:07.416451  9599 leader_election.cc:343] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:07.418350  9684 ts_tablet_manager.cc:1434] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.037s	user 0.007s	sys 0.003s
I20260808 01:49:07.418545  9374 catalog_manager.cc:5697] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:07.426177  9479 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:07.427716  9978 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:07.429095  9601 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:49:07.433401  9426 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:49:07.434507  9945 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:07.435700  9945 raft_consensus.cc:697] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:07.437624  9945 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.417289  9741 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:07.507025  9477 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:07.536538  9688 tablet_bootstrap.cc:654] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:07.544296  9682 tablet_bootstrap.cc:654] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:07.545987  9915 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:07.550000  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:07.553930  9684 tablet_bootstrap.cc:492] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
W20260808 01:49:07.570947  9601 leader_election.cc:343] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:07.571759  9601 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:07.576790  9945 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:07.584010  9372 catalog_manager.cc:5697] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:07.584394  9817 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:07.581130 10000 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:07.607626  9684 tablet_bootstrap.cc:654] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:07.631009 10000 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260808 01:49:07.642074  9688 tablet_bootstrap.cc:492] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:07.642833  9688 ts_tablet_manager.cc:1403] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.271s	user 0.039s	sys 0.016s
I20260808 01:49:07.684206  9688 raft_consensus.cc:359] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.685271  9688 raft_consensus.cc:385] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:07.685671  9688 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:07.686576  9688 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.699141  9688 ts_tablet_manager.cc:1434] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.056s	user 0.007s	sys 0.004s
I20260808 01:49:07.690732  9915 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260808 01:49:07.690521 10001 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260808 01:49:07.717718  9688 tablet_bootstrap.cc:492] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:07.704195  9682 tablet_bootstrap.cc:492] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:07.727349  9682 ts_tablet_manager.cc:1403] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.326s	user 0.010s	sys 0.022s
I20260808 01:49:07.738595  9997 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:07.739823  9997 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.750581  9373 catalog_manager.cc:5697] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:07.763103  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:07.762789  9682 raft_consensus.cc:359] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.765352  9682 raft_consensus.cc:385] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:07.765987  9682 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:07.766062  9997 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:07.767211  9682 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.771114  9817 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:07.782944  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:07.788024  9836 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:07.789134  9836 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.807893  9682 ts_tablet_manager.cc:1434] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.080s	user 0.010s	sys 0.001s
I20260808 01:49:07.810920  9521 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:07.812068  9560 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:07.822702  9836 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:07.831161  9682 tablet_bootstrap.cc:492] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:07.834197 10001 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:07.881521  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:07.893678  9688 tablet_bootstrap.cc:654] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:07.900964  9684 tablet_bootstrap.cc:492] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:07.901517  9684 ts_tablet_manager.cc:1403] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.348s	user 0.021s	sys 0.017s
I20260808 01:49:07.907959  9684 raft_consensus.cc:359] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.908661  9684 raft_consensus.cc:385] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:07.908981  9684 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:07.909641  9684 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.892700  9557 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:07.917303  9684 ts_tablet_manager.cc:1434] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.000s
I20260808 01:49:07.927039  9599 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:07.927699  9684 tablet_bootstrap.cc:492] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:07.930635  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:07.932391  9481 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:07.954236  9688 tablet_bootstrap.cc:492] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:07.954905  9688 ts_tablet_manager.cc:1403] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.238s	user 0.016s	sys 0.027s
I20260808 01:49:07.932694 10021 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:07.964221  9688 raft_consensus.cc:359] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.972550  9688 raft_consensus.cc:385] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:07.972910  9688 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:07.985033  9688 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:07.993842  9688 ts_tablet_manager.cc:1434] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.039s	user 0.007s	sys 0.002s
W20260808 01:49:07.999245  9599 consensus_peers.cc:597] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:07.985327 10021 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:08.003621  9688 tablet_bootstrap.cc:492] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:08.002943 10021 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:08.026300  9968 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:49:08.033126  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50587.8us
I20260808 01:49:08.057410  9682 tablet_bootstrap.cc:654] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:08.060012  9978 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:08.067296  9684 tablet_bootstrap.cc:654] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:08.071211 10021 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.087555  9688 tablet_bootstrap.cc:654] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:08.091892 10021 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:08.093456  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:08.094247  9480 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:08.093231  9945 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:08.104770  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:08.105151  9684 tablet_bootstrap.cc:492] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:08.105494  9560 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:08.105625  9684 ts_tablet_manager.cc:1403] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.178s	user 0.008s	sys 0.016s
I20260808 01:49:08.108080  9480 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:08.108701  9684 raft_consensus.cc:359] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.109520  9684 raft_consensus.cc:385] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:08.109813  9684 raft_consensus.cc:740] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:08.111575  9601 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:08.112394 10031 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:08.119493 10031 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:08.120468 10031 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.119305  9684 consensus_queue.cc:260] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.133555  9684 ts_tablet_manager.cc:1434] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.028s	user 0.006s	sys 0.002s
I20260808 01:49:08.143605  9684 tablet_bootstrap.cc:492] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:08.156724  9688 tablet_bootstrap.cc:492] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:08.157306  9688 ts_tablet_manager.cc:1403] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.159s	user 0.014s	sys 0.001s
I20260808 01:49:08.161800  9684 tablet_bootstrap.cc:654] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:08.167754  9560 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:08.168686 10031 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:08.168808  9682 tablet_bootstrap.cc:492] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:08.169585  9682 ts_tablet_manager.cc:1403] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.339s	user 0.014s	sys 0.025s
I20260808 01:49:08.169260 10031 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.178646  9688 raft_consensus.cc:359] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.179517  9688 raft_consensus.cc:385] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:08.179800  9688 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:08.180487  9688 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.183171  9688 ts_tablet_manager.cc:1434] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.026s	user 0.009s	sys 0.000s
I20260808 01:49:08.184275  9688 tablet_bootstrap.cc:492] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:08.197662  9377 catalog_manager.cc:5697] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:08.201586 10031 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:08.204095  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:08.205672  9688 tablet_bootstrap.cc:654] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
W20260808 01:49:08.206005  9599 leader_election.cc:343] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:08.210027  9684 tablet_bootstrap.cc:492] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:08.210939  9684 ts_tablet_manager.cc:1403] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.068s	user 0.009s	sys 0.009s
I20260808 01:49:08.215207  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:08.216586  9682 raft_consensus.cc:359] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.217648  9682 raft_consensus.cc:385] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:08.218009  9682 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:08.219235  9682 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:08.225354  9601 leader_election.cc:343] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:08.225884  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:08.227684 10021 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:08.228123  9688 tablet_bootstrap.cc:492] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:08.228610  9688 ts_tablet_manager.cc:1403] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.045s	user 0.015s	sys 0.013s
I20260808 01:49:08.227638  9684 raft_consensus.cc:359] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.229138  9684 raft_consensus.cc:385] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:08.229439  9684 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:08.230841  9684 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.231948  9688 raft_consensus.cc:359] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.232719  9688 raft_consensus.cc:385] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:08.233134  9684 ts_tablet_manager.cc:1434] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.000s
I20260808 01:49:08.232980  9688 raft_consensus.cc:740] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:08.234274  9684 tablet_bootstrap.cc:492] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:08.234891  9688 consensus_queue.cc:260] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.237648  9688 ts_tablet_manager.cc:1434] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260808 01:49:08.237943  9682 ts_tablet_manager.cc:1434] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.068s	user 0.010s	sys 0.001s
I20260808 01:49:08.238736  9688 tablet_bootstrap.cc:492] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:08.238967  9682 tablet_bootstrap.cc:492] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:08.248797  9688 tablet_bootstrap.cc:654] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:08.255226 10028 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:08.256206 10028 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.259770  9684 tablet_bootstrap.cc:654] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:08.260937 10028 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:08.261252  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:08.268132  9682 tablet_bootstrap.cc:654] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:08.281213  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:08.293581  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:08.304576  9688 tablet_bootstrap.cc:492] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:08.305373 10028 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:08.306761  9684 tablet_bootstrap.cc:492] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:08.307421  9684 ts_tablet_manager.cc:1403] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.073s	user 0.012s	sys 0.038s
I20260808 01:49:08.309616  9682 tablet_bootstrap.cc:492] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:08.310141  9682 ts_tablet_manager.cc:1403] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.071s	user 0.008s	sys 0.010s
I20260808 01:49:08.310971  9688 ts_tablet_manager.cc:1403] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.073s	user 0.013s	sys 0.009s
I20260808 01:49:08.321095  9688 raft_consensus.cc:359] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.321846  9688 raft_consensus.cc:385] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:08.322165  9688 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:08.322866  9688 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.318249  9684 raft_consensus.cc:359] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.323838  9684 raft_consensus.cc:385] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:08.324137  9684 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:08.324854  9684 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.327639  9684 ts_tablet_manager.cc:1434] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.020s	user 0.009s	sys 0.000s
I20260808 01:49:08.328965  9684 tablet_bootstrap.cc:492] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:08.334524  9688 ts_tablet_manager.cc:1434] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.000s
I20260808 01:49:08.336516  9682 raft_consensus.cc:359] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.337450  9682 raft_consensus.cc:385] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:08.337769  9682 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:08.338590  9682 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.341735  9682 ts_tablet_manager.cc:1434] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.031s	user 0.007s	sys 0.003s
I20260808 01:49:08.342657  9688 tablet_bootstrap.cc:492] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:08.343086  9682 tablet_bootstrap.cc:492] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:08.352654  9682 tablet_bootstrap.cc:654] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:08.354306  9688 tablet_bootstrap.cc:654] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:08.357245  9817 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:08.358295  9817 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.366353  9817 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:08.366811  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:08.367368  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:08.368536  9632 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
W20260808 01:49:08.369015  9444 leader_election.cc:343] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:08.370396  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:08.371359  9978 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:08.372253  9978 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.372757  9817 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:08.373487  9817 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:08.377200  9978 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:08.382236  9684 tablet_bootstrap.cc:654] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:08.387887  9919 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:08.387566  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:08.388497  9919 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.389181  9632 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:08.391239  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:08.393054  9919 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:08.374439  9817 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:08.402225 10021 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:08.403306 10021 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.406322  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:08.407307  9559 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
W20260808 01:49:08.409049  9444 leader_election.cc:343] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:08.409772  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:08.410899  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:08.412204  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:08.415483  9633 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:08.414633  9919 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:08.418742  9919 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:08.419328  9919 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:08.423434 10021 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:08.439083  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:08.439949  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:08.440591  9559 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:08.442389  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:08.443699  9945 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:08.444440  9945 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:08.415344  9978 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:08.448885  9945 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:08.448690  9978 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:08.470337  9978 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:08.520371  9684 tablet_bootstrap.cc:492] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:08.521216  9684 ts_tablet_manager.cc:1403] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.193s	user 0.028s	sys 0.021s
I20260808 01:49:08.535964  9559 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:49:08.529247  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 115119us
I20260808 01:49:08.550351  9688 tablet_bootstrap.cc:492] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:08.551461  9688 ts_tablet_manager.cc:1403] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.209s	user 0.022s	sys 0.035s
W20260808 01:49:08.566120  9426 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:49:08.527693  9919 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.571609  9817 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.564597 10021 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:08.571875  9978 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.579553  9480 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 1: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:08.582207  9688 raft_consensus.cc:359] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.583350  9688 raft_consensus.cc:385] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:08.583707  9688 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:08.582827 10021 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.587023  9688 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.574653  9817 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:08.584115  9684 raft_consensus.cc:359] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.600265  9684 raft_consensus.cc:385] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:08.600673  9684 raft_consensus.cc:740] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:08.604207  9682 tablet_bootstrap.cc:492] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:08.604949  9682 ts_tablet_manager.cc:1403] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.262s	user 0.033s	sys 0.051s
I20260808 01:49:08.606931  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:08.601344  9684 consensus_queue.cc:260] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.587492 10021 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:08.621310  9684 ts_tablet_manager.cc:1434] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.100s	user 0.009s	sys 0.005s
I20260808 01:49:08.619093  9688 ts_tablet_manager.cc:1434] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.067s	user 0.006s	sys 0.004s
W20260808 01:49:08.621048  9601 leader_election.cc:343] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:08.626176  9688 tablet_bootstrap.cc:492] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:08.628325  9684 tablet_bootstrap.cc:492] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:08.630854  9978 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:08.648201  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 82846.6us
I20260808 01:49:08.651753  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:08.650408  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:08.658838  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:08.663506  9682 raft_consensus.cc:359] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.687274  9682 raft_consensus.cc:385] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:08.687623  9682 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:08.688464  9682 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.698472  9682 ts_tablet_manager.cc:1434] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.092s	user 0.009s	sys 0.004s
I20260808 01:49:08.675729  9633 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:49:08.695483  9444 leader_election.cc:343] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:49:08.700505  9444 leader_election.cc:343] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:08.678603  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:08.704159  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:49:08.706815  9599 leader_election.cc:343] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:08.707409  9599 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:08.708662 10031 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:08.706367  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:08.711167  9555 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:08.677938  9919 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:08.691147 10021 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:08.717661 10021 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.706300  9945 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:08.749996  9741 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:08.781522  9682 tablet_bootstrap.cc:492] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:08.779398  9637 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 1: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:49:08.786808  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:08.791172  9601 leader_election.cc:343] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:08.774798  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:08.777832 10032 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20260808 01:49:08.843072  9684 tablet_bootstrap.cc:654] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
W20260808 01:49:08.843924 10054 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260808 01:49:08.787151 10021 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:49:08.845170  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 80530us
I20260808 01:49:08.836721  9632 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:08.881456  9688 tablet_bootstrap.cc:654] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:08.869328  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:08.894097  9477 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 1: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:08.889225  9945 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:08.935222  9632 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:08.936908  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:08.944159  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:49:08.945740  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 86446.4us
I20260808 01:49:08.971858  9633 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:08.946894  9978 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:08.973807  9978 raft_consensus.cc:697] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:08.975370  9555 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:08.978259  9978 consensus_queue.cc:237] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.002458  9684 tablet_bootstrap.cc:492] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:09.002628  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:09.003743  9919 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:09.004395  9919 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:09.006172  9684 ts_tablet_manager.cc:1403] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.378s	user 0.015s	sys 0.016s
I20260808 01:49:09.005468  9919 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.004419  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:09.013404  9682 tablet_bootstrap.cc:654] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:09.014832  9688 tablet_bootstrap.cc:492] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:09.015458  9688 ts_tablet_manager.cc:1403] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.390s	user 0.021s	sys 0.022s
I20260808 01:49:09.016775 10048 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:09.019362 10048 raft_consensus.cc:697] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:09.020016  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:09.022291  9555 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 1: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:09.020646  9688 raft_consensus.cc:359] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.025956  9688 raft_consensus.cc:385] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:09.026249  9688 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:09.032135  9688 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.020196 10048 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.038851  9684 raft_consensus.cc:359] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.043816  9377 catalog_manager.cc:5697] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:09.046864  9684 raft_consensus.cc:385] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:09.041320  9688 ts_tablet_manager.cc:1434] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.001s
I20260808 01:49:09.048496  9684 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:09.049369  9684 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.051050  9688 tablet_bootstrap.cc:492] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
W20260808 01:49:09.060436  9599 leader_election.cc:343] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:09.061770  9682 tablet_bootstrap.cc:492] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:09.063035  9682 ts_tablet_manager.cc:1403] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.282s	user 0.012s	sys 0.006s
I20260808 01:49:09.063086  9684 ts_tablet_manager.cc:1434] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.057s	user 0.008s	sys 0.000s
I20260808 01:49:09.064343  9684 tablet_bootstrap.cc:492] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:09.065997  9682 raft_consensus.cc:359] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.062361  9599 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:09.067621  9682 raft_consensus.cc:385] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:09.071766 10032 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:09.073151  9682 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:09.074611  9682 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.071776  9688 tablet_bootstrap.cc:654] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:09.079705  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:09.083755 10058 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:49:09.078541 10032 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:09.089287 10032 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.092793  9682 ts_tablet_manager.cc:1434] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.029s	user 0.008s	sys 0.000s
I20260808 01:49:09.097194  9682 tablet_bootstrap.cc:492] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:09.098824  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:09.098903  9978 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:09.099601  9978 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:09.100610  9601 leader_election.cc:343] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:09.106897 10032 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:09.111308  9945 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:09.111852  9945 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.113847  9945 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:09.114765  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:09.115481  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:09.116014  9480 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:09.114786  9978 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:09.117182  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:09.117480  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:09.118114  9558 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:09.118255  9945 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:09.118659  9945 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:09.119004  9945 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:09.121656  9684 tablet_bootstrap.cc:654] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:09.124012  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:09.125684  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:09.127328  9444 leader_election.cc:343] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:49:09.128633  9444 leader_election.cc:343] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:09.129070  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:49:09.129209  9599 leader_election.cc:343] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:09.129861  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:09.130867  9741 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:09.133175 10053 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:09.141120  9945 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.146860  9688 tablet_bootstrap.cc:492] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:09.147585  9688 ts_tablet_manager.cc:1403] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.097s	user 0.014s	sys 0.001s
I20260808 01:49:09.149389  9945 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:09.150002  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:09.150595  9480 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:09.153008  9682 tablet_bootstrap.cc:654] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:09.159379  9480 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:09.159693  9684 tablet_bootstrap.cc:492] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:09.160817  9684 ts_tablet_manager.cc:1403] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.097s	user 0.013s	sys 0.004s
I20260808 01:49:09.160696  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:09.164057  9945 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:09.163944  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:09.165182  9684 raft_consensus.cc:359] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.165508  9555 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:09.169910  9688 raft_consensus.cc:359] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.170660  9688 raft_consensus.cc:385] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:09.165895  9684 raft_consensus.cc:385] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:09.164757  9945 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:09.174789  9684 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:09.175693  9684 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.170972  9688 raft_consensus.cc:740] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:09.174015  9945 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.177577  9688 consensus_queue.cc:260] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.180158  9684 ts_tablet_manager.cc:1434] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.019s	user 0.003s	sys 0.004s
I20260808 01:49:09.181363  9684 tablet_bootstrap.cc:492] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:09.183046  9688 ts_tablet_manager.cc:1434] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.035s	user 0.005s	sys 0.001s
I20260808 01:49:09.184134  9688 tablet_bootstrap.cc:492] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:09.190081  9688 tablet_bootstrap.cc:654] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:09.204964  9684 tablet_bootstrap.cc:654] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:09.212407  9555 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:09.217991  9682 tablet_bootstrap.cc:492] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:09.218709  9682 ts_tablet_manager.cc:1403] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.122s	user 0.016s	sys 0.021s
I20260808 01:49:09.223114  9377 catalog_manager.cc:5697] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:09.223824  9682 raft_consensus.cc:359] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.224726  9682 raft_consensus.cc:385] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:09.229251  9682 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:09.230058  9682 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.232647  9682 ts_tablet_manager.cc:1434] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260808 01:49:09.233772  9682 tablet_bootstrap.cc:492] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:09.240651  9682 tablet_bootstrap.cc:654] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:09.289217  9684 tablet_bootstrap.cc:492] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:09.289744  9684 ts_tablet_manager.cc:1403] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.109s	user 0.008s	sys 0.008s
I20260808 01:49:09.289132  9374 catalog_manager.cc:5697] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:09.291466  9374 catalog_manager.cc:5697] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:09.296914  9684 raft_consensus.cc:359] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.297963  9684 raft_consensus.cc:385] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:09.298317  9684 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:09.299355  9684 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.300390  9688 tablet_bootstrap.cc:492] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:09.301100  9688 ts_tablet_manager.cc:1403] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.117s	user 0.026s	sys 0.026s
I20260808 01:49:09.303396  9684 ts_tablet_manager.cc:1434] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.007s
I20260808 01:49:09.305058  9684 tablet_bootstrap.cc:492] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:09.306612  9688 raft_consensus.cc:359] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.307530  9688 raft_consensus.cc:385] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:09.307894  9688 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:09.319814  9688 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.322154  9688 ts_tablet_manager.cc:1434] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.021s	user 0.017s	sys 0.002s
I20260808 01:49:09.324527  9688 tablet_bootstrap.cc:492] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:09.332147  9682 tablet_bootstrap.cc:492] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:09.333340  9682 ts_tablet_manager.cc:1403] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.100s	user 0.019s	sys 0.026s
I20260808 01:49:09.334316 10053 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:09.341004  9682 raft_consensus.cc:359] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.342191  9682 raft_consensus.cc:385] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:09.342550  9682 raft_consensus.cc:740] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:09.343384  9682 consensus_queue.cc:260] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.342978 10053 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.347908  9682 ts_tablet_manager.cc:1434] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.001s
I20260808 01:49:09.348453  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:09.349717  9555 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:09.351074  9682 tablet_bootstrap.cc:492] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:09.351948  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:09.353852 10048 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:09.354413 10048 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:09.355088 10048 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:09.367750 10053 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:09.375576  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:09.376221  9684 tablet_bootstrap.cc:654] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:09.376344  9637 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:09.389916  9688 tablet_bootstrap.cc:654] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:09.404327 10048 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.412871  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:09.413503  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:09.414114  9637 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:09.414366  9682 tablet_bootstrap.cc:654] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:09.413513  9558 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:09.414108  9688 tablet_bootstrap.cc:492] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:09.415380  9688 ts_tablet_manager.cc:1403] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.091s	user 0.014s	sys 0.009s
I20260808 01:49:09.422518  9558 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:09.423910  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:09.425195  9978 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:09.425760  9688 raft_consensus.cc:359] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.425766  9978 raft_consensus.cc:697] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:09.427709  9978 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.430148  9637 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:09.430349  9684 tablet_bootstrap.cc:492] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:09.431282  9684 ts_tablet_manager.cc:1403] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.127s	user 0.004s	sys 0.032s
I20260808 01:49:09.432914 10048 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:09.437712  9684 raft_consensus.cc:359] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.438443  9684 raft_consensus.cc:385] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:09.426717  9688 raft_consensus.cc:385] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:09.438642  9684 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:09.438970  9688 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:09.439785  9688 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.442492  9688 ts_tablet_manager.cc:1434] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.027s	user 0.008s	sys 0.004s
I20260808 01:49:09.443265  9555 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:09.444721  9688 tablet_bootstrap.cc:492] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:09.439491  9684 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.449476  9682 tablet_bootstrap.cc:492] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:09.453159  9682 ts_tablet_manager.cc:1403] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.102s	user 0.012s	sys 0.028s
I20260808 01:49:09.457808  9684 ts_tablet_manager.cc:1434] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.026s	user 0.009s	sys 0.000s
I20260808 01:49:09.463740  9684 tablet_bootstrap.cc:492] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:09.466719  9682 raft_consensus.cc:359] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.468138  9682 raft_consensus.cc:385] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:09.468781  9682 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:09.469424  9682 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.471798  9682 ts_tablet_manager.cc:1434] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.000s
I20260808 01:49:09.476555 10048 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:09.482282  9682 tablet_bootstrap.cc:492] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:09.549052  9688 tablet_bootstrap.cc:654] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:09.546489  9374 catalog_manager.cc:5697] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:09.591851  9978 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:09.580214 10048 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260808 01:49:09.607717  9682 tablet_bootstrap.cc:654] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:09.609653  9684 tablet_bootstrap.cc:654] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:09.612135  9741 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:09.612624  9741 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.610581  9919 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260808 01:49:09.618577  9741 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:09.629668  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:09.629760 10053 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260808 01:49:09.640698  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:09.643127  9477 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
W20260808 01:49:09.656399  9599 leader_election.cc:343] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:09.664244  9601 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:09.669118  9558 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:09.686038 10093 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:09.686301 10088 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:09.716109 10088 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.718909 10052 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:09.673002 10032 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:09.727507  9688 tablet_bootstrap.cc:492] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:09.728119  9688 ts_tablet_manager.cc:1403] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.284s	user 0.033s	sys 0.018s
I20260808 01:49:09.730101  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:09.718472 10088 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:09.673718  9945 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.069s
W20260808 01:49:09.737358 10091 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260808 01:49:09.692453 10093 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.727084 10032 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:09.778707 10000 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:09.775424 10032 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:09.728902 10052 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:09.793555  9426 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:49:09.807081  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:09.842967 10093 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:09.792965 10000 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.801636 10094 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:09.881347  9682 tablet_bootstrap.cc:492] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:09.882109  9682 ts_tablet_manager.cc:1403] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.400s	user 0.026s	sys 0.046s
I20260808 01:49:09.891326  9688 raft_consensus.cc:359] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.893919  9688 raft_consensus.cc:385] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:09.894325  9688 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:09.895236  9688 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.907580  9688 ts_tablet_manager.cc:1434] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.179s	user 0.009s	sys 0.001s
I20260808 01:49:09.911715  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:09.913370 10000 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:09.914284  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:09.916410  9637 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:09.921207  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:09.939221  9682 raft_consensus.cc:359] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.940235  9682 raft_consensus.cc:385] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:09.940585  9682 raft_consensus.cc:740] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:09.944770  9682 consensus_queue.cc:260] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.909039  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:09.947024  9632 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:09.883605 10094 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:09.947793  9682 ts_tablet_manager.cc:1434] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.065s	user 0.008s	sys 0.002s
I20260808 01:49:09.958429 10094 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:09.957257  9521 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:09.960608 10093 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:09.961213 10093 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:09.898604 10052 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:09.961711 10093 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:09.960633  9682 tablet_bootstrap.cc:492] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:09.980238  9688 tablet_bootstrap.cc:492] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:10.062047 10032 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:10.064271 10032 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:49:10.100497  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 353678us
W20260808 01:49:10.106724  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 171417us
W20260808 01:49:10.108273  9524 leader_election.cc:343] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:10.110316  9524 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:10.112282  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:10.114670  9684 tablet_bootstrap.cc:492] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:10.115276  9684 ts_tablet_manager.cc:1403] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.652s	user 0.039s	sys 0.059s
I20260808 01:49:10.115191  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:10.123122  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:10.124527  9478 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:49:10.125900  9601 leader_election.cc:343] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:10.142020  9968 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:10.167114  9684 raft_consensus.cc:359] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:10.167822  9684 raft_consensus.cc:385] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:10.168370  9684 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:10.168992  9684 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:10.156023  9997 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:10.187026  9684 ts_tablet_manager.cc:1434] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.071s	user 0.010s	sys 0.004s
I20260808 01:49:10.190503  9684 tablet_bootstrap.cc:492] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:10.190050  9997 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:10.195538  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 63656.1us
I20260808 01:49:10.197818  9997 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:10.214985  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 54121.7us
W20260808 01:49:10.215945  9599 leader_election.cc:343] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:10.266924  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:10.269081  9521 leader_election.cc:343] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:10.270995 10113 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260808 01:49:10.278060  9688 tablet_bootstrap.cc:654] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:10.282701  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:10.285290  9682 tablet_bootstrap.cc:654] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:10.288205  9478 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:10.244516  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:10.296140  9601 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:10.297958  9945 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:10.299007  9945 raft_consensus.cc:697] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:49:10.300257  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.039s	sys 0.017s
W20260808 01:49:10.300665  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.039s	sys 0.017s
W20260808 01:49:10.300998  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 509015us
I20260808 01:49:10.300803  9945 consensus_queue.cc:237] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:10.313906  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 60234.6us
W20260808 01:49:10.314898  9599 leader_election.cc:343] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:10.315678  9599 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:49:10.316205  9524 leader_election.cc:343] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:10.316485  9741 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:10.316848  9524 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:49:10.310909  9444 leader_election.cc:343] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:10.319579 10107 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260808 01:49:10.350663  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:10.377055  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:49:10.408207  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 77116.9us
I20260808 01:49:10.355175 10093 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:10.484151 10117 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:10.481659 10108 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:10.504007  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:10.505347  9478 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:10.501794 10108 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:10.564781 10108 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:49:10.580178 10090 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.129s	user 0.008s	sys 0.000s
I20260808 01:49:10.579375 10093 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:10.603227  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 193636us
I20260808 01:49:10.611828  9684 tablet_bootstrap.cc:654] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:10.680917  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:10.683418  9631 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:10.698005  9478 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:10.699606 10108 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:10.726493 10108 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:10.720315 10119 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:10.751808 10119 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:10.754630 10119 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:10.763023  9524 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:49:10.767831  9444 leader_election.cc:343] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:10.719386 10104 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:10.785991 10100 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:10.796482 10100 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:10.811718 10048 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:10.834405  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:10.835281  9637 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:10.836229  9684 tablet_bootstrap.cc:492] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:10.836993  9684 ts_tablet_manager.cc:1403] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.647s	user 0.023s	sys 0.057s
I20260808 01:49:10.860627  9684 raft_consensus.cc:359] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:10.861790  9684 raft_consensus.cc:385] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:10.862231  9684 raft_consensus.cc:740] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:10.799937 10100 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:10.807670 10104 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:10.873694 10116 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:10.874716 10116 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:10.884756  9374 catalog_manager.cc:5697] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:10.892761  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:10.899227 10000 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:10.872735  9684 consensus_queue.cc:260] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:10.900048 10000 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:10.906282 10000 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:10.910566  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:10.925838  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:10.930755 10116 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:10.948251  9637 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:10.951704  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:10.958125  9524 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:10.961122  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:10.962847 10000 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:10.966629  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:10.984002 10048 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:10.984604 10048 raft_consensus.cc:697] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:10.985229 10107 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:10.985663 10048 consensus_queue.cc:237] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:10.986002 10107 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:10.987221  9688 tablet_bootstrap.cc:492] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:10.987901  9688 ts_tablet_manager.cc:1403] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 1.022s	user 0.030s	sys 0.013s
I20260808 01:49:10.989392 10107 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:11.004262  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:11.005234  9636 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:10.999164  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:11.012162  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:11.023630  9684 ts_tablet_manager.cc:1434] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.186s	user 0.010s	sys 0.003s
I20260808 01:49:11.022190  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:49:11.033591  9601 leader_election.cc:343] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:11.034461  9524 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:11.035590 10000 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:11.037213 10104 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:11.027155 10107 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:11.039052 10107 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:11.039311  9631 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:11.039419 10107 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:11.034515  9684 tablet_bootstrap.cc:492] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:11.039718 10116 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:11.040248  9688 raft_consensus.cc:359] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.041937  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:11.042091  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:11.046626  9688 raft_consensus.cc:385] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:11.050932  9688 raft_consensus.cc:740] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:11.060737  9521 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:11.062378 10000 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:11.055083  9688 consensus_queue.cc:260] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.079973  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:11.087375  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:49:11.096112  9444 leader_election.cc:343] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:11.097599 10105 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:11.098326  9682 tablet_bootstrap.cc:492] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:11.096637  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:11.102492 10105 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:11.115841  9682 ts_tablet_manager.cc:1403] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 1.155s	user 0.051s	sys 0.112s
I20260808 01:49:11.115625 10097 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:49:11.136176  9599 leader_election.cc:343] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:11.136726  9599 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:49:11.138265  9444 leader_election.cc:343] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:11.117477 10097 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.142334  9688 ts_tablet_manager.cc:1434] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.154s	user 0.006s	sys 0.005s
I20260808 01:49:11.143761  9682 raft_consensus.cc:359] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.144819  9682 raft_consensus.cc:385] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:11.145181  9682 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:11.145998  9682 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.140120 10125 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:49:11.151661  9444 leader_election.cc:343] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:49:11.151336  9599 consensus_peers.cc:597] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:11.166389 10097 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:11.171487  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:11.181159 10134 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:11.181835 10134 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.193568  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:11.195873  9601 leader_election.cc:343] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:11.177915  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:11.236729  9524 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:11.246841  9688 tablet_bootstrap.cc:492] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:11.254344 10104 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:11.303400  9682 ts_tablet_manager.cc:1434] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.187s	user 0.009s	sys 0.001s
I20260808 01:49:11.348966 10134 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:49:11.354002 10148 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
I20260808 01:49:11.409914  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:11.410733  9558 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:11.246239 10107 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.429385  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:11.436638 10131 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:11.438344 10131 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:11.438741 10131 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:11.456372  9682 tablet_bootstrap.cc:492] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:11.461208 10107 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:11.467260  9684 tablet_bootstrap.cc:654] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:11.494092  9373 catalog_manager.cc:5697] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
W20260808 01:49:11.505033  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 64724.5us
W20260808 01:49:11.517184  9444 consensus_peers.cc:597] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:11.519956  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:11.532581  9377 catalog_manager.cc:5697] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:11.533419  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:11.522866  9636 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:11.552220 10119 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20260808 01:49:11.556630  9444 leader_election.cc:343] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:11.558799 10131 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.570966  9968 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:11.551277 10097 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:11.582625 10131 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:11.595047  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:11.595806  9554 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:11.588932  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:11.588662 10097 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:11.611658  9601 leader_election.cc:343] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:11.616544  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:11.619205  9524 leader_election.cc:343] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:11.624770  9682 tablet_bootstrap.cc:654] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:11.628614  9688 tablet_bootstrap.cc:654] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:11.632124 10097 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:11.634479  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:11.636170  9521 leader_election.cc:343] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:11.636631  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:11.637874 10000 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:11.641767  9554 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:11.643568  9684 tablet_bootstrap.cc:492] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:11.643955  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:11.648254  9684 ts_tablet_manager.cc:1403] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.614s	user 0.010s	sys 0.012s
I20260808 01:49:11.649717 10116 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:11.651381 10116 raft_consensus.cc:697] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:11.653860 10116 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.661888  9684 raft_consensus.cc:359] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.665470  9684 raft_consensus.cc:385] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:11.667742  9684 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:11.667234  9636 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:11.668638  9684 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.683017  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:11.685376 10157 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:11.685923 10157 raft_consensus.cc:697] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:49:11.687106  9426 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:49:11.686762 10157 consensus_queue.cc:237] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.698645  9682 tablet_bootstrap.cc:492] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:11.699250  9682 ts_tablet_manager.cc:1403] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.243s	user 0.027s	sys 0.001s
I20260808 01:49:11.706944 10153 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:11.709831  9682 raft_consensus.cc:359] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.710440  9682 raft_consensus.cc:385] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:11.710748  9682 raft_consensus.cc:740] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:11.711563  9682 consensus_queue.cc:260] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.712965  9688 tablet_bootstrap.cc:492] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:11.713475  9688 ts_tablet_manager.cc:1403] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.467s	user 0.033s	sys 0.026s
I20260808 01:49:11.713788  9682 ts_tablet_manager.cc:1434] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260808 01:49:11.717442  9688 raft_consensus.cc:359] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.718438  9688 raft_consensus.cc:385] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:11.718825  9688 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:11.722003  9688 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.706992  9684 ts_tablet_manager.cc:1434] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.058s	user 0.008s	sys 0.000s
I20260808 01:49:11.726285  9688 ts_tablet_manager.cc:1434] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.000s
I20260808 01:49:11.727321  9684 tablet_bootstrap.cc:492] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:11.728518  9688 tablet_bootstrap.cc:492] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:11.738102  9682 tablet_bootstrap.cc:492] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:11.744956  9684 tablet_bootstrap.cc:654] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:11.763114  9688 tablet_bootstrap.cc:654] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:11.779980  9374 catalog_manager.cc:5697] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:11.785679  9684 tablet_bootstrap.cc:492] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:11.786839  9684 ts_tablet_manager.cc:1403] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.060s	user 0.019s	sys 0.013s
I20260808 01:49:11.790580  9684 raft_consensus.cc:359] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.791548  9684 raft_consensus.cc:385] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:11.792200  9684 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:11.793062  9684 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.795549  9684 ts_tablet_manager.cc:1434] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260808 01:49:11.796139  9682 tablet_bootstrap.cc:654] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:11.796730  9684 tablet_bootstrap.cc:492] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:11.802798 10131 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:11.803311 10131 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.804147  9684 tablet_bootstrap.cc:654] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:11.805521 10131 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:11.807404  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:11.809209  9601 leader_election.cc:343] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:11.809579  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:11.810242  9554 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:11.811555  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:11.813160 10116 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:11.813557 10116 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:11.813961 10116 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:11.814401  9377 catalog_manager.cc:5697] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:11.818811  9688 tablet_bootstrap.cc:492] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:11.819365  9688 ts_tablet_manager.cc:1403] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.091s	user 0.013s	sys 0.002s
I20260808 01:49:11.822213  9688 raft_consensus.cc:359] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.822918  9688 raft_consensus.cc:385] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:11.823195  9688 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:11.832039  9684 tablet_bootstrap.cc:492] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:11.834447  9684 ts_tablet_manager.cc:1403] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.038s	user 0.023s	sys 0.012s
I20260808 01:49:11.837255  9684 raft_consensus.cc:359] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.837996  9684 raft_consensus.cc:385] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:11.838281  9684 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:11.827147 10116 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.842084  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:49:11.844010  9601 leader_election.cc:343] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:11.843978  9684 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.845171  9688 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.846539  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:11.847246  9555 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:11.847478  9688 ts_tablet_manager.cc:1434] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.028s	user 0.006s	sys 0.001s
I20260808 01:49:11.851780  9684 ts_tablet_manager.cc:1434] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.017s	user 0.004s	sys 0.002s
I20260808 01:49:11.853034  9684 tablet_bootstrap.cc:492] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:11.858083  9688 tablet_bootstrap.cc:492] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:11.864650  9682 tablet_bootstrap.cc:492] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:11.865211  9682 ts_tablet_manager.cc:1403] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.130s	user 0.006s	sys 0.006s
I20260808 01:49:11.867964  9688 tablet_bootstrap.cc:654] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:11.868657  9682 raft_consensus.cc:359] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.869545  9682 raft_consensus.cc:385] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:11.869815  9682 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:11.870563  9682 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.877612  9682 ts_tablet_manager.cc:1434] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.000s
I20260808 01:49:11.878865  9682 tablet_bootstrap.cc:492] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:11.880146  9684 tablet_bootstrap.cc:654] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:11.877177 10116 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:11.890852  9555 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:11.897037  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:11.899050  9682 tablet_bootstrap.cc:654] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:11.902661 10116 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:11.903622 10116 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:11.904652 10116 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.908003  9688 tablet_bootstrap.cc:492] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:11.912171 10097 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:11.912853 10097 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.915278 10097 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:11.919049  9684 tablet_bootstrap.cc:492] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:11.919649  9684 ts_tablet_manager.cc:1403] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.067s	user 0.008s	sys 0.013s
I20260808 01:49:11.922919  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:11.923087  9684 raft_consensus.cc:359] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.923907  9684 raft_consensus.cc:385] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:11.924217  9684 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:11.924968  9684 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.927829  9684 ts_tablet_manager.cc:1434] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260808 01:49:11.928695  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:11.929714  9634 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 1: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:11.931384  9684 tablet_bootstrap.cc:492] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:11.933357  9521 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:11.936997 10100 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:11.938277  9684 tablet_bootstrap.cc:654] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:11.943395  9688 ts_tablet_manager.cc:1403] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.086s	user 0.014s	sys 0.000s
I20260808 01:49:11.946527  9688 raft_consensus.cc:359] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.961840  9688 raft_consensus.cc:385] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:11.962191  9688 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:11.963805  9688 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:11.978333  9688 ts_tablet_manager.cc:1434] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.034s	user 0.008s	sys 0.000s
I20260808 01:49:11.984817  9688 tablet_bootstrap.cc:492] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:11.990574 10100 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:11.997627  9682 tablet_bootstrap.cc:492] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:11.998373  9682 ts_tablet_manager.cc:1403] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.120s	user 0.009s	sys 0.007s
I20260808 01:49:12.001410  9688 tablet_bootstrap.cc:654] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:12.010514  9684 tablet_bootstrap.cc:492] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:12.011154  9684 ts_tablet_manager.cc:1403] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.080s	user 0.028s	sys 0.018s
I20260808 01:49:12.014467  9684 raft_consensus.cc:359] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.015329  9684 raft_consensus.cc:385] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:12.015656  9684 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:12.016427  9684 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.019039  9684 ts_tablet_manager.cc:1434] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260808 01:49:12.003532  9374 catalog_manager.cc:5697] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:12.007047 10131 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:12.019740 10131 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.021930 10131 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:12.033283  9682 raft_consensus.cc:359] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.034188  9682 raft_consensus.cc:385] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:12.034499  9682 raft_consensus.cc:740] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:12.035274  9682 consensus_queue.cc:260] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.035669  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:12.037643  9682 ts_tablet_manager.cc:1434] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.038s	user 0.005s	sys 0.001s
I20260808 01:49:12.039565  9684 tablet_bootstrap.cc:492] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
W20260808 01:49:12.044564  9601 leader_election.cc:343] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:12.046649  9684 tablet_bootstrap.cc:654] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:12.048905  9682 tablet_bootstrap.cc:492] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:12.034013  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:12.066640  9682 tablet_bootstrap.cc:654] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:12.066787  9688 tablet_bootstrap.cc:492] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:12.067991  9688 ts_tablet_manager.cc:1403] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.083s	user 0.017s	sys 0.005s
W20260808 01:49:12.068553  9599 leader_election.cc:343] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:12.069054  9599 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:12.071300 10048 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:12.071523 10134 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:12.071862 10048 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.074374 10048 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:12.081295  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:12.089231  9688 raft_consensus.cc:359] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.090020  9688 raft_consensus.cc:385] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:12.090310  9688 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:12.095719  9688 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.097507  9688 ts_tablet_manager.cc:1434] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.000s
I20260808 01:49:12.098605  9688 tablet_bootstrap.cc:492] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:12.085031  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:12.104068  9554 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 1: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:49:12.107363  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:12.109644 10048 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:49:12.111727  9601 consensus_peers.cc:597] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:12.112047 10116 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:12.112622 10116 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.114413 10116 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:12.115897  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:12.116671  9478 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:12.117857  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:12.123317 10116 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:12.123816  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:12.125399  9684 tablet_bootstrap.cc:492] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:12.117892 10048 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:12.125910  9684 ts_tablet_manager.cc:1403] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.087s	user 0.011s	sys 0.011s
W20260808 01:49:12.127139  9599 leader_election.cc:343] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:12.119372 10096 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:12.123812 10116 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:12.128567 10133 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:12.126539 10131 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:12.135998 10131 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.130200 10096 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.144906 10131 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:12.149978  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:12.152712  9634 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:12.151631  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:12.150904  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:12.153769  9474 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:12.153978  9556 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:12.130975 10116 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:12.155650  9601 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:12.155864  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:12.146770 10096 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:12.161412 10105 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:12.164342 10105 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:12.164790 10105 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:12.162936  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:12.163188 10183 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:12.172055 10183 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:12.172456 10183 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:12.178170  9688 tablet_bootstrap.cc:654] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
W20260808 01:49:12.191296  9599 leader_election.cc:343] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:12.229251 10183 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.230957 10116 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.232542 10096 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:12.242121 10096 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.247483 10105 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.251677  9682 tablet_bootstrap.cc:492] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:12.252338  9682 ts_tablet_manager.cc:1403] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.204s	user 0.007s	sys 0.021s
I20260808 01:49:12.254370 10097 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:12.255013 10097 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.258275  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:12.259735  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:12.263388  9444 leader_election.cc:343] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:12.267017 10105 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:12.268730 10097 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:12.269443 10116 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:12.270743  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:12.271430  9474 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:12.273160  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:12.273680  9559 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:12.279904  9684 raft_consensus.cc:359] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.280671  9684 raft_consensus.cc:385] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:12.280990  9684 raft_consensus.cc:740] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:12.283787  9684 consensus_queue.cc:260] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.285686  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:12.287799  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:12.289944  9599 leader_election.cc:343] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:12.303951  9684 ts_tablet_manager.cc:1434] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.177s	user 0.004s	sys 0.006s
I20260808 01:49:12.315510 10183 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:12.319540  9559 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:12.293087 10096 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:12.325636  9682 raft_consensus.cc:359] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:12.326223  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 58912.4us
I20260808 01:49:12.326588  9682 raft_consensus.cc:385] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:12.328353  9682 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:12.329097  9682 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.333736  9521 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:12.333757  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:12.350729  9682 ts_tablet_manager.cc:1434] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.098s	user 0.007s	sys 0.002s
I20260808 01:49:12.339252 10000 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:12.343246  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:12.358834  9684 tablet_bootstrap.cc:492] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:12.332998 10096 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20260808 01:49:12.342896  9474 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:12.372762  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:12.360854 10048 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:12.378196  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:12.379024  9479 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:12.368165 10157 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
W20260808 01:49:12.392467  9599 leader_election.cc:343] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:12.398809  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:12.408011  9682 tablet_bootstrap.cc:492] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:12.411432  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:12.414626  9556 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:12.415584 10048 raft_consensus.cc:697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:12.417761 10133 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:12.413192  9634 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:12.423229 10133 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:49:12.430362  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 58398.3us
W20260808 01:49:12.435628  9444 leader_election.cc:343] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:12.436123  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:12.443066 10157 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:12.444880 10100 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:12.445482 10100 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.447700 10100 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:12.448372 10133 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.455291  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:12.456991  9524 leader_election.cc:343] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:12.466123 10048 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.469010 10193 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:12.478796 10193 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.495208 10193 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:12.498363  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:12.503252  9688 tablet_bootstrap.cc:492] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:12.503891  9688 ts_tablet_manager.cc:1403] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.406s	user 0.031s	sys 0.046s
I20260808 01:49:12.507885  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:12.507182  9444 leader_election.cc:343] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:12.517073 10134 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:12.517664 10134 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.521291  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:12.520025 10134 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:12.523965  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:12.550100  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:12.551959  9688 raft_consensus.cc:359] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.552721  9688 raft_consensus.cc:385] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:12.552982  9688 raft_consensus.cc:740] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:12.553403  9682 tablet_bootstrap.cc:654] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:12.553702  9688 consensus_queue.cc:260] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.554775  9631 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:12.564307  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:12.567442 10133 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:12.569332  9634 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:12.550911  9478 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:12.571545  9479 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:12.572033 10194 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260808 01:49:12.574841  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 50283.3us
W20260808 01:49:12.577559  9521 leader_election.cc:343] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:12.577836  9684 tablet_bootstrap.cc:654] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:12.572502 10194 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:12.574633  9601 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:12.578063  9373 catalog_manager.cc:5697] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:12.589202  9688 ts_tablet_manager.cc:1434] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.085s	user 0.006s	sys 0.002s
I20260808 01:49:12.596850 10189 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:49:12.631317  9599 leader_election.cc:343] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:12.632524  9521 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:49:12.630889  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 56380.1us
I20260808 01:49:12.634848  9601 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:12.640614 10134 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:12.641168 10134 raft_consensus.cc:697] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:12.642052 10134 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:12.645120  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 67596us
I20260808 01:49:12.633705 10194 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:12.653954  9688 tablet_bootstrap.cc:492] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:12.643319 10183 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:12.655380 10183 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:12.655781 10183 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:12.659387 10204 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:12.633940 10189 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.663170 10189 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:12.673375  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:12.676601  9682 tablet_bootstrap.cc:492] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:12.677450  9682 ts_tablet_manager.cc:1403] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.302s	user 0.018s	sys 0.020s
W20260808 01:49:12.683173  9601 leader_election.cc:343] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:12.688575 10184 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:12.669770 10105 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:12.720928  9682 raft_consensus.cc:359] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.724614  9682 raft_consensus.cc:385] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:12.724920  9682 raft_consensus.cc:740] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
W20260808 01:49:12.730103  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46018.5us
I20260808 01:49:12.735381  9682 consensus_queue.cc:260] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.706974 10105 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.742635 10105 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:12.745855  9682 ts_tablet_manager.cc:1434] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.068s	user 0.010s	sys 0.000s
I20260808 01:49:12.747289  9682 tablet_bootstrap.cc:492] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:12.747184  9684 tablet_bootstrap.cc:492] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:12.749058  9684 ts_tablet_manager.cc:1403] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.396s	user 0.012s	sys 0.029s
I20260808 01:49:12.719700  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:12.755594  9688 tablet_bootstrap.cc:654] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
W20260808 01:49:12.764717  9599 leader_election.cc:343] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:12.766342  9599 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:12.763135  9684 raft_consensus.cc:359] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.768862  9684 raft_consensus.cc:385] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:12.769850  9684 raft_consensus.cc:740] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:12.792306 10197 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:12.762544 10194 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.769945 10183 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.867553 10194 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:12.871695  9684 consensus_queue.cc:260] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.875993 10183 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:12.887266  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:12.887423  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:12.888147  9555 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
W20260808 01:49:12.888772  9426 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:49:12.891525  9684 ts_tablet_manager.cc:1434] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.142s	user 0.019s	sys 0.002s
I20260808 01:49:12.892699  9684 tablet_bootstrap.cc:492] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
W20260808 01:49:12.893653  9444 leader_election.cc:343] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:12.894487  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:12.895829 10193 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:12.897891 10193 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:12.898285 10193 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:12.952797  9688 tablet_bootstrap.cc:492] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:12.953653  9688 ts_tablet_manager.cc:1403] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.306s	user 0.010s	sys 0.015s
I20260808 01:49:12.945245  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:12.957417  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:12.958699  9634 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 1: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:49:12.960493 10214 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.081s	user 0.002s	sys 0.000s
I20260808 01:49:12.961658  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:12.962477  9481 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 1: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:12.965864  9688 raft_consensus.cc:359] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:12.968546  9688 raft_consensus.cc:385] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:12.993402  9682 tablet_bootstrap.cc:654] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:12.968819  9688 raft_consensus.cc:740] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:13.029201  9688 consensus_queue.cc:260] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:13.044845  9631 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:13.048278  9688 ts_tablet_manager.cc:1434] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.094s	user 0.004s	sys 0.005s
I20260808 01:49:13.050089  9688 tablet_bootstrap.cc:492] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
W20260808 01:49:13.056269  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 78965us
W20260808 01:49:13.072314  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 145286us
I20260808 01:49:13.095438  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:13.134972  9688 tablet_bootstrap.cc:654] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
W20260808 01:49:13.169551  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48449.6us
W20260808 01:49:13.177817  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 75482.6us
I20260808 01:49:13.178464  9684 tablet_bootstrap.cc:654] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:13.178627  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:13.173141 10193 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:13.234848  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 59199.1us
W20260808 01:49:13.238839  9599 leader_election.cc:343] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:13.257346  9599 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:13.267890 10184 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:49:13.321650 10225 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:49:13.351049 10153 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:13.371093 10153 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:13.376601 10231 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260808 01:49:13.377378 10153 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:13.380648 10193 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:13.399559  9682 tablet_bootstrap.cc:492] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:13.400467  9682 ts_tablet_manager.cc:1403] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.653s	user 0.024s	sys 0.048s
I20260808 01:49:13.404176  9682 raft_consensus.cc:359] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:13.405144  9682 raft_consensus.cc:385] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:13.405491  9682 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
W20260808 01:49:13.405853  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 143627us
I20260808 01:49:13.407720  9682 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:13.410347  9682 ts_tablet_manager.cc:1434] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.005s
I20260808 01:49:13.435832  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:13.436201  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:13.436581  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:13.436604  9631 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:13.437309  9560 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:13.412472  9682 tablet_bootstrap.cc:492] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
W20260808 01:49:13.452152  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 319942us
I20260808 01:49:13.500327  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:13.501055  9631 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:49:13.492478  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 43618us
I20260808 01:49:13.502888  9374 catalog_manager.cc:5697] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:13.507220  9560 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:13.510638  9684 tablet_bootstrap.cc:492] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:13.511292  9684 ts_tablet_manager.cc:1403] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.619s	user 0.026s	sys 0.002s
I20260808 01:49:13.506471 10234 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260808 01:49:13.516477  9552 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:13.536471  9684 raft_consensus.cc:359] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:13.537128  9684 raft_consensus.cc:385] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:13.537590  9684 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:13.538228  9684 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:13.539789  9688 tablet_bootstrap.cc:492] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:13.540320  9688 ts_tablet_manager.cc:1403] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.491s	user 0.013s	sys 0.012s
I20260808 01:49:13.540822  9684 ts_tablet_manager.cc:1434] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.000s
I20260808 01:49:13.530730  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:13.602698  9684 tablet_bootstrap.cc:492] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:13.606576 10229 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:13.607316 10229 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:13.607772 10229 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:13.581112 10157 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.105s
I20260808 01:49:13.713251  9631 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:13.891175 10229 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:13.901934  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 234889us
W20260808 01:49:13.902423  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 314436us
I20260808 01:49:13.926468 10213 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260808 01:49:13.903375  9444 leader_election.cc:343] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:13.943113 10235 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:13.949446  9688 raft_consensus.cc:359] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:13.955206  9688 raft_consensus.cc:385] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:13.955602  9688 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:13.965044  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:13.965626 10235 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.002949  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:13.990600 10225 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:14.008615 10225 raft_consensus.cc:697] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:14.008692 10235 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:14.009613 10225 consensus_queue.cc:237] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:13.981974 10229 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:14.013202  9688 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.021785 10242 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:14.023488  9682 tablet_bootstrap.cc:654] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:14.036908  9688 ts_tablet_manager.cc:1434] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.496s	user 0.017s	sys 0.001s
I20260808 01:49:14.017807 10157 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:14.022656 10242 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:14.064113  9444 leader_election.cc:343] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:49:14.069741  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 109516us
I20260808 01:49:14.056576 10157 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.094906  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:14.106815  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:14.111562  9638 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:14.116565 10157 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:14.126384  9684 tablet_bootstrap.cc:654] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
W20260808 01:49:14.136853  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 69152.2us
I20260808 01:49:14.141170  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:14.141922  9557 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:14.163659  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:14.169647 10193 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:14.170086 10193 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:14.170414 10193 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:14.138190 10189 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:14.175138 10189 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.178053 10193 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.180830 10189 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:14.181034 10193 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:14.185520  9638 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:14.190081  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:14.191020  9555 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:14.201975 10105 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:14.202733 10105 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.205295 10105 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:14.162138 10227 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.046s
I20260808 01:49:14.209905  9688 tablet_bootstrap.cc:492] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Bootstrap starting.
I20260808 01:49:14.210210  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:14.213160  9638 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:14.215109  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:14.217295  9521 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:14.219278  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:14.220341  9599 consensus_peers.cc:597] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:14.259816  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:14.260835  9630 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
W20260808 01:49:14.253226  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 54354.8us
W20260808 01:49:14.286490  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 64567.6us
I20260808 01:49:14.288026  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:14.291952  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:14.301261 10252 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:14.301874 10252 raft_consensus.cc:697] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:14.303143 10252 consensus_queue.cc:237] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.324635  9555 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:14.328207 10160 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:14.331120  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:14.308060  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:14.333168  9635 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:49:14.310431  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 92950.4us
W20260808 01:49:14.339073  9444 leader_election.cc:343] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:14.342308  9684 tablet_bootstrap.cc:492] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:14.343209  9684 ts_tablet_manager.cc:1403] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.768s	user 0.041s	sys 0.018s
I20260808 01:49:14.356781 10160 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:14.357627 10160 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:14.358070  9688 tablet_bootstrap.cc:654] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:14.365499  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:49:14.368609  9601 leader_election.cc:343] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:14.375550 10242 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:14.378439  9684 raft_consensus.cc:359] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.381707  9684 raft_consensus.cc:385] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:14.382107  9684 raft_consensus.cc:740] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:14.411587  9684 consensus_queue.cc:260] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.368923 10240 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
W20260808 01:49:14.451333  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 63225.4us
W20260808 01:49:14.496213  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 150854us
I20260808 01:49:14.499960  9684 ts_tablet_manager.cc:1434] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.156s	user 0.009s	sys 0.000s
I20260808 01:49:14.505811  9635 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:14.476872 10240 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:14.508152 10240 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.533921 10131 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260808 01:49:14.534169  9599 leader_election.cc:343] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:14.547230  9599 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:14.552541 10131 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:14.560750 10160 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.563946 10160 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:14.569298  9688 tablet_bootstrap.cc:492] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: No bootstrap required, opened a new log
I20260808 01:49:14.569854  9688 ts_tablet_manager.cc:1403] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Time spent bootstrapping tablet: real 0.360s	user 0.017s	sys 0.005s
I20260808 01:49:14.572754  9688 raft_consensus.cc:359] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.574265  9688 raft_consensus.cc:385] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:14.574693  9688 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Initialized, Role: FOLLOWER
I20260808 01:49:14.573786  9559 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:14.575793  9688 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.588295  9688 ts_tablet_manager.cc:1434] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.000s
I20260808 01:49:14.594364  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:14.597661  9684 tablet_bootstrap.cc:492] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Bootstrap starting.
I20260808 01:49:14.625123 10272 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:14.625726 10272 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.627027  9481 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:14.631472 10272 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:49:14.633011  9444 consensus_peers.cc:597] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:49:14.635818 10157 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:14.636176  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:14.637051  9477 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 0.
I20260808 01:49:14.637821 10157 raft_consensus.cc:3055] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:14.638222 10157 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:14.643316 10238 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:14.646915  9601 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:14.653851 10265 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:14.655725 10265 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:14.658116 10265 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:14.658669  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:49:14.663511  9426 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:49:14.675420  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:14.656662  9374 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50084 (request call id 112) took 1227 ms. Trace:
I20260808 01:49:14.656778 10157 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.684434 10157 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:14.686702  9377 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50068 (request call id 121) took 1162 ms. Trace:
I20260808 01:49:14.687137  9377 rpcz_store.cc:276] 0808 01:49:13.524073 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:13.524279 (+   206us) service_pool.cc:224] Handling call
0808 01:49:14.686614 (+1162335us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39064,"reported_tablets":5}
I20260808 01:49:14.669508  9373 catalog_manager.cc:5697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
W20260808 01:49:14.691808  9444 leader_election.cc:343] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:14.692306  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:14.695611 10252 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:14.697499  9684 tablet_bootstrap.cc:654] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:14.678500  9682 tablet_bootstrap.cc:492] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:14.698114  9682 ts_tablet_manager.cc:1403] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 1.286s	user 0.027s	sys 0.034s
I20260808 01:49:14.702317  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:14.703903  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:14.701246  9682 raft_consensus.cc:359] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.706876  9682 raft_consensus.cc:385] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260808 01:49:14.699648  9599 leader_election.cc:343] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:14.680588  9374 rpcz_store.cc:276] 0808 01:49:13.429649 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:13.429793 (+   144us) service_pool.cc:224] Handling call
0808 01:49:14.656560 (+1226767us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:49:13.896690 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:49:13.897304 (+   614us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:49:13.897338 (+    34us) write_op.cc:435] Acquired schema lock
0808 01:49:13.897354 (+    16us) tablet.cc:662] Decoding operations
0808 01:49:13.901165 (+  3811us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:49:13.901243 (+    78us) write_op.cc:641] Partition lock acquired for write op
0808 01:49:13.901275 (+    32us) tablet.cc:685] Acquiring locks for 1 operations
0808 01:49:13.901475 (+   200us) tablet.cc:701] Row locks acquired
0808 01:49:13.901493 (+    18us) write_op.cc:260] PREPARE: finished
0808 01:49:13.901644 (+   151us) write_op.cc:270] Start()
0808 01:49:13.901760 (+   116us) write_op.cc:276] Timestamp: P: 1786153753901602 usec, L: 0
0808 01:49:13.901800 (+    40us) op_driver.cc:348] REPLICATION: starting
0808 01:49:13.931431 (+ 29631us) log.cc:844] Serialized 540 byte log entry
0808 01:49:14.127543 (+196112us) op_driver.cc:464] REPLICATION: finished
0808 01:49:14.291753 (+164210us) write_op.cc:301] APPLY: starting
0808 01:49:14.291839 (+    86us) tablet.cc:1367] starting BulkCheckPresence
0808 01:49:14.291947 (+   108us) tablet.cc:1370] finished BulkCheckPresence
0808 01:49:14.300643 (+  8696us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:49:14.300962 (+   319us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:49:14.301199 (+   237us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 01:49:14.301269 (+    70us) write_op.cc:312] APPLY: finished
0808 01:49:14.301576 (+   307us) log.cc:844] Serialized 22 byte log entry
0808 01:49:14.301817 (+   241us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:14.301927 (+   110us) write_op.cc:454] Released schema lock
0808 01:49:14.301974 (+    47us) write_op.cc:341] FINISH: Updating metrics
0808 01:49:14.302409 (+   435us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:14.302461 (+    52us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":13,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":163931,"apply.run_cpu_time_us":2333,"apply.run_wall_time_us":10588,"num_ops":1,"prepare.queue_time_us":339845,"prepare.run_cpu_time_us":3214,"prepare.run_wall_time_us":62835,"replication_time_us":225575,"thread_start_us":135822,"threads_started":2}]]}
I20260808 01:49:14.721315 10265 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.707258  9682 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:14.735127  9682 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.739581  9682 ts_tablet_manager.cc:1434] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.041s	user 0.004s	sys 0.003s
I20260808 01:49:14.740862  9682 tablet_bootstrap.cc:492] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:14.745478  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:14.746850  9475 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:14.792703  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:49:14.797189  9599 leader_election.cc:343] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:49:14.812062 10277 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
W20260808 01:49:14.820468  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 44577.9us
I20260808 01:49:14.774854 10265 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:14.830497 10252 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260808 01:49:14.846426 10160 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:14.847642 10160 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.864466  9552 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:49:14.878032  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 102418us
I20260808 01:49:14.926841  9684 tablet_bootstrap.cc:492] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: No bootstrap required, opened a new log
I20260808 01:49:14.927400  9684 ts_tablet_manager.cc:1403] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Time spent bootstrapping tablet: real 0.330s	user 0.022s	sys 0.012s
W20260808 01:49:14.929448  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 50584.5us
I20260808 01:49:14.930357  9521 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:14.956193 10000 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:14.954084 10160 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:14.971100  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:14.988320  9684 raft_consensus.cc:359] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:14.989321  9684 raft_consensus.cc:385] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:14.989632  9684 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Initialized, Role: FOLLOWER
I20260808 01:49:15.006999  9475 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
W20260808 01:49:15.026346  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 96056.8us
I20260808 01:49:15.027251  9682 tablet_bootstrap.cc:654] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:15.026119  9684 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.065395  9684 ts_tablet_manager.cc:1434] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Time spent starting tablet: real 0.138s	user 0.010s	sys 0.001s
I20260808 01:49:15.086190  9474 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:15.097980 10194 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:15.098572 10194 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.100769 10194 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:15.107918  9601 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:15.115392 10189 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:15.115934 10189 raft_consensus.cc:697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:15.116901 10189 consensus_queue.cc:237] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.117451  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:15.118090  9637 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 0.
I20260808 01:49:15.119181  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:15.120934  9521 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:15.143186  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:15.142802 10229 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260808 01:49:15.131117 10287 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:15.160633  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:49:15.164510  9444 consensus_peers.cc:597] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:15.168009 10243 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:15.168622 10243 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:15.169160 10243 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:15.161835 10227 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260808 01:49:15.192577  9682 tablet_bootstrap.cc:492] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:15.193310  9682 ts_tablet_manager.cc:1403] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.453s	user 0.024s	sys 0.012s
I20260808 01:49:15.196094 10096 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:15.197134  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:15.199342 10289 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:15.200410 10289 raft_consensus.cc:697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:15.200500  9682 raft_consensus.cc:359] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.201531  9682 raft_consensus.cc:385] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:15.201869  9682 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:15.202943  9682 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.202069 10289 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.219262  9682 ts_tablet_manager.cc:1434] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.026s	user 0.019s	sys 0.005s
I20260808 01:49:15.220561  9682 tablet_bootstrap.cc:492] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:15.226825  9374 catalog_manager.cc:5697] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:49:15.233143  9682 tablet_bootstrap.cc:654] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
W20260808 01:49:15.265764  9444 leader_election.cc:343] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:15.270826  9373 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50064 (request call id 121) took 1871 ms. Trace:
W20260808 01:49:15.330547  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46020.4us
I20260808 01:49:15.271474  9373 rpcz_store.cc:276] 0808 01:49:13.399666 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:13.399830 (+   164us) service_pool.cc:224] Handling call
0808 01:49:15.270729 (+1870899us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:49:14.835337 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:49:14.843650 (+  8313us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:49:14.843683 (+    33us) write_op.cc:435] Acquired schema lock
0808 01:49:14.843699 (+    16us) tablet.cc:662] Decoding operations
0808 01:49:14.844043 (+   344us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:49:14.844142 (+    99us) write_op.cc:641] Partition lock acquired for write op
0808 01:49:14.844174 (+    32us) tablet.cc:685] Acquiring locks for 1 operations
0808 01:49:14.844410 (+   236us) tablet.cc:701] Row locks acquired
0808 01:49:14.844428 (+    18us) write_op.cc:260] PREPARE: finished
0808 01:49:14.844572 (+   144us) write_op.cc:270] Start()
0808 01:49:14.844679 (+   107us) write_op.cc:276] Timestamp: P: 1786153754844528 usec, L: 0
0808 01:49:14.844700 (+    21us) op_driver.cc:348] REPLICATION: starting
0808 01:49:14.845419 (+   719us) log.cc:844] Serialized 540 byte log entry
0808 01:49:14.932270 (+ 86851us) op_driver.cc:464] REPLICATION: finished
0808 01:49:15.153044 (+220774us) write_op.cc:301] APPLY: starting
0808 01:49:15.153196 (+   152us) tablet.cc:1367] starting BulkCheckPresence
0808 01:49:15.153500 (+   304us) tablet.cc:1370] finished BulkCheckPresence
0808 01:49:15.155365 (+  1865us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:49:15.155706 (+   341us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:49:15.155921 (+   215us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 01:49:15.155989 (+    68us) write_op.cc:312] APPLY: finished
0808 01:49:15.178573 (+ 22584us) log.cc:844] Serialized 22 byte log entry
0808 01:49:15.182090 (+  3517us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:15.183069 (+   979us) write_op.cc:454] Released schema lock
0808 01:49:15.184517 (+  1448us) write_op.cc:341] FINISH: Updating metrics
0808 01:49:15.191579 (+  7062us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:15.191653 (+    74us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":213843,"apply.run_cpu_time_us":4388,"apply.run_wall_time_us":40530,"num_ops":1,"prepare.queue_time_us":105718,"prepare.run_cpu_time_us":3061,"prepare.run_wall_time_us":24664,"replication_time_us":87288,"thread_start_us":144606,"threads_started":2}]]}
I20260808 01:49:15.355808  9682 tablet_bootstrap.cc:492] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:15.356504  9682 ts_tablet_manager.cc:1403] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.136s	user 0.010s	sys 0.012s
I20260808 01:49:15.381889  9682 raft_consensus.cc:359] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.382905  9682 raft_consensus.cc:385] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:15.383261  9682 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:15.384449  9682 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.386621  9682 ts_tablet_manager.cc:1434] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.030s	user 0.007s	sys 0.000s
I20260808 01:49:15.390411  9682 tablet_bootstrap.cc:492] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:15.411662  9372 catalog_manager.cc:5697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 1 to 2, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 2 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:15.417909 10160 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:15.420276 10160 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.428242 10160 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:15.429306  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 71076.6us
I20260808 01:49:15.436632  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:15.437297  9638 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 0.
I20260808 01:49:15.438453  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:15.442328 10160 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:49:15.442948 10160 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:15.443514 10160 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:15.447458 10243 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.457082 10157 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:15.459482  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:15.459497 10157 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.464661  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:15.465251  9638 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:15.464231 10243 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:15.460521  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:15.477691  9638 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:15.479218  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:15.479466  9473 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 1: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:15.491531 10252 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:49:15.492105 10252 raft_consensus.cc:697] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:15.493122 10252 consensus_queue.cc:237] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.495170  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:15.480453  9373 catalog_manager.cc:5697] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:49:15.498229  9373 catalog_manager.cc:5697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:15.500772  9373 catalog_manager.cc:5697] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:49:15.505828  9444 leader_election.cc:343] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:15.516602  9682 tablet_bootstrap.cc:654] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:15.518509 10157 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:15.532570  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:15.533221  9636 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:15.491178 10160 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.545171  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:15.549654  9555 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 1: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:49:15.537511 10100 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:15.550949 10299 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:15.551198 10100 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.547061 10194 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:15.553431 10100 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:15.553457 10194 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:15.555191 10194 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:15.554715  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:15.562741  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:15.565079  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:15.576726  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:15.588436 10210 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:15.595809 10160 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:15.600967  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:15.602061  9629 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 1: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:15.603989  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:15.606387 10160 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:49:15.616542  9682 tablet_bootstrap.cc:492] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:15.617096  9682 ts_tablet_manager.cc:1403] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.227s	user 0.014s	sys 0.028s
I20260808 01:49:15.606067 10194 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.619760 10194 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:15.621537  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:15.625663  9682 raft_consensus.cc:359] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.626423  9682 raft_consensus.cc:385] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:15.623436  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:15.626730  9682 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:15.626987  9629 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:15.633548  9682 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.636008  9682 ts_tablet_manager.cc:1434] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.000s
I20260808 01:49:15.637053  9682 tablet_bootstrap.cc:492] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
W20260808 01:49:15.639336  9444 leader_election.cc:343] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:49:15.644838  9637 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:15.647409  9682 tablet_bootstrap.cc:654] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:15.667014  9551 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:15.668706  9637 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:15.669023  9372 catalog_manager.cc:5697] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:15.660542  9629 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:49:15.680081  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:15.680992 10194 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:15.681936 10160 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:15.686841 10194 raft_consensus.cc:697] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:15.695910 10194 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.739888  9682 tablet_bootstrap.cc:492] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:15.740437  9682 ts_tablet_manager.cc:1403] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.104s	user 0.018s	sys 0.025s
I20260808 01:49:15.751693  9682 raft_consensus.cc:359] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.752425  9682 raft_consensus.cc:385] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:15.752720  9682 raft_consensus.cc:740] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:15.753450  9682 consensus_queue.cc:260] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.755801  9682 ts_tablet_manager.cc:1434] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.001s
I20260808 01:49:15.757735  9682 tablet_bootstrap.cc:492] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Bootstrap starting.
I20260808 01:49:15.757496  9472 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:15.760835 10260 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:49:15.766080 10210 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:15.771968  9682 tablet_bootstrap.cc:654] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Neither blocks nor log segments found. Creating new log.
I20260808 01:49:15.826625 10160 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:15.827291 10160 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.839502 10160 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:15.839586  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:15.843495  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:15.844837  9638 raft_consensus.cc:2368] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2.
I20260808 01:49:15.855654  9521 leader_election.cc:400] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2.
I20260808 01:49:15.856010  9521 leader_election.cc:403] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:49:15.856598 10210 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 2
I20260808 01:49:15.857649  9373 catalog_manager.cc:5697] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 0 to 2, leader changed from <none> to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 2 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
W20260808 01:49:15.871583  9426 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:49:15.873890  9682 tablet_bootstrap.cc:492] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: No bootstrap required, opened a new log
I20260808 01:49:15.874495  9682 ts_tablet_manager.cc:1403] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent bootstrapping tablet: real 0.117s	user 0.019s	sys 0.016s
I20260808 01:49:15.893524 10210 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2.
I20260808 01:49:15.896457  9682 raft_consensus.cc:359] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.897147  9682 raft_consensus.cc:385] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:49:15.897461  9682 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Initialized, Role: FOLLOWER
I20260808 01:49:15.898489  9682 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:15.906736  9682 ts_tablet_manager.cc:1434] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent starting tablet: real 0.032s	user 0.006s	sys 0.000s
I20260808 01:49:15.939185  9551 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:15.964597 10252 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:16.019230 10240 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:16.043324 10160 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:16.043958 10160 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:16.048448  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:16.048460 10160 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:16.048885  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:16.050839  9521 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:16.055083 10210 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:49:16.062098  9550 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:49:16.304448  9550 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260808 01:49:16.306175 10238 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:16.323071  9637 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260808 01:49:16.326611 10238 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:49:16.680819  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:49:16.689604 10322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:49:16.690323 10323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:49:16.692996  7831 server_base.cc:1061] running on GCE node
W20260808 01:49:16.694654 10326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:49:16.696377  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:49:16.696846  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:49:16.697188  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786153756697165 us; error 0 us; skew 500 ppm
I20260808 01:49:16.697808  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:49:16.702297  7831 webserver.cc:533] Webserver started at http://127.7.165.196:33675/ using document root <none> and password file <none>
I20260808 01:49:16.703624  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:49:16.704936  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:49:16.705480  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:49:16.707399  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-3-root/instance:
uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
format_stamp: "Formatted at 2026-08-08 01:49:16 on dist-test-slave-8tdl"
I20260808 01:49:16.712925  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20260808 01:49:16.717777 10330 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:49:16.718786  7831 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20260808 01:49:16.719116  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-3-root
uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
format_stamp: "Formatted at 2026-08-08 01:49:16 on dist-test-slave-8tdl"
I20260808 01:49:16.719395  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:49:16.745596  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:49:16.747773  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:49:16.752132  7831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:49:16.754649  7831 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:49:16.755031  7831 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.001s
I20260808 01:49:16.755368  7831 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:49:16.755590  7831 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:49:16.866906  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.196:33455
I20260808 01:49:16.867089 10393 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.196:33455 every 8 connection(s)
I20260808 01:49:16.872877  7831 auto_leader_rebalancer-test.cc:581] add a tserver: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:16.903342 10394 heartbeater.cc:344] Connected to a master server at 127.7.165.254:43565
I20260808 01:49:16.903908 10394 heartbeater.cc:461] Registering TS with master...
I20260808 01:49:16.904923 10394 heartbeater.cc:507] Master 127.7.165.254:43565 requested a full tablet report, sending...
I20260808 01:49:16.907361  9373 ts_manager.cc:194] Registered new tserver with Master: 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:16.909353  9373 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55058
I20260808 01:49:17.912822 10394 heartbeater.cc:499] Master 127.7.165.254:43565 was elected leader, sending a full tablet report...
I20260808 01:49:18.017805  9474 consensus_queue.cc:237] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:18.026618  9551 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:18.028201  9637 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:18.029366 10238 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:18.030377 10227 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:18.048995 10405 raft_consensus.cc:2955] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:18.066041  9629 raft_consensus.cc:2955] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:18.081126  9551 raft_consensus.cc:2955] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:18.087278  9373 catalog_manager.cc:5697] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260808 01:49:18.116371  9445 consensus_peers.cc:597] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32724e13cbcb49e991b11d18699e899a. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:18.188194  9637 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:18.211669  9472 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:18.211671  9551 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:18.214699 10399 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260808 01:49:18.221481 10260 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:18.338953 10420 raft_consensus.cc:2955] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:18.342096  9474 raft_consensus.cc:2955] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:18.342530  9551 raft_consensus.cc:2955] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
W20260808 01:49:18.343654  9600 consensus_peers.cc:597] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2069a355aad47b48173a35b4efc6f9d. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:18.368235  9374 catalog_manager.cc:5697] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:18.385803  9629 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:18.411746  9551 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:18.411927  9474 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 01:49:18.412472  9600 consensus_peers.cc:597] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3354328ed023401e83a2361925b02e81. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:18.418589 10418 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:18.424320 10418 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:18.447121 10399 raft_consensus.cc:2955] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:18.453351  9474 raft_consensus.cc:2955] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:18.452636  9550 raft_consensus.cc:2955] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:18.488562  9373 catalog_manager.cc:5697] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:18.509631 10426 ts_tablet_manager.cc:933] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359)
I20260808 01:49:18.557547  9637 consensus_queue.cc:237] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:18.587827 10426 tablet_copy_client.cc:323] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.193:39359
I20260808 01:49:18.659745  9550 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:18.661782 10420 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:18.662928  9474 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 01:49:18.661453  9600 consensus_peers.cc:597] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cbead77343f44dd82199bebcfb2def7. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:49:18.666052  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 100013us
I20260808 01:49:18.664999 10418 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260808 01:49:18.815106  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 113362us
W20260808 01:49:18.882037  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 217973us
I20260808 01:49:18.906600  9491 tablet_copy_service.cc:140] P 6e7856bed1be43acaa821290d506ee97: Received BeginTabletCopySession request for tablet 32724e13cbcb49e991b11d18699e899a from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:37248)
I20260808 01:49:18.907642  9491 tablet_copy_service.cc:161] P 6e7856bed1be43acaa821290d506ee97: Beginning new tablet copy session on tablet 32724e13cbcb49e991b11d18699e899a from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:37248: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-32724e13cbcb49e991b11d18699e899a
W20260808 01:49:18.943147 10440 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.061s	user 0.000s	sys 0.002s
I20260808 01:49:18.963706  9491 tablet_copy_source_session.cc:215] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:18.964708 10418 raft_consensus.cc:2955] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:18.972929 10441 ts_tablet_manager.cc:933] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:18.974709 10443 ts_tablet_manager.cc:933] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:18.973417  9474 raft_consensus.cc:2955] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:19.000759  9552 raft_consensus.cc:2955] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:19.018571 10426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32724e13cbcb49e991b11d18699e899a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 01:49:19.093061  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 104943us
I20260808 01:49:19.095137  9373 catalog_manager.cc:5697] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:19.105118 10441 tablet_copy_client.cc:323] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.195:34565
W20260808 01:49:19.107972  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 131201us
I20260808 01:49:19.125577 10443 tablet_copy_client.cc:323] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.195:34565
I20260808 01:49:19.147979 10426 tablet_copy_client.cc:806] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:19.149703 10426 tablet_copy_client.cc:670] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:19.213390 10426 tablet_copy_client.cc:538] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:19.270252  9648 tablet_copy_service.cc:140] P 259bcb3f0d254e8b9effac3277e654e0: Received BeginTabletCopySession request for tablet c2069a355aad47b48173a35b4efc6f9d from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:45658)
I20260808 01:49:19.271680  9648 tablet_copy_service.cc:161] P 259bcb3f0d254e8b9effac3277e654e0: Beginning new tablet copy session on tablet c2069a355aad47b48173a35b4efc6f9d from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:45658: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-c2069a355aad47b48173a35b4efc6f9d
I20260808 01:49:19.274715  9647 tablet_copy_service.cc:140] P 259bcb3f0d254e8b9effac3277e654e0: Received BeginTabletCopySession request for tablet 3354328ed023401e83a2361925b02e81 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:45658)
I20260808 01:49:19.275759  9647 tablet_copy_service.cc:161] P 259bcb3f0d254e8b9effac3277e654e0: Beginning new tablet copy session on tablet 3354328ed023401e83a2361925b02e81 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:45658: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-3354328ed023401e83a2361925b02e81
I20260808 01:49:19.290598 10426 tablet_bootstrap.cc:492] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:19.307693  9648 tablet_copy_source_session.cc:215] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:19.310164  9647 tablet_copy_source_session.cc:215] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:19.311403 10452 ts_tablet_manager.cc:933] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:19.314209 10441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2069a355aad47b48173a35b4efc6f9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:19.321491 10443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3354328ed023401e83a2361925b02e81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:19.394660 10452 tablet_copy_client.cc:323] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.195:34565
I20260808 01:49:19.396943  9647 tablet_copy_service.cc:140] P 259bcb3f0d254e8b9effac3277e654e0: Received BeginTabletCopySession request for tablet 8cbead77343f44dd82199bebcfb2def7 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:45658)
I20260808 01:49:19.397701  9647 tablet_copy_service.cc:161] P 259bcb3f0d254e8b9effac3277e654e0: Beginning new tablet copy session on tablet 8cbead77343f44dd82199bebcfb2def7 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:45658: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-8cbead77343f44dd82199bebcfb2def7
I20260808 01:49:19.404688 10443 tablet_copy_client.cc:806] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:19.405408  9647 tablet_copy_source_session.cc:215] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:19.405484 10443 tablet_copy_client.cc:670] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:19.411146 10443 tablet_copy_client.cc:538] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:19.422305 10452 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cbead77343f44dd82199bebcfb2def7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:19.430747 10441 tablet_copy_client.cc:806] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:19.432415 10441 tablet_copy_client.cc:670] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:19.456722 10443 tablet_bootstrap.cc:492] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:19.478724 10426 tablet_bootstrap.cc:492] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:19.481133 10426 tablet_bootstrap.cc:492] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:19.482587 10426 ts_tablet_manager.cc:1403] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.192s	user 0.026s	sys 0.023s
I20260808 01:49:19.495489 10426 raft_consensus.cc:359] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:19.497330 10426 raft_consensus.cc:740] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:19.498852 10426 consensus_queue.cc:260] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:19.505946 10441 tablet_copy_client.cc:538] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:19.532428 10426 ts_tablet_manager.cc:1434] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.049s	user 0.008s	sys 0.015s
I20260808 01:49:19.535159  9491 tablet_copy_service.cc:342] P 6e7856bed1be43acaa821290d506ee97: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-32724e13cbcb49e991b11d18699e899a received from {username='slave'} at 127.0.0.1:37248
I20260808 01:49:19.536075  9491 tablet_copy_service.cc:434] P 6e7856bed1be43acaa821290d506ee97: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-32724e13cbcb49e991b11d18699e899a on tablet 32724e13cbcb49e991b11d18699e899a with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:19.556962 10452 tablet_copy_client.cc:806] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:19.557869 10452 tablet_copy_client.cc:670] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:19.571743 10452 tablet_copy_client.cc:538] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:19.572784 10441 tablet_bootstrap.cc:492] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:19.627193 10452 tablet_bootstrap.cc:492] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:19.652863 10443 tablet_bootstrap.cc:492] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:19.654325 10443 tablet_bootstrap.cc:492] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:19.655426 10443 ts_tablet_manager.cc:1403] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.214s	user 0.034s	sys 0.005s
I20260808 01:49:19.701866 10443 raft_consensus.cc:359] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:19.703622 10443 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:19.714272 10443 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:19.724083 10452 tablet_bootstrap.cc:492] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:19.726301 10452 tablet_bootstrap.cc:492] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:19.727506 10452 ts_tablet_manager.cc:1403] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.101s	user 0.023s	sys 0.033s
I20260808 01:49:19.744427 10452 raft_consensus.cc:359] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:19.746381 10452 raft_consensus.cc:740] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:19.747972 10452 consensus_queue.cc:260] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:19.864667 10441 tablet_bootstrap.cc:492] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:19.865741 10441 tablet_bootstrap.cc:492] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:19.866544 10441 ts_tablet_manager.cc:1403] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.294s	user 0.101s	sys 0.089s
I20260808 01:49:19.871407 10441 raft_consensus.cc:359] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:19.872680 10441 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:19.873575 10443 ts_tablet_manager.cc:1434] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.217s	user 0.013s	sys 0.003s
I20260808 01:49:19.877676  9647 tablet_copy_service.cc:342] P 259bcb3f0d254e8b9effac3277e654e0: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-3354328ed023401e83a2361925b02e81 received from {username='slave'} at 127.0.0.1:45658
I20260808 01:49:19.878409  9647 tablet_copy_service.cc:434] P 259bcb3f0d254e8b9effac3277e654e0: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-3354328ed023401e83a2361925b02e81 on tablet 3354328ed023401e83a2361925b02e81 with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:19.880851 10368 raft_consensus.cc:1217] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260808 01:49:19.873833 10441 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:19.898635 10441 ts_tablet_manager.cc:1434] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.032s	user 0.005s	sys 0.006s
I20260808 01:49:19.900857 10452 ts_tablet_manager.cc:1434] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.173s	user 0.040s	sys 0.058s
I20260808 01:49:19.907377  9647 tablet_copy_service.cc:342] P 259bcb3f0d254e8b9effac3277e654e0: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-c2069a355aad47b48173a35b4efc6f9d received from {username='slave'} at 127.0.0.1:45658
I20260808 01:49:19.908013  9647 tablet_copy_service.cc:434] P 259bcb3f0d254e8b9effac3277e654e0: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-c2069a355aad47b48173a35b4efc6f9d on tablet c2069a355aad47b48173a35b4efc6f9d with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:19.909360  9648 tablet_copy_service.cc:342] P 259bcb3f0d254e8b9effac3277e654e0: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-8cbead77343f44dd82199bebcfb2def7 received from {username='slave'} at 127.0.0.1:45658
I20260808 01:49:19.910065  9648 tablet_copy_service.cc:434] P 259bcb3f0d254e8b9effac3277e654e0: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-8cbead77343f44dd82199bebcfb2def7 on tablet 8cbead77343f44dd82199bebcfb2def7 with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:19.910130 10368 raft_consensus.cc:1217] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260808 01:49:19.999218 10368 raft_consensus.cc:1217] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260808 01:49:20.118278 10368 raft_consensus.cc:1217] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260808 01:49:20.231822 10399 raft_consensus.cc:1064] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:20.235167 10399 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:20.246896 10368 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:20.247133  9554 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:20.248665 10265 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:49:20.250437 10446 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:20.252837  9473 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:20.254715 10399 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:20.283563  9473 raft_consensus.cc:2955] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:20.283303 10446 raft_consensus.cc:2955] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:20.287551  9554 raft_consensus.cc:2955] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:20.306304 10368 raft_consensus.cc:2955] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:20.316337  9372 catalog_manager.cc:5697] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:20.412133  9629 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:20.418318  9472 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:20.420228 10446 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:20.423552 10368 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:20.433056 10418 raft_consensus.cc:2955] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:20.438817 10399 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:20.441844 10434 raft_consensus.cc:1064] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:20.443898  9472 raft_consensus.cc:2955] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:20.451898  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 3354328ed023401e83a2361925b02e81 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:20.464391 10265 raft_consensus.cc:1064] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:20.476231 10265 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:20.469583  9374 catalog_manager.cc:5697] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:20.466771 10434 consensus_queue.cc:237] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:20.462627 10368 raft_consensus.cc:2955] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:20.531319 10369 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:20.540772 10399 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:20.544904  9476 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:20.546808  9547 tablet_service.cc:1558] Processing DeleteTablet for tablet 3354328ed023401e83a2361925b02e81 with delete_type TABLET_DATA_TOMBSTONED (TS 52ccf4dcca1843f7bb29e9172e2d4ab4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53202
I20260808 01:49:20.550894 10418 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:20.574039 10367 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:20.581247 10238 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260808 01:49:20.609373  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 79432.4us
I20260808 01:49:20.615015  9629 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260808 01:49:20.628051 10473 log.cc:927] Time spent T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.061s	user 0.000s	sys 0.003s
I20260808 01:49:20.771162  9552 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:20.772974  9554 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:20.786216 10238 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:20.787703 10474 tablet_replica.cc:333] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:49:20.830782 10474 raft_consensus.cc:2243] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:20.842993 10446 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260808 01:49:20.883584  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 70334.1us
I20260808 01:49:20.920483 10369 raft_consensus.cc:2955] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:20.983959 10474 raft_consensus.cc:2272] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:20.930002 10488 raft_consensus.cc:2955] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:20.991992 10474 ts_tablet_manager.cc:1916] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:21.023483 10486 raft_consensus.cc:1064] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
W20260808 01:49:21.028409  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 214292us
W20260808 01:49:21.061659  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 45529.3us
I20260808 01:49:21.104686  9473 raft_consensus.cc:2955] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.093912 10486 consensus_queue.cc:237] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:49:21.155951  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 434959us
I20260808 01:49:21.239305  9374 catalog_manager.cc:5697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.242339 10238 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260808 01:49:21.246594  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 72934.8us
I20260808 01:49:21.339968 10369 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:21.345775  9554 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:21.349730  9555 raft_consensus.cc:2955] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.360333 10486 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:21.376713 10474 ts_tablet_manager.cc:1929] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:49:21.383961 10436 raft_consensus.cc:2955] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.387346  9477 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:21.401258 10500 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:21.405697 10488 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:21.411731 10474 log.cc:1199] T 3354328ed023401e83a2361925b02e81 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-1-root/wals/3354328ed023401e83a2361925b02e81
I20260808 01:49:21.436791  9631 raft_consensus.cc:2955] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.448743  9360 catalog_manager.cc:5028] TS 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): tablet 3354328ed023401e83a2361925b02e81 (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:21.437302 10369 raft_consensus.cc:2955] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.495788  9374 catalog_manager.cc:5697] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:21.512059 10488 raft_consensus.cc:2955] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.518919  9550 raft_consensus.cc:2955] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.535234  9473 raft_consensus.cc:2955] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.549001  9373 catalog_manager.cc:5697] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.545266 10369 raft_consensus.cc:2955] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.571825  9631 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:21.593777  9477 consensus_queue.cc:237] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:21.647435  9550 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:21.648525 10369 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:21.649250 10260 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260808 01:49:21.657858 10500 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:21.693580  9550 raft_consensus.cc:2955] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.704936 10369 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:21.710749 10436 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:21.704955  9554 raft_consensus.cc:2955] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.709291 10488 raft_consensus.cc:2955] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.760995  9631 consensus_queue.cc:237] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:21.769285 10470 raft_consensus.cc:2955] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.776393 10367 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:21.783555 10369 raft_consensus.cc:2955] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.787040  9551 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:21.793885 10260 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:21.798842 10500 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260808 01:49:21.801580  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 66537.2us
I20260808 01:49:21.803977 10367 raft_consensus.cc:2955] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.810204 10500 raft_consensus.cc:2955] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.817734  9550 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:21.821972  9362 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 32724e13cbcb49e991b11d18699e899a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:21.830897  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet c2069a355aad47b48173a35b4efc6f9d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:21.830556  9550 raft_consensus.cc:2955] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.832825 10368 raft_consensus.cc:2955] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.838997  9373 catalog_manager.cc:5697] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 reported cstate change: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:21.851102  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8cbead77343f44dd82199bebcfb2def7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:21.853997 10489 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:21.855367  9624 tablet_service.cc:1558] Processing DeleteTablet for tablet 32724e13cbcb49e991b11d18699e899a with delete_type TABLET_DATA_TOMBSTONED (TS 259bcb3f0d254e8b9effac3277e654e0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42810
I20260808 01:49:21.856679  9373 catalog_manager.cc:5697] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:21.868891  9554 raft_consensus.cc:2955] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.893508 10512 tablet_replica.cc:333] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:49:21.894930 10512 raft_consensus.cc:2243] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:21.896132  9374 catalog_manager.cc:5697] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:21.899685 10512 raft_consensus.cc:2272] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:21.904132  9465 tablet_service.cc:1558] Processing DeleteTablet for tablet c2069a355aad47b48173a35b4efc6f9d with delete_type TABLET_DATA_TOMBSTONED (TS 6e7856bed1be43acaa821290d506ee97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50120
I20260808 01:49:21.910620 10514 tablet_replica.cc:333] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:49:21.912050 10514 raft_consensus.cc:2243] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:21.913015  9465 tablet_service.cc:1558] Processing DeleteTablet for tablet 8cbead77343f44dd82199bebcfb2def7 with delete_type TABLET_DATA_TOMBSTONED (TS 6e7856bed1be43acaa821290d506ee97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50120
I20260808 01:49:21.913010 10514 raft_consensus.cc:2272] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:21.925896 10512 ts_tablet_manager.cc:1916] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:21.934871 10514 ts_tablet_manager.cc:1916] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:21.949391 10512 ts_tablet_manager.cc:1929] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:49:21.950012 10512 log.cc:1199] T 32724e13cbcb49e991b11d18699e899a P 259bcb3f0d254e8b9effac3277e654e0: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-2-root/wals/32724e13cbcb49e991b11d18699e899a
I20260808 01:49:21.952242  9360 catalog_manager.cc:5028] TS 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): tablet 32724e13cbcb49e991b11d18699e899a (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:21.959074 10514 ts_tablet_manager.cc:1929] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:49:21.959683 10514 log.cc:1199] T c2069a355aad47b48173a35b4efc6f9d P 6e7856bed1be43acaa821290d506ee97: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-0-root/wals/c2069a355aad47b48173a35b4efc6f9d
I20260808 01:49:21.961835 10514 tablet_replica.cc:333] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:49:21.962704  9362 catalog_manager.cc:5028] TS 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): tablet c2069a355aad47b48173a35b4efc6f9d (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:21.963292 10514 raft_consensus.cc:2243] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:21.964250 10514 raft_consensus.cc:2272] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:21.969501 10514 ts_tablet_manager.cc:1916] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:21.989488 10514 ts_tablet_manager.cc:1929] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:49:21.989912 10514 log.cc:1199] T 8cbead77343f44dd82199bebcfb2def7 P 6e7856bed1be43acaa821290d506ee97: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-0-root/wals/8cbead77343f44dd82199bebcfb2def7
I20260808 01:49:21.991694  9362 catalog_manager.cc:5028] TS 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): tablet 8cbead77343f44dd82199bebcfb2def7 (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:23.035391  9554 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:23.045820  9636 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:23.048033 10482 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:23.049144  9473 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:23.050833 10482 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 01:49:23.086395  9523 consensus_peers.cc:597] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11c99c174e234329801ee9b1ceae68d9. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:23.098946 10495 raft_consensus.cc:2955] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:23.125643  9477 raft_consensus.cc:2955] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:23.136821  9631 raft_consensus.cc:2955] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:23.152961  9629 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:23.164762  9377 catalog_manager.cc:5697] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:23.180168  9554 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:23.181872  9477 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 01:49:23.182082  9600 consensus_peers.cc:597] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b1e1cf1fbe745feb69372fed0557ce0. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:23.186177 10418 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:23.195168 10529 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:23.212918 10418 raft_consensus.cc:2955] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:23.215444  9473 raft_consensus.cc:2955] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:23.234156  9631 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:23.246906  9554 raft_consensus.cc:2955] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:23.250896  9373 catalog_manager.cc:5697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260808 01:49:23.262295  9600 consensus_peers.cc:597] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c49fd321f8b4f89868246239530e78b. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:23.268188  9554 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:23.270727  9473 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:23.270771 10529 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:23.273161 10529 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:23.287971 10529 raft_consensus.cc:2955] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:23.289848  9477 raft_consensus.cc:2955] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:23.295445  9555 raft_consensus.cc:2955] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:23.311169  9373 catalog_manager.cc:5697] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:23.311215  9473 consensus_queue.cc:237] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:23.326529  9629 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 01:49:23.326701  9445 consensus_peers.cc:597] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 360b0cf59e7749058ca972bb487cbd10. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:23.328809 10510 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:23.328996  9554 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:23.331284 10510 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:23.343575 10515 raft_consensus.cc:2955] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:23.362079  9629 raft_consensus.cc:2955] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:23.405074  9554 raft_consensus.cc:2955] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:23.406409  9373 catalog_manager.cc:5697] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:23.538590 10542 ts_tablet_manager.cc:933] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:23.542217 10542 tablet_copy_client.cc:323] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.194:36095
I20260808 01:49:23.557590  9571 tablet_copy_service.cc:140] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Received BeginTabletCopySession request for tablet 11c99c174e234329801ee9b1ceae68d9 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:60552)
I20260808 01:49:23.558264  9571 tablet_copy_service.cc:161] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Beginning new tablet copy session on tablet 11c99c174e234329801ee9b1ceae68d9 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:60552: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-11c99c174e234329801ee9b1ceae68d9
I20260808 01:49:23.564189  9571 tablet_copy_source_session.cc:215] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:23.567883 10542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11c99c174e234329801ee9b1ceae68d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:23.586833 10542 tablet_copy_client.cc:806] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:23.587577 10542 tablet_copy_client.cc:670] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:23.596163 10542 tablet_copy_client.cc:538] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:23.606709 10542 tablet_bootstrap.cc:492] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:23.610417 10546 ts_tablet_manager.cc:933] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:23.612491 10546 tablet_copy_client.cc:323] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.195:34565
I20260808 01:49:23.614522  9648 tablet_copy_service.cc:140] P 259bcb3f0d254e8b9effac3277e654e0: Received BeginTabletCopySession request for tablet 7b1e1cf1fbe745feb69372fed0557ce0 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:45658)
I20260808 01:49:23.615134  9648 tablet_copy_service.cc:161] P 259bcb3f0d254e8b9effac3277e654e0: Beginning new tablet copy session on tablet 7b1e1cf1fbe745feb69372fed0557ce0 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:45658: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-7b1e1cf1fbe745feb69372fed0557ce0
I20260808 01:49:23.624261  9648 tablet_copy_source_session.cc:215] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:23.627358 10546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b1e1cf1fbe745feb69372fed0557ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:23.636571 10542 tablet_bootstrap.cc:492] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:23.637816 10542 tablet_bootstrap.cc:492] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:23.638669 10542 ts_tablet_manager.cc:1403] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.032s	user 0.014s	sys 0.020s
I20260808 01:49:23.642323 10542 raft_consensus.cc:359] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:23.643435 10542 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:23.644213 10542 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:23.651803 10546 tablet_copy_client.cc:806] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:23.652654 10546 tablet_copy_client.cc:670] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:23.655588 10542 ts_tablet_manager.cc:1434] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.016s	user 0.010s	sys 0.004s
I20260808 01:49:23.660954  9571 tablet_copy_service.cc:342] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-11c99c174e234329801ee9b1ceae68d9 received from {username='slave'} at 127.0.0.1:60552
I20260808 01:49:23.661504  9571 tablet_copy_service.cc:434] P 52ccf4dcca1843f7bb29e9172e2d4ab4: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-11c99c174e234329801ee9b1ceae68d9 on tablet 11c99c174e234329801ee9b1ceae68d9 with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:23.689829 10546 tablet_copy_client.cc:538] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:23.703276 10546 tablet_bootstrap.cc:492] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:23.728636 10546 tablet_bootstrap.cc:492] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:23.730367 10546 tablet_bootstrap.cc:492] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:23.731302 10546 ts_tablet_manager.cc:1403] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.028s	user 0.024s	sys 0.000s
I20260808 01:49:23.735023 10546 raft_consensus.cc:359] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:23.736140 10546 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:23.737535 10546 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:23.740020 10546 ts_tablet_manager.cc:1434] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260808 01:49:23.745478  9648 tablet_copy_service.cc:342] P 259bcb3f0d254e8b9effac3277e654e0: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-7b1e1cf1fbe745feb69372fed0557ce0 received from {username='slave'} at 127.0.0.1:45658
I20260808 01:49:23.746104  9648 tablet_copy_service.cc:434] P 259bcb3f0d254e8b9effac3277e654e0: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-7b1e1cf1fbe745feb69372fed0557ce0 on tablet 7b1e1cf1fbe745feb69372fed0557ce0 with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:23.827697 10546 ts_tablet_manager.cc:933] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:23.829476 10546 tablet_copy_client.cc:323] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.195:34565
I20260808 01:49:23.831226  9648 tablet_copy_service.cc:140] P 259bcb3f0d254e8b9effac3277e654e0: Received BeginTabletCopySession request for tablet 8c49fd321f8b4f89868246239530e78b from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:45658)
I20260808 01:49:23.831844  9648 tablet_copy_service.cc:161] P 259bcb3f0d254e8b9effac3277e654e0: Beginning new tablet copy session on tablet 8c49fd321f8b4f89868246239530e78b from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:45658: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-8c49fd321f8b4f89868246239530e78b
I20260808 01:49:23.840752  9648 tablet_copy_source_session.cc:215] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:23.843900 10546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c49fd321f8b4f89868246239530e78b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:23.856268 10542 ts_tablet_manager.cc:933] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359)
I20260808 01:49:23.860472 10542 tablet_copy_client.cc:323] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.193:39359
I20260808 01:49:23.860661 10546 tablet_copy_client.cc:806] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:23.861455 10546 tablet_copy_client.cc:670] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:23.863226  9491 tablet_copy_service.cc:140] P 6e7856bed1be43acaa821290d506ee97: Received BeginTabletCopySession request for tablet 360b0cf59e7749058ca972bb487cbd10 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:37248)
I20260808 01:49:23.863853  9491 tablet_copy_service.cc:161] P 6e7856bed1be43acaa821290d506ee97: Beginning new tablet copy session on tablet 360b0cf59e7749058ca972bb487cbd10 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:37248: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-360b0cf59e7749058ca972bb487cbd10
I20260808 01:49:23.869334 10546 tablet_copy_client.cc:538] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:23.872334  9491 tablet_copy_source_session.cc:215] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:23.878216 10542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 360b0cf59e7749058ca972bb487cbd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:23.895334 10546 tablet_bootstrap.cc:492] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:23.911214 10542 tablet_copy_client.cc:806] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:23.913343 10542 tablet_copy_client.cc:670] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:23.920504 10542 tablet_copy_client.cc:538] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:23.925642 10546 tablet_bootstrap.cc:492] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:23.926848 10546 tablet_bootstrap.cc:492] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:23.927695 10546 ts_tablet_manager.cc:1403] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.033s	user 0.022s	sys 0.004s
I20260808 01:49:23.931761 10542 tablet_bootstrap.cc:492] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:23.932454 10546 raft_consensus.cc:359] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:23.933466 10546 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:23.934235 10546 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:23.944231 10546 ts_tablet_manager.cc:1434] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260808 01:49:23.946265  9648 tablet_copy_service.cc:342] P 259bcb3f0d254e8b9effac3277e654e0: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-8c49fd321f8b4f89868246239530e78b received from {username='slave'} at 127.0.0.1:45658
I20260808 01:49:23.946863  9648 tablet_copy_service.cc:434] P 259bcb3f0d254e8b9effac3277e654e0: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-8c49fd321f8b4f89868246239530e78b on tablet 8c49fd321f8b4f89868246239530e78b with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:23.979867 10542 tablet_bootstrap.cc:492] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:23.981499 10542 tablet_bootstrap.cc:492] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:23.983716 10542 ts_tablet_manager.cc:1403] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.052s	user 0.027s	sys 0.007s
I20260808 01:49:23.992749 10542 raft_consensus.cc:359] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:23.993835 10542 raft_consensus.cc:740] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:23.995206 10542 consensus_queue.cc:260] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:23.999936 10542 ts_tablet_manager.cc:1434] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.016s	user 0.000s	sys 0.010s
I20260808 01:49:24.007400  9491 tablet_copy_service.cc:342] P 6e7856bed1be43acaa821290d506ee97: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-360b0cf59e7749058ca972bb487cbd10 received from {username='slave'} at 127.0.0.1:37248
I20260808 01:49:24.008358  9491 tablet_copy_service.cc:434] P 6e7856bed1be43acaa821290d506ee97: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-360b0cf59e7749058ca972bb487cbd10 on tablet 360b0cf59e7749058ca972bb487cbd10 with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:24.043977 10368 raft_consensus.cc:1217] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260808 01:49:24.156806 10368 raft_consensus.cc:1217] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260808 01:49:24.234791 10368 raft_consensus.cc:1217] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260808 01:49:24.451202 10368 raft_consensus.cc:1217] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260808 01:49:24.545290 10526 raft_consensus.cc:1064] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:24.548920 10526 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:24.553726 10556 raft_consensus.cc:1064] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:24.561164 10556 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:24.574388 10368 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:24.577185 10525 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:49:24.580142  9550 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:24.582028  9473 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:24.583566 10486 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:24.585665 10500 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:49:24.590736 10368 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:24.593314 10500 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:24.595784  9629 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:24.602991 10495 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:24.664860  9476 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:24.671398 10486 raft_consensus.cc:1064] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:24.680413 10486 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:24.715646 10564 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:24.736599 10529 raft_consensus.cc:2955] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:24.757186  9554 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:24.759124 10557 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:24.767539 10525 raft_consensus.cc:2955] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:24.789870  9477 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:24.793910  9629 raft_consensus.cc:2955] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:24.796957  9476 raft_consensus.cc:2955] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:24.800396 10369 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:24.813236 10556 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:24.808079 10368 raft_consensus.cc:2955] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:24.871543  9477 raft_consensus.cc:2955] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:24.827103 10500 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260808 01:49:24.890754  9377 catalog_manager.cc:5697] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:24.902951  9373 catalog_manager.cc:5697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:49:24.933384  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 88746.5us
I20260808 01:49:24.970197  9555 raft_consensus.cc:2955] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:24.975853 10557 raft_consensus.cc:2955] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:49:24.980957  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 49450.6us
I20260808 01:49:24.988237 10369 raft_consensus.cc:2955] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:24.999644  9476 raft_consensus.cc:2955] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.011704 10369 raft_consensus.cc:2955] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.021710  9555 raft_consensus.cc:2955] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.050268 10575 raft_consensus.cc:1064] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:25.053095 10575 consensus_queue.cc:237] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:25.057389  9372 catalog_manager.cc:5697] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97 reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.069859  9555 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:25.078830 10369 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:25.079260 10368 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:25.077695  9477 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:25.081195  9631 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:25.082906  9555 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:25.083252 10525 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260808 01:49:25.088567 10510 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:25.089015 10495 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:25.089781 10489 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:49:25.099594 10515 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:25.104719 10565 raft_consensus.cc:2955] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.113018 10369 raft_consensus.cc:2955] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.145812 10515 raft_consensus.cc:2955] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.150478  9629 raft_consensus.cc:2955] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.157352  9635 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:25.164193  9473 raft_consensus.cc:2955] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.168792  9480 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:25.171435 10367 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:25.174381 10556 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:25.181226  9638 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:25.185263 10500 raft_consensus.cc:2955] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.198994  9554 raft_consensus.cc:2955] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.203981 10568 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260808 01:49:25.203078  9377 catalog_manager.cc:5697] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:25.212514  9555 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:25.243269 10486 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:25.222148  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 7b1e1cf1fbe745feb69372fed0557ce0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:25.244690 10369 raft_consensus.cc:2955] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.230310  9373 catalog_manager.cc:5697] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.224253 10368 raft_consensus.cc:2955] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.213016 10367 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:25.255424 10568 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:25.256377  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 11c99c174e234329801ee9b1ceae68d9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:25.255342 10529 raft_consensus.cc:2955] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.257040  9555 raft_consensus.cc:2955] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.271399 10367 raft_consensus.cc:2955] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.282325  9473 raft_consensus.cc:2955] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.290396  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8c49fd321f8b4f89868246239530e78b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:25.297274  9624 tablet_service.cc:1558] Processing DeleteTablet for tablet 11c99c174e234329801ee9b1ceae68d9 with delete_type TABLET_DATA_TOMBSTONED (TS 259bcb3f0d254e8b9effac3277e654e0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42810
W20260808 01:49:25.302878  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 48852.3us
I20260808 01:49:25.312485  9371 catalog_manager.cc:5697] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.331146  9377 catalog_manager.cc:5697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:25.376682  9473 consensus_queue.cc:237] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:25.378116 10589 tablet_replica.cc:333] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:49:25.379365  9465 tablet_service.cc:1558] Processing DeleteTablet for tablet 8c49fd321f8b4f89868246239530e78b with delete_type TABLET_DATA_TOMBSTONED (TS 6e7856bed1be43acaa821290d506ee97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50120
I20260808 01:49:25.392375  9547 tablet_service.cc:1558] Processing DeleteTablet for tablet 7b1e1cf1fbe745feb69372fed0557ce0 with delete_type TABLET_DATA_TOMBSTONED (TS 52ccf4dcca1843f7bb29e9172e2d4ab4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53202
I20260808 01:49:25.393057 10590 tablet_replica.cc:333] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:49:25.395303 10590 raft_consensus.cc:2243] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:25.396946 10590 raft_consensus.cc:2272] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:25.408293 10591 tablet_replica.cc:333] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:49:25.409920 10590 ts_tablet_manager.cc:1916] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:25.412186 10589 raft_consensus.cc:2243] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:25.412415 10367 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:25.419927 10591 raft_consensus.cc:2243] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:25.423056 10510 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:25.422345 10591 raft_consensus.cc:2272] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:25.415036 10589 raft_consensus.cc:2272] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:25.429566 10589 ts_tablet_manager.cc:1916] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:25.431923  9629 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:25.441980 10489 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20260808 01:49:25.440927 10515 raft_consensus.cc:2955] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.443928 10369 raft_consensus.cc:2955] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.457427  9629 raft_consensus.cc:2955] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.456744 10590 ts_tablet_manager.cc:1929] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:49:25.459091 10590 log.cc:1199] T 8c49fd321f8b4f89868246239530e78b P 6e7856bed1be43acaa821290d506ee97: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-0-root/wals/8c49fd321f8b4f89868246239530e78b
I20260808 01:49:25.461198  9362 catalog_manager.cc:5028] TS 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): tablet 8c49fd321f8b4f89868246239530e78b (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:25.435997 10591 ts_tablet_manager.cc:1916] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:25.479682  9373 catalog_manager.cc:5697] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:25.487088  9362 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 360b0cf59e7749058ca972bb487cbd10 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:25.502221  9547 tablet_service.cc:1558] Processing DeleteTablet for tablet 360b0cf59e7749058ca972bb487cbd10 with delete_type TABLET_DATA_TOMBSTONED (TS 52ccf4dcca1843f7bb29e9172e2d4ab4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53202
I20260808 01:49:25.503628 10591 ts_tablet_manager.cc:1929] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:49:25.504339 10591 log.cc:1199] T 7b1e1cf1fbe745feb69372fed0557ce0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-1-root/wals/7b1e1cf1fbe745feb69372fed0557ce0
I20260808 01:49:25.507921 10591 tablet_replica.cc:333] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:49:25.511519  9360 catalog_manager.cc:5028] TS 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): tablet 7b1e1cf1fbe745feb69372fed0557ce0 (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:25.550390 10591 raft_consensus.cc:2243] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:25.551231 10589 ts_tablet_manager.cc:1929] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:49:25.551857 10589 log.cc:1199] T 11c99c174e234329801ee9b1ceae68d9 P 259bcb3f0d254e8b9effac3277e654e0: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-2-root/wals/11c99c174e234329801ee9b1ceae68d9
I20260808 01:49:25.551800 10591 raft_consensus.cc:2272] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:25.555433  9360 catalog_manager.cc:5028] TS 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): tablet 11c99c174e234329801ee9b1ceae68d9 (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:25.568126 10591 ts_tablet_manager.cc:1916] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:25.581169 10591 ts_tablet_manager.cc:1929] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:49:25.581575 10591 log.cc:1199] T 360b0cf59e7749058ca972bb487cbd10 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-1-root/wals/360b0cf59e7749058ca972bb487cbd10
I20260808 01:49:25.583079  9360 catalog_manager.cc:5028] TS 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): tablet 360b0cf59e7749058ca972bb487cbd10 (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:26.584656  9550 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:26.593451  9629 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:26.594761  9473 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 01:49:26.595580  9523 consensus_peers.cc:597] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2b2a88358244f93970835ea3120c380. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:26.596232 10565 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:26.597301 10564 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:26.615916 10564 raft_consensus.cc:2955] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:26.623261  9629 raft_consensus.cc:2955] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:26.647840  9373 catalog_manager.cc:5697] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:26.647895  9473 raft_consensus.cc:2955] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:26.674705  9477 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
W20260808 01:49:26.686491  9445 consensus_peers.cc:597] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20a4f92418934341a6ac4927db61dc87. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:26.690033  9554 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:26.690091  9629 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:26.692467 10515 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:26.693676 10575 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:26.730882 10575 raft_consensus.cc:2955] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:26.733978  9629 raft_consensus.cc:2955] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:26.747377  9554 raft_consensus.cc:2955] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:26.755394  9377 catalog_manager.cc:5697] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:26.769001  9476 consensus_queue.cc:237] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
W20260808 01:49:26.777634  9445 consensus_peers.cc:597] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 626249dbd6eb4d66bfa8cce2e716cbe2. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:26.784332  9629 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:26.786492 10555 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:26.790218  9550 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:26.793177 10555 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:26.805960 10515 raft_consensus.cc:2955] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:26.808257  9629 raft_consensus.cc:2955] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:26.812518  9550 raft_consensus.cc:2955] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:26.828024  9629 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:26.830353  9377 catalog_manager.cc:5697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
W20260808 01:49:26.868852  9600 consensus_peers.cc:597] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4bcbe89d624b472f8be5abc6b34c9026. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:26.872148  9473 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:26.874760 10556 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:26.878729  9550 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:26.880491 10557 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:26.888389 10556 raft_consensus.cc:2955] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:26.889894  9473 raft_consensus.cc:2955] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:26.923578  9555 raft_consensus.cc:2955] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:26.948267  9377 catalog_manager.cc:5697] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:27.028157 10618 ts_tablet_manager.cc:933] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:27.030098 10618 tablet_copy_client.cc:323] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.194:36095
I20260808 01:49:27.031970  9571 tablet_copy_service.cc:140] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Received BeginTabletCopySession request for tablet d2b2a88358244f93970835ea3120c380 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:60552)
I20260808 01:49:27.032605  9571 tablet_copy_service.cc:161] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Beginning new tablet copy session on tablet d2b2a88358244f93970835ea3120c380 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:60552: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-d2b2a88358244f93970835ea3120c380
I20260808 01:49:27.042320  9571 tablet_copy_source_session.cc:215] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:27.045851 10618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b2a88358244f93970835ea3120c380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:27.062220 10618 tablet_copy_client.cc:806] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:27.063023 10618 tablet_copy_client.cc:670] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:27.067444 10618 tablet_copy_client.cc:538] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:27.075197 10618 tablet_bootstrap.cc:492] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:27.093600 10618 tablet_bootstrap.cc:492] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:27.094633 10618 tablet_bootstrap.cc:492] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:27.095508 10618 ts_tablet_manager.cc:1403] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.020s	user 0.016s	sys 0.004s
I20260808 01:49:27.098390 10618 raft_consensus.cc:359] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:27.099368 10618 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:27.100171 10618 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:27.107583 10618 ts_tablet_manager.cc:1434] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.012s	user 0.012s	sys 0.000s
I20260808 01:49:27.112447  9571 tablet_copy_service.cc:342] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-d2b2a88358244f93970835ea3120c380 received from {username='slave'} at 127.0.0.1:60552
I20260808 01:49:27.112980  9571 tablet_copy_service.cc:434] P 52ccf4dcca1843f7bb29e9172e2d4ab4: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-d2b2a88358244f93970835ea3120c380 on tablet d2b2a88358244f93970835ea3120c380 with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:27.113662 10622 ts_tablet_manager.cc:933] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359)
I20260808 01:49:27.161290 10622 tablet_copy_client.cc:323] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.193:39359
I20260808 01:49:27.163657  9491 tablet_copy_service.cc:140] P 6e7856bed1be43acaa821290d506ee97: Received BeginTabletCopySession request for tablet 20a4f92418934341a6ac4927db61dc87 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:37248)
I20260808 01:49:27.164255  9491 tablet_copy_service.cc:161] P 6e7856bed1be43acaa821290d506ee97: Beginning new tablet copy session on tablet 20a4f92418934341a6ac4927db61dc87 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:37248: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-20a4f92418934341a6ac4927db61dc87
I20260808 01:49:27.176980  9491 tablet_copy_source_session.cc:215] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:27.183030 10622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20a4f92418934341a6ac4927db61dc87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:27.204177 10622 tablet_copy_client.cc:806] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:27.205001 10622 tablet_copy_client.cc:670] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:27.209894 10622 tablet_copy_client.cc:538] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:27.220750 10622 tablet_bootstrap.cc:492] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:27.255548 10622 tablet_bootstrap.cc:492] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:27.256811 10622 tablet_bootstrap.cc:492] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:27.257704 10622 ts_tablet_manager.cc:1403] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.037s	user 0.018s	sys 0.009s
I20260808 01:49:27.261374 10622 raft_consensus.cc:359] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:27.263036 10622 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:27.264001 10622 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:27.267238 10622 ts_tablet_manager.cc:1434] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260808 01:49:27.269553  9491 tablet_copy_service.cc:342] P 6e7856bed1be43acaa821290d506ee97: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-20a4f92418934341a6ac4927db61dc87 received from {username='slave'} at 127.0.0.1:37248
I20260808 01:49:27.270056  9491 tablet_copy_service.cc:434] P 6e7856bed1be43acaa821290d506ee97: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-20a4f92418934341a6ac4927db61dc87 on tablet 20a4f92418934341a6ac4927db61dc87 with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:27.317849 10622 ts_tablet_manager.cc:933] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359)
I20260808 01:49:27.319921 10622 tablet_copy_client.cc:323] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.193:39359
I20260808 01:49:27.321915  9491 tablet_copy_service.cc:140] P 6e7856bed1be43acaa821290d506ee97: Received BeginTabletCopySession request for tablet 626249dbd6eb4d66bfa8cce2e716cbe2 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:37248)
I20260808 01:49:27.322515  9491 tablet_copy_service.cc:161] P 6e7856bed1be43acaa821290d506ee97: Beginning new tablet copy session on tablet 626249dbd6eb4d66bfa8cce2e716cbe2 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:37248: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-626249dbd6eb4d66bfa8cce2e716cbe2
I20260808 01:49:27.329879  9491 tablet_copy_source_session.cc:215] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:27.333770 10622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 626249dbd6eb4d66bfa8cce2e716cbe2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:27.349727 10622 tablet_copy_client.cc:806] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:27.350571 10622 tablet_copy_client.cc:670] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:27.356348 10622 tablet_copy_client.cc:538] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:27.365845 10622 tablet_bootstrap.cc:492] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:27.373523 10618 ts_tablet_manager.cc:933] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:27.397711 10618 tablet_copy_client.cc:323] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.195:34565
I20260808 01:49:27.399457 10622 tablet_bootstrap.cc:492] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:27.399582  9648 tablet_copy_service.cc:140] P 259bcb3f0d254e8b9effac3277e654e0: Received BeginTabletCopySession request for tablet 4bcbe89d624b472f8be5abc6b34c9026 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:45658)
I20260808 01:49:27.400260  9648 tablet_copy_service.cc:161] P 259bcb3f0d254e8b9effac3277e654e0: Beginning new tablet copy session on tablet 4bcbe89d624b472f8be5abc6b34c9026 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:45658: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-4bcbe89d624b472f8be5abc6b34c9026
I20260808 01:49:27.400723 10622 tablet_bootstrap.cc:492] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:27.401356 10622 ts_tablet_manager.cc:1403] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.036s	user 0.029s	sys 0.005s
I20260808 01:49:27.404592 10622 raft_consensus.cc:359] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:27.405390 10622 raft_consensus.cc:740] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:27.406124 10622 consensus_queue.cc:260] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:27.408679  9648 tablet_copy_source_session.cc:215] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:27.409893 10622 ts_tablet_manager.cc:1434] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260808 01:49:27.411908  9491 tablet_copy_service.cc:342] P 6e7856bed1be43acaa821290d506ee97: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-626249dbd6eb4d66bfa8cce2e716cbe2 received from {username='slave'} at 127.0.0.1:37248
I20260808 01:49:27.412482  9491 tablet_copy_service.cc:434] P 6e7856bed1be43acaa821290d506ee97: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-626249dbd6eb4d66bfa8cce2e716cbe2 on tablet 626249dbd6eb4d66bfa8cce2e716cbe2 with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:27.412835 10618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bcbe89d624b472f8be5abc6b34c9026. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:27.427870 10369 raft_consensus.cc:1217] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260808 01:49:27.441464 10618 tablet_copy_client.cc:806] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:27.442188 10618 tablet_copy_client.cc:670] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:27.448010 10618 tablet_copy_client.cc:538] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:27.462313 10618 tablet_bootstrap.cc:492] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:27.484517 10618 tablet_bootstrap.cc:492] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:27.485404 10618 tablet_bootstrap.cc:492] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:27.486084 10618 ts_tablet_manager.cc:1403] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.024s	user 0.022s	sys 0.000s
I20260808 01:49:27.489527 10618 raft_consensus.cc:359] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:27.490563 10618 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:27.491452 10618 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:27.494582 10618 ts_tablet_manager.cc:1434] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.004s
I20260808 01:49:27.496824  9648 tablet_copy_service.cc:342] P 259bcb3f0d254e8b9effac3277e654e0: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-4bcbe89d624b472f8be5abc6b34c9026 received from {username='slave'} at 127.0.0.1:45658
I20260808 01:49:27.497344  9648 tablet_copy_service.cc:434] P 259bcb3f0d254e8b9effac3277e654e0: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-4bcbe89d624b472f8be5abc6b34c9026 on tablet 4bcbe89d624b472f8be5abc6b34c9026 with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:27.597052 10369 raft_consensus.cc:1217] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260808 01:49:27.919943 10369 raft_consensus.cc:1217] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260808 01:49:27.941133 10369 raft_consensus.cc:1217] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260808 01:49:27.971365 10600 raft_consensus.cc:1064] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:27.974184 10600 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:27.986274 10369 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:27.988694  9638 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:27.990208  9477 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:27.992054 10631 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:27.993454 10564 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:49:27.996344 10564 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:49:28.013329 10600 raft_consensus.cc:2955] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.016160  9477 raft_consensus.cc:2955] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.055529  9638 raft_consensus.cc:2955] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.060354 10369 raft_consensus.cc:2955] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.086462  9377 catalog_manager.cc:5697] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:28.128140  9555 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:28.134955 10369 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:28.142143 10564 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:28.142885  9477 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:28.143481 10612 raft_consensus.cc:1064] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:28.148368 10600 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:28.146247 10612 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:28.162585 10632 raft_consensus.cc:2955] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.167730 10369 raft_consensus.cc:2955] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.171798  9473 raft_consensus.cc:2955] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.185353 10367 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:28.186147  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet d2b2a88358244f93970835ea3120c380 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:28.187314  9638 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:28.188894 10612 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:28.190440  9371 catalog_manager.cc:5697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.201004  9554 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:28.201238 10612 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:28.213112 10515 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:28.252185 10612 raft_consensus.cc:2955] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.254568 10369 raft_consensus.cc:2955] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.262339  9554 raft_consensus.cc:2955] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.277096  9624 tablet_service.cc:1558] Processing DeleteTablet for tablet d2b2a88358244f93970835ea3120c380 with delete_type TABLET_DATA_TOMBSTONED (TS 259bcb3f0d254e8b9effac3277e654e0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42810
I20260808 01:49:28.279933  9638 raft_consensus.cc:2955] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.289386 10645 tablet_replica.cc:333] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:49:28.292667  9371 catalog_manager.cc:5697] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.295799 10645 raft_consensus.cc:2243] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:28.303965 10645 raft_consensus.cc:2272] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:28.308413 10645 ts_tablet_manager.cc:1916] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:28.335405 10645 ts_tablet_manager.cc:1929] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:49:28.336061 10645 log.cc:1199] T d2b2a88358244f93970835ea3120c380 P 259bcb3f0d254e8b9effac3277e654e0: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-2-root/wals/d2b2a88358244f93970835ea3120c380
I20260808 01:49:28.340308  9476 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:28.343060  9360 catalog_manager.cc:5028] TS 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): tablet d2b2a88358244f93970835ea3120c380 (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:28.354176 10369 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:28.356137 10612 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:28.364567  9638 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:28.366638 10612 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:28.369467 10515 raft_consensus.cc:2955] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.379331 10369 raft_consensus.cc:2955] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.390020  9629 raft_consensus.cc:2955] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.398291  9373 catalog_manager.cc:5697] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.409020  9362 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 20a4f92418934341a6ac4927db61dc87 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:28.412254 10556 raft_consensus.cc:1064] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:28.412295  9547 tablet_service.cc:1558] Processing DeleteTablet for tablet 20a4f92418934341a6ac4927db61dc87 with delete_type TABLET_DATA_TOMBSTONED (TS 52ccf4dcca1843f7bb29e9172e2d4ab4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53202
I20260808 01:49:28.418188 10556 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:28.439850 10646 tablet_replica.cc:333] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:49:28.442960 10646 raft_consensus.cc:2243] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:28.447957 10369 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:28.449052  9551 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:28.450157  9476 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:28.469875 10557 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:49:28.473057 10615 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:28.482751 10557 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:28.512686 10646 raft_consensus.cc:2272] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:28.517534 10646 ts_tablet_manager.cc:1916] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:28.539860 10648 raft_consensus.cc:1064] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:28.545356 10648 consensus_queue.cc:237] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:28.577477 10556 raft_consensus.cc:2955] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.591595 10646 ts_tablet_manager.cc:1929] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:49:28.592288 10646 log.cc:1199] T 20a4f92418934341a6ac4927db61dc87 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-1-root/wals/20a4f92418934341a6ac4927db61dc87
I20260808 01:49:28.594975  9476 raft_consensus.cc:2955] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.600904  9551 raft_consensus.cc:2955] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.625711 10369 raft_consensus.cc:2955] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.634085  9360 catalog_manager.cc:5028] TS 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): tablet 20a4f92418934341a6ac4927db61dc87 (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:28.639817  9373 catalog_manager.cc:5697] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.654158 10367 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:28.665174 10555 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:28.681497  9631 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:28.688079 10555 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:28.714984  9631 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:28.733973  9555 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:28.741767 10369 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:28.743461  9554 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:28.744360 10650 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:28.757357 10648 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:28.751600 10654 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20260808 01:49:28.779778 10653 raft_consensus.cc:2955] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.805380  9550 raft_consensus.cc:2955] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.786607 10367 raft_consensus.cc:2955] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.812084 10555 raft_consensus.cc:2955] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.821709  9554 raft_consensus.cc:2955] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.835602  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 4bcbe89d624b472f8be5abc6b34c9026 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:28.849686  9373 catalog_manager.cc:5697] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.863549 10369 raft_consensus.cc:2955] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.882356  9372 catalog_manager.cc:5697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.905288  9465 tablet_service.cc:1558] Processing DeleteTablet for tablet 4bcbe89d624b472f8be5abc6b34c9026 with delete_type TABLET_DATA_TOMBSTONED (TS 6e7856bed1be43acaa821290d506ee97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50120
I20260808 01:49:28.940732 10663 tablet_replica.cc:333] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:49:28.951303 10663 raft_consensus.cc:2243] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:28.953824 10663 raft_consensus.cc:2272] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:28.963846  9473 consensus_queue.cc:237] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:28.979133 10369 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:28.982353 10663 ts_tablet_manager.cc:1916] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:28.983790 10664 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:28.992311 10612 raft_consensus.cc:2955] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:28.994856  9550 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:28.997782 10658 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:29.000609 10369 raft_consensus.cc:2955] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:29.004534  9550 raft_consensus.cc:2955] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:29.017951  9362 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 626249dbd6eb4d66bfa8cce2e716cbe2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:29.060789  9372 catalog_manager.cc:5697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:29.148327 10663 ts_tablet_manager.cc:1929] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:49:29.150970 10663 log.cc:1199] T 4bcbe89d624b472f8be5abc6b34c9026 P 6e7856bed1be43acaa821290d506ee97: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-0-root/wals/4bcbe89d624b472f8be5abc6b34c9026
I20260808 01:49:29.167098  9624 tablet_service.cc:1558] Processing DeleteTablet for tablet 626249dbd6eb4d66bfa8cce2e716cbe2 with delete_type TABLET_DATA_TOMBSTONED (TS 259bcb3f0d254e8b9effac3277e654e0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42810
I20260808 01:49:29.167207  9362 catalog_manager.cc:5028] TS 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): tablet 4bcbe89d624b472f8be5abc6b34c9026 (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:29.187942 10666 tablet_replica.cc:333] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:49:29.189172 10666 raft_consensus.cc:2243] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:29.189929 10666 pending_rounds.cc:70] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0: Trying to abort 1 pending ops.
I20260808 01:49:29.190192 10666 pending_rounds.cc:77] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7316085835947003904 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } } }
I20260808 01:49:29.191504 10666 raft_consensus.cc:2917] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20260808 01:49:29.192046 10666 raft_consensus.cc:2889] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260808 01:49:29.192577 10666 raft_consensus.cc:2272] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:29.197470 10666 ts_tablet_manager.cc:1916] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:29.223356 10666 ts_tablet_manager.cc:1929] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:49:29.223888 10666 log.cc:1199] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 259bcb3f0d254e8b9effac3277e654e0: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-2-root/wals/626249dbd6eb4d66bfa8cce2e716cbe2
I20260808 01:49:29.225538  9360 catalog_manager.cc:5028] TS 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): tablet 626249dbd6eb4d66bfa8cce2e716cbe2 (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:30.233984  9555 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:30.246232  9638 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:49:30.247917 10631 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260808 01:49:30.252853  9523 consensus_peers.cc:597] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5cf7e7eadc2347aa84c98413849ebfc9. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:30.255679  9473 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:49:30.257522 10631 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:49:30.261143 10600 raft_consensus.cc:2955] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:30.264122  9631 raft_consensus.cc:2955] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:30.269234  9477 raft_consensus.cc:2955] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:30.279063  9473 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:30.291806  9635 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 01:49:30.291863  9445 consensus_peers.cc:597] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54a2156aecd649d391b0acd6dc223843. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:30.295050 10667 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:30.300650  9373 catalog_manager.cc:5697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index -1 to 3, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 2 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:30.308684  9550 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:30.312436 10664 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:30.321406 10678 raft_consensus.cc:2955] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:30.323939  9638 raft_consensus.cc:2955] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:30.352339  9550 raft_consensus.cc:2955] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:30.355710  9555 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:30.371129  9372 catalog_manager.cc:5697] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:30.376461  9473 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260808 01:49:30.377004  9523 consensus_peers.cc:597] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f04078e3beb42eda250da45f7485537. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:30.378093 10600 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:30.378756  9638 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:30.382872 10600 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:30.393540 10600 raft_consensus.cc:2955] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:30.396173  9473 raft_consensus.cc:2955] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:30.396979  9638 raft_consensus.cc:2955] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:30.410835  9555 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:30.419327  9372 catalog_manager.cc:5697] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:30.430711  9638 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:30.432266 10631 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 01:49:30.434612  9523 consensus_peers.cc:597] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df6982ec010444a8854140f7b64e985d. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:30.436589  9477 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:30.442921 10600 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:30.448835 10631 raft_consensus.cc:2955] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:30.456912  9631 raft_consensus.cc:2955] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:30.507097  9371 catalog_manager.cc:5697] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:30.476222  9476 raft_consensus.cc:2955] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:30.723951 10691 ts_tablet_manager.cc:933] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359)
I20260808 01:49:30.726068 10691 tablet_copy_client.cc:323] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.193:39359
I20260808 01:49:30.727880  9491 tablet_copy_service.cc:140] P 6e7856bed1be43acaa821290d506ee97: Received BeginTabletCopySession request for tablet 54a2156aecd649d391b0acd6dc223843 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:37248)
I20260808 01:49:30.728493  9491 tablet_copy_service.cc:161] P 6e7856bed1be43acaa821290d506ee97: Beginning new tablet copy session on tablet 54a2156aecd649d391b0acd6dc223843 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:37248: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-54a2156aecd649d391b0acd6dc223843
I20260808 01:49:30.737871  9491 tablet_copy_source_session.cc:215] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:30.742023 10691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54a2156aecd649d391b0acd6dc223843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:30.765820 10691 tablet_copy_client.cc:806] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:30.766623 10691 tablet_copy_client.cc:670] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:30.771373 10691 tablet_copy_client.cc:538] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:30.789539 10691 tablet_bootstrap.cc:492] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:30.798588 10693 ts_tablet_manager.cc:933] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:30.807919 10693 tablet_copy_client.cc:323] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.194:36095
I20260808 01:49:30.812870  9571 tablet_copy_service.cc:140] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Received BeginTabletCopySession request for tablet 5cf7e7eadc2347aa84c98413849ebfc9 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:60552)
I20260808 01:49:30.813632  9571 tablet_copy_service.cc:161] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Beginning new tablet copy session on tablet 5cf7e7eadc2347aa84c98413849ebfc9 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:60552: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-5cf7e7eadc2347aa84c98413849ebfc9
I20260808 01:49:30.825275 10691 tablet_bootstrap.cc:492] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:30.826721 10691 tablet_bootstrap.cc:492] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:30.827685 10691 ts_tablet_manager.cc:1403] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.038s	user 0.019s	sys 0.015s
I20260808 01:49:30.849932 10691 raft_consensus.cc:359] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:30.851176 10691 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:30.851985 10691 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:30.853224  9571 tablet_copy_source_session.cc:215] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:30.864037 10693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cf7e7eadc2347aa84c98413849ebfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:30.878244 10691 ts_tablet_manager.cc:1434] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.050s	user 0.014s	sys 0.005s
I20260808 01:49:30.883114  9491 tablet_copy_service.cc:342] P 6e7856bed1be43acaa821290d506ee97: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-54a2156aecd649d391b0acd6dc223843 received from {username='slave'} at 127.0.0.1:37248
I20260808 01:49:30.883736  9491 tablet_copy_service.cc:434] P 6e7856bed1be43acaa821290d506ee97: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-54a2156aecd649d391b0acd6dc223843 on tablet 54a2156aecd649d391b0acd6dc223843 with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:30.896888 10693 tablet_copy_client.cc:806] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:30.897779 10693 tablet_copy_client.cc:670] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:30.903257 10693 tablet_copy_client.cc:538] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:30.915124 10693 tablet_bootstrap.cc:492] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:30.950150 10693 tablet_bootstrap.cc:492] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:30.950881 10691 ts_tablet_manager.cc:933] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:30.951308 10693 tablet_bootstrap.cc:492] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:30.952154 10693 ts_tablet_manager.cc:1403] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.037s	user 0.015s	sys 0.011s
I20260808 01:49:30.956558 10693 raft_consensus.cc:359] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:30.957628 10693 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:30.958367 10693 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:30.962474 10693 ts_tablet_manager.cc:1434] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.001s
I20260808 01:49:30.963654 10691 tablet_copy_client.cc:323] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.194:36095
I20260808 01:49:30.965718  9571 tablet_copy_service.cc:140] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Received BeginTabletCopySession request for tablet df6982ec010444a8854140f7b64e985d from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:60552)
I20260808 01:49:30.966354  9571 tablet_copy_service.cc:161] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Beginning new tablet copy session on tablet df6982ec010444a8854140f7b64e985d from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:60552: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-df6982ec010444a8854140f7b64e985d
I20260808 01:49:30.968257  9570 tablet_copy_service.cc:342] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-5cf7e7eadc2347aa84c98413849ebfc9 received from {username='slave'} at 127.0.0.1:60552
I20260808 01:49:30.968807  9570 tablet_copy_service.cc:434] P 52ccf4dcca1843f7bb29e9172e2d4ab4: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-5cf7e7eadc2347aa84c98413849ebfc9 on tablet 5cf7e7eadc2347aa84c98413849ebfc9 with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:30.977677  9571 tablet_copy_source_session.cc:215] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:30.981379 10691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df6982ec010444a8854140f7b64e985d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:30.987093 10693 ts_tablet_manager.cc:933] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:30.995707 10693 tablet_copy_client.cc:323] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.194:36095
I20260808 01:49:30.997745  9571 tablet_copy_service.cc:140] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Received BeginTabletCopySession request for tablet 4f04078e3beb42eda250da45f7485537 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:60552)
I20260808 01:49:30.998477  9571 tablet_copy_service.cc:161] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Beginning new tablet copy session on tablet 4f04078e3beb42eda250da45f7485537 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:60552: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-4f04078e3beb42eda250da45f7485537
I20260808 01:49:31.005918 10691 tablet_copy_client.cc:806] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:31.006904 10691 tablet_copy_client.cc:670] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:31.099033 10691 tablet_copy_client.cc:538] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:31.136065 10368 raft_consensus.cc:1217] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
W20260808 01:49:31.165556  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 55418.6us
I20260808 01:49:31.190409  9571 tablet_copy_source_session.cc:215] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:31.229785 10691 tablet_bootstrap.cc:492] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:31.232621 10693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f04078e3beb42eda250da45f7485537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:31.431298 10368 raft_consensus.cc:1217] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260808 01:49:31.451282 10693 tablet_copy_client.cc:806] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:31.452206 10693 tablet_copy_client.cc:670] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:31.478065 10693 tablet_copy_client.cc:538] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:31.500087 10691 tablet_bootstrap.cc:492] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:31.504364 10691 tablet_bootstrap.cc:492] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:31.505333 10691 ts_tablet_manager.cc:1403] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.276s	user 0.020s	sys 0.021s
I20260808 01:49:31.512144 10691 raft_consensus.cc:359] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:31.513724 10691 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:31.514565 10691 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:31.518743 10693 tablet_bootstrap.cc:492] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:31.527456 10691 ts_tablet_manager.cc:1434] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.004s
I20260808 01:49:31.531558  9571 tablet_copy_service.cc:342] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-df6982ec010444a8854140f7b64e985d received from {username='slave'} at 127.0.0.1:60552
I20260808 01:49:31.532188  9571 tablet_copy_service.cc:434] P 52ccf4dcca1843f7bb29e9172e2d4ab4: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-df6982ec010444a8854140f7b64e985d on tablet df6982ec010444a8854140f7b64e985d with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:31.556041 10693 tablet_bootstrap.cc:492] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:31.558877 10693 tablet_bootstrap.cc:492] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:31.559814 10693 ts_tablet_manager.cc:1403] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.041s	user 0.024s	sys 0.006s
I20260808 01:49:31.563741 10693 raft_consensus.cc:359] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:31.564803 10693 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:31.565578 10693 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:31.568490 10693 ts_tablet_manager.cc:1434] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260808 01:49:31.570785  9571 tablet_copy_service.cc:342] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-4f04078e3beb42eda250da45f7485537 received from {username='slave'} at 127.0.0.1:60552
I20260808 01:49:31.571269  9571 tablet_copy_service.cc:434] P 52ccf4dcca1843f7bb29e9172e2d4ab4: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-4f04078e3beb42eda250da45f7485537 on tablet 4f04078e3beb42eda250da45f7485537 with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:31.637039 10714 raft_consensus.cc:1064] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:31.640152 10714 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:31.647807 10368 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:31.647837  9635 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:31.651652  9551 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:31.651610 10678 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:31.653810 10696 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:31.656410 10678 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:49:31.671964 10696 raft_consensus.cc:2955] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:31.675308  9551 raft_consensus.cc:2955] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:31.675374  9635 raft_consensus.cc:2955] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:31.709046 10369 raft_consensus.cc:2955] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:31.706521  9371 catalog_manager.cc:5697] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:31.780275  9476 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:31.800644 10369 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:31.811277 10664 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:31.824021  9635 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:31.827808 10678 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:31.833575 10714 raft_consensus.cc:2955] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:31.855609 10369 raft_consensus.cc:2955] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:31.868602 10368 raft_consensus.cc:1217] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260808 01:49:31.886574  9635 raft_consensus.cc:2955] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:31.889310  9362 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 54a2156aecd649d391b0acd6dc223843 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:31.903203  9373 catalog_manager.cc:5697] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:31.921578  9547 tablet_service.cc:1558] Processing DeleteTablet for tablet 54a2156aecd649d391b0acd6dc223843 with delete_type TABLET_DATA_TOMBSTONED (TS 52ccf4dcca1843f7bb29e9172e2d4ab4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53202
I20260808 01:49:31.925540 10725 tablet_replica.cc:333] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:49:31.927057 10725 raft_consensus.cc:2243] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:31.929040 10725 raft_consensus.cc:2272] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:31.935781 10725 ts_tablet_manager.cc:1916] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:31.955812 10725 ts_tablet_manager.cc:1929] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:49:31.956406 10725 log.cc:1199] T 54a2156aecd649d391b0acd6dc223843 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-1-root/wals/54a2156aecd649d391b0acd6dc223843
I20260808 01:49:31.959355  9360 catalog_manager.cc:5028] TS 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): tablet 54a2156aecd649d391b0acd6dc223843 (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:32.011242 10369 raft_consensus.cc:1217] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260808 01:49:32.024529 10712 raft_consensus.cc:1064] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:32.027539 10712 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:32.036046 10369 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEARNER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:49:32.036993  9638 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:49:32.038249  9473 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:49:32.038102 10600 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:32.040184 10713 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:32.042306 10713 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:32.062057 10712 raft_consensus.cc:2955] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.065204  9476 raft_consensus.cc:2955] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.070006  9638 raft_consensus.cc:2955] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.080293 10369 raft_consensus.cc:2955] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.096243  9371 catalog_manager.cc:5697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: config changed from index 3 to 4, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:32.136684  9551 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:32.141815 10369 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:49:32.141911  9476 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:49:32.143361 10703 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:49:32.144271 10712 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:49:32.153406 10713 raft_consensus.cc:2955] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.155747 10369 raft_consensus.cc:2955] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.162221  9476 raft_consensus.cc:2955] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.179744  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5cf7e7eadc2347aa84c98413849ebfc9 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:32.179057  9373 catalog_manager.cc:5697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 4 to 5, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New cstate: current_term: 2 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.192413  9624 tablet_service.cc:1558] Processing DeleteTablet for tablet 5cf7e7eadc2347aa84c98413849ebfc9 with delete_type TABLET_DATA_TOMBSTONED (TS 259bcb3f0d254e8b9effac3277e654e0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42810
I20260808 01:49:32.211403 10730 tablet_replica.cc:333] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:49:32.213112 10730 raft_consensus.cc:2243] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:32.214970 10730 raft_consensus.cc:2272] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:32.218956 10730 ts_tablet_manager.cc:1916] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:32.246591 10730 ts_tablet_manager.cc:1929] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260808 01:49:32.247166 10730 log.cc:1199] T 5cf7e7eadc2347aa84c98413849ebfc9 P 259bcb3f0d254e8b9effac3277e654e0: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-2-root/wals/5cf7e7eadc2347aa84c98413849ebfc9
I20260808 01:49:32.250458  9360 catalog_manager.cc:5028] TS 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): tablet 5cf7e7eadc2347aa84c98413849ebfc9 (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:32.401669 10703 raft_consensus.cc:1064] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:32.404873 10703 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:32.415483 10368 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:32.417450 10713 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:32.430869  9635 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:32.433151 10631 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:32.438311  9473 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:32.441588 10712 raft_consensus.cc:1064] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:32.444723 10712 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:32.446374 10600 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:32.469022 10703 raft_consensus.cc:2955] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.492265 10368 raft_consensus.cc:2955] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.501276  9631 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:32.502112 10369 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:32.503024 10600 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:49:32.505012 10600 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:32.498755  9638 raft_consensus.cc:2955] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.515021  9477 raft_consensus.cc:2955] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.551826  9473 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:32.562356  9377 catalog_manager.cc:5697] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:32.567512 10631 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:32.595059  9554 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:32.607115 10368 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:32.609552  9638 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:32.609062 10631 raft_consensus.cc:2955] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.611845  9635 raft_consensus.cc:2955] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.613003 10703 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:32.620651 10600 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:32.641333 10703 raft_consensus.cc:2955] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.643929 10369 raft_consensus.cc:2955] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.645067  9477 raft_consensus.cc:2955] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.645116 10368 raft_consensus.cc:2955] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.666456  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 4f04078e3beb42eda250da45f7485537 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:32.666469  9371 catalog_manager.cc:5697] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.642841  9373 catalog_manager.cc:5697] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:32.679612  9631 raft_consensus.cc:2955] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.697866  9554 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:32.710742  9465 tablet_service.cc:1558] Processing DeleteTablet for tablet 4f04078e3beb42eda250da45f7485537 with delete_type TABLET_DATA_TOMBSTONED (TS 6e7856bed1be43acaa821290d506ee97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50120
I20260808 01:49:32.714648  9638 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:32.717808 10713 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:32.719076 10739 tablet_replica.cc:333] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:49:32.720399 10739 raft_consensus.cc:2243] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:32.728205 10739 raft_consensus.cc:2272] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:32.730620 10600 raft_consensus.cc:2955] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.744704 10739 ts_tablet_manager.cc:1916] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:32.732285  9638 raft_consensus.cc:2955] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.730049 10369 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:32.753924 10631 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:32.765816 10369 raft_consensus.cc:2955] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.772150 10739 ts_tablet_manager.cc:1929] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:49:32.772938 10739 log.cc:1199] T 4f04078e3beb42eda250da45f7485537 P 6e7856bed1be43acaa821290d506ee97: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-0-root/wals/4f04078e3beb42eda250da45f7485537
I20260808 01:49:32.775271  9362 catalog_manager.cc:5028] TS 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): tablet 4f04078e3beb42eda250da45f7485537 (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:32.786643  9373 catalog_manager.cc:5697] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:32.789280  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet df6982ec010444a8854140f7b64e985d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:32.803447  9465 tablet_service.cc:1558] Processing DeleteTablet for tablet df6982ec010444a8854140f7b64e985d with delete_type TABLET_DATA_TOMBSTONED (TS 6e7856bed1be43acaa821290d506ee97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50120
I20260808 01:49:32.807515 10739 tablet_replica.cc:333] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:49:32.808683 10739 raft_consensus.cc:2243] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:32.809605 10739 raft_consensus.cc:2272] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:32.815320 10739 ts_tablet_manager.cc:1916] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:32.829753 10739 ts_tablet_manager.cc:1929] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:49:32.830148 10739 log.cc:1199] T df6982ec010444a8854140f7b64e985d P 6e7856bed1be43acaa821290d506ee97: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-0-root/wals/df6982ec010444a8854140f7b64e985d
I20260808 01:49:32.831708  9362 catalog_manager.cc:5028] TS 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): tablet df6982ec010444a8854140f7b64e985d (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:33.886116  9554 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:33.896050  9631 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:33.899128 10600 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:33.899153  9477 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:33.902762 10600 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 01:49:33.905189  9523 consensus_peers.cc:597] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e57f5706d264426eb123cb26c6f3b89c. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:33.911912 10631 raft_consensus.cc:2955] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:33.914089  9635 raft_consensus.cc:2955] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:33.920264  9473 raft_consensus.cc:2955] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:33.929298  9635 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:33.931921  9373 catalog_manager.cc:5697] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:33.938805  9554 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:33.938982  9473 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:33.941260 10702 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260808 01:49:33.944034  9600 consensus_peers.cc:597] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea7d6b03a61944ea8a8201984886f30a. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:33.946240 10702 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:33.959903 10702 raft_consensus.cc:2955] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:33.961097  9551 raft_consensus.cc:2955] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:33.968017  9473 raft_consensus.cc:2955] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:34.031041  9473 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:34.036131  9373 catalog_manager.cc:5697] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:34.066252  9638 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:34.067623  9554 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260808 01:49:34.067696  9445 consensus_peers.cc:597] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cab93716156d46d3b0bda8d98d1fc76c. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:34.069270 10740 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:34.070441 10664 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:34.079897 10678 raft_consensus.cc:2955] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:34.083606  9635 raft_consensus.cc:2955] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:34.083559  9551 raft_consensus.cc:2955] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:34.097236  9372 catalog_manager.cc:5697] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 2 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:34.102643  9638 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:34.114977  9551 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:34.116061  9473 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:34.117105 10758 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:34.119179 10758 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 01:49:34.122200  9600 consensus_peers.cc:597] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f358d8403bf440059a6753dc4ed0be38. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:34.138350 10755 raft_consensus.cc:2955] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:34.146796  9554 raft_consensus.cc:2955] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:34.148572  9473 raft_consensus.cc:2955] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:34.161881  9372 catalog_manager.cc:5697] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:34.479573 10767 ts_tablet_manager.cc:933] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:34.482378 10767 tablet_copy_client.cc:323] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.194:36095
I20260808 01:49:34.484915  9571 tablet_copy_service.cc:140] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Received BeginTabletCopySession request for tablet e57f5706d264426eb123cb26c6f3b89c from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:60552)
I20260808 01:49:34.485798  9571 tablet_copy_service.cc:161] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Beginning new tablet copy session on tablet e57f5706d264426eb123cb26c6f3b89c from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:60552: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-e57f5706d264426eb123cb26c6f3b89c
I20260808 01:49:34.498067  9571 tablet_copy_source_session.cc:215] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:34.501652 10767 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e57f5706d264426eb123cb26c6f3b89c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:34.517789 10767 tablet_copy_client.cc:806] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:34.518460 10767 tablet_copy_client.cc:670] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:34.524031 10767 tablet_copy_client.cc:538] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:34.533557 10767 tablet_bootstrap.cc:492] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:34.551590 10770 ts_tablet_manager.cc:933] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:34.556478 10767 tablet_bootstrap.cc:492] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:34.558045 10767 tablet_bootstrap.cc:492] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:34.558990 10767 ts_tablet_manager.cc:1403] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.026s	user 0.021s	sys 0.004s
I20260808 01:49:34.561672 10770 tablet_copy_client.cc:323] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.195:34565
I20260808 01:49:34.562413 10767 raft_consensus.cc:359] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:34.563499 10767 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:34.564105  9648 tablet_copy_service.cc:140] P 259bcb3f0d254e8b9effac3277e654e0: Received BeginTabletCopySession request for tablet ea7d6b03a61944ea8a8201984886f30a from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:45658)
I20260808 01:49:34.564736  9648 tablet_copy_service.cc:161] P 259bcb3f0d254e8b9effac3277e654e0: Beginning new tablet copy session on tablet ea7d6b03a61944ea8a8201984886f30a from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:45658: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-ea7d6b03a61944ea8a8201984886f30a
I20260808 01:49:34.564330 10767 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:34.578712  9648 tablet_copy_source_session.cc:215] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:34.583755 10767 ts_tablet_manager.cc:1434] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.024s	user 0.011s	sys 0.012s
I20260808 01:49:34.587083  9571 tablet_copy_service.cc:342] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-e57f5706d264426eb123cb26c6f3b89c received from {username='slave'} at 127.0.0.1:60552
I20260808 01:49:34.587639  9571 tablet_copy_service.cc:434] P 52ccf4dcca1843f7bb29e9172e2d4ab4: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-e57f5706d264426eb123cb26c6f3b89c on tablet e57f5706d264426eb123cb26c6f3b89c with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:34.588510 10770 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea7d6b03a61944ea8a8201984886f30a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:34.619349 10770 tablet_copy_client.cc:806] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:34.620129 10770 tablet_copy_client.cc:670] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:34.626276 10767 ts_tablet_manager.cc:933] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:34.657426 10770 tablet_copy_client.cc:538] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:34.666893 10775 ts_tablet_manager.cc:933] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359)
I20260808 01:49:34.697010 10770 tablet_bootstrap.cc:492] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:34.694453 10775 tablet_copy_client.cc:323] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.193:39359
I20260808 01:49:34.705332 10767 tablet_copy_client.cc:323] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.195:34565
I20260808 01:49:34.714547  9491 tablet_copy_service.cc:140] P 6e7856bed1be43acaa821290d506ee97: Received BeginTabletCopySession request for tablet cab93716156d46d3b0bda8d98d1fc76c from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:37248)
I20260808 01:49:34.714967  9648 tablet_copy_service.cc:140] P 259bcb3f0d254e8b9effac3277e654e0: Received BeginTabletCopySession request for tablet f358d8403bf440059a6753dc4ed0be38 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:45658)
I20260808 01:49:34.715397  9491 tablet_copy_service.cc:161] P 6e7856bed1be43acaa821290d506ee97: Beginning new tablet copy session on tablet cab93716156d46d3b0bda8d98d1fc76c from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:37248: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-cab93716156d46d3b0bda8d98d1fc76c
I20260808 01:49:34.715793  9648 tablet_copy_service.cc:161] P 259bcb3f0d254e8b9effac3277e654e0: Beginning new tablet copy session on tablet f358d8403bf440059a6753dc4ed0be38 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:45658: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-f358d8403bf440059a6753dc4ed0be38
I20260808 01:49:34.727468  9648 tablet_copy_source_session.cc:215] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:34.732702 10767 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f358d8403bf440059a6753dc4ed0be38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:34.734709  9491 tablet_copy_source_session.cc:215] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:34.769246 10775 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cab93716156d46d3b0bda8d98d1fc76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:34.780114 10767 tablet_copy_client.cc:806] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:34.782943 10767 tablet_copy_client.cc:670] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:34.801955 10767 tablet_copy_client.cc:538] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:34.802009 10775 tablet_copy_client.cc:806] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:34.807031 10775 tablet_copy_client.cc:670] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:34.817315 10775 tablet_copy_client.cc:538] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:34.828593 10767 tablet_bootstrap.cc:492] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:34.841655 10770 tablet_bootstrap.cc:492] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:34.844643 10770 tablet_bootstrap.cc:492] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:34.847281 10770 ts_tablet_manager.cc:1403] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.151s	user 0.025s	sys 0.021s
I20260808 01:49:34.859977 10775 tablet_bootstrap.cc:492] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:34.864629 10770 raft_consensus.cc:359] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:34.867691 10770 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:34.870563 10770 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:34.921473 10770 ts_tablet_manager.cc:1434] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.073s	user 0.009s	sys 0.006s
I20260808 01:49:34.931917  9648 tablet_copy_service.cc:342] P 259bcb3f0d254e8b9effac3277e654e0: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-ea7d6b03a61944ea8a8201984886f30a received from {username='slave'} at 127.0.0.1:45658
I20260808 01:49:34.932942  9648 tablet_copy_service.cc:434] P 259bcb3f0d254e8b9effac3277e654e0: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-ea7d6b03a61944ea8a8201984886f30a on tablet ea7d6b03a61944ea8a8201984886f30a with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:34.953307 10767 tablet_bootstrap.cc:492] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:34.954623 10767 tablet_bootstrap.cc:492] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:34.955811 10767 ts_tablet_manager.cc:1403] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.128s	user 0.030s	sys 0.051s
I20260808 01:49:34.960862 10767 raft_consensus.cc:359] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:34.962173 10767 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:34.962983 10767 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:34.965975 10767 ts_tablet_manager.cc:1434] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260808 01:49:34.968384  9648 tablet_copy_service.cc:342] P 259bcb3f0d254e8b9effac3277e654e0: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-f358d8403bf440059a6753dc4ed0be38 received from {username='slave'} at 127.0.0.1:45658
I20260808 01:49:34.968873  9648 tablet_copy_service.cc:434] P 259bcb3f0d254e8b9effac3277e654e0: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-f358d8403bf440059a6753dc4ed0be38 on tablet f358d8403bf440059a6753dc4ed0be38 with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:34.976475 10775 tablet_bootstrap.cc:492] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:34.977856 10775 tablet_bootstrap.cc:492] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:34.978785 10775 ts_tablet_manager.cc:1403] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.119s	user 0.016s	sys 0.028s
I20260808 01:49:34.988504 10775 raft_consensus.cc:359] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:34.989437 10775 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:34.990211 10775 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:34.993570 10775 ts_tablet_manager.cc:1434] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.014s	user 0.003s	sys 0.005s
I20260808 01:49:34.997251  9491 tablet_copy_service.cc:342] P 6e7856bed1be43acaa821290d506ee97: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-cab93716156d46d3b0bda8d98d1fc76c received from {username='slave'} at 127.0.0.1:37248
I20260808 01:49:34.997749  9491 tablet_copy_service.cc:434] P 6e7856bed1be43acaa821290d506ee97: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-cab93716156d46d3b0bda8d98d1fc76c on tablet cab93716156d46d3b0bda8d98d1fc76c with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:35.033413 10368 raft_consensus.cc:1217] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260808 01:49:35.044612 10368 raft_consensus.cc:1217] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2]   Dedup: 2.2->[]
I20260808 01:49:35.058982 10368 raft_consensus.cc:1217] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260808 01:49:35.060258 10367 raft_consensus.cc:1217] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260808 01:49:35.497160 10600 raft_consensus.cc:1064] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:35.500554 10600 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:35.518263 10368 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:35.518723  9477 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:35.520160 10631 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:49:35.527875 10600 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:49:35.531765  9638 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:35.538641 10631 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:49:35.549750 10740 raft_consensus.cc:1064] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:35.552439 10755 raft_consensus.cc:1064] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:35.554003 10740 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:35.560999 10755 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:35.567157 10631 raft_consensus.cc:2955] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:35.595702 10369 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEARNER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:49:35.598336 10774 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:35.598393 10368 raft_consensus.cc:2955] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:35.614282  9372 catalog_manager.cc:5697] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:35.581199  9477 raft_consensus.cc:2955] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:35.714138  9635 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:49:35.704538  9638 raft_consensus.cc:2955] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:35.727109 10774 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:35.730116  9550 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:49:35.845110 10791 raft_consensus.cc:1064] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
W20260808 01:49:35.871361  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 145943us
I20260808 01:49:35.872644 10791 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:35.879292  9551 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:35.880221 10368 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:35.889815 10816 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:35.897804 10794 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:35.928089  9476 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:35.930780 10816 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:36.028968 10814 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:36.053268 10804 raft_consensus.cc:2955] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:36.059110  9550 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:36.087965  9635 raft_consensus.cc:2955] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:36.119168 10369 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:36.162756 10367 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260808 01:49:36.157244 10817 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260808 01:49:36.175014 10808 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260808 01:49:36.181015  9481 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:36.182245 10824 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260808 01:49:36.214208  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 241875us
I20260808 01:49:36.214938 10791 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:36.228175  9556 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260808 01:49:36.267930  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 152158us
W20260808 01:49:36.320350  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 190819us
W20260808 01:49:36.340780  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 93119.8us
W20260808 01:49:36.345844 10784 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.119s	user 0.000s	sys 0.012s
I20260808 01:49:36.351286  9371 catalog_manager.cc:5697] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:36.400861  9476 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:36.423514  9555 raft_consensus.cc:2955] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:36.420893  9481 raft_consensus.cc:2955] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:36.466550 10807 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260808 01:49:36.497278 10368 raft_consensus.cc:2955] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:36.501878 10600 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:36.508370  9551 raft_consensus.cc:2955] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:36.486903 10829 raft_consensus.cc:2955] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:36.383363 10816 raft_consensus.cc:2955] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:36.555440 10366 raft_consensus.cc:2955] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:36.660176 10369 raft_consensus.cc:2955] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:49:36.692759  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 75484.9us
W20260808 01:49:36.765105  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 157014us
W20260808 01:49:36.776258  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 110342us
I20260808 01:49:36.779274 10833 raft_consensus.cc:2955] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:36.778213  9477 raft_consensus.cc:2955] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:36.832948 10834 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:36.833750 10834 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:36.835191  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet e57f5706d264426eb123cb26c6f3b89c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:36.836261 10834 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:36.859691  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:36.864514  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:36.877736  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 274923us
I20260808 01:49:36.786718  9475 raft_consensus.cc:2955] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:36.903599  9524 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:36.912714  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:49:36.928212 10824 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:36.932446 10368 raft_consensus.cc:2955] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:36.954437  9373 catalog_manager.cc:5697] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:36.956116  9372 catalog_manager.cc:5697] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New cstate: current_term: 1 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:36.960287  9372 catalog_manager.cc:5697] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:37.045814  9624 tablet_service.cc:1558] Processing DeleteTablet for tablet e57f5706d264426eb123cb26c6f3b89c with delete_type TABLET_DATA_TOMBSTONED (TS 259bcb3f0d254e8b9effac3277e654e0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42810
W20260808 01:49:37.057937  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 73083.2us
I20260808 01:49:37.038291  9480 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:37.216147 10368 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:49:37.260002  9556 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:49:37.262051 10795 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:37.272974 10811 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:37.297747 10848 tablet_replica.cc:333] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:49:37.299031  9556 raft_consensus.cc:2955] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:37.299042 10848 raft_consensus.cc:2243] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260808 01:49:37.313390 10839 raft_consensus.cc:670] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopping
I20260808 01:49:37.298014 10827 raft_consensus.cc:2955] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:37.316800 10848 raft_consensus.cc:2272] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:37.317709  9555 raft_consensus.cc:2955] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:37.319072 10368 raft_consensus.cc:2955] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:37.342963 10848 ts_tablet_manager.cc:1916] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:37.351753  9636 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:37.400055 10369 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:37.402534  9555 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:40060
I20260808 01:49:37.403462  9555 raft_consensus.cc:606] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:37.406428  9477 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:37.418421 10848 ts_tablet_manager.cc:1929] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:49:37.419059 10848 log.cc:1199] T e57f5706d264426eb123cb26c6f3b89c P 259bcb3f0d254e8b9effac3277e654e0: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-2-root/wals/e57f5706d264426eb123cb26c6f3b89c
I20260808 01:49:37.429678 10839 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:37.431900 10839 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:37.443221  9360 catalog_manager.cc:5028] TS 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): tablet e57f5706d264426eb123cb26c6f3b89c (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:37.450965 10807 raft_consensus.cc:2955] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:37.453011  9480 raft_consensus.cc:2955] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:37.465690 10369 raft_consensus.cc:2955] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:37.479425  9635 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3759209338e14437aff5f92a2fff36da"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:49:37.480149  9635 raft_consensus.cc:606] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:37.480399  9373 catalog_manager.cc:5697] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 reported cstate change: config changed from index 3 to 4, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:37.497656  9372 catalog_manager.cc:5697] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: config changed from index 3 to 4, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New cstate: current_term: 2 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:37.501129  9636 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:37.510401 10834 raft_consensus.cc:993] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: : Instructing follower 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:49:37.514917 10834 raft_consensus.cc:1081] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Signalling peer 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:49:37.544327  9634 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:42824
I20260808 01:49:37.545228  9634 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:37.545766  9634 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:37.548548  9362 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet cab93716156d46d3b0bda8d98d1fc76c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:37.554193  9547 tablet_service.cc:1558] Processing DeleteTablet for tablet ea7d6b03a61944ea8a8201984886f30a with delete_type TABLET_DATA_TOMBSTONED (TS 52ccf4dcca1843f7bb29e9172e2d4ab4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53202
I20260808 01:49:37.555183  9480 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
 from {username='slave'} at 127.0.0.1:38182
I20260808 01:49:37.557843  9480 raft_consensus.cc:606] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Received request to transfer leadership to TS 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:37.561195  9624 tablet_service.cc:1558] Processing DeleteTablet for tablet cab93716156d46d3b0bda8d98d1fc76c with delete_type TABLET_DATA_TOMBSTONED (TS 259bcb3f0d254e8b9effac3277e654e0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42810
I20260808 01:49:37.561467  9631 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:49:37.562181  9631 raft_consensus.cc:606] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Received request to transfer leadership to TS 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:37.563766 10848 tablet_replica.cc:333] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:49:37.561504  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet ea7d6b03a61944ea8a8201984886f30a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:37.575899 10848 raft_consensus.cc:2243] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:37.576680  9631 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:49:37.577476  9631 raft_consensus.cc:606] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:37.581210 10848 raft_consensus.cc:2272] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:37.581830 10820 raft_consensus.cc:993] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:49:37.582391 10820 raft_consensus.cc:1081] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Signalling peer 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:49:37.584372  9555 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3759209338e14437aff5f92a2fff36da"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:53208
I20260808 01:49:37.584990  9555 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:37.585395  9555 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:37.586158 10854 tablet_replica.cc:333] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:49:37.586561 10848 ts_tablet_manager.cc:1916] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:37.587152  9631 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:49:37.587828  9631 raft_consensus.cc:606] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Received request to transfer leadership to TS 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:37.594626  9635 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fca874793205463dbb922993f57bf7f5"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:49:37.596163  9635 raft_consensus.cc:606] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:37.598207  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 7
I20260808 01:49:37.600176  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:49:37.602722  9555 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:37.605151 10367 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:37.606279  9555 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:37.607561  9554 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:37.610114  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:49:37.609988 10807 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:49:37.612677 10816 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:37.623059 10820 raft_consensus.cc:2955] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:49:37.637404  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 68769.4us
I20260808 01:49:37.646415 10367 raft_consensus.cc:2955] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:37.636154  9554 raft_consensus.cc:2955] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:37.653481  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:37.665674  9631 raft_consensus.cc:3055] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:37.666069  9631 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:37.666862  9631 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:37.667707  9634 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:37.668160  9631 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:37.735126  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:37.736049  9475 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:37.739789 10795 raft_consensus.cc:993] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97: : Instructing follower 7201605bb2f54c949a4ebe6e9b3b4b2a to start an election
I20260808 01:49:37.740433 10795 raft_consensus.cc:1081] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Signalling peer 7201605bb2f54c949a4ebe6e9b3b4b2a to start an election
I20260808 01:49:37.741044 10816 raft_consensus.cc:993] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:49:37.741621 10833 raft_consensus.cc:1081] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Signalling peer 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:49:37.744671  9558 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fca874793205463dbb922993f57bf7f5"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:53208
I20260808 01:49:37.745072  9475 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:37.745323  9558 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:37.745887  9558 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:37.749588  9634 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:37.751717 10367 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
 from {username='slave'} at 127.0.0.1:56258
I20260808 01:49:37.752316 10367 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:37.752641 10367 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:37.734447  9631 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:37.759971  9521 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:37.760838 10834 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:37.760732  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:37.761570 10834 raft_consensus.cc:697] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:37.762573 10834 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:37.761734  9475 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:37.763586  9558 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:37.765575  9558 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:37.769532  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:37.770321  9554 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:37.778558  9475 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:49:37.770641  9554 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:37.784387  9554 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:37.789760 10848 ts_tablet_manager.cc:1929] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260808 01:49:37.790628 10848 log.cc:1199] T cab93716156d46d3b0bda8d98d1fc76c P 259bcb3f0d254e8b9effac3277e654e0: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-2-root/wals/cab93716156d46d3b0bda8d98d1fc76c
I20260808 01:49:37.794430 10854 raft_consensus.cc:2243] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:37.795421  9554 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:37.795527 10854 raft_consensus.cc:2272] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:37.799360 10854 ts_tablet_manager.cc:1916] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:37.805337  9554 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
W20260808 01:49:37.807459  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47344.7us
I20260808 01:49:37.811225  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:37.811896  9636 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:37.812219  9636 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:37.812942  9636 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:37.816484  9360 catalog_manager.cc:5028] TS 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): tablet cab93716156d46d3b0bda8d98d1fc76c (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
W20260808 01:49:37.808632  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 49433.4us
I20260808 01:49:37.817440  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet f358d8403bf440059a6753dc4ed0be38 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:37.823844  9636 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:37.829880 10808 raft_consensus.cc:993] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 7201605bb2f54c949a4ebe6e9b3b4b2a to start an election
I20260808 01:49:37.810506  9601 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:37.830628 10808 raft_consensus.cc:1081] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Signalling peer 7201605bb2f54c949a4ebe6e9b3b4b2a to start an election
I20260808 01:49:37.831071 10803 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:37.831713 10803 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:37.832196 10820 raft_consensus.cc:993] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 7201605bb2f54c949a4ebe6e9b3b4b2a to start an election
I20260808 01:49:37.832763 10803 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:37.837672 10752 raft_consensus.cc:1081] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Signalling peer 7201605bb2f54c949a4ebe6e9b3b4b2a to start an election
I20260808 01:49:37.838354  9636 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:37.840317 10367 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:37.843315 10369 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
 from {username='slave'} at 127.0.0.1:56266
I20260808 01:49:37.844038 10369 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:37.843930  9521 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:37.844937 10368 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
 from {username='slave'} at 127.0.0.1:56266
I20260808 01:49:37.845333 10843 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:37.845551 10368 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:37.845918 10843 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:37.846009 10368 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:37.847062 10843 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:37.844473 10369 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:37.872398 10368 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:37.875222 10369 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:37.880960 10367 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:37.887253  9377 catalog_manager.cc:5697] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:37.899893 10369 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:37.893005 10368 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:38.341121 10854 ts_tablet_manager.cc:1929] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:49:38.736867 10854 log.cc:1199] T ea7d6b03a61944ea8a8201984886f30a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-1-root/wals/ea7d6b03a61944ea8a8201984886f30a
I20260808 01:49:38.738458 10854 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53202 (request call id 70) took 1184 ms. Trace:
I20260808 01:49:38.738936 10854 rpcz_store.cc:276] 0808 01:49:37.553797 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:37.553957 (+   160us) service_pool.cc:224] Handling call
0808 01:49:37.585778 (+ 31821us) ts_tablet_manager.cc:1151] Deleting tablet ea7d6b03a61944ea8a8201984886f30a
0808 01:49:37.585851 (+    73us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0808 01:49:37.853315 (+267464us) tablet_metadata.cc:665] Metadata flushed
0808 01:49:38.340891 (+487576us) tablet_metadata.cc:665] Metadata flushed
0808 01:49:38.738348 (+397457us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":206701,"tablet-delete.queue_time_us":28006,"thread_start_us":19889,"threads_started":1}
I20260808 01:49:38.758260  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:38.759039  9634 raft_consensus.cc:3055] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:38.759419  9634 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:38.760389  9634 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:38.761987  9634 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:38.763072  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:38.763772  9636 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:38.764155  9636 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:38.765053  9636 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:38.774590  9634 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:49:38.776611 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:49:38.814374 10861 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:38.821631  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:38.822328  9554 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:38.825413 10863 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:38.826009 10863 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:38.827129  9360 catalog_manager.cc:5028] TS 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): tablet ea7d6b03a61944ea8a8201984886f30a (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:38.827762  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:38.828395  9555 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:38.835201 10858 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:38.836386 10858 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:38.840430 10863 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:38.841048 10795 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:38.842128 10795 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:38.842525 10858 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:38.844147 10795 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:38.844883  9555 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:49:38.850396 10332 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:49:38.853911  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:38.858619 10866 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:38.859850 10866 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:38.863534 10866 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:38.864919 10861 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:49:38.864857  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:38.865365 10864 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:38.866274 10864 raft_consensus.cc:697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:49:38.866612  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:38.867439 10864 consensus_queue.cc:237] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:38.869267  9631 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:38.867440 10861 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:49:38.878232 10803 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.035s	user 0.023s	sys 0.027s
W20260808 01:49:38.878753 10803 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.035s	user 0.023s	sys 0.027s
I20260808 01:49:38.846495  9554 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:49:38.889626 10864 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:38.890516 10864 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:38.894503 10837 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:38.895205 10837 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:38.895197 10864 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:38.896934 10864 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:38.897094  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:38.899659 10864 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:38.903294 10860 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:38.904196 10864 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:38.904348 10860 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:38.905081 10864 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:38.907459 10864 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:38.897718 10837 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:38.913650 10863 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:38.914613 10860 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:38.914408 10863 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:38.913652 10864 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:38.916873  9636 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:38.918531  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:38.919114 10860 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:38.919819 10863 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:38.918569  9631 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:38.921952 10863 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:38.922621 10863 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:38.923741  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:38.926236  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:38.927232 10863 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:38.927788  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:38.928339  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:38.929879  9377 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50084 (request call id 209) took 1101 ms. Trace:
I20260808 01:49:38.933346  9371 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50068 (request call id 216) took 1158 ms. Trace:
I20260808 01:49:38.930289  9377 rpcz_store.cc:276] 0808 01:49:37.828938 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:37.833220 (+  4282us) service_pool.cc:224] Handling call
0808 01:49:38.929784 (+1096564us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:49:38.770848 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:49:38.771428 (+   580us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:49:38.771470 (+    42us) write_op.cc:435] Acquired schema lock
0808 01:49:38.771488 (+    18us) tablet.cc:662] Decoding operations
0808 01:49:38.771820 (+   332us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:49:38.771874 (+    54us) write_op.cc:641] Partition lock acquired for write op
0808 01:49:38.771908 (+    34us) tablet.cc:685] Acquiring locks for 1 operations
0808 01:49:38.772103 (+   195us) tablet.cc:701] Row locks acquired
0808 01:49:38.772121 (+    18us) write_op.cc:260] PREPARE: finished
0808 01:49:38.772270 (+   149us) write_op.cc:270] Start()
0808 01:49:38.772384 (+   114us) write_op.cc:276] Timestamp: P: 1786153778772223 usec, L: 0
0808 01:49:38.772406 (+    22us) op_driver.cc:348] REPLICATION: starting
0808 01:49:38.773056 (+   650us) log.cc:844] Serialized 535 byte log entry
0808 01:49:38.889229 (+116173us) op_driver.cc:464] REPLICATION: finished
0808 01:49:38.891759 (+  2530us) write_op.cc:301] APPLY: starting
0808 01:49:38.891846 (+    87us) tablet.cc:1367] starting BulkCheckPresence
0808 01:49:38.891951 (+   105us) tablet.cc:1370] finished BulkCheckPresence
0808 01:49:38.892110 (+   159us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:49:38.892340 (+   230us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:49:38.892565 (+   225us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 01:49:38.892617 (+    52us) write_op.cc:312] APPLY: finished
0808 01:49:38.892872 (+   255us) log.cc:844] Serialized 22 byte log entry
0808 01:49:38.911218 (+ 18346us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:38.911337 (+   119us) write_op.cc:454] Released schema lock
0808 01:49:38.911379 (+    42us) write_op.cc:341] FINISH: Updating metrics
0808 01:49:38.911823 (+   444us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:38.911859 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":871619,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":2261,"apply.run_cpu_time_us":2224,"apply.run_wall_time_us":20016,"num_ops":1,"prepare.queue_time_us":5226,"prepare.run_cpu_time_us":2468,"prepare.run_wall_time_us":2466,"replication_time_us":116653,"thread_start_us":4120,"threads_started":1}]]}
I20260808 01:49:38.933826  9371 rpcz_store.cc:276] 0808 01:49:37.775437 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:37.776082 (+   645us) service_pool.cc:224] Handling call
0808 01:49:37.840491 (+ 64409us) spinlock_profiling.cc:243] Waited 62.5 ms on lock 0x7b5800001720. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d41855a62 00007f5d4a821df9 00007f5d4a7bad56 00007f5d4a78c248 00007f5d4a888bfe 00007f5d48d424a4 00007f5d48d42432 00007f5d48d42361 00007f5d48d422dc
0808 01:49:38.933268 (+1092777us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45426,"reported_tablets":1,"spinlock_wait_cycles":145098496}
I20260808 01:49:38.940222  9636 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:49:38.942930 10794 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:38.943747 10794 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:38.944556  9372 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55058 (request call id 97) took 1214 ms. Trace:
I20260808 01:49:38.945003  9372 rpcz_store.cc:276] 0808 01:49:37.729843 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:37.730179 (+   336us) service_pool.cc:224] Handling call
0808 01:49:37.851020 (+120841us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b5800001720. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d41855a62 00007f5d4a821df9 00007f5d4a7bad56 00007f5d4a78c248 00007f5d4a888bfe 00007f5d48d424a4 00007f5d48d42432 00007f5d48d42361 00007f5d48d422dc
0808 01:49:38.944464 (+1093444us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36174,"reported_tablets":1,"spinlock_wait_cycles":233472896}
W20260808 01:49:38.946044  9409 debug-util.cc:398] Leaking SignalData structure 0x7b0800027960 after lost signal to thread 7834
W20260808 01:49:38.947387  9409 debug-util.cc:398] Leaking SignalData structure 0x7b08001e2240 after lost signal to thread 9412
I20260808 01:49:38.948195 10864 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:49:38.948803  9409 debug-util.cc:398] Leaking SignalData structure 0x7b080031a080 after lost signal to thread 9506
I20260808 01:49:38.948719 10774 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:38.949419 10774 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:49:38.950170  9409 debug-util.cc:398] Leaking SignalData structure 0x7b080033c360 after lost signal to thread 9585
I20260808 01:49:38.878060 10834 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:38.948797 10864 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:38.946373 10794 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:38.951519  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:49:38.970126 10858 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:38.972184 10858 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:38.973629 10866 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:38.973083 10858 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:38.960544 10865 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:38.974339 10866 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:38.976325 10865 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:38.978284 10866 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:38.980373  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:49:38.950650  9409 debug-util.cc:398] Leaking SignalData structure 0x7b080033c4e0 after lost signal to thread 9586
I20260808 01:49:38.998040  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:38.981360 10332 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:49:38.978349 10807 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:38.982524 10864 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:38.961447 10774 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:49:39.000396  9409 debug-util.cc:398] Leaking SignalData structure 0x7b0800324fa0 after lost signal to thread 9663
I20260808 01:49:39.000567 10864 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:39.000506 10807 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:39.001605 10864 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:49:39.000859  9409 debug-util.cc:398] Leaking SignalData structure 0x7b0800324dc0 after lost signal to thread 9664
I20260808 01:49:39.002588 10864 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:39.003403 10807 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:39.003569  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:39.004604  9474 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:38.964684 10863 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:39.005553 10863 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:39.002264 10833 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:38.994199  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:38.965512  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1126) took 1139 ms. Trace:
I20260808 01:49:38.967917  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:39.007251 10833 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:39.012030  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.013149 10833 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:39.013806  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:38.968508  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:38.982324 10865 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:39.017731 10863 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:39.019447  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:39.014241  9409 debug-util.cc:398] Leaking SignalData structure 0x7b08003259c0 after lost signal to thread 10393
W20260808 01:49:39.021317  9409 debug-util.cc:398] Leaking SignalData structure 0x7b080029a3c0 after lost signal to thread 10394
I20260808 01:49:39.021684  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.008931  9480 rpcz_store.cc:276] 0808 01:49:37.825838 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:38.964253 (+1138415us) service_pool.cc:224] Handling call
0808 01:49:38.964721 (+   468us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:38.965009 (+   288us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:38.965046 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:38.965118 (+    72us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:49:38.965154 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:38.965322 (+   168us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:38.965445 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:49:39.023619  9474 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:49:39.000425  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.029920  9481 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:38.969514  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:38.919773 10860 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:38.990164  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:38.992625  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:39.031972  9477 raft_consensus.cc:3055] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:39.032335  9477 raft_consensus.cc:740] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:39.025840  9480 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:39.033318 10860 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:39.033156  9477 consensus_queue.cc:260] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:38.993863 10858 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:39.035550 10334 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.036625 10866 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.037236 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:38.994189  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.038430 10866 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.034577  9477 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:39.039819 10858 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:39.040402 10858 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:38.996034  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.043433 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.044415 10866 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:39.045075 10858 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:39.051241 10877 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:39.051885 10877 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:39.058051  9477 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:49:39.062956  9480 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:39.074216  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:39.079777  9477 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:39.081405 10861 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:39.082178 10860 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:39.082086 10861 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:39.080632  9372 catalog_manager.cc:5697] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 1 to 2, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 2 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:39.087415 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:39.083088 10860 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:39.086714  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.099767  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:39.101320  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:39.103291  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.103199 10871 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.075470 10807 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:39.105722  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:39.105928  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 158711us
I20260808 01:49:39.106426  9631 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:39.106805  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:39.107329  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.108515  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.088121 10861 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:39.108726  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:39.109521  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:39.098052  9372 catalog_manager.cc:5697] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 1 to 2, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 2 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:39.085786 10334 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.114101 10807 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:39.123155 10801 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:49:39.123574 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 60295.5us
I20260808 01:49:39.124565 10369 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:39.115408  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:39.116245  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:39.117712 10860 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:39.096236 10833 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:39.121254  9372 catalog_manager.cc:5697] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 1 to 2, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 2 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:49:39.125385  9635 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:39.128907  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.136814  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.138032 10807 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:39.144428  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:39.128934 10833 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:39.123795 10801 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:39.128911  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:39.146170  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:39.149299  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:39.149441  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.150012  9559 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:39.147545  9521 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:39.126622 10861 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.151652  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.153704  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.155973  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:39.156294  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.156636  9637 raft_consensus.cc:1813] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Handling vote request from an unknown peer 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.157183  9637 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 2: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:49:39.154289 10858 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:39.160210 10858 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:39.163270  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:39.166586 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.150943 10833 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:39.170766 10860 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:39.154735  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:39.171283 10860 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:39.171695 10860 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:39.172183 10801 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:39.173089  9521 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:39.173748  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:39.174948  9524 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.134449 10334 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:49:39.176638 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50210.3us
I20260808 01:49:39.178462 10865 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:39.178949  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.183188 10864 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.185992 10826 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.175235 10861 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.186614 10824 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.188182 10334 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.189523 10803 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:39.184465 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:39.173846 10858 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:39.195372  9444 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:39.196769  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.199666  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.178962 10865 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:39.197144  9599 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.202145  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:39.202183 10827 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:39.191761 10861 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.191866 10774 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.209173 10876 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:39.209182 10862 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:39.207504  9599 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.204216  9372 catalog_manager.cc:5697] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 1 to 2, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 2 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:39.211823  9372 catalog_manager.cc:5697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 1 to 2, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 2 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:39.202662 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:39.214109  9372 catalog_manager.cc:5697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 1 to 2, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 2 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:39.205402 10865 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:39.228667  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:39.228823  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:39.210940  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:39.237087  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 51557.9us
I20260808 01:49:39.238940  9524 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.240124 10834 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.241711  9524 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.242620 10824 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:39.243670  9524 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.244645 10824 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.244657  9524 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.245591 10824 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:39.246615  9465 tablet_service.cc:1558] Processing DeleteTablet for tablet f358d8403bf440059a6753dc4ed0be38 with delete_type TABLET_DATA_TOMBSTONED (TS 6e7856bed1be43acaa821290d506ee97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50120
I20260808 01:49:39.247110  9524 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.247310 10834 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.248142 10834 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.249001 10834 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.223877 10877 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:39.225641 10752 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.251293  9524 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.214637 10876 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:39.255873  9524 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.235543 10833 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:39.252869  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:39.223347 10862 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:39.264724 10824 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.276796 10859 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:39.277495 10859 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:39.278002 10859 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:39.263581 10834 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:49:39.290228  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 100127us
W20260808 01:49:39.291375  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 93586.3us
W20260808 01:49:39.300050  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 51621.9us
I20260808 01:49:39.272270 10833 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:39.322180  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:39.311522 10816 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:39.341516  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:39.355786  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 55638.6us
I20260808 01:49:39.357229 10816 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:39.395882 10863 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:39.398910 10863 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:39.399238  9523 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:39.400112 10858 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.401528 10863 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:39.405448  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 170781us
I20260808 01:49:39.406247  9599 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.403610 10859 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:39.431926 10894 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:49:39.433633  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 115990us
W20260808 01:49:39.435039  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 198575us
I20260808 01:49:39.435479  9599 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.436698  9472 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260808 01:49:39.436892  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 51876.8us
I20260808 01:49:39.437429  9599 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.441568  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.443575  9524 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.426116 10862 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:39.424038 10861 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:39.448282 10892 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.450177  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.441350 10839 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.459154 10862 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.457352 10864 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260808 01:49:39.432641 10894 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:39.475239 10876 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:39.453886 10803 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.458137 10833 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:39.451719 10861 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:39.557229 10880 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:39.557723 10861 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:39.564932  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.566042 10897 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:49:39.616719  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 178254us
I20260808 01:49:39.625303  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.628041  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.629917  9629 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:49:39.631458  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:39.632151  9635 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:39.634939 10827 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:49:39.638216 10901 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260808 01:49:39.617687 10816 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:39.647778  9480 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:49:39.566655 10897 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:39.661933 10865 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:39.672748  9635 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260808 01:49:39.704088 10899 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260808 01:49:39.711835 10861 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:49:39.664080 10871 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:49:39.636905 10860 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:39.731938 10866 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:49:39.739817 10860 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:39.764580 10897 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:39.765065 10894 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:39.766448  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.771287  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:39.743865 10865 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:39.820629  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:39.834779  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 250531us
W20260808 01:49:39.838563  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 388454us
I20260808 01:49:39.864298  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:39.905268  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:39.909873 10890 tablet_replica.cc:333] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:49:39.914006  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:39.923677  9634 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:39.913295 10890 raft_consensus.cc:2243] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:39.932251  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:49:39.956141  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:39.919759  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:49:39.965368  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 47871.5us
I20260808 01:49:39.972350  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:49:39.977896  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.010s	sys 0.026s
W20260808 01:49:39.978338  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.010s	sys 0.026s
W20260808 01:49:39.976944  9475 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.023s	sys 0.043s
I20260808 01:49:39.994328  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:39.976908 10890 raft_consensus.cc:2272] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:39.996510  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:40.000361  9599 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:39.997524 10776 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260808 01:49:40.014889  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:40.030400 10839 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:40.030964 10827 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:40.031625 10827 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:40.041169 10859 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:40.045840  9599 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:40.048465 10890 ts_tablet_manager.cc:1916] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:40.051199  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:40.033149 10827 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:40.068977  9521 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:49:40.090845  9523 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.007s	sys 0.026s
W20260808 01:49:40.091274  9523 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.007s	sys 0.026s
I20260808 01:49:40.068449 10801 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:49:40.100256  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 49149.7us
I20260808 01:49:40.101179  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:49:40.103987  9524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.006s	sys 0.015s
W20260808 01:49:40.104447  9524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.006s	sys 0.015s
W20260808 01:49:40.107108  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 543714us
I20260808 01:49:40.115458  9599 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:40.099583 10827 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:40.100780 10808 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:40.129513  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:40.127362  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:40.117087 10803 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:40.130429 10801 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:40.173993  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:40.187597  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:40.173600 10884 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:40.288962  9550 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:40.205229 10884 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:40.290395  9557 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 2: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:49:40.300125 10914 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
W20260808 01:49:40.302779 10899 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
W20260808 01:49:40.303514 10893 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260808 01:49:40.318014  9599 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:49:40.298411  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 92641.4us
I20260808 01:49:40.325018  9524 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:49:40.326145  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 175534us
W20260808 01:49:40.310587 10912 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.136s	user 0.000s	sys 0.002s
I20260808 01:49:40.318415 10884 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:49:40.345031  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:49:40.341465 10910 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:49:40.335632 10894 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:49:40.347891  9552 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:40.384403  9551 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260808 01:49:40.404246 10917 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260808 01:49:40.474666  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:40.477761  9556 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:49:40.393397  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1153) took 1084 ms. Trace:
I20260808 01:49:40.483811 10903 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:49:40.484679 10912 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
W20260808 01:49:40.464891 10901 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260808 01:49:40.506563  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 129268us
I20260808 01:49:40.500620 10903 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:40.568670 10877 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:40.572009 10877 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:40.569720  9638 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:49:40.577232  9638 raft_consensus.cc:606] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:40.583712 10903 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:40.482188  9475 rpcz_store.cc:276] 0808 01:49:39.302228 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:39.303648 (+  1420us) service_pool.cc:224] Handling call
0808 01:49:39.329510 (+ 25862us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:49:39.419223 (+ 89713us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:39.421093 (+  1870us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:49:39.450608 (+ 29515us) log.cc:844] Serialized 26 byte log entry
0808 01:49:40.007365 (+556757us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:49:40.201843 (+194478us) log.cc:844] Serialized 14 byte log entry
0808 01:49:40.209831 (+  7988us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:40.309607 (+ 99776us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:49:40.366780 (+ 57173us) raft_consensus.cc:1685] finished
0808 01:49:40.370205 (+  3425us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:40.382250 (+ 12045us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17669,"prepare.queue_time_us":280676,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":45540,"raft.run_cpu_time_us":398,"raft.run_wall_time_us":81817,"thread_start_us":620733,"threads_started":2,"wal-append.queue_time_us":662891}
I20260808 01:49:40.523542 10928 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
I20260808 01:49:40.606113 10884 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:40.608325  9558 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:40.610584 10884 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:40.612208 10877 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:40.640491  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 28) took 1079 ms. Trace:
W20260808 01:49:40.642920  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 250638us
I20260808 01:49:40.648746  9638 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:49:40.656522  9475 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:40.649513  9638 raft_consensus.cc:606] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:40.672785 10910 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:40.685998  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:49:40.677420  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:49:40.641321  9478 rpcz_store.cc:276] 0808 01:49:39.560576 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:39.562469 (+  1893us) service_pool.cc:224] Handling call
0808 01:49:39.562864 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:49:39.743873 (+181009us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:39.743905 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:49:39.760505 (+ 16600us) log.cc:844] Serialized 26 byte log entry
0808 01:49:40.105022 (+344517us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:49:40.105085 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:40.105188 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:49:40.633649 (+528461us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b2400013a78. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41713594 00007f5d4171169b 00007f5d418cd6d9 00007f5d418cd365 00007f5d418cc8c0 00007f5d4455a638 00007f5d4455ad98 00007f5d44532588 00007f5d4452d95a 00007f5d4452cda6
0808 01:49:40.637648 (+  3999us) raft_consensus.cc:1685] finished
0808 01:49:40.637686 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:40.640374 (+  2688us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":247025,"raft.queue_time_us":191576,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":157,"spinlock_wait_cycles":28296448,"thread_start_us":338161,"threads_started":1,"wal-append.queue_time_us":412237}
I20260808 01:49:40.682340  9481 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:40.684827  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:40.695475  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:40.697800  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 47140.5us
I20260808 01:49:40.686506 10827 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:40.718843  9629 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:49:40.719556  9629 raft_consensus.cc:606] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Received request to transfer leadership to TS 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:40.697125 10776 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:40.705165 10880 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:40.697227 10910 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:40.730244 10827 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:40.736418 10776 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:40.749086 10827 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:40.744400 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 233446us
I20260808 01:49:40.750064 10910 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:40.731575 10880 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:40.759820 10939 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:49:40.749159 10776 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:40.763615  9630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:49:40.771001  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:40.778780  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:49:40.779909 10880 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:40.765204  9630 raft_consensus.cc:606] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Received request to transfer leadership to TS 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:40.781282 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:40.796284  9480 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
 from {username='slave'} at 127.0.0.1:38182
I20260808 01:49:40.797065  9480 raft_consensus.cc:606] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Received request to transfer leadership to TS 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:40.806756  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:40.821679  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:40.818897 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 67241us
I20260808 01:49:40.828038 10332 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:40.829602  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:40.851024  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:40.856659 10890 ts_tablet_manager.cc:1929] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:49:40.857609 10890 log.cc:1199] T f358d8403bf440059a6753dc4ed0be38 P 6e7856bed1be43acaa821290d506ee97: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-0-root/wals/f358d8403bf440059a6753dc4ed0be38
I20260808 01:49:40.860141 10807 raft_consensus.cc:993] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:49:40.869009 10925 raft_consensus.cc:1081] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Signalling peer 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:49:40.918891 10890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50120 (request call id 71) took 1672 ms. Trace:
I20260808 01:49:40.930843  9521 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:40.931336 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:40.936684  9551 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d2b2a88358244f93970835ea3120c380"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
 from {username='slave'} at 127.0.0.1:40060
I20260808 01:49:40.937979  9551 raft_consensus.cc:606] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Received request to transfer leadership to TS 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:40.954759 10946 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:40.968394  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260808 01:49:40.938901 10903 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:40.972260  9523 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:40.978065 10858 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:49:40.982966  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 283959us
I20260808 01:49:40.995502  9599 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:40.994222  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:40.994222 10839 raft_consensus.cc:993] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 7201605bb2f54c949a4ebe6e9b3b4b2a to start an election
I20260808 01:49:41.016405  9524 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:40.919660 10890 rpcz_store.cc:276] 0808 01:49:39.246226 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:39.246418 (+   192us) service_pool.cc:224] Handling call
0808 01:49:39.298049 (+ 51631us) ts_tablet_manager.cc:1151] Deleting tablet f358d8403bf440059a6753dc4ed0be38
0808 01:49:39.298719 (+   670us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0808 01:49:39.827477 (+528758us) spinlock_profiling.cc:243] Waited 334 ms on lock 0x7b540038c7d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4499c3d9 00007f5d499db292 00007f5d49a0c0e0 00007f5d499f264d 00007f5d499f25f9 00007f5d499f2589 00007f5d499f2551 00007f5d499f177d 00000000003aa554 00000000003aa399
0808 01:49:40.598047 (+770570us) tablet_metadata.cc:665] Metadata flushed
0808 01:49:40.856423 (+258376us) tablet_metadata.cc:665] Metadata flushed
0808 01:49:40.918722 (+ 62299us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":400,"spinlock_wait_cycles":767720704,"tablet-delete.queue_time_us":44880,"thread_start_us":17865,"threads_started":1}
I20260808 01:49:41.020105 10794 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:41.021062 10794 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:41.025081  9521 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:41.032055 10794 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:41.033221 10794 raft_consensus.cc:993] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: : Instructing follower 7201605bb2f54c949a4ebe6e9b3b4b2a to start an election
I20260808 01:49:41.034140 10794 raft_consensus.cc:1081] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Signalling peer 7201605bb2f54c949a4ebe6e9b3b4b2a to start an election
I20260808 01:49:41.006317 10925 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:40.996690 10897 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:41.037426 10897 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:41.058246  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 44323us
I20260808 01:49:41.066252 10365 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
 from {username='slave'} at 127.0.0.1:56258
I20260808 01:49:41.067209 10365 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:41.067728 10365 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:41.070959 10891 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:41.072800  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:49:41.073764  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
W20260808 01:49:41.095415 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 267547us
I20260808 01:49:41.037026  9362 catalog_manager.cc:5028] TS 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): tablet f358d8403bf440059a6753dc4ed0be38 (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:41.092851 10873 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election won for term 2
W20260808 01:49:41.099661  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 88664.8us
W20260808 01:49:41.110669  9362 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 73705us
I20260808 01:49:41.111881  9599 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:41.115909 10803 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:41.118893 10859 raft_consensus.cc:993] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:49:41.119807 10859 raft_consensus.cc:1081] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Signalling peer 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
W20260808 01:49:41.102028  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 61703.2us
I20260808 01:49:41.064110 10839 raft_consensus.cc:1081] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Signalling peer 7201605bb2f54c949a4ebe6e9b3b4b2a to start an election
I20260808 01:49:41.073649 10940 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
I20260808 01:49:41.136130 10369 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
 from {username='slave'} at 127.0.0.1:56266
I20260808 01:49:41.136816 10369 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:41.137547 10369 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:49:41.187747  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 85944.6us
I20260808 01:49:41.188145  9551 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:53208
I20260808 01:49:41.189110  9551 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:41.189683  9551 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:41.146796  9557 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:53208
I20260808 01:49:41.144845 10839 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:41.143824 10873 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:49:41.204615  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 61635us
I20260808 01:49:41.212755 10927 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:49:41.214870 10839 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:41.217689 10897 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:41.219223 10808 raft_consensus.cc:993] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 7201605bb2f54c949a4ebe6e9b3b4b2a to start an election
I20260808 01:49:41.195511  9557 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:41.221272 10808 raft_consensus.cc:1081] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Signalling peer 7201605bb2f54c949a4ebe6e9b3b4b2a to start an election
I20260808 01:49:41.221607  9557 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:41.225482  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:41.221990 10873 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:41.244531  9551 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:41.245620 10367 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
 from {username='slave'} at 127.0.0.1:56266
I20260808 01:49:41.243366 10839 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:41.246441 10367 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:41.246872  9551 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:41.246869 10367 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:41.231662 10905 consensus_queue.cc:1059] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.152s
I20260808 01:49:41.254367 10369 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:41.256915 10369 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:41.280397 10891 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:41.280524 10366 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:41.281108 10891 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:41.285542 10888 raft_consensus.cc:3055] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:41.286298 10888 raft_consensus.cc:740] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:41.291324  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:41.291541 10859 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:49:41.292958  9632 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:41.293588  9632 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:41.301319 10367 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:41.294937  9632 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:41.282294 10891 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:41.305209  9632 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:41.306398 10891 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:41.307397 10367 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:41.295004 10365 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:41.312983 10365 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:41.312983 10888 consensus_queue.cc:260] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:41.315773 10888 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:49:41.388717  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 199840us
I20260808 01:49:41.293813  9557 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:41.409384  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:41.439489 10833 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:41.440960 10833 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:41.442993  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:41.444208  9629 raft_consensus.cc:3055] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:41.444644  9629 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:49:41.439834  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 292307us
W20260808 01:49:41.477962  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:49:41.500783  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:41.506258  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:41.498521  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:41.530714  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:41.532197  9631 raft_consensus.cc:3055] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:41.532753  9631 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:41.534106 10833 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:49:41.540071  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 44519.4us
I20260808 01:49:41.546602  9629 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:41.566977  9557 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:41.582698 10367 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3. Ops: []
W20260808 01:49:41.593039  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:49:41.593713 10365 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:41.599362  9479 raft_consensus.cc:3055] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:49:41.600010  9479 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:41.601706  9479 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:41.599673 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:41.607113 10795 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:41.608099 10795 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:41.610129  9629 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:41.619064  9631 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:41.615172  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:41.629878  9479 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:41.631992  9551 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:41.632164  9631 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:49:41.634775  9445 consensus_peers.cc:597] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c094b8e92d5e449aa2641c359d46d622. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:49:41.635504  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 53011.5us
W20260808 01:49:41.636570  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 51316.7us
I20260808 01:49:41.608260  9632 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:49:41.653182  9631 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
W20260808 01:49:41.654281 10859 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260808 01:49:41.677350  9550 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:41.678304  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:41.682950 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:49:41.682871 10795 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:41.684924  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:41.685761  9481 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:41.685983 10947 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:41.687014 10947 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:49:41.693462  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:41.694350  9476 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:41.699172 10898 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:41.711663  9629 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:49:41.712010  9479 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:49:41.688189 10947 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:41.733489  9476 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:41.733613 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:49:41.700208 10898 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:41.734929 10951 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:49:41.747828  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:41.749037  9550 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:41.751356 10951 raft_consensus.cc:697] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:49:41.753217 10951 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:41.731142 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:49:41.772516 10939 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:41.777545  9481 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:41.778188  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:41.791205  9481 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:41.794487 10873 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:41.794255  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:49:41.798823  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59615.2us
I20260808 01:49:41.805339  9551 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:41.807374 10833 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260808 01:49:41.796634 10873 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:41.820946  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:41.802429 10898 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:41.824420  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:49:41.833245  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:41.837604  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:41.841068 10953 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:41.829842 10858 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:41.844122  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:41.863765  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:41.866178  9556 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:41.870332 10858 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:41.848484 10953 raft_consensus.cc:697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:41.872714  9633 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:41.876375 10911 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:41.876564  9632 raft_consensus.cc:3055] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:41.877044  9632 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:41.878930 10873 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:41.881383  9556 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:49:41.883487  9550 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:49:41.884255 10858 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:41.887542  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:41.878093  9632 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:41.906141  9600 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:49:41.908257  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 73368.4us
I20260808 01:49:41.907050  9551 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:49:41.904183 10939 raft_consensus.cc:697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:49:41.928205 10955 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:41.877122 10911 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:41.916961 10873 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:41.964267 10803 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:41.965083 10803 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:41.964931 10939 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:41.967947 10803 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:41.969388 10911 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:41.969959 10898 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:41.970315 10963 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:41.970752 10898 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:41.971125 10963 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:41.973554 10898 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:41.973807 10963 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:41.976419 10966 raft_consensus.cc:993] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: : Instructing follower 7201605bb2f54c949a4ebe6e9b3b4b2a to start an election
I20260808 01:49:41.986418 10966 raft_consensus.cc:1081] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Signalling peer 7201605bb2f54c949a4ebe6e9b3b4b2a to start an election
I20260808 01:49:41.977337 10929 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:41.983021 10953 consensus_queue.cc:237] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:41.973559 10776 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260808 01:49:41.990861 10873 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:41.955862  9632 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:49:42.012352  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 74615.3us
I20260808 01:49:41.987315  9371 catalog_manager.cc:5697] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 1 to 2, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 2 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:42.021699  9371 catalog_manager.cc:5697] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 2 to 3, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 3 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:42.040311 10873 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:41.958956 10955 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:42.058859  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:49:42.062191  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 102010us
W20260808 01:49:42.071861  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 127264us
W20260808 01:49:42.087224  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 96910us
I20260808 01:49:42.078009 10947 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:42.088851 10947 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:42.091445 10898 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:42.092232 10898 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:42.098497  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:42.075079 10859 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:42.098830  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:42.100621  9558 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:42.101540  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:42.103112 10955 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:42.104918  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 173354us
I20260808 01:49:42.105901  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:42.113642 10972 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:42.078590  9524 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:42.122895 10911 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:42.123514 10911 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:42.125597 10911 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:42.128268  9551 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3. Ops: []
I20260808 01:49:42.137917 10947 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:42.139022  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:42.143149  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:42.148676 10955 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:42.149309 10955 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:42.151152 10898 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:42.155936  9481 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260808 01:49:42.169253  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 51948us
I20260808 01:49:42.171887  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:42.177273  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:42.179409  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 57104us
I20260808 01:49:42.185217  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:42.185866  9556 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:42.189064  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:42.191023  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:42.188406  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:42.194145  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:42.194763 10929 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:42.195537 10906 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:42.195477 10929 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:42.197808 10929 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:42.201611  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:42.202502  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:42.202824  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:42.203801 10795 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:42.188550 10366 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d2b2a88358244f93970835ea3120c380"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
 from {username='slave'} at 127.0.0.1:56276
I20260808 01:49:42.204380 10906 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:42.204946 10366 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:42.205185  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:42.185087  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:42.206329 10795 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:42.205406 10366 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:49:42.215988  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:49:42.220660  9632 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
W20260808 01:49:42.221436 10808 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260808 01:49:42.228889 10955 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:42.228677  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:42.233707 10332 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:42.237425 10947 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:42.237645  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:42.247747  9444 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:42.252175  9599 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:42.257791 10803 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:42.260908 10803 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:42.260946  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:42.262494  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:42.263228  9476 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:49:42.261377 10803 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:42.270460 10971 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:42.271674 10971 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:42.256533 10906 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:42.280691 10971 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:42.283752  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:42.286268  9476 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:42.288805 10366 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:42.291615  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:42.293344  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:42.294431 10367 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:42.294881  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:42.294273 10776 raft_consensus.cc:2955] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:42.298506  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:42.300567  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:42.301311  9476 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:42.298553 10986 ts_tablet_manager.cc:933] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359)
I20260808 01:49:42.308495  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:42.307554  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:42.309762  9475 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:49:42.313349  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:42.313957 10939 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:49:42.317106 10982 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:49:42.318768 10963 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:42.319257 10963 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:42.320272 10963 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:42.323029 10983 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:42.329931  9556 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:49:42.330318  9476 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:49:42.335330 10366 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:42.339885  9632 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:42.337854  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:42.340891 10334 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:49:42.341243  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:42.342803 10939 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:42.343453 10939 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:49:42.347434  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:42.357400  9524 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:42.358879 10955 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:42.330399 10963 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:42.360881  9558 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:49:42.361786 10951 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:49:42.365952  9524 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:42.367172 10983 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:42.367709 10983 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:42.358155 10939 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:42.368223 10983 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:42.381682  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:42.403388 10947 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260808 01:49:42.416765  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:42.417755  9630 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:42.427758 10803 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:42.429189  9631 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:42.432210 10803 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:42.432327 10794 consensus_queue.cc:1059] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:42.439669  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:42.448371 10986 tablet_copy_client.cc:323] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.193:39359
I20260808 01:49:42.491036  9521 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:42.494655  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:42.499765  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:42.500918  9473 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:49:42.506829  9491 tablet_copy_service.cc:140] P 6e7856bed1be43acaa821290d506ee97: Received BeginTabletCopySession request for tablet c094b8e92d5e449aa2641c359d46d622 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:37248)
I20260808 01:49:42.507721  9491 tablet_copy_service.cc:161] P 6e7856bed1be43acaa821290d506ee97: Beginning new tablet copy session on tablet c094b8e92d5e449aa2641c359d46d622 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:37248: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-c094b8e92d5e449aa2641c359d46d622
I20260808 01:49:42.510058  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:42.511698  9521 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:42.525017  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:42.525550  9521 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:42.529795  9521 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:42.531894  9521 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:42.519196 10972 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:42.517683 10963 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:42.545619  9559 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:49:42.532757 10833 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:42.532204 10858 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:42.531716 10953 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:42.538450 10972 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:42.518115 10983 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:42.585602 10975 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:42.585346  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:42.549702 10833 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:42.618367 10776 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:37234 (request call id 6) took 1968 ms. Trace:
I20260808 01:49:42.619771  9558 raft_consensus.cc:2368] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3.
I20260808 01:49:42.562448 10972 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:42.616567  9633 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:49:42.619388 10833 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:42.664456 10965 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:49:42.619089 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 115935us
W20260808 01:49:42.674062 10991 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260808 01:49:42.627221 10973 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:42.720607  9558 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260808 01:49:42.723333  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 191521us
W20260808 01:49:42.701301 10967 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
W20260808 01:49:42.723693  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 106155us
W20260808 01:49:42.739719 10997 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260808 01:49:42.741645 10999 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:49:42.742484 10947 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260808 01:49:42.744987 10995 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260808 01:49:42.779413  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 53471us
I20260808 01:49:42.780612  9521 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:42.781281  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:42.791149 11015 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:42.792480  9521 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:42.793246 10911 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:42.779739 10858 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260808 01:49:42.803622  9599 leader_election.cc:400] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3.
I20260808 01:49:42.807991 10833 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:42.871657 10833 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:42.620368 10776 rpcz_store.cc:276] 0808 01:49:40.650320 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:40.653213 (+  2893us) service_pool.cc:224] Handling call
0808 01:49:40.746899 (+ 93686us) log.cc:844] Serialized 506 byte log entry
0808 01:49:40.978403 (+231504us) spinlock_profiling.cc:243] Waited 2.98 ms on lock 0x7b1800359140. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:49:41.072641 (+ 94238us) spinlock_profiling.cc:243] Waited 7.78 ms on lock 0x7b1800359140. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4185ac56 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44913f06 00007f5d4490d87a 00007f5d4490caf4
0808 01:49:41.238104 (+165463us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b1800359140. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4185ac56 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490c522 00007f5d448ed28a 00007f5d448f4133
0808 01:49:41.239260 (+  1156us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b5400128ca0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490c522 00007f5d448ed28a 00007f5d448f4133 00007f5d448f40b9 00007f5d448f4049 00007f5d448f4011 00007f5d448f31dd 00000000003aa554 00000000003aa399 00007f5d4188cdb5
0808 01:49:41.246002 (+  6742us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x7b5400128ca0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44913f06 00007f5d4490d87a 00007f5d4490caf4 00007f5d448ed28a 00007f5d448f4133 00007f5d448f40b9 00007f5d448f4049 00007f5d448f4011 00007f5d448f31dd 00000000003aa554
0808 01:49:42.618103 (+1372101us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1551,"raft.queue_time_us":33479,"raft.run_cpu_time_us":12042,"raft.run_wall_time_us":686029,"spinlock_wait_cycles":85679872,"thread_start_us":171869,"threads_started":1,"wal-append.queue_time_us":189591,"wal-append.run_cpu_time_us":3805,"wal-append.run_wall_time_us":1097007}
I20260808 01:49:42.813417 10808 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:49:42.881527 10972 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:42.881804  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:42.883288  9475 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:42.868827 10983 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:42.915436 10827 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:42.916374 10827 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:42.903224 10990 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260808 01:49:42.932533  9377 catalog_manager.cc:5697] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 1 to 2, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 2 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:42.942929  9474 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:49:42.947481 10827 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:42.937620  9377 catalog_manager.cc:5697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 1 to 2, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 2 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:42.966300 10776 raft_consensus.cc:3055] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:42.967438 10776 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:42.968751 10776 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:42.949831 10972 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:42.970525 10776 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:42.956004  9491 tablet_copy_source_session.cc:215] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:42.958211 10999 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:49:42.987330 10986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c094b8e92d5e449aa2641c359d46d622. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:42.987528  9475 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
W20260808 01:49:42.988572  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 178910us
W20260808 01:49:43.009871  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 185275us
W20260808 01:49:43.023402  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 227732us
I20260808 01:49:43.019383  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:43.026602  9551 raft_consensus.cc:2368] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3.
I20260808 01:49:43.044914 11010 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:43.045672 11010 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:43.048146 11010 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:43.054553  9558 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:49:43.061085  9631 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:43.064561  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:43.065438  9637 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:43.066527  9631 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260808 01:49:43.071182  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:43.076340  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 290621us
I20260808 01:49:43.093394  9481 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:43.103325  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:49:43.104815  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:49:43.106462  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:43.107486  9475 raft_consensus.cc:2368] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3.
W20260808 01:49:43.120885  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:49:43.132570  9630 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:43.135377 10999 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:49:43.141114  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:43.141885  9479 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:43.154287  9631 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:43.155972  9637 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:49:43.157712  9551 raft_consensus.cc:2955] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:43.168120  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:43.169243  9550 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:49:43.169510  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:43.170564  9629 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:43.181783  9476 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:43.187374  9601 leader_election.cc:400] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3.
I20260808 01:49:43.187945  9601 leader_election.cc:403] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260808 01:49:43.194609  9524 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:43.159838  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:43.160288 10973 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20260808 01:49:43.242843 11022 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.083s
I20260808 01:49:43.256392 11010 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260808 01:49:43.271569  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:43.273281 10794 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:43.273825 10794 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:43.251916 10963 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:49:43.274437 10794 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:43.273576 10925 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3.
W20260808 01:49:43.289033  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52847.3us
W20260808 01:49:43.291860  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 86839.8us
I20260808 01:49:43.309715  9633 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:49:43.363315 10858 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260808 01:49:43.302832 10963 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:43.389670 10794 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:43.403187  9524 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:49:43.410841 11032 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260808 01:49:43.441475 10986 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:56258 (request call id 189) took 1197 ms. Trace:
I20260808 01:49:43.452728  9635 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:43.456122 10963 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:43.417325 10965 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:43.462067 10965 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:43.462530 10965 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:49:43.472487  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 55180us
I20260808 01:49:43.477363 10911 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:49:43.477936 10911 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:43.486414  9599 leader_election.cc:400] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3.
I20260808 01:49:43.494529  9599 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:43.442003 10986 rpcz_store.cc:276] 0808 01:49:42.244412 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:42.244593 (+   181us) service_pool.cc:224] Handling call
0808 01:49:42.302976 (+ 58383us) ts_tablet_manager.cc:934] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359)
0808 01:49:42.985710 (+682734us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:49:43.271740 (+286030us) tablet_metadata.cc:665] Metadata flushed
0808 01:49:43.271875 (+   135us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:49:43.440465 (+168590us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:49:43.440894 (+   429us) tablet_replica.cc:206] Creating consensus instance
0808 01:49:43.441304 (+   410us) ts_tablet_manager.cc:989] Replica registered.
0808 01:49:43.441382 (+    78us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":62095,"tablet-copy.queue_time_us":53107,"thread_start_us":118463,"threads_started":2}
I20260808 01:49:43.537669 10986 tablet_copy_client.cc:806] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:43.538493 10986 tablet_copy_client.cc:670] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:43.492467 10982 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260808 01:49:43.561103 11034 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
W20260808 01:49:43.562348 11041 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
I20260808 01:49:43.561360 10937 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:49:43.571372 10794 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:43.572113  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:43.575692  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:43.576577  9630 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 3
I20260808 01:49:43.538905 10911 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:43.579988 10911 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:43.589406 10965 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:43.591130  9557 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:49:43.616204 10986 tablet_copy_client.cc:538] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:43.638509 10965 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:43.639885  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:43.640764  9476 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:43.667428  9630 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:49:43.670378 10986 tablet_bootstrap.cc:492] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
W20260808 01:49:43.684190  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 120356us
I20260808 01:49:43.688038 10972 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:43.688783 10972 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:43.693704 10972 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:43.712145  9473 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:43.715700  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:43.716686  9634 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:49:43.717624  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 142685us
I20260808 01:49:43.737910  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:43.744637  9521 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:43.749478 11021 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:49:43.750216 11021 raft_consensus.cc:697] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:43.744572 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:43.757733 10953 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260808 01:49:43.762475  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 108001us
I20260808 01:49:43.784336  9559 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:43.799247 10972 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:43.800335 11021 consensus_queue.cc:237] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:43.808264  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:43.810464  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:43.857260 10833 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:43.883445  9373 catalog_manager.cc:5697] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260808 01:49:43.905308  9600 consensus_peers.cc:597] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bf65b9272854b479e73f2cea21e93f2. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:43.882261  9377 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50068 (request call id 220) took 1596 ms. Trace:
I20260808 01:49:43.883579  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:49:43.909085  9371 catalog_manager.cc:5697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 1 to 2, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 2 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:43.923548  9377 rpcz_store.cc:276] 0808 01:49:42.286141 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:42.287357 (+  1216us) service_pool.cc:224] Handling call
0808 01:49:42.312087 (+ 24730us) spinlock_profiling.cc:243] Waited 4.56 ms on lock 0x7b540005c3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fd470 00007f5d4499c31a 00007f5d4a78c38c 00007f5d4a888bfe 00007f5d48d424a4 00007f5d48d42432 00007f5d48d42361 00007f5d48d422dc 00007f5d48d41592 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:49:43.882074 (+1569987us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:49:43.380008 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:49:43.380584 (+   576us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:49:43.380621 (+    37us) write_op.cc:435] Acquired schema lock
0808 01:49:43.380637 (+    16us) tablet.cc:662] Decoding operations
0808 01:49:43.381041 (+   404us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:49:43.381094 (+    53us) write_op.cc:641] Partition lock acquired for write op
0808 01:49:43.381130 (+    36us) tablet.cc:685] Acquiring locks for 2 operations
0808 01:49:43.381441 (+   311us) tablet.cc:701] Row locks acquired
0808 01:49:43.381472 (+    31us) write_op.cc:260] PREPARE: finished
0808 01:49:43.381616 (+   144us) write_op.cc:270] Start()
0808 01:49:43.381731 (+   115us) write_op.cc:276] Timestamp: P: 1786153783381571 usec, L: 0
0808 01:49:43.381749 (+    18us) op_driver.cc:348] REPLICATION: starting
0808 01:49:43.382411 (+   662us) log.cc:844] Serialized 928 byte log entry
0808 01:49:43.627154 (+244743us) op_driver.cc:464] REPLICATION: finished
0808 01:49:43.790142 (+162988us) write_op.cc:301] APPLY: starting
0808 01:49:43.790232 (+    90us) tablet.cc:1367] starting BulkCheckPresence
0808 01:49:43.790453 (+   221us) tablet.cc:1370] finished BulkCheckPresence
0808 01:49:43.790626 (+   173us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:49:43.791272 (+   646us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:49:43.791445 (+   173us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0808 01:49:43.791488 (+    43us) write_op.cc:312] APPLY: finished
0808 01:49:43.791817 (+   329us) log.cc:844] Serialized 28 byte log entry
0808 01:49:43.806333 (+ 14516us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:43.808580 (+  2247us) write_op.cc:454] Released schema lock
0808 01:49:43.813109 (+  4529us) write_op.cc:341] FINISH: Updating metrics
0808 01:49:43.923471 (+110362us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:43.926565 (+  3094us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":43269,"reported_tablets":4,"spinlock_wait_cycles":10498944,"child_traces":[["op",{"apply.queue_time_us":161188,"apply.run_cpu_time_us":5262,"apply.run_wall_time_us":38293,"num_ops":2,"prepare.queue_time_us":351450,"prepare.run_cpu_time_us":3016,"prepare.run_wall_time_us":22823,"replication_time_us":244028,"spinlock_wait_cycles":1271296,"thread_start_us":309729,"threads_started":2}]]}
I20260808 01:49:43.926201  9371 catalog_manager.cc:5697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 1 to 2, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 2 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 01:49:43.916658  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 120935us
I20260808 01:49:43.971390 11042 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:43.984467 11028 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:44.052507  9480 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:49:44.032253 11042 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:44.097482 11052 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.067s	user 0.002s	sys 0.001s
W20260808 01:49:44.169034  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52144.1us
I20260808 01:49:44.158056 11010 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20260808 01:49:44.254701  9634 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:49:44.252701 11021 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:49:44.296754 11058 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
I20260808 01:49:44.298873 11021 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:44.375949 11021 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:44.377455 10937 raft_consensus.cc:2955] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:44.380165 11063 ts_tablet_manager.cc:933] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:44.389319 11063 tablet_copy_client.cc:323] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.195:34565
I20260808 01:49:44.392347  9648 tablet_copy_service.cc:140] P 259bcb3f0d254e8b9effac3277e654e0: Received BeginTabletCopySession request for tablet 9bf65b9272854b479e73f2cea21e93f2 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:45658)
I20260808 01:49:44.393168  9648 tablet_copy_service.cc:161] P 259bcb3f0d254e8b9effac3277e654e0: Beginning new tablet copy session on tablet 9bf65b9272854b479e73f2cea21e93f2 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:45658: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-9bf65b9272854b479e73f2cea21e93f2
I20260808 01:49:44.435879 10986 tablet_bootstrap.cc:492] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:44.437050 10937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:45650 (request call id 8) took 1245 ms. Trace:
I20260808 01:49:44.437580 10937 rpcz_store.cc:276] 0808 01:49:43.191768 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:43.192018 (+   250us) service_pool.cc:224] Handling call
0808 01:49:43.451611 (+259593us) spinlock_profiling.cc:243] Waited 32.3 ms on lock 0x7b58002501d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d448ef48a 00007f5d44971827 00007f5d4498eaea 00007f5d4498fb4a 00007f5d4498ecfb 00007f5d4499a41c 00007f5d44999a93 00007f5d4998bb39 00007f5d42ed8334 00007f5d42ed82c2
0808 01:49:43.461167 (+  9556us) log.cc:844] Serialized 506 byte log entry
0808 01:49:43.676132 (+214965us) spinlock_profiling.cc:243] Waited 2.76 ms on lock 0x7b18003135a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d448eceeb 00007f5d44971f43 00007f5d44999b32 00007f5d4998bb39
0808 01:49:44.436931 (+760799us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":208867,"raft.queue_time_us":138433,"raft.run_cpu_time_us":2217,"raft.run_wall_time_us":29732,"spinlock_wait_cycles":80546432,"thread_start_us":91585,"threads_started":1,"wal-append.queue_time_us":151380}
I20260808 01:49:44.439900 10986 tablet_bootstrap.cc:492] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:44.440814 10986 ts_tablet_manager.cc:1403] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.771s	user 0.035s	sys 0.031s
I20260808 01:49:44.444953 10986 raft_consensus.cc:359] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:44.445907 10986 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:44.446650 10986 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:44.449810 10986 ts_tablet_manager.cc:1434] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.004s
I20260808 01:49:44.539189 11042 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:49:44.583676 11055 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.319s	user 0.003s	sys 0.001s
I20260808 01:49:44.691642 11010 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:44.692487 11010 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:44.704463 11005 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:44.705196 11005 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:44.711607 11005 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:44.702353 11021 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:44.714784 11021 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:44.717558 11021 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:44.720623 11010 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:44.733449  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:44.734616  9631 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 2: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:49:44.839959  9648 tablet_copy_source_session.cc:215] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:44.841329 10965 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:44.953315  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:44.956156  9478 raft_consensus.cc:2955] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:44.959210 11008 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:44.959882 11008 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:44.962430 11008 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:44.977653  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:44.978425  9476 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:49:44.980288 11063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bf65b9272854b479e73f2cea21e93f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:44.981160  9491 tablet_copy_service.cc:342] P 6e7856bed1be43acaa821290d506ee97: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-c094b8e92d5e449aa2641c359d46d622 received from {username='slave'} at 127.0.0.1:37248
I20260808 01:49:44.981676  9491 tablet_copy_service.cc:434] P 6e7856bed1be43acaa821290d506ee97: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-c094b8e92d5e449aa2641c359d46d622 on tablet c094b8e92d5e449aa2641c359d46d622 with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:44.985380  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:44.985870  9559 raft_consensus.cc:2955] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:44.996645 11062 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:44.997350 11062 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:45.004446 11028 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:45.007445 11028 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:45.017529 11062 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:45.023322 11028 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:45.025328 11017 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:45.026326 11017 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:45.031921 11017 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:45.048419 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:45.063270  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:45.064117  9557 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:45.066265  9371 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55058 (request call id 103) took 1762 ms. Trace:
I20260808 01:49:45.068459  9373 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50064 (request call id 218) took 2082 ms. Trace:
I20260808 01:49:45.070443  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:45.071223  9632 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:49:45.068881  9373 rpcz_store.cc:276] 0808 01:49:42.986060 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:42.986239 (+   179us) service_pool.cc:224] Handling call
0808 01:49:45.068397 (+2082158us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:49:44.295171 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:49:44.314534 (+ 19363us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:49:44.314597 (+    63us) write_op.cc:435] Acquired schema lock
0808 01:49:44.314617 (+    20us) tablet.cc:662] Decoding operations
0808 01:49:44.315181 (+   564us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:49:44.315282 (+   101us) write_op.cc:641] Partition lock acquired for write op
0808 01:49:44.315323 (+    41us) tablet.cc:685] Acquiring locks for 1 operations
0808 01:49:44.315648 (+   325us) tablet.cc:701] Row locks acquired
0808 01:49:44.315692 (+    44us) write_op.cc:260] PREPARE: finished
0808 01:49:44.315990 (+   298us) write_op.cc:270] Start()
0808 01:49:44.316216 (+   226us) write_op.cc:276] Timestamp: P: 1786153784315901 usec, L: 0
0808 01:49:44.316263 (+    47us) op_driver.cc:348] REPLICATION: starting
0808 01:49:44.317165 (+   902us) log.cc:844] Serialized 598 byte log entry
0808 01:49:44.614960 (+297795us) op_driver.cc:464] REPLICATION: finished
0808 01:49:44.616401 (+  1441us) write_op.cc:301] APPLY: starting
0808 01:49:44.616503 (+   102us) tablet.cc:1367] starting BulkCheckPresence
0808 01:49:44.616611 (+   108us) tablet.cc:1370] finished BulkCheckPresence
0808 01:49:44.616742 (+   131us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:49:44.617208 (+   466us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:49:44.617440 (+   232us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 01:49:44.617492 (+    52us) write_op.cc:312] APPLY: finished
0808 01:49:44.617718 (+   226us) log.cc:844] Serialized 22 byte log entry
0808 01:49:44.617931 (+   213us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:44.618027 (+    96us) write_op.cc:454] Released schema lock
0808 01:49:44.618110 (+    83us) write_op.cc:341] FINISH: Updating metrics
0808 01:49:44.618515 (+   405us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:44.618548 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":16346,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":1157,"apply.run_cpu_time_us":1994,"apply.run_wall_time_us":2067,"num_ops":1,"prepare.queue_time_us":348742,"prepare.run_cpu_time_us":4837,"prepare.run_wall_time_us":43747,"replication_time_us":298514,"thread_start_us":234615,"threads_started":2}]]}
W20260808 01:49:45.075677 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 65260.2us
I20260808 01:49:45.076704  9372 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50084 (request call id 214) took 1355 ms. Trace:
I20260808 01:49:45.077173  9372 rpcz_store.cc:276] 0808 01:49:43.720900 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:43.721075 (+   175us) service_pool.cc:224] Handling call
0808 01:49:43.800284 (+ 79209us) spinlock_profiling.cc:243] Waited 74.1 ms on lock 0x7b540005c3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fd470 00007f5d4499c31a 00007f5d4a78c38c 00007f5d4a888bfe 00007f5d48d424a4 00007f5d48d42432 00007f5d48d42361 00007f5d48d422dc 00007f5d48d41592 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:49:45.076474 (+1276190us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":170495744}
I20260808 01:49:45.082156 10906 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:45.082859 10906 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:45.083513 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:45.067006  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:45.084386  9556 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:45.085114 10906 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:45.085611  9599 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:45.086658 10803 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:45.087097 10803 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:45.087486 10803 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:45.087796  9371 rpcz_store.cc:276] 0808 01:49:43.303527 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:43.303692 (+   165us) service_pool.cc:224] Handling call
0808 01:49:45.066196 (+1762504us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:49:44.332068 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:49:44.352387 (+ 20319us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:49:44.354900 (+  2513us) write_op.cc:435] Acquired schema lock
0808 01:49:44.356229 (+  1329us) tablet.cc:662] Decoding operations
0808 01:49:44.362096 (+  5867us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:49:44.363461 (+  1365us) write_op.cc:641] Partition lock acquired for write op
0808 01:49:44.365650 (+  2189us) tablet.cc:685] Acquiring locks for 2 operations
0808 01:49:44.377397 (+ 11747us) tablet.cc:701] Row locks acquired
0808 01:49:44.377431 (+    34us) write_op.cc:260] PREPARE: finished
0808 01:49:44.377598 (+   167us) write_op.cc:270] Start()
0808 01:49:44.377744 (+   146us) write_op.cc:276] Timestamp: P: 1786153784377538 usec, L: 0
0808 01:49:44.377762 (+    18us) op_driver.cc:348] REPLICATION: starting
0808 01:49:44.378922 (+  1160us) log.cc:844] Serialized 918 byte log entry
0808 01:49:44.616458 (+237536us) op_driver.cc:464] REPLICATION: finished
0808 01:49:44.617869 (+  1411us) write_op.cc:301] APPLY: starting
0808 01:49:44.618084 (+   215us) tablet.cc:1367] starting BulkCheckPresence
0808 01:49:44.618404 (+   320us) tablet.cc:1370] finished BulkCheckPresence
0808 01:49:44.618560 (+   156us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:49:44.619070 (+   510us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:49:44.619248 (+   178us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0808 01:49:44.619281 (+    33us) write_op.cc:312] APPLY: finished
0808 01:49:44.619509 (+   228us) log.cc:844] Serialized 28 byte log entry
0808 01:49:44.619705 (+   196us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:44.619799 (+    94us) write_op.cc:454] Released schema lock
0808 01:49:44.619854 (+    55us) write_op.cc:341] FINISH: Updating metrics
0808 01:49:44.620662 (+   808us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:44.620683 (+    21us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":434215,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":1159,"apply.run_cpu_time_us":2164,"apply.run_wall_time_us":2759,"num_ops":2,"prepare.queue_time_us":343472,"prepare.run_cpu_time_us":9061,"prepare.run_wall_time_us":49875,"replication_time_us":238546,"spinlock_wait_cycles":2304,"thread_start_us":730,"threads_started":1}]]}
I20260808 01:49:45.095997  9551 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:45.102039 10906 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:45.102658 10906 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:45.105041 10906 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:45.074285  9377 catalog_manager.cc:5697] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 1 to 2, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 2 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:45.113862  9377 catalog_manager.cc:5697] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 1 to 3, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 3 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
W20260808 01:49:45.132058  9523 consensus_peers.cc:597] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fca874793205463dbb922993f57bf7f5. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:45.160113  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:45.185935  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 182458us
I20260808 01:49:45.189797  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:45.190703  9478 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
W20260808 01:49:45.203346  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 68102.9us
I20260808 01:49:45.200248 11013 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:45.202414  9601 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:45.182929 10803 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:45.213742 10803 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:45.216879  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:45.218693  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:45.218914 11063 tablet_copy_client.cc:806] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:45.219409  9550 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:49:45.219625 11063 tablet_copy_client.cc:670] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:45.220650  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:45.220485  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:45.228907  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:45.229832  9636 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:49:45.230388  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:45.231048  9476 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:45.233351 11026 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:45.244313  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:45.245682  9633 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:49:45.248673 11023 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:49:45.235174 11005 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:45.252358 11005 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:45.236464  9634 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:49:45.244091  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:45.259409  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:45.260139  9557 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 3
I20260808 01:49:45.233935 11026 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:45.264694  9601 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:45.207011 11013 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:45.265242 10983 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:45.280889 10983 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:45.222100 11007 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:45.274482  9476 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:49:45.289582  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:45.254482 11005 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:45.327183  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 51619.9us
I20260808 01:49:45.340788  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:45.275600  9556 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:40060
I20260808 01:49:45.284271 11036 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:49:45.343554  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 119128us
I20260808 01:49:45.344313  9556 raft_consensus.cc:606] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:45.348019  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:45.315701 10929 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:49:45.355142 10906 raft_consensus.cc:2764] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260808 01:49:45.361156 11017 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:45.373211 10983 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:45.376394 10906 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
W20260808 01:49:45.388301  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 60010.6us
W20260808 01:49:45.442667  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 101986us
I20260808 01:49:45.446076 11036 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:45.377902 11063 tablet_copy_client.cc:538] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:45.364795 10929 raft_consensus.cc:697] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:49:45.452190  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 122117us
I20260808 01:49:45.437068 11026 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:45.457332  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:45.457504  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:45.436805 11013 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:45.456972  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:45.395031 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:45.465186 10827 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:45.465677 10827 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:45.466041 10827 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:45.446825 11036 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:45.464282  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:45.475311 10929 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:45.482308  9557 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:49:45.486989  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:45.490839  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:45.489416 11080 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:45.489996  9524 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:45.495136  9632 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:49:45.497818  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:45.498353  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:45.510234  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:45.513013  9557 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:49:45.516144  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:45.518738  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:45.514163  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:45.531865 11080 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:45.538578  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:45.539917  9521 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:45.544329  9473 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:49:45.501026 11062 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:49:45.541369  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:45.505136 11022 raft_consensus.cc:2764] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260808 01:49:45.548939  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:45.556162  9524 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:45.559185 10776 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:45.547762 10833 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:45.551010 11021 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260808 01:49:45.585722 10833 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:45.589874 10833 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:45.574757 11086 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:45.590489 10982 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:45.562252 10906 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:45.592939  9556 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:40060
I20260808 01:49:45.598479  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:45.576992 11083 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:49:45.598698  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:45.542361 11080 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:45.622117  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 73294.7us
I20260808 01:49:45.645484  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:45.647204  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:45.608676 11083 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:49:45.658588  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:45.660400 10827 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:45.640174  9660 debug-util.cc:398] Leaking SignalData structure 0x7b0800114080 after lost signal to thread 7834
I20260808 01:49:45.662523  9477 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
W20260808 01:49:45.663209  9660 debug-util.cc:398] Leaking SignalData structure 0x7b080013c8a0 after lost signal to thread 9412
W20260808 01:49:45.665344  9660 debug-util.cc:398] Leaking SignalData structure 0x7b080027cd20 after lost signal to thread 9506
I20260808 01:49:45.667632 11063 tablet_bootstrap.cc:492] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:45.671653  9601 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:45.673867  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:45.683341 10962 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:45.672603 11083 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:49:45.699242  9660 debug-util.cc:398] Leaking SignalData structure 0x7b080035ae80 after lost signal to thread 9585
W20260808 01:49:45.701889  9660 debug-util.cc:398] Leaking SignalData structure 0x7b0800344c40 after lost signal to thread 9663
I20260808 01:49:45.694802 11008 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:45.712916 10827 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:45.714341 11093 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:45.687768 11021 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:45.669982 10982 raft_consensus.cc:2955] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:45.736701 11036 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:45.739321  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:45.744747  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:49:45.747776  9660 debug-util.cc:398] Leaking SignalData structure 0x7b08003628a0 after lost signal to thread 10393
I20260808 01:49:45.736426 11083 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:45.778443  9557 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:49:45.779053  9556 raft_consensus.cc:606] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:45.779531  9556 raft_consensus.cc:614] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:49:45.789103 11080 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:45.791292  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:45.765935  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:45.698423 10962 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:45.798734  9632 raft_consensus.cc:2955] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:45.809571 11095 ts_tablet_manager.cc:933] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:45.815223  9599 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:45.823743 10962 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:45.826395  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:45.827930 10911 raft_consensus.cc:993] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: : Instructing follower 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:49:45.828913 10911 raft_consensus.cc:1081] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Signalling peer 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:49:45.829728  9633 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:49:45.831543 10808 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:49:45.839159 11083 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:45.849953 11023 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:45.863816  9636 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:42824
I20260808 01:49:45.864691  9636 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:45.865176  9636 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:45.868361  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:45.878854  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:45.881043 10827 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:49:45.865722 11023 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:45.902611  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:45.903016  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:45.903388  9473 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:49:45.906958  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:45.888082 11086 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:45.933400  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:45.937376  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:45.937451  9377 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50068 (request call id 221) took 1220 ms. Trace:
I20260808 01:49:45.942258  9377 rpcz_store.cc:276] 0808 01:49:44.716693 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:44.717715 (+  1022us) service_pool.cc:224] Handling call
0808 01:49:45.937377 (+1219662us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:49:45.196894 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:49:45.197522 (+   628us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:49:45.197562 (+    40us) write_op.cc:435] Acquired schema lock
0808 01:49:45.197582 (+    20us) tablet.cc:662] Decoding operations
0808 01:49:45.198031 (+   449us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:49:45.198131 (+   100us) write_op.cc:641] Partition lock acquired for write op
0808 01:49:45.198165 (+    34us) tablet.cc:685] Acquiring locks for 2 operations
0808 01:49:45.198604 (+   439us) tablet.cc:701] Row locks acquired
0808 01:49:45.198622 (+    18us) write_op.cc:260] PREPARE: finished
0808 01:49:45.202031 (+  3409us) write_op.cc:270] Start()
0808 01:49:45.207867 (+  5836us) write_op.cc:276] Timestamp: P: 1786153785200258 usec, L: 0
0808 01:49:45.207902 (+    35us) op_driver.cc:348] REPLICATION: starting
0808 01:49:45.261957 (+ 54055us) log.cc:844] Serialized 928 byte log entry
0808 01:49:45.359541 (+ 97584us) op_driver.cc:464] REPLICATION: finished
0808 01:49:45.480888 (+121347us) spinlock_profiling.cc:243] Waited 3.49 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4186fa5c 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d483e98d0
0808 01:49:45.550767 (+ 69879us) write_op.cc:301] APPLY: starting
0808 01:49:45.550930 (+   163us) tablet.cc:1367] starting BulkCheckPresence
0808 01:49:45.551203 (+   273us) tablet.cc:1370] finished BulkCheckPresence
0808 01:49:45.551260 (+    57us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:49:45.615751 (+ 64491us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:49:45.633214 (+ 17463us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0808 01:49:45.647056 (+ 13842us) write_op.cc:312] APPLY: finished
0808 01:49:45.692554 (+ 45498us) log.cc:844] Serialized 28 byte log entry
0808 01:49:45.705775 (+ 13221us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:45.731058 (+ 25283us) write_op.cc:454] Released schema lock
0808 01:49:45.741436 (+ 10378us) write_op.cc:341] FINISH: Updating metrics
0808 01:49:45.760607 (+ 19171us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:45.760700 (+    93us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":340217,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":190862,"apply.run_cpu_time_us":4939,"apply.run_wall_time_us":197372,"num_ops":2,"prepare.queue_time_us":69178,"prepare.run_cpu_time_us":3699,"prepare.run_wall_time_us":65433,"replication_time_us":148696,"spinlock_wait_cycles":8024960,"thread_start_us":249122,"threads_started":2}]]}
I20260808 01:49:45.914088 11023 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:45.946110  9372 catalog_manager.cc:5697] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:45.966149  9372 catalog_manager.cc:5697] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
W20260808 01:49:45.997488  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 83178.4us
I20260808 01:49:46.013268  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:49:46.013810  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 70573.9us
I20260808 01:49:45.949640 11086 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:46.043362  9524 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:46.050926 11023 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:46.021422 11013 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:46.051260 11086 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:46.070168  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:46.080989 11098 log.cc:927] Time spent T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.146s	user 0.002s	sys 0.000s
I20260808 01:49:46.072257  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:46.086614 10332 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:49:46.093079  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 67065.3us
I20260808 01:49:46.091468 11074 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:46.120224  9636 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:46.100818 11095 tablet_copy_client.cc:323] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.194:36095
I20260808 01:49:46.150477  9473 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
W20260808 01:49:46.306999 10982 consensus_meta.cc:287] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.557s	user 0.010s	sys 0.008s
W20260808 01:49:46.345798 10808 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.513s	user 0.004s	sys 0.014s
W20260808 01:49:46.375675  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 177821us
I20260808 01:49:46.370839 10808 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:49:46.436456  9560 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:49:46.436578 10365 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Advancing to term 3
W20260808 01:49:45.849138  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 4bce6ea2a8244811bf007638181a417a (from 52ccf4dcca1843f7bb29e9172e2d4ab4 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:49:46.466778  9481 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:38182
I20260808 01:49:46.467489  9481 raft_consensus.cc:606] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:46.474355  9632 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:49:46.499691  9636 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:46.506829 11104 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:46.507526 11104 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:46.506071  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:46.513170  9560 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:46.513545  9560 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:46.518369 10982 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:60546 (request call id 6) took 1418 ms. Trace:
I20260808 01:49:46.525208 10982 rpcz_store.cc:276] 0808 01:49:45.093755 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:45.093918 (+   163us) service_pool.cc:224] Handling call
0808 01:49:45.101671 (+  7753us) log.cc:844] Serialized 506 byte log entry
0808 01:49:46.511833 (+1410162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12946,"raft.queue_time_us":17358,"raft.run_cpu_time_us":1739,"raft.run_wall_time_us":1738,"spinlock_wait_cycles":543104,"thread_start_us":12866,"threads_started":1,"wal-append.queue_time_us":11232,"wal-append.run_cpu_time_us":1832,"wal-append.run_wall_time_us":1002504}
I20260808 01:49:46.531342 10365 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 LEARNER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:49:46.479789 11019 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260808 01:49:46.533207  9557 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:46.536783  9560 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:46.545563 11104 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:49:46.547472 11111 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.269s	user 0.001s	sys 0.007s
I20260808 01:49:46.555809 10833 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:46.563402 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:46.573383  9480 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:38182
I20260808 01:49:46.574378  9480 raft_consensus.cc:606] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:46.575085  9480 raft_consensus.cc:614] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:49:46.567979  9571 tablet_copy_service.cc:140] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Received BeginTabletCopySession request for tablet fca874793205463dbb922993f57bf7f5 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:60552)
W20260808 01:49:46.582767  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet c094b8e92d5e449aa2641c359d46d622 (from 6e7856bed1be43acaa821290d506ee97 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:49:46.595862  9571 tablet_copy_service.cc:161] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Beginning new tablet copy session on tablet fca874793205463dbb922993f57bf7f5 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:60552: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-fca874793205463dbb922993f57bf7f5
I20260808 01:49:46.596342  9636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:49:46.597110  9636 raft_consensus.cc:606] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Received request to transfer leadership to TS 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:46.602739  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260808 01:49:46.607638  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:46.620040  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:46.620855  9599 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:46.631793  9524 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:46.633984  9477 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:46.639715  9524 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:46.629232 11104 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:46.666255 11084 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:49:46.651062 11017 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:49:46.637568  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 125858us
I20260808 01:49:46.679066  9473 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:49:46.683456  9560 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:46.686035  9571 tablet_copy_source_session.cc:215] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:46.699430  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:49:46.701037  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:49:46.701397 11084 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:46.714249  9372 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50064 (request call id 219) took 1373 ms. Trace:
I20260808 01:49:46.716137  9636 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:46.703855  9476 raft_consensus.cc:2955] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:46.757917 11104 raft_consensus.cc:993] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 7201605bb2f54c949a4ebe6e9b3b4b2a to start an election
I20260808 01:49:46.702881 11084 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:46.759725 11104 raft_consensus.cc:1081] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Signalling peer 7201605bb2f54c949a4ebe6e9b3b4b2a to start an election
I20260808 01:49:46.738313 11036 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260808 01:49:46.762732 10365 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
 from {username='slave'} at 127.0.0.1:56266
I20260808 01:49:46.763640 10365 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:46.764245 10365 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:46.710863  9373 catalog_manager.cc:5697] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index 2 to -1, term changed from 1 to 3, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195), NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) evicted, now has a pending config: VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193), VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194), VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195), NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 3 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
W20260808 01:49:46.786697  9523 consensus_peers.cc:597] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03b17f21031f4ff4828c85b7332fcc2d. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:46.804296  9560 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:49:46.811556  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:46.782127  9477 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:49:46.825770  9480 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:46.827370  9476 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260808 01:49:46.836438 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 166766us
I20260808 01:49:46.848750 11073 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:46.849709 11073 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:46.835274 11086 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260808 01:49:46.715082  9372 rpcz_store.cc:276] 0808 01:49:45.340796 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:45.340982 (+   186us) service_pool.cc:224] Handling call
0808 01:49:46.714173 (+1373191us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:49:46.154620 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:49:46.155775 (+  1155us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:49:46.155826 (+    51us) write_op.cc:435] Acquired schema lock
0808 01:49:46.155849 (+    23us) tablet.cc:662] Decoding operations
0808 01:49:46.157610 (+  1761us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:49:46.157745 (+   135us) write_op.cc:641] Partition lock acquired for write op
0808 01:49:46.157776 (+    31us) tablet.cc:685] Acquiring locks for 2 operations
0808 01:49:46.158090 (+   314us) tablet.cc:701] Row locks acquired
0808 01:49:46.158105 (+    15us) write_op.cc:260] PREPARE: finished
0808 01:49:46.158262 (+   157us) write_op.cc:270] Start()
0808 01:49:46.158381 (+   119us) write_op.cc:276] Timestamp: P: 1786153786158216 usec, L: 0
0808 01:49:46.158395 (+    14us) op_driver.cc:348] REPLICATION: starting
0808 01:49:46.159138 (+   743us) log.cc:844] Serialized 986 byte log entry
0808 01:49:46.341527 (+182389us) op_driver.cc:464] REPLICATION: finished
0808 01:49:46.578992 (+237465us) write_op.cc:301] APPLY: starting
0808 01:49:46.581630 (+  2638us) tablet.cc:1367] starting BulkCheckPresence
0808 01:49:46.581883 (+   253us) tablet.cc:1370] finished BulkCheckPresence
0808 01:49:46.582259 (+   376us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:49:46.587784 (+  5525us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:49:46.588103 (+   319us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0808 01:49:46.588176 (+    73us) write_op.cc:312] APPLY: finished
0808 01:49:46.588416 (+   240us) log.cc:844] Serialized 28 byte log entry
0808 01:49:46.588708 (+   292us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:46.588827 (+   119us) write_op.cc:454] Released schema lock
0808 01:49:46.588898 (+    71us) write_op.cc:341] FINISH: Updating metrics
0808 01:49:46.589465 (+   567us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:46.589515 (+    50us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":61626,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":235961,"apply.run_cpu_time_us":3547,"apply.run_wall_time_us":10335,"num_ops":2,"prepare.queue_time_us":131429,"prepare.run_cpu_time_us":3802,"prepare.run_wall_time_us":89403,"replication_time_us":182801,"thread_start_us":271485,"threads_started":2}]]}
I20260808 01:49:46.770390 11019 raft_consensus.cc:2955] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:46.864631 10365 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:46.900300 11095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fca874793205463dbb922993f57bf7f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:49:46.890319  9630 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260808 01:49:46.951254 11063 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20260808 01:49:46.960595 10365 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:47.083976  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 131428us
I20260808 01:49:47.081775 11063 tablet_bootstrap.cc:492] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:46.921245 11073 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:47.091312  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:46.954474  9371 catalog_manager.cc:5697] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index -1 to 2, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added, no longer has a pending config: VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193), VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194), VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195), NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 3 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
W20260808 01:49:47.090456  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 121312us
I20260808 01:49:47.112669  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:47.114959  9472 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:47.118558  9373 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50084 (request call id 215) took 1255 ms. Trace:
I20260808 01:49:47.124318  9630 raft_consensus.cc:3055] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:47.133067  9630 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:47.119092  9373 rpcz_store.cc:276] 0808 01:49:45.863205 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:45.863400 (+   195us) service_pool.cc:224] Handling call
0808 01:49:47.118493 (+1255093us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:49:46.774302 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:49:46.775125 (+   823us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:49:46.775179 (+    54us) write_op.cc:435] Acquired schema lock
0808 01:49:46.775209 (+    30us) tablet.cc:662] Decoding operations
0808 01:49:46.777223 (+  2014us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:49:46.777301 (+    78us) write_op.cc:641] Partition lock acquired for write op
0808 01:49:46.777336 (+    35us) tablet.cc:685] Acquiring locks for 1 operations
0808 01:49:46.777617 (+   281us) tablet.cc:701] Row locks acquired
0808 01:49:46.777634 (+    17us) write_op.cc:260] PREPARE: finished
0808 01:49:46.777836 (+   202us) write_op.cc:270] Start()
0808 01:49:46.778013 (+   177us) write_op.cc:276] Timestamp: P: 1786153786777782 usec, L: 0
0808 01:49:46.778034 (+    21us) op_driver.cc:348] REPLICATION: starting
0808 01:49:46.785367 (+  7333us) log.cc:844] Serialized 792 byte log entry
0808 01:49:46.790513 (+  5146us) op_driver.cc:464] REPLICATION: finished
0808 01:49:46.791062 (+   549us) write_op.cc:301] APPLY: starting
0808 01:49:46.791147 (+    85us) tablet.cc:1367] starting BulkCheckPresence
0808 01:49:46.791283 (+   136us) tablet.cc:1370] finished BulkCheckPresence
0808 01:49:46.794073 (+  2790us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:49:46.794340 (+   267us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:49:46.794556 (+   216us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 01:49:46.794635 (+    79us) write_op.cc:312] APPLY: finished
0808 01:49:46.799599 (+  4964us) log.cc:844] Serialized 23 byte log entry
0808 01:49:46.799857 (+   258us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:46.799968 (+   111us) write_op.cc:454] Released schema lock
0808 01:49:46.800022 (+    54us) write_op.cc:341] FINISH: Updating metrics
0808 01:49:46.808998 (+  8976us) spinlock_profiling.cc:243] Waited 6.05 ms on lock 0x7b18002c6fe0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188ce62 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:49:46.809194 (+   196us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:46.809238 (+    44us) write_op.cc:454] Released schema lock
Metrics: {"dns_us":20644,"mutex_wait_us":71271,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":193,"apply.run_cpu_time_us":2149,"apply.run_wall_time_us":9380,"num_ops":1,"prepare.queue_time_us":630,"prepare.run_cpu_time_us":3993,"prepare.run_wall_time_us":21050,"replication_time_us":12297,"spinlock_wait_cycles":13924224}]]}
I20260808 01:49:47.136400  9477 raft_consensus.cc:2955] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:47.153542  9472 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:49:47.153802 10955 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20260808 01:49:47.162194 11063 tablet_bootstrap.cc:492] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:47.134459  9630 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:47.215655 11063 ts_tablet_manager.cc:1403] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 1.549s	user 0.038s	sys 0.055s
W20260808 01:49:47.259902  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 113579us
I20260808 01:49:47.219756 11104 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:47.194561 10334 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:49:47.291021  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 144801us
I20260808 01:49:47.269836 11104 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:47.319980  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:49:47.222784 11119 raft_consensus.cc:2955] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
W20260808 01:49:47.330358  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 47950.7us
W20260808 01:49:47.338939 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 55850.6us
W20260808 01:49:47.345556  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 100615us
I20260808 01:49:47.324070 11134 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:47.415416  9477 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260808 01:49:47.457257  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 84338.1us
I20260808 01:49:47.461068 11158 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:47.464931 11155 raft_consensus.cc:993] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: : Instructing follower 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
W20260808 01:49:47.474630  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 125936us
W20260808 01:49:47.484439  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 72998.6us
W20260808 01:49:47.487416  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 68153.7us
W20260808 01:49:47.430717 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 244097us
W20260808 01:49:47.497597  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 150782us
I20260808 01:49:47.488265 11155 raft_consensus.cc:1081] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Signalling peer 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:49:47.534837 11104 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:49:47.540721 11019 consensus_meta.cc:287] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.620s	user 0.015s	sys 0.002s
W20260808 01:49:47.526299 11137 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.236s	user 0.001s	sys 0.000s
I20260808 01:49:47.589373 11063 raft_consensus.cc:359] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:47.590803 11063 raft_consensus.cc:740] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:47.591908 11063 consensus_queue.cc:260] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:47.629493  9632 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:49:47.612865  9630 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:47.608047 11134 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:49:47.622437 11095 consensus_meta.cc:287] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.526s	user 0.011s	sys 0.007s
W20260808 01:49:47.637205  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 170206us
I20260808 01:49:47.634552  9372 catalog_manager.cc:5697] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: config changed from index -1 to 3, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 2 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:47.644032  9372 catalog_manager.cc:5697] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:47.647387  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:47.666919 11063 ts_tablet_manager.cc:1434] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.411s	user 0.005s	sys 0.007s
I20260808 01:49:47.675482  9557 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:53216
I20260808 01:49:47.684190  9557 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:47.684679  9557 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:47.700851 11084 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:47.706234 11095 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:56276 (request call id 190) took 2032 ms. Trace:
I20260808 01:49:47.740764  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:47.745227  9476 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:47.761700  9557 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:47.768755  9557 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:47.775903  9599 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:47.777137  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:47.777990  9629 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:47.767912 11119 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:60546 (request call id 7) took 1235 ms. Trace:
I20260808 01:49:47.791085  9648 tablet_copy_service.cc:342] P 259bcb3f0d254e8b9effac3277e654e0: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-9bf65b9272854b479e73f2cea21e93f2 received from {username='slave'} at 127.0.0.1:45658
I20260808 01:49:47.798648  9557 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260808 01:49:47.793474 11163 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260808 01:49:47.797317 11168 ts_tablet_manager.cc:933] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:47.788565 10808 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:47.808606 10358 tablet_service.cc:1558] Processing DeleteTablet for tablet c094b8e92d5e449aa2641c359d46d622 with delete_type TABLET_DATA_TOMBSTONED (TS 7201605bb2f54c949a4ebe6e9b3b4b2a not found in new config with opid_index -1) from {username='slave'} at 127.0.0.1:49490
I20260808 01:49:47.820267  9630 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:49:47.717907 11095 rpcz_store.cc:276] 0808 01:49:45.674098 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:45.674259 (+   161us) service_pool.cc:224] Handling call
0808 01:49:45.810151 (+135892us) ts_tablet_manager.cc:934] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
0808 01:49:46.858736 (+1048585us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:49:47.096091 (+237355us) tablet_metadata.cc:665] Metadata flushed
0808 01:49:47.096293 (+   202us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:49:47.658846 (+562553us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:49:47.705605 (+ 46759us) tablet_replica.cc:206] Creating consensus instance
0808 01:49:47.706091 (+   486us) ts_tablet_manager.cc:989] Replica registered.
0808 01:49:47.706158 (+    67us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":44678,"tablet-copy.queue_time_us":75905,"thread_start_us":247894,"threads_started":2}
I20260808 01:49:47.764971 11134 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:47.831149  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:47.833280  9479 raft_consensus.cc:3055] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:47.833710  9479 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:47.812224  9648 tablet_copy_service.cc:434] P 259bcb3f0d254e8b9effac3277e654e0: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-9bf65b9272854b479e73f2cea21e93f2 on tablet 9bf65b9272854b479e73f2cea21e93f2 with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:47.795152 10937 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260808 01:49:47.861398  9629 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:47.899907  9479 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:47.793828 11119 rpcz_store.cc:276] 0808 01:49:46.531184 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:46.531364 (+   180us) service_pool.cc:224] Handling call
0808 01:49:46.538007 (+  6643us) log.cc:844] Serialized 506 byte log entry
0808 01:49:47.766032 (+1228025us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":123342,"raft.queue_time_us":127487,"raft.run_cpu_time_us":1894,"raft.run_wall_time_us":1897,"thread_start_us":140176,"threads_started":1,"wal-append.queue_time_us":181205,"wal-append.run_cpu_time_us":1785,"wal-append.run_wall_time_us":1015672}
I20260808 01:49:47.767891  9633 raft_consensus.cc:2955] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:47.933341  9479 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:49:47.951824  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 167285us
W20260808 01:49:47.955924  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 171146us
I20260808 01:49:47.997557  9521 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:47.955907 10925 raft_consensus.cc:1064] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
W20260808 01:49:48.017952  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 233313us
I20260808 01:49:48.027500 11095 tablet_copy_client.cc:806] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:48.028609 11095 tablet_copy_client.cc:670] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:48.053546 11152 raft_consensus.cc:1064] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
W20260808 01:49:48.065485  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 68059.4us
I20260808 01:49:48.067262 11146 raft_consensus.cc:1064] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:48.092422  9479 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
W20260808 01:49:48.103040  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:49:48.106276  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 83291.7us
I20260808 01:49:48.066190 11165 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:48.111735  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:49:48.118575 11165 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:48.123291 10925 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:48.122830 11165 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:48.131474  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 158766us
W20260808 01:49:48.137357  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 139013us
I20260808 01:49:48.162720 11104 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260808 01:49:48.200269 11095 tablet_copy_client.cc:538] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260808 01:49:48.263082  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:49:48.316706  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 89850.8us
I20260808 01:49:48.310272  9634 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
W20260808 01:49:48.329113 10358 thread.cc:641] tablet-delete [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.014s	sys 0.044s
W20260808 01:49:48.383782  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:49:48.459985  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 98811.8us
W20260808 01:49:48.460587  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 98303.2us
W20260808 01:49:48.472326 10358 thread.cc:608] tablet-delete [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.014s	sys 0.044s
W20260808 01:49:48.493131 11168 thread.cc:641] tablet-download-pool [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.016s	sys 0.036s
I20260808 01:49:48.547219 11160 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:49:48.550949  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 216831us
W20260808 01:49:48.533564 11168 thread.cc:608] tablet-download-pool [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.016s	sys 0.036s
I20260808 01:49:48.547848 11160 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:48.595664 11168 tablet_copy_client.cc:323] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.194:36095
W20260808 01:49:48.651295  9599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.002s	sys 0.024s
W20260808 01:49:48.651810  9599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.002s	sys 0.024s
W20260808 01:49:48.652150  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 513994us
I20260808 01:49:48.656538 11160 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:48.657989 11160 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:48.663468 11176 tablet_replica.cc:333] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
W20260808 01:49:48.646871  9633 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.714s	user 0.010s	sys 0.007s
I20260808 01:49:48.665493  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1267) took 1065 ms. Trace:
I20260808 01:49:48.665678 11150 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:48.673718 11150 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:48.665967  9633 rpcz_store.cc:276] 0808 01:49:47.600046 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:47.600175 (+   129us) service_pool.cc:224] Handling call
0808 01:49:47.614187 (+ 14012us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:49:47.614897 (+   710us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:47.614998 (+   101us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:49:47.617200 (+  2202us) log.cc:844] Serialized 506 byte log entry
0808 01:49:47.764040 (+146840us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:49:48.664782 (+900742us) log.cc:844] Serialized 14 byte log entry
0808 01:49:48.665071 (+   289us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:48.665154 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:49:48.665202 (+    48us) raft_consensus.cc:1685] finished
0808 01:49:48.665218 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:48.665390 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3554,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":117,"thread_start_us":69237,"threads_started":1,"wal-append.queue_time_us":115546}
W20260808 01:49:48.680261 11178 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
W20260808 01:49:48.690819 11165 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.034s	sys 0.019s
I20260808 01:49:48.658527 11160 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:48.740087 11165 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.034s	sys 0.020s
W20260808 01:49:48.765584  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 260182us
I20260808 01:49:48.800668 11150 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:48.803944 11175 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.311s	user 0.004s	sys 0.001s
I20260808 01:49:48.812849 11176 raft_consensus.cc:2243] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 LEARNER]: Raft consensus shutting down.
I20260808 01:49:48.813905 11176 raft_consensus.cc:2272] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 LEARNER]: Raft consensus is shut down!
I20260808 01:49:48.817319 11160 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:48.841995  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 89775.6us
I20260808 01:49:48.846794  9480 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260808 01:49:48.843636  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 70892.3us
I20260808 01:49:48.861968  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1277) took 1087 ms. Trace:
I20260808 01:49:48.862068  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:48.862545  9632 rpcz_store.cc:276] 0808 01:49:47.774949 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:47.775649 (+   700us) service_pool.cc:224] Handling call
0808 01:49:47.775947 (+   298us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:49:47.779618 (+  3671us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:47.779645 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:49:47.791729 (+ 12084us) log.cc:844] Serialized 52 byte log entry
0808 01:49:48.180055 (+388326us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:49:48.233144 (+ 53089us) log.cc:844] Serialized 14 byte log entry
0808 01:49:48.286773 (+ 53629us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:48.286857 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:49:48.683005 (+396148us) spinlock_profiling.cc:243] Waited 33.6 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:49:48.849464 (+166459us) raft_consensus.cc:1685] finished
0808 01:49:48.861740 (+ 12276us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:48.861887 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58711,"prepare.queue_time_us":59162,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":57338,"raft.run_cpu_time_us":324,"raft.run_wall_time_us":17000,"spinlock_wait_cycles":77356928,"thread_start_us":386463,"threads_started":1,"wal-append.queue_time_us":536746}
W20260808 01:49:48.889675  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46179.6us
I20260808 01:49:48.867569 10365 raft_consensus.cc:1217] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260808 01:49:48.868820  9571 tablet_copy_service.cc:140] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Received BeginTabletCopySession request for tablet 03b17f21031f4ff4828c85b7332fcc2d from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:60552)
I20260808 01:49:48.929251 11147 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:49:48.932518  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:48.914490  9633 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:49:48.934962  9571 tablet_copy_service.cc:161] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Beginning new tablet copy session on tablet 03b17f21031f4ff4828c85b7332fcc2d from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:60552: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-03b17f21031f4ff4828c85b7332fcc2d
I20260808 01:49:48.936643  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:48.946973  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 89568.1us
W20260808 01:49:48.956604  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 44724.7us
W20260808 01:49:48.940912 11190 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260808 01:49:48.932567 11095 tablet_bootstrap.cc:492] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
W20260808 01:49:48.964303  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:49:48.978585  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 84825.9us
W20260808 01:49:49.003399  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 58413us
I20260808 01:49:49.008415 11176 ts_tablet_manager.cc:1916] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:49.010592 11097 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:49:49.017856  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 75397.1us
W20260808 01:49:49.020015  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 49684.9us
I20260808 01:49:49.040419  9571 tablet_copy_source_session.cc:215] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:49.035305 11097 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:49.045128 11097 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:49.049084 11168 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03b17f21031f4ff4828c85b7332fcc2d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 01:49:49.052908  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 82265.5us
W20260808 01:49:49.053953  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 74183.7us
I20260808 01:49:49.077761 11209 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:49.078433 11209 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:49.065812 11149 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260808 01:49:49.081524 11209 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:49.091193  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:49.106014  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:49.118597  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:49.127274  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 57754.1us
I20260808 01:49:49.130370  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:49.132743 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:49:49.137286  9600 consensus_peers.cc:597] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:49.180771  9373 catalog_manager.cc:5697] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 1 to 2, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 2 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:49.188354  9372 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50068 (request call id 223) took 2296 ms. Trace:
I20260808 01:49:49.191533  9377 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50064 (request call id 220) took 2243 ms. Trace:
I20260808 01:49:49.192085  9377 rpcz_store.cc:276] 0808 01:49:46.948419 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:46.948559 (+   140us) service_pool.cc:224] Handling call
0808 01:49:49.191464 (+2242905us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36307,"reported_tablets":3}
I20260808 01:49:49.188834  9372 rpcz_store.cc:276] 0808 01:49:46.886499 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:46.886619 (+   120us) service_pool.cc:224] Handling call
0808 01:49:49.182352 (+2295733us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:49:47.873045 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:49:47.897163 (+ 24118us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:49:47.897207 (+    44us) write_op.cc:435] Acquired schema lock
0808 01:49:47.897232 (+    25us) tablet.cc:662] Decoding operations
0808 01:49:47.899606 (+  2374us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:49:47.899680 (+    74us) write_op.cc:641] Partition lock acquired for write op
0808 01:49:47.899705 (+    25us) tablet.cc:685] Acquiring locks for 2 operations
0808 01:49:47.900149 (+   444us) tablet.cc:701] Row locks acquired
0808 01:49:47.900192 (+    43us) write_op.cc:260] PREPARE: finished
0808 01:49:47.900399 (+   207us) write_op.cc:270] Start()
0808 01:49:47.900514 (+   115us) write_op.cc:276] Timestamp: P: 1786153787900352 usec, L: 0
0808 01:49:47.900534 (+    20us) op_driver.cc:348] REPLICATION: starting
0808 01:49:47.959431 (+ 58897us) log.cc:844] Serialized 987 byte log entry
0808 01:49:48.609655 (+650224us) op_driver.cc:464] REPLICATION: finished
0808 01:49:48.890264 (+280609us) write_op.cc:301] APPLY: starting
0808 01:49:48.890388 (+   124us) tablet.cc:1367] starting BulkCheckPresence
0808 01:49:48.890595 (+   207us) tablet.cc:1370] finished BulkCheckPresence
0808 01:49:48.891273 (+   678us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:49:48.892201 (+   928us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:49:48.892456 (+   255us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0808 01:49:48.892527 (+    71us) write_op.cc:312] APPLY: finished
0808 01:49:48.892827 (+   300us) log.cc:844] Serialized 29 byte log entry
0808 01:49:48.894364 (+  1537us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:48.894516 (+   152us) write_op.cc:454] Released schema lock
0808 01:49:48.894573 (+    57us) write_op.cc:341] FINISH: Updating metrics
0808 01:49:48.895355 (+   782us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:48.895399 (+    44us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":24486,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":247829,"apply.run_cpu_time_us":3041,"apply.run_wall_time_us":4996,"num_ops":2,"prepare.queue_time_us":163608,"prepare.run_cpu_time_us":5041,"prepare.run_wall_time_us":148625,"replication_time_us":663031,"thread_start_us":306017,"threads_started":2}]]}
I20260808 01:49:49.182422 11104 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:49.214643  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:49.214931 10803 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:49.227356 10803 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:49.200286  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:49.233250 11162 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:49.251289  9552 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:40060
I20260808 01:49:49.268904  9552 raft_consensus.cc:606] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:49.219080 11104 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:49.313484 11208 raft_consensus.cc:1064] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:49.325996  9552 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "439040d168824496acc076b1c280bde3"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:40060
I20260808 01:49:49.326825  9552 raft_consensus.cc:606] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:49.303373  9521 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:49.343719  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:49.346330  9477 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:38182
I20260808 01:49:49.346262 11160 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:49.347259  9477 raft_consensus.cc:606] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:49:49.355261  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 58988.4us
I20260808 01:49:49.356052  9634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
W20260808 01:49:49.372684 11152 raft_consensus.cc:1068] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Unable to promote non-voter 7201605bb2f54c949a4ebe6e9b3b4b2a: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }.
  Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:49.356894 10803 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:49.373791  9634 raft_consensus.cc:606] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:49.364928 11104 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:49.366427  9521 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:49.396715  9560 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:40060
I20260808 01:49:49.397490  9560 raft_consensus.cc:606] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:49.399761  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260808 01:49:49.400858  9601 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:49.401820  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:49:49.402786  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:49:49.406507  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:49.411116 11183 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:49.415560  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:49:49.373862 11146 raft_consensus.cc:1068] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Unable to promote non-voter 7201605bb2f54c949a4ebe6e9b3b4b2a: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }.
  Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:49.399639 11168 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:56276 (request call id 204) took 1697 ms. Trace:
I20260808 01:49:49.431427 11151 raft_consensus.cc:993] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: : Instructing follower 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:49:49.431977 11151 raft_consensus.cc:1081] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Signalling peer 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:49:49.439168 11200 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:49.439635  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:49.452706 11195 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:49.483666  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:49.491190  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:49.501024 11189 raft_consensus.cc:993] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 6e7856bed1be43acaa821290d506ee97 to start an election
I20260808 01:49:49.528523 11172 raft_consensus.cc:1081] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Signalling peer 6e7856bed1be43acaa821290d506ee97 to start an election
I20260808 01:49:49.548390 11017 raft_consensus.cc:993] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: : Instructing follower 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:49:49.549041 11017 raft_consensus.cc:1081] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Signalling peer 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:49:49.539919  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:49.577133 11189 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:49.495944 11208 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:49.581154  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:49.554618 11176 ts_tablet_manager.cc:1929] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.3
I20260808 01:49:49.605468  9635 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:42824
I20260808 01:49:49.606341  9635 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:49.606927  9635 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:49.637557  9476 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:50140
I20260808 01:49:49.638439  9476 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:49.638845  9476 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:49.587173 10925 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 01:49:49.641852  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 50565.4us
I20260808 01:49:49.604611 11176 log.cc:1199] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-3-root/wals/c094b8e92d5e449aa2641c359d46d622
I20260808 01:49:49.429936 11168 rpcz_store.cc:276] 0808 01:49:47.701829 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:47.703327 (+  1498us) service_pool.cc:224] Handling call
0808 01:49:47.830535 (+127208us) ts_tablet_manager.cc:934] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
0808 01:49:49.047728 (+1217193us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:49:49.152821 (+105093us) tablet_metadata.cc:665] Metadata flushed
0808 01:49:49.152951 (+   130us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:49:49.361518 (+208567us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:49:49.361996 (+   478us) tablet_replica.cc:206] Creating consensus instance
0808 01:49:49.396375 (+ 34379us) ts_tablet_manager.cc:989] Replica registered.
0808 01:49:49.399373 (+  2998us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":31728,"tablet-copy.queue_time_us":91208,"thread_start_us":697395,"threads_started":2}
I20260808 01:49:49.657419 11168 tablet_copy_client.cc:806] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:49.658121 11168 tablet_copy_client.cc:670] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:49:49.664606  9635 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:49.676044 11151 raft_consensus.cc:993] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: : Instructing follower 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:49:49.694231  9635 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:49.718257  9476 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:49.730412 11176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49490 (request call id 0) took 1966 ms. Trace:
I20260808 01:49:49.730863 11176 rpcz_store.cc:276] 0808 01:49:47.774005 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:47.797625 (+ 23620us) service_pool.cc:224] Handling call
0808 01:49:48.563278 (+765653us) ts_tablet_manager.cc:1151] Deleting tablet c094b8e92d5e449aa2641c359d46d622
0808 01:49:48.602036 (+ 38758us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0808 01:49:49.233108 (+631072us) tablet_metadata.cc:665] Metadata flushed
0808 01:49:49.554462 (+321354us) tablet_metadata.cc:665] Metadata flushed
0808 01:49:49.730320 (+175858us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":93572,"tablet-delete.queue_time_us":704200,"thread_start_us":680363,"threads_started":1}
I20260808 01:49:49.732681  9361 catalog_manager.cc:5028] TS 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): tablet c094b8e92d5e449aa2641c359d46d622 (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:49.716643 11119 raft_consensus.cc:1081] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Signalling peer 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:49:49.758234  9476 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:49.785526 11168 tablet_copy_client.cc:538] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260808 01:49:49.779002 11225 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.052s	user 0.001s	sys 0.001s
I20260808 01:49:49.798010  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:49.798874  9632 raft_consensus.cc:3055] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:49.799351  9632 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:49.803157  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:49.804221  9555 raft_consensus.cc:3055] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:49.804659  9555 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:49.809934  9632 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:49.818217  9632 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:49.815279  9555 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:49.820144  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:49:49.849712 10365 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 LEARNER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:49.858222  9635 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:49.858882  9629 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:42824
I20260808 01:49:49.859537  9629 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:49.859890  9629 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:49.863430  9633 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "439040d168824496acc076b1c280bde3"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:42824
I20260808 01:49:49.864174  9633 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:49.864636  9633 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:49:49.901851  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 50257.5us
I20260808 01:49:49.903931 11095 tablet_bootstrap.cc:492] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:49.905193 11095 tablet_bootstrap.cc:492] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:49.906113 11095 ts_tablet_manager.cc:1403] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.974s	user 0.052s	sys 0.010s
I20260808 01:49:49.907980  9550 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260808 01:49:49.908740  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 119739us
I20260808 01:49:49.915252  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:49.915890  9476 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:49.921000 11210 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260808 01:49:49.940479  9555 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:49.942142 11095 raft_consensus.cc:359] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:49.943109 11095 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:49.943799 11095 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:49.948028  9632 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:49:49.951519  9629 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:49.953181  9555 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:49:49.965447  9630 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: []
I20260808 01:49:49.969154 11168 tablet_bootstrap.cc:492] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
W20260808 01:49:49.970657  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 149755us
I20260808 01:49:49.978121  9629 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:50.003015  9476 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:49:50.010819  9599 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:50.012786  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:50.016789  9477 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:50.015779 10833 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260808 01:49:50.011718  9633 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:50.046828  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:50.047761  9552 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:50.051605 11232 ts_tablet_manager.cc:933] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:50.051903 11095 ts_tablet_manager.cc:1434] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.145s	user 0.007s	sys 0.000s
W20260808 01:49:50.045871  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 54803.1us
I20260808 01:49:50.056470  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:50.057152  9557 raft_consensus.cc:3055] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:49:50.057469  9557 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:50.053210 11188 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:50.027812 11208 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:50.068101  9552 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:49:50.069362  9633 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:50.058249  9557 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:50.073066  9557 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:49:50.079679  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 69244.7us
I20260808 01:49:50.087234  9557 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:49:50.089834 11188 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:50.095817  9599 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:50.106206 11170 raft_consensus.cc:993] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: : Instructing follower 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:49:50.110211 11099 raft_consensus.cc:1081] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Signalling peer 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:49:50.121500 11238 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:49:50.122176  9550 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:53216
I20260808 01:49:50.122840  9550 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:50.123224  9550 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:49:50.124508  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 44081.1us
I20260808 01:49:50.126579 11242 raft_consensus.cc:2955] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 2 to 4, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:50.129112  9636 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:50.134205  9550 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:50.144378  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:50.144562  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:50.145097  9476 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:50.145211  9557 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:50.145572  9477 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260808 01:49:50.145638  9557 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:50.146852  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:50.122287 11238 raft_consensus.cc:697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:50.091279 11208 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:50.156821  9479 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:50.157019  9557 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:50.162856  9477 raft_consensus.cc:2955] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 2 to 4, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:50.177016  9550 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:50.181988  9373 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55058 (request call id 106) took 1845 ms. Trace:
I20260808 01:49:50.182485  9373 rpcz_store.cc:276] 0808 01:49:48.337073 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:48.337239 (+   166us) service_pool.cc:224] Handling call
0808 01:49:50.181919 (+1844680us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:49:49.437668 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:49:49.474915 (+ 37247us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:49:49.474952 (+    37us) write_op.cc:435] Acquired schema lock
0808 01:49:49.474972 (+    20us) tablet.cc:662] Decoding operations
0808 01:49:49.475333 (+   361us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:49:49.475386 (+    53us) write_op.cc:641] Partition lock acquired for write op
0808 01:49:49.475426 (+    40us) tablet.cc:685] Acquiring locks for 1 operations
0808 01:49:49.475621 (+   195us) tablet.cc:701] Row locks acquired
0808 01:49:49.475640 (+    19us) write_op.cc:260] PREPARE: finished
0808 01:49:49.475793 (+   153us) write_op.cc:270] Start()
0808 01:49:49.475905 (+   112us) write_op.cc:276] Timestamp: P: 1786153789475751 usec, L: 0
0808 01:49:49.475934 (+    29us) op_driver.cc:348] REPLICATION: starting
0808 01:49:49.476674 (+   740us) log.cc:844] Serialized 536 byte log entry
0808 01:49:49.745841 (+269167us) op_driver.cc:464] REPLICATION: finished
0808 01:49:49.928688 (+182847us) write_op.cc:301] APPLY: starting
0808 01:49:49.928786 (+    98us) tablet.cc:1367] starting BulkCheckPresence
0808 01:49:49.928918 (+   132us) tablet.cc:1370] finished BulkCheckPresence
0808 01:49:49.929078 (+   160us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:49:49.929386 (+   308us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:49:49.929587 (+   201us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 01:49:49.929661 (+    74us) write_op.cc:312] APPLY: finished
0808 01:49:49.929895 (+   234us) log.cc:844] Serialized 23 byte log entry
0808 01:49:49.930087 (+   192us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:49.930176 (+    89us) write_op.cc:454] Released schema lock
0808 01:49:49.930213 (+    37us) write_op.cc:341] FINISH: Updating metrics
0808 01:49:49.930589 (+   376us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:49.930620 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":90482,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":180353,"apply.run_cpu_time_us":1868,"apply.run_wall_time_us":1868,"num_ops":1,"prepare.queue_time_us":216947,"prepare.run_cpu_time_us":2898,"prepare.run_wall_time_us":51926,"replication_time_us":258118,"spinlock_wait_cycles":2432,"thread_start_us":191231,"threads_started":2}]]}
I20260808 01:49:50.187613  9476 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:49:50.189550 11238 consensus_queue.cc:237] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:50.196228  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:50.196960  9480 raft_consensus.cc:3055] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:50.197264  9480 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:50.186265  9371 catalog_manager.cc:5697] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 1 to 2, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 2 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:50.192703 11172 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260808 01:49:50.220494 11208 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:50.227118 10364 raft_consensus.cc:1217] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260808 01:49:50.228847 11188 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:50.232121  9571 tablet_copy_service.cc:342] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-fca874793205463dbb922993f57bf7f5 received from {username='slave'} at 127.0.0.1:60552
I20260808 01:49:50.232792  9571 tablet_copy_service.cc:434] P 52ccf4dcca1843f7bb29e9172e2d4ab4: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-fca874793205463dbb922993f57bf7f5 on tablet fca874793205463dbb922993f57bf7f5 with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:50.215953  9480 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:50.235116  9480 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:50.242693  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:50.243395  9629 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:50.257885  9557 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:50.281534  9479 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:49:50.282624  9557 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
W20260808 01:49:50.283972 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46166.1us
I20260808 01:49:50.288568  9629 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:50.290259  9599 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:50.292361 11189 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:50.293020 11189 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:50.294215 11189 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:50.266075 11232 tablet_copy_client.cc:323] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.195:34565
I20260808 01:49:50.314487  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:50.322849 11105 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election won for term 2
W20260808 01:49:50.342296 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 53114.8us
I20260808 01:49:50.382890  9648 tablet_copy_service.cc:140] P 259bcb3f0d254e8b9effac3277e654e0: Received BeginTabletCopySession request for tablet c094b8e92d5e449aa2641c359d46d622 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:45658)
I20260808 01:49:50.383571  9648 tablet_copy_service.cc:161] P 259bcb3f0d254e8b9effac3277e654e0: Beginning new tablet copy session on tablet c094b8e92d5e449aa2641c359d46d622 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:45658: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-c094b8e92d5e449aa2641c359d46d622
W20260808 01:49:50.388245  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 43515.3us
I20260808 01:49:50.392084  9480 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:50.404417 11105 raft_consensus.cc:697] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:50.412571 11105 consensus_queue.cc:237] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:50.484745  9474 raft_consensus.cc:2955] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:50.504808 10803 raft_consensus.cc:2955] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:50.564281 10364 raft_consensus.cc:2955] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:50.593566  9648 tablet_copy_source_session.cc:215] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:49:50.623683 11105 raft_consensus.cc:1064] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:50.637277  9557 raft_consensus.cc:2955] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 2 to 4, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:50.657860 11168 tablet_bootstrap.cc:492] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:50.663724 11168 tablet_bootstrap.cc:492] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:50.664141 11232 ts_tablet_manager.cc:1916] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260808 01:49:50.665220 11168 ts_tablet_manager.cc:1403] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.696s	user 0.043s	sys 0.020s
I20260808 01:49:50.671749  9560 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:50.673504  9552 raft_consensus.cc:2955] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:50.689086  9560 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:50.700335  9371 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50084 (request call id 216) took 3060 ms. Trace:
I20260808 01:49:50.705348 11105 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:50.707820  9372 catalog_manager.cc:5697] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 reported cstate change: config changed from index -1 to 3, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 3 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:49:50.716104  9637 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:50.710187  9372 catalog_manager.cc:5697] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 1 to 2, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 2 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:49:50.723683 11205 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:50.700819  9371 rpcz_store.cc:276] 0808 01:49:47.639609 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:47.639825 (+   216us) service_pool.cc:224] Handling call
0808 01:49:50.700265 (+3060440us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:49:50.417082 (+     0us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b540005c228. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4ad89c21 00007f5d4ad8888c 00007f5d483fd4bc 00007f5d483e8a86 00007f5d483e8738 00007f5d483ed3d1 00007f5d483ed389 00007f5d483ed319 00007f5d483ed2e1 00007f5d483ec50d 00000000003aa554
0808 01:49:50.417183 (+   101us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:49:50.417740 (+   557us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:49:50.417778 (+    38us) write_op.cc:435] Acquired schema lock
0808 01:49:50.417797 (+    19us) tablet.cc:662] Decoding operations
0808 01:49:50.418168 (+   371us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:49:50.418225 (+    57us) write_op.cc:641] Partition lock acquired for write op
0808 01:49:50.418254 (+    29us) tablet.cc:685] Acquiring locks for 1 operations
0808 01:49:50.418479 (+   225us) tablet.cc:701] Row locks acquired
0808 01:49:50.418498 (+    19us) write_op.cc:260] PREPARE: finished
0808 01:49:50.418643 (+   145us) write_op.cc:270] Start()
0808 01:49:50.461548 (+ 42905us) write_op.cc:276] Timestamp: P: 1786153790418602 usec, L: 0
0808 01:49:50.461588 (+    40us) op_driver.cc:348] REPLICATION: starting
0808 01:49:50.462319 (+   731us) log.cc:844] Serialized 541 byte log entry
0808 01:49:50.542109 (+ 79790us) op_driver.cc:464] REPLICATION: finished
0808 01:49:50.682848 (+140739us) write_op.cc:301] APPLY: starting
0808 01:49:50.682986 (+   138us) tablet.cc:1367] starting BulkCheckPresence
0808 01:49:50.683098 (+   112us) tablet.cc:1370] finished BulkCheckPresence
0808 01:49:50.683120 (+    22us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:49:50.683544 (+   424us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:49:50.683827 (+   283us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 01:49:50.683924 (+    97us) write_op.cc:312] APPLY: finished
0808 01:49:50.684253 (+   329us) log.cc:844] Serialized 23 byte log entry
0808 01:49:50.684532 (+   279us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:50.684679 (+   147us) write_op.cc:454] Released schema lock
0808 01:49:50.684742 (+    63us) write_op.cc:341] FINISH: Updating metrics
0808 01:49:50.685453 (+   711us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:50.685503 (+    50us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":246867,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":140335,"apply.run_cpu_time_us":2462,"apply.run_wall_time_us":2462,"num_ops":1,"prepare.queue_time_us":140192,"prepare.run_cpu_time_us":17451,"prepare.run_wall_time_us":117147,"replication_time_us":80334,"spinlock_wait_cycles":25199744,"thread_start_us":171399,"threads_started":2}]]}
I20260808 01:49:50.724303  9557 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:50.748026 11242 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:50.760258  9474 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:49:50.773062  9474 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:50.765974 11146 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260808 01:49:50.780464 11121 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:50.783170 11132 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:50.786507  9560 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:49:50.783936 11104 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260808 01:49:50.816936  9372 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50064 (request call id 221) took 1600 ms. Trace:
I20260808 01:49:50.822638  9480 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:50.825415 11017 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:50.830435 10364 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEARNER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:49:50.817564  9372 rpcz_store.cc:276] 0808 01:49:49.216811 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:49.217111 (+   300us) service_pool.cc:224] Handling call
0808 01:49:50.816856 (+1599745us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:49:50.722668 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:49:50.723916 (+  1248us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:49:50.723984 (+    68us) write_op.cc:435] Acquired schema lock
0808 01:49:50.723998 (+    14us) tablet.cc:662] Decoding operations
0808 01:49:50.724994 (+   996us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:49:50.725073 (+    79us) write_op.cc:641] Partition lock acquired for write op
0808 01:49:50.725112 (+    39us) tablet.cc:685] Acquiring locks for 2 operations
0808 01:49:50.725701 (+   589us) tablet.cc:701] Row locks acquired
0808 01:49:50.725757 (+    56us) write_op.cc:260] PREPARE: finished
0808 01:49:50.726091 (+   334us) write_op.cc:270] Start()
0808 01:49:50.726344 (+   253us) write_op.cc:276] Timestamp: P: 1786153790725972 usec, L: 0
0808 01:49:50.726408 (+    64us) op_driver.cc:348] REPLICATION: starting
0808 01:49:50.738812 (+ 12404us) log.cc:844] Serialized 987 byte log entry
0808 01:49:50.751356 (+ 12544us) op_driver.cc:464] REPLICATION: finished
0808 01:49:50.763758 (+ 12402us) write_op.cc:301] APPLY: starting
0808 01:49:50.766056 (+  2298us) tablet.cc:1367] starting BulkCheckPresence
0808 01:49:50.778816 (+ 12760us) tablet.cc:1370] finished BulkCheckPresence
0808 01:49:50.780637 (+  1821us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:49:50.781032 (+   395us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:49:50.781193 (+   161us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0808 01:49:50.781259 (+    66us) write_op.cc:312] APPLY: finished
0808 01:49:50.781450 (+   191us) log.cc:844] Serialized 29 byte log entry
0808 01:49:50.781619 (+   169us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:50.781778 (+   159us) write_op.cc:454] Released schema lock
0808 01:49:50.781824 (+    46us) write_op.cc:341] FINISH: Updating metrics
0808 01:49:50.782355 (+   531us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:50.782404 (+    49us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":5694,"apply.run_cpu_time_us":2108,"apply.run_wall_time_us":19888,"num_ops":2,"prepare.queue_time_us":977,"prepare.run_cpu_time_us":5885,"prepare.run_wall_time_us":16680,"replication_time_us":24682}]]}
W20260808 01:49:50.845082  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 55087.6us
I20260808 01:49:50.848690  9480 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260808 01:49:50.802136 11168 raft_consensus.cc:359] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:50.859010  9377 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50068 (request call id 224) took 1446 ms. Trace:
I20260808 01:49:50.859752  9377 rpcz_store.cc:276] 0808 01:49:49.412680 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:49.412817 (+   137us) service_pool.cc:224] Handling call
0808 01:49:50.858919 (+1446102us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32397,"reported_tablets":5}
W20260808 01:49:50.861586  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 55898.4us
I20260808 01:49:50.865285 11168 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:49:50.867650 11165 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:50.850430  9371 catalog_manager.cc:5697] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 2 to 3, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:50.891376  9638 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20260808 01:49:50.898545  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 52537.9us
I20260808 01:49:50.942763 11160 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:50.897570 11168 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:50.963804  9552 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53202
I20260808 01:49:50.976274  9552 raft_consensus.cc:606] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Received request to transfer leadership
I20260808 01:49:50.917237 10833 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
W20260808 01:49:51.017391  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 101863us
I20260808 01:49:51.022926 11151 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:51.024902  9472 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:51.042066 11242 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 01:49:51.109189  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 75701.5us
W20260808 01:49:51.137835 11267 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.089s	user 0.000s	sys 0.002s
W20260808 01:49:51.155081 11266 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
W20260808 01:49:51.157742 11264 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260808 01:49:51.168141 11168 ts_tablet_manager.cc:1434] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.502s	user 0.013s	sys 0.005s
W20260808 01:49:51.168691 11168 ts_tablet_manager.cc:1467] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Tablet startup took 502ms
W20260808 01:49:51.208500  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 130365us
W20260808 01:49:51.169045 11168 ts_tablet_manager.cc:1469] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Trace:
0808 01:49:47.701829 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:47.703327 (+  1498us) service_pool.cc:224] Handling call
0808 01:49:47.830535 (+127208us) ts_tablet_manager.cc:934] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
0808 01:49:49.047728 (+1217193us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:49:49.152821 (+105093us) tablet_metadata.cc:665] Metadata flushed
0808 01:49:49.152951 (+   130us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:49:49.361518 (+208567us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:49:49.361996 (+   478us) tablet_replica.cc:206] Creating consensus instance
0808 01:49:49.396375 (+ 34379us) ts_tablet_manager.cc:989] Replica registered.
0808 01:49:49.399373 (+  2998us) inbound_call.cc:177] Queueing success response
0808 01:49:49.968748 (+569375us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0808 01:49:50.666176 (+697428us) ts_tablet_manager.cc:1435] Starting tablet replica
0808 01:49:50.731104 (+ 64928us) tablet_replica.cc:261] Starting instrumentation
0808 01:49:50.732355 (+  1251us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":31728,"mutex_wait_us":1718,"raft.queue_time_us":6839,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":15661,"tablet-copy.queue_time_us":91208,"tablet-download-pool.queue_time_us":2768,"tablet-download-pool.run_cpu_time_us":1742,"tablet-download-pool.run_wall_time_us":112727,"thread_start_us":697395,"threads_started":2}
I20260808 01:49:51.225589 10369 raft_consensus.cc:1217] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3]   Dedup: 3.3->[]
I20260808 01:49:51.241379  9571 tablet_copy_service.cc:342] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-03b17f21031f4ff4828c85b7332fcc2d received from {username='slave'} at 127.0.0.1:60552
I20260808 01:49:51.241977  9571 tablet_copy_service.cc:434] P 52ccf4dcca1843f7bb29e9172e2d4ab4: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-03b17f21031f4ff4828c85b7332fcc2d on tablet 03b17f21031f4ff4828c85b7332fcc2d with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:51.249060 11232 ts_tablet_manager.cc:1929] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 3.3
I20260808 01:49:51.296944  9629 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:51.187085  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
W20260808 01:49:51.350529  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48862.9us
I20260808 01:49:51.418290  9550 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53202
I20260808 01:49:51.419029  9550 raft_consensus.cc:606] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Received request to transfer leadership
W20260808 01:49:51.421491  9360 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 03b17f21031f4ff4828c85b7332fcc2d already in progress [suppressed 4 similar messages]
I20260808 01:49:51.409632 11232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c094b8e92d5e449aa2641c359d46d622. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 01:49:51.480154  9360 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 59482.9us
W20260808 01:49:51.483661 11265 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.429s	user 0.007s	sys 0.000s
I20260808 01:49:51.447453 11017 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
W20260808 01:49:51.489826  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 98945.8us
I20260808 01:49:51.500281  9555 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:40060
I20260808 01:49:51.501266  9555 raft_consensus.cc:606] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:49:51.628470  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 222600us
I20260808 01:49:51.711568 11232 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:56266 (request call id 338) took 1904 ms. Trace:
I20260808 01:49:51.712265 11232 rpcz_store.cc:276] 0808 01:49:49.807262 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:49.808867 (+  1605us) service_pool.cc:224] Handling call
0808 01:49:50.051974 (+243107us) ts_tablet_manager.cc:934] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
0808 01:49:50.643086 (+591112us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:49:50.985214 (+342128us) tablet_metadata.cc:665] Metadata flushed
0808 01:49:51.248835 (+263621us) tablet_metadata.cc:665] Metadata flushed
0808 01:49:51.345412 (+ 96577us) tablet_copy_client.cc:463] Replaced tombstoned tablet metadata.
0808 01:49:51.710316 (+364904us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:49:51.710802 (+   486us) tablet_replica.cc:206] Creating consensus instance
0808 01:49:51.711367 (+   565us) ts_tablet_manager.cc:989] Replica registered.
0808 01:49:51.711467 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":91112,"tablet-copy.queue_time_us":180366,"thread_start_us":185728,"threads_started":2}
I20260808 01:49:51.714512 11232 tablet_copy_client.cc:806] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:49:51.716468 11232 tablet_copy_client.cc:670] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
W20260808 01:49:51.717350 11284 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.205s	user 0.004s	sys 0.000s
W20260808 01:49:51.760636  7831 auto_leader_rebalancer.cc:456] leader transfer for tablet 03b17f21031f4ff4828c85b7332fcc2d from 52ccf4dcca1843f7bb29e9172e2d4ab4 to 259bcb3f0d254e8b9effac3277e654e0 failed: Remote error: Service unavailable: leadership transfer for 03b17f21031f4ff4828c85b7332fcc2d already in progress
W20260808 01:49:51.778448  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 141296us
I20260808 01:49:51.855150  9559 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:53202
I20260808 01:49:51.856280  9559 raft_consensus.cc:606] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Received request to transfer leadership
W20260808 01:49:51.957930  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 53675.6us
W20260808 01:49:51.939160  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 273593us
W20260808 01:49:52.014006  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 136365us
W20260808 01:49:51.953301 11282 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.482s	user 0.004s	sys 0.000s
I20260808 01:49:51.959518 11183 raft_consensus.cc:2955] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:51.996750 10364 raft_consensus.cc:2955] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:51.964058  9373 catalog_manager.cc:5697] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index 2 to 4, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260808 01:49:52.033991 11266 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.080s	user 0.001s	sys 0.001s
I20260808 01:49:52.037132  9373 catalog_manager.cc:5697] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 1 to 2, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 2 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:52.044803 11308 raft_consensus.cc:1064] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
W20260808 01:49:52.051414 11308 raft_consensus.cc:1068] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Unable to promote non-voter 7201605bb2f54c949a4ebe6e9b3b4b2a: Service unavailable: leader transfer in progress
I20260808 01:49:52.052114  9373 catalog_manager.cc:5697] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 1 to 2, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 2 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:52.063153  9373 catalog_manager.cc:5697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 2 to 3, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 3 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
W20260808 01:49:52.066984 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 180121us
I20260808 01:49:52.100661 11232 tablet_copy_client.cc:538] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:49:52.116708 11232 tablet_bootstrap.cc:492] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:49:52.117319 10965 raft_consensus.cc:993] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4: : Instructing follower 6e7856bed1be43acaa821290d506ee97 to start an election
I20260808 01:49:52.122596 11105 raft_consensus.cc:1081] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Signalling peer 6e7856bed1be43acaa821290d506ee97 to start an election
I20260808 01:49:52.137521 11305 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:52.138298 11305 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:52.154439 11305 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:52.166469  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 172155us
I20260808 01:49:52.163914  9552 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:40060
I20260808 01:49:52.183894  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:52.170117  9552 raft_consensus.cc:606] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:52.187311  9476 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:50124
I20260808 01:49:52.188148  9476 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:52.188597  9476 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:49:52.188602  9629 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:49:52.191380  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:52.189658  9629 raft_consensus.cc:606] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:52.197578  9472 raft_consensus.cc:2955] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:52.221895  9634 raft_consensus.cc:2955] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:52.237882  9476 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:52.244652  9559 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:40060
I20260808 01:49:52.244989  9476 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:52.245432  9559 raft_consensus.cc:606] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:52.247102  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260808 01:49:52.257303 11105 raft_consensus.cc:1064] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:52.257769  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:49:52.261914  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
W20260808 01:49:52.258064 11105 raft_consensus.cc:1068] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Unable to promote non-voter 7201605bb2f54c949a4ebe6e9b3b4b2a: Service unavailable: leader transfer in progress
I20260808 01:49:52.281212 11017 raft_consensus.cc:993] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: : Instructing follower 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:49:52.281955 11017 raft_consensus.cc:1081] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Signalling peer 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:49:52.288416 11200 raft_consensus.cc:993] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 6e7856bed1be43acaa821290d506ee97 to start an election
I20260808 01:49:52.289149 11200 raft_consensus.cc:1081] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Signalling peer 6e7856bed1be43acaa821290d506ee97 to start an election
I20260808 01:49:52.291971  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:52.293186 11121 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:52.295047  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:52.295740  9472 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:50140
I20260808 01:49:52.296280  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:52.296418  9472 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:52.296895  9633 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:49:52.296958  9472 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:52.295780  9559 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:49:52.298591  9632 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:42824
I20260808 01:49:52.298998  9559 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:52.299445  9632 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:52.300158  9559 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:52.300173  9632 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:49:52.319679  9559 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:49:52.335808  9633 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
I20260808 01:49:52.337682  9472 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:52.341228  9559 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
I20260808 01:49:52.342446  9472 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:52.343500  9632 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:52.348482  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:52.349355  9472 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:49:52.348949  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:52.351610  9559 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:49:52.359903  9472 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:49:52.361708  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:52.365334  9371 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55058 (request call id 109) took 1020 ms. Trace:
I20260808 01:49:52.368530 10925 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:49:52.369210 10925 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:52.370165 10925 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:52.373009  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:52.374441  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:52.375180  9633 raft_consensus.cc:3055] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:52.375608  9633 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:52.376611  9633 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:52.379334 11310 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:49:52.380064 11310 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 4 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:52.381069  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:52.381753  9552 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:52.388299  9633 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:52.381129 11310 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:49:52.398845  9637 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:52.403208  9377 catalog_manager.cc:5697] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 1 to 2, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 2 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:52.366042  9371 rpcz_store.cc:276] 0808 01:49:51.344900 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:51.345164 (+   264us) service_pool.cc:224] Handling call
0808 01:49:52.365059 (+1019895us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5712,"reported_tablets":1}
I20260808 01:49:52.392400  9373 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50084 (request call id 217) took 1548 ms. Trace:
I20260808 01:49:52.408998  9559 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:52.409353  9633 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:49:52.410482  9632 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:52.405220  9377 catalog_manager.cc:5697] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 1 to 2, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 2 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:49:52.413892  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:52.417372  9552 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:49:52.423534  9559 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:52.432077  9559 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:49:52.447103  9555 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:52.454382  9559 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:49:52.437106 11305 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:52.480013 11238 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260808 01:49:52.476555 11305 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:52.485100 11305 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:52.406070  9373 rpcz_store.cc:276] 0808 01:49:50.844327 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:50.844679 (+   352us) service_pool.cc:224] Handling call
0808 01:49:52.392305 (+1547626us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:49:52.145071 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:49:52.145689 (+   618us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:49:52.145753 (+    64us) write_op.cc:435] Acquired schema lock
0808 01:49:52.145789 (+    36us) tablet.cc:662] Decoding operations
0808 01:49:52.146574 (+   785us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:49:52.146743 (+   169us) write_op.cc:641] Partition lock acquired for write op
0808 01:49:52.146772 (+    29us) tablet.cc:685] Acquiring locks for 4 operations
0808 01:49:52.156725 (+  9953us) tablet.cc:701] Row locks acquired
0808 01:49:52.156757 (+    32us) write_op.cc:260] PREPARE: finished
0808 01:49:52.156979 (+   222us) write_op.cc:270] Start()
0808 01:49:52.157122 (+   143us) write_op.cc:276] Timestamp: P: 1786153792156904 usec, L: 0
0808 01:49:52.157163 (+    41us) op_driver.cc:348] REPLICATION: starting
0808 01:49:52.173165 (+ 16002us) log.cc:844] Serialized 1764 byte log entry
0808 01:49:52.319705 (+146540us) op_driver.cc:464] REPLICATION: finished
0808 01:49:52.344182 (+ 24477us) write_op.cc:301] APPLY: starting
0808 01:49:52.344279 (+    97us) tablet.cc:1367] starting BulkCheckPresence
0808 01:49:52.344584 (+   305us) tablet.cc:1370] finished BulkCheckPresence
0808 01:49:52.344821 (+   237us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:49:52.345974 (+  1153us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:49:52.346262 (+   288us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0808 01:49:52.346423 (+   161us) write_op.cc:312] APPLY: finished
0808 01:49:52.347854 (+  1431us) log.cc:844] Serialized 41 byte log entry
0808 01:49:52.348733 (+   879us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:52.348835 (+   102us) write_op.cc:454] Released schema lock
0808 01:49:52.348914 (+    79us) write_op.cc:341] FINISH: Updating metrics
0808 01:49:52.349396 (+   482us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:49:52.349429 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":296716,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":24064,"apply.run_cpu_time_us":3481,"apply.run_wall_time_us":5190,"num_ops":4,"prepare.queue_time_us":70310,"prepare.run_cpu_time_us":41150,"prepare.run_wall_time_us":113719,"raft.queue_time_us":2697,"raft.run_cpu_time_us":6909,"raft.run_wall_time_us":45223,"replication_time_us":148776,"spinlock_wait_cycles":1977728,"thread_start_us":163365,"threads_started":4,"wal-append.queue_time_us":90819}]]}
I20260808 01:49:52.571184 10364 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260808 01:49:52.613011 11208 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
W20260808 01:49:52.625977  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48625.4us
I20260808 01:49:52.692041 11232 tablet_bootstrap.cc:492] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:49:52.704331 11232 tablet_bootstrap.cc:492] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:49:52.705236 11232 ts_tablet_manager.cc:1403] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.589s	user 0.054s	sys 0.104s
W20260808 01:49:52.713712  9426 auto_rebalancer.cc:912] Removing replace marker failed after 3 inline attempts; will retry on next rebalancer loop iteration: Replica 52ccf4dcca1843f7bb29e9172e2d4ab4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
W20260808 01:49:52.714324  9426 auto_rebalancer.cc:921] Could not move replica: Remote error: Service unavailable: leadership transfer for 03b17f21031f4ff4828c85b7332fcc2d already in progress
W20260808 01:49:52.714598  9426 auto_rebalancer.cc:342] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 03b17f21031f4ff4828c85b7332fcc2d already in progress
I20260808 01:49:52.731424 11017 raft_consensus.cc:993] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: : Instructing follower 6e7856bed1be43acaa821290d506ee97 to start an election
I20260808 01:49:52.732074 11017 raft_consensus.cc:1081] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Signalling peer 6e7856bed1be43acaa821290d506ee97 to start an election
I20260808 01:49:52.736770 11232 raft_consensus.cc:359] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:52.737890 11232 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: FOLLOWER
I20260808 01:49:52.738981 11232 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:52.741219 11232 ts_tablet_manager.cc:1434] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.036s	user 0.007s	sys 0.000s
I20260808 01:49:52.743548  9373 catalog_manager.cc:5697] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 3 to 4, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:52.749912 10366 raft_consensus.cc:1217] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4]   Dedup: 3.4->[]
I20260808 01:49:52.752895  9476 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:50124
I20260808 01:49:52.753539  9476 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:52.753916  9476 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:52.766659  9474 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:49:52.772532  9476 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:52.775403  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:52.776005  9555 raft_consensus.cc:3055] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:52.776298  9555 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:52.777195  9555 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:52.788523  9555 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:52.789649  9648 tablet_copy_service.cc:342] P 259bcb3f0d254e8b9effac3277e654e0: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-c094b8e92d5e449aa2641c359d46d622 received from {username='slave'} at 127.0.0.1:45658
I20260808 01:49:52.790282  9648 tablet_copy_service.cc:434] P 259bcb3f0d254e8b9effac3277e654e0: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-c094b8e92d5e449aa2641c359d46d622 on tablet c094b8e92d5e449aa2641c359d46d622 with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:52.797446 11200 raft_consensus.cc:2955] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:52.803519 10364 raft_consensus.cc:2955] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:52.804750 11189 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:52.811015  9555 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:49:52.813791 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:49:52.814527 10365 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:52.814158  9552 raft_consensus.cc:2955] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:52.825300  9373 catalog_manager.cc:5697] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 3 to 4, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New cstate: current_term: 1 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:52.826802  9476 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:52.829507 10365 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:49:52.825477  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:52.836855 11310 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:52.837512 11310 raft_consensus.cc:697] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:52.845819  9559 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:49:52.845639 11310 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:52.847913 11132 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:52.838790  9377 catalog_manager.cc:5697] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 4 to 5, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 5 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:49:52.869189  9372 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50068 (request call id 225) took 1286 ms. Trace:
I20260808 01:49:52.878857  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9bf65b9272854b479e73f2cea21e93f2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:52.889948  9371 catalog_manager.cc:5697] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 3 to 4, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 4 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:52.894496  9465 tablet_service.cc:1558] Processing DeleteTablet for tablet 9bf65b9272854b479e73f2cea21e93f2 with delete_type TABLET_DATA_TOMBSTONED (TS 6e7856bed1be43acaa821290d506ee97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50120
I20260808 01:49:52.906493  9371 catalog_manager.cc:5697] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 1 to 2, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 2 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:52.906713  9559 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:49:52.914476  9633 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:52.917812 10803 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:52.922633 11334 tablet_replica.cc:333] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:49:52.926735 11334 raft_consensus.cc:2243] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:52.899523  9372 rpcz_store.cc:276] 0808 01:49:51.582816 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:51.583005 (+   189us) service_pool.cc:224] Handling call
0808 01:49:52.868872 (+1285867us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18121,"reported_tablets":9}
I20260808 01:49:52.940876  9476 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:49:52.943284 10803 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:49:52.931612 11334 raft_consensus.cc:2272] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:52.987685 11334 ts_tablet_manager.cc:1916] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:52.990015 10366 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:49:53.002489 10803 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:49:53.067374  9557 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:53.067675 11205 raft_consensus.cc:2955] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:53.069767 11132 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:49:53.076752  9474 raft_consensus.cc:2955] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:53.088763  9633 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:49:53.095177 10366 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 LEARNER]: Advancing to term 4
I20260808 01:49:53.095623 11310 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:53.097254 11334 ts_tablet_manager.cc:1929] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260808 01:49:53.097770 11334 log.cc:1199] T 9bf65b9272854b479e73f2cea21e93f2 P 6e7856bed1be43acaa821290d506ee97: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-0-root/wals/9bf65b9272854b479e73f2cea21e93f2
I20260808 01:49:53.105614  9371 catalog_manager.cc:5697] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 1 to 2, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 2 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:49:53.112097  9362 catalog_manager.cc:5028] TS 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): tablet 9bf65b9272854b479e73f2cea21e93f2 (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:53.114303 10366 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEARNER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:49:53.119923 11278 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:49:53.130630  9637 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:53.136950 11323 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:53.146293  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet c094b8e92d5e449aa2641c359d46d622 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:53.183995  9557 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:53.186362 10364 raft_consensus.cc:2955] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:53.212046  9371 catalog_manager.cc:5697] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index 4 to 5, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New cstate: current_term: 3 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260808 01:49:53.287745  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46824.2us
I20260808 01:49:53.293576  9547 tablet_service.cc:1558] Processing DeleteTablet for tablet c094b8e92d5e449aa2641c359d46d622 with delete_type TABLET_DATA_TOMBSTONED (TS 52ccf4dcca1843f7bb29e9172e2d4ab4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:53202
I20260808 01:49:53.296478 10364 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:49:53.305249  9474 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:49:53.307941 11308 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:49:53.340649 11305 raft_consensus.cc:1064] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:49:53.342540 11160 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:49:53.364449 11345 tablet_replica.cc:333] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:49:53.377178 11305 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:53.383279 11345 raft_consensus.cc:2243] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:53.384318 11345 raft_consensus.cc:2272] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:53.404122 10365 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEARNER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260808 01:49:53.409905 11323 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:49:53.416155 11345 ts_tablet_manager.cc:1916] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:53.432307  9638 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260808 01:49:53.442281  9552 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260808 01:49:53.446368 11160 raft_consensus.cc:2955] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:53.449156 10365 raft_consensus.cc:2955] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:53.450904 11132 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:49:53.458930 11132 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:49:53.462100  9552 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:49:53.470497 10364 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:49:53.473091 11310 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:49:53.482725  9372 catalog_manager.cc:5697] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 4 to 5, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New cstate: current_term: 2 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:53.484210 11310 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:49:53.484469  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet fca874793205463dbb922993f57bf7f5 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:53.493274 11310 raft_consensus.cc:2955] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:53.496083  9474 raft_consensus.cc:2955] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:53.515727  9552 raft_consensus.cc:2955] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:53.523293 10365 raft_consensus.cc:2955] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:53.524652  9624 tablet_service.cc:1558] Processing DeleteTablet for tablet fca874793205463dbb922993f57bf7f5 with delete_type TABLET_DATA_TOMBSTONED (TS 259bcb3f0d254e8b9effac3277e654e0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42810
I20260808 01:49:53.540714  9374 catalog_manager.cc:5697] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 reported cstate change: config changed from index 3 to 5, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:53.550595  9638 raft_consensus.cc:2955] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:53.646306 11345 ts_tablet_manager.cc:1929] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20260808 01:49:53.650260 11354 tablet_replica.cc:333] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:49:53.651805 11354 raft_consensus.cc:2243] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:53.652913 11354 raft_consensus.cc:2272] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:53.673924 11354 ts_tablet_manager.cc:1916] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:53.658183 11345 log.cc:1199] T c094b8e92d5e449aa2641c359d46d622 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-1-root/wals/c094b8e92d5e449aa2641c359d46d622
I20260808 01:49:53.678656  9360 catalog_manager.cc:5028] TS 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): tablet c094b8e92d5e449aa2641c359d46d622 (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:53.679520  9481 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:53.699049 10365 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:49:53.700958 11310 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:49:53.701804  9638 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:49:53.709653 11268 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:49:53.718591  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:49:53.724367 11132 raft_consensus.cc:2955] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:53.734876  9637 raft_consensus.cc:2955] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:53.739655 11354 ts_tablet_manager.cc:1929] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260808 01:49:53.740383 11354 log.cc:1199] T fca874793205463dbb922993f57bf7f5 P 259bcb3f0d254e8b9effac3277e654e0: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-2-root/wals/fca874793205463dbb922993f57bf7f5
I20260808 01:49:53.743088  9360 catalog_manager.cc:5028] TS 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): tablet fca874793205463dbb922993f57bf7f5 (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:53.744428  9362 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 03b17f21031f4ff4828c85b7332fcc2d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:49:53.744488 10365 raft_consensus.cc:2955] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:49:53.759370  9371 catalog_manager.cc:5697] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 reported cstate change: config changed from index 5 to 6, VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) evicted. New cstate: current_term: 4 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:49:53.791141  9547 tablet_service.cc:1558] Processing DeleteTablet for tablet 03b17f21031f4ff4828c85b7332fcc2d with delete_type TABLET_DATA_TOMBSTONED (TS 52ccf4dcca1843f7bb29e9172e2d4ab4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53202
I20260808 01:49:53.792332 11345 tablet_replica.cc:333] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:49:53.793568 11345 raft_consensus.cc:2243] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260808 01:49:53.794401 11345 raft_consensus.cc:2272] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260808 01:49:53.803085 11345 ts_tablet_manager.cc:1916] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260808 01:49:53.847599 11345 ts_tablet_manager.cc:1929] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.5
I20260808 01:49:53.848498 11345 log.cc:1199] T 03b17f21031f4ff4828c85b7332fcc2d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Deleting WAL directory at /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786153669593643-7831-0/minicluster-data/ts-1-root/wals/03b17f21031f4ff4828c85b7332fcc2d
I20260808 01:49:53.850966  9360 catalog_manager.cc:5028] TS 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): tablet 03b17f21031f4ff4828c85b7332fcc2d (table test-workload [id=5ed6eaf552d849608150254f71c99d76]) successfully deleted
I20260808 01:49:53.871083  9481 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "030eacff61b14455978d2f0de262a089"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:38182
I20260808 01:49:53.871822  9481 raft_consensus.cc:606] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:53.880218  9637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:49:53.881477  9637 raft_consensus.cc:606] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:53.892129  9474 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:38182
I20260808 01:49:53.892838  9474 raft_consensus.cc:606] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:53.898936  9637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:49:53.899670  9637 raft_consensus.cc:606] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:53.901288  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260808 01:49:53.906082  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:49:53.907001  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:49:53.909313 11268 raft_consensus.cc:993] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: : Instructing follower 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:49:53.910029 11268 raft_consensus.cc:1081] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Signalling peer 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:49:53.918289  9637 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "030eacff61b14455978d2f0de262a089"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:42840
I20260808 01:49:53.921164  9637 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:53.933768  9637 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:53.982195  9637 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:53.985067  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:53.985675  9474 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:53.985978  9474 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:53.986783  9474 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:53.987890  9474 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:53.989624  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:53.990223  9552 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:54.000211  9474 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:49:54.001801  9552 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:49:54.001948  9601 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:54.006156 10803 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:54.014112 10803 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:54.016059 10803 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:54.002799  9637 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:54.041069  9371 catalog_manager.cc:5697] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 1 to 2, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 2 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
W20260808 01:49:54.450971 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 78271.9us
I20260808 01:49:55.203986 11371 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.204632 11371 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:55.205823 11370 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.206534 11370 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.207180 11371 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:55.208853 11370 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:55.215451 11373 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:55.216176 11373 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.232218 11373 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:55.232910 11370 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.233708 11370 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:55.236130 11370 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:55.258870 11370 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.259567 11370 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.265256 11370 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:55.274194 11372 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.275313 11372 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.280164 11365 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:55.280967 11365 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:55.284960 11365 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:55.289465 11376 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.290257 11376 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.292882 11376 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:55.294102 11372 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:55.295969 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:55.300814 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:55.314014 11372 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.314762 11372 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.318354 11372 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:55.435595 11378 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:55.436414 11378 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:55.490163  9474 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:55.507651  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:55.502267 11382 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:55.438931 11371 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.460323 11365 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:55.471021 11360 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:55.524986 11377 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:55.525684 11377 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.528851 11377 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:49:55.531946 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 50425.1us
I20260808 01:49:55.506276 11184 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:55.477499 11375 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:55.460667 11372 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.532256 11384 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.511351 11368 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:55.516214 11376 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.535434 11378 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:55.536592 11365 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:55.537859 11377 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.513139 11382 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:55.533529 11371 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.572144 11372 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.537699 11360 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.562582 11184 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.604796 11378 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.590854 11383 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:55.573354 11375 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:55.582734 11376 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:55.615418 11370 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.572921 11384 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:55.598467 11377 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.611682 11381 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:55.578527 11368 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:55.648231  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:55.656237 11386 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.657846 11393 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:55.659013 11392 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:55.659103 11184 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:55.660244 11184 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:55.659946 11392 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.661263 11184 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.644239 11383 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:55.665197 11382 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:55.666777 11184 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:55.668020 11184 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:55.668672 11360 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:55.659005 11393 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.657958 11380 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.662892 11388 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:55.659281  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:55.674450 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:55.657140 11387 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:55.685372 11377 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:55.686254 11383 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:55.648552 11370 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.688086 11375 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:55.681010 11368 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:55.681499 11385 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.653318 11381 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:55.689762 11370 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:55.683895 11372 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:55.688930 11366 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:55.669176 11184 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.714071 11366 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:55.685365 11387 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:55.691149 11391 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:55.692898 11381 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:55.640334 11378 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.694353 11380 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.694063 11385 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:55.657338 11386 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.692986 11365 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:55.696696 11392 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:55.699203 11388 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:55.670619 11384 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:55.707453  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:55.705439 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:55.682911 11393 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:55.686162 11376 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:55.815472  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:55.839949 10365 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
W20260808 01:49:55.815775  9523 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.321s	user 0.057s	sys 0.062s
I20260808 01:49:55.719125  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:55.841758  9523 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.347s	user 0.061s	sys 0.063s
I20260808 01:49:55.842276  9634 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
W20260808 01:49:55.842715  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 1.34805e+06us
I20260808 01:49:55.836460  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:55.838537  9631 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:55.720861  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:55.843741  9632 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:49:55.813710  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:55.894170  9633 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:55.897639  9638 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:49:55.891336  9635 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:55.800738 11184 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:55.728600 11383 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.815300  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:55.908515 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:55.733773 11366 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:55.723256 11377 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:55.869102 11384 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:55.865751 11393 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:55.883342 11392 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:55.928500 11393 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.929078 11392 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.923449 11377 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.919866 11383 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.929991 11394 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:55.913652 11365 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:55.869683 11391 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.954002  9629 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:55.948565 11393 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:49:55.950188  7836 debug-util.cc:398] Leaking SignalData structure 0x7b080000f340 after lost signal to thread 9520
I20260808 01:49:55.949966 11359 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:55.951103 11395 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.952661 11377 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:55.963311 11386 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:55.958237 11380 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:55.927356 11384 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:55.973323 11385 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:55.980264 11394 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.988471 11394 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:55.988791 11359 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:55.989852 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:55.952450 11401 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:55.935838 11382 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:55.923552  9630 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:49:55.925958 11373 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:55.958235 11383 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:49:56.033739  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 215751us
I20260808 01:49:55.964075 11253 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:55.964044 11396 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:56.016584 11366 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:55.963493 11403 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.960042 11387 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:55.979322 11388 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:56.077793  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:56.081293 11394 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:56.027434 11375 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:55.996982 11371 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:55.993986 11392 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:56.022874 11393 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:56.081365 11359 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:55.980621 11365 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:56.097078  9634 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:49:56.099385 11396 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:56.039352 11391 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:49:56.162753  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 220396us
I20260808 01:49:56.082749 11253 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:55.969830 11378 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:56.274739  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:55.996845 11385 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:56.084718 11366 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:56.027601 11401 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:56.040391 11373 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:56.279692  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.281625 11373 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:56.282259 11385 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:56.125988  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:56.040601 11380 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:56.040484 11397 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:56.286041 11380 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:56.055836 11357 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:56.286615 11397 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:56.055722 11384 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:56.167402  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:56.291390  9474 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:49:56.158847 11361 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:56.104496 11394 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:56.292594  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:49:56.156643 11368 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:56.034880 11395 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:56.157379 11359 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:56.144618 11392 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:49:56.156240  9502 debug-util.cc:398] Leaking SignalData structure 0x7b08000b0ba0 after lost signal to thread 7834
I20260808 01:49:56.157953 11375 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:56.171543 11393 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:56.034518 11382 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:56.103734 11403 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:56.283432 11401 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:56.359589  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:56.353050 11360 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:56.361856 11359 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:56.328038 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:56.322649 11394 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:56.368393 11395 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:56.363626 11360 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:56.180346  9445 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:49:56.261602 11184 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:56.261847 10365 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
W20260808 01:49:56.263981  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 96128.1us
I20260808 01:49:56.421805 11184 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:56.266637 11388 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:49:56.267767  9520 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.773s	user 0.041s	sys 0.051s
I20260808 01:49:56.427258 11388 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:56.273357  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:56.277143  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.430338  9555 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:49:56.432368  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.432948 11388 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:56.426349 11184 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:56.094954 11408 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:56.284543 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:56.294413  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.289568 11357 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:56.311630 11369 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:56.366503  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.155179  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:56.443410  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.443698  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.445254  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.325129 11368 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:56.445767 11394 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:49:56.443708  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 288758us
I20260808 01:49:56.377334  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:56.376206 11360 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:56.317128 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:56.340104 11385 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:56.447803 11388 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:56.456565 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:56.456759 11388 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:56.378587 11382 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:56.381738  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.379477 11392 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:56.281701 11366 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:56.384511  9502 debug-util.cc:398] Leaking SignalData structure 0x7b08000eafa0 after lost signal to thread 9412
I20260808 01:49:56.446264 11390 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:56.384786  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.463299 11390 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:56.464248 11390 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:56.464525 11357 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:56.385130 11396 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:56.338420 11380 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:56.385807  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.467801  9523 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:49:56.387486 11375 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:56.273936 11411 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:56.479398  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.479815 11411 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:56.480692 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:56.481824  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.391686 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:56.391839 10334 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:49:56.324476 11361 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:56.399677 11253 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:56.371444 11160 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:56.403599 11365 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:56.406642 11397 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:56.408706  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.409418  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.411830 11403 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:56.415288 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:56.418704 11359 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:56.265828 11383 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:49:56.428227  9520 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.934s	user 0.042s	sys 0.052s
I20260808 01:49:56.270668  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:56.435191 11408 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:56.437970  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 345141us
I20260808 01:49:56.438874  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.440580 11373 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:56.261287 11387 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:56.376610  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:56.454119  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:56.455289  9557 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 1.
I20260808 01:49:56.460894  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.447134 11394 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:56.313831 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:56.465003  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:49:56.516018  9502 debug-util.cc:398] Leaking SignalData structure 0x7b080028e5e0 after lost signal to thread 9506
I20260808 01:49:56.484234  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:49:56.559185  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 91239.3us
I20260808 01:49:56.487073 11411 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:49:56.576121  9502 debug-util.cc:398] Leaking SignalData structure 0x7b08003504c0 after lost signal to thread 9585
I20260808 01:49:56.487828  9559 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:49:56.579713 11375 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:56.580627 11375 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:56.581318 11375 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:56.581866 11383 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:56.530122 11386 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
W20260808 01:49:56.583462  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 129540us
I20260808 01:49:56.555001 11392 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:56.485548 11393 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:56.592394  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:56.592984 11361 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:49:56.594069 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 202560us
I20260808 01:49:56.586807  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.606593  9559 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 1.
I20260808 01:49:56.549849 11388 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:56.574868 11387 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:56.578601  9556 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:49:56.595057 11366 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:56.610637 11366 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:56.611577 11366 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:56.473922 11308 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:56.578485 11373 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:56.603901  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.451001 11369 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:56.605222 11390 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:56.618294  9552 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 1.
W20260808 01:49:56.616963  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 51201.4us
I20260808 01:49:56.595260 11160 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:56.627485  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:49:56.629235  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 161585us
I20260808 01:49:56.585119 11383 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:56.648934  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:56.502578 11368 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:56.652268 10332 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:56.654196  9600 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:56.619601 11308 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:56.620023 11373 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:56.639314 11392 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:56.638782 11401 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 01:49:56.711908 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 59774.6us
I20260808 01:49:56.731859  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.738626 10332 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:56.650326 11404 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:56.746397  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:56.646265 11408 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:56.746851 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:56.699371 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:56.752628 11387 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:56.744802 11401 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:56.765460 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:49:56.767267  9502 debug-util.cc:398] Leaking SignalData structure 0x7b08002ad620 after lost signal to thread 9663
I20260808 01:49:56.799114  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:56.727782 11392 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:56.739763 11365 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:56.813176 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:56.814008  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:56.708169 11383 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:49:56.700816 11372 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:56.755084 11357 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:56.740949 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:49:56.807919 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:56.817049 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:49:56.826323 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 85566.2us
I20260808 01:49:56.838795 11160 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:56.841709  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:56.845664 11394 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:56.850845 11385 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:56.852010 11385 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:56.852780 11385 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:56.853387  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.854499  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.855234  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.801976 11382 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:56.855744 11308 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:56.856452 11382 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:56.857323 11382 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:56.856834 11372 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:56.858310  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.802927 11401 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:56.859779  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.860366 11372 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:56.861413 11357 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:56.856715  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.861254  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.787966 11404 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:56.862069 11357 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:56.877774 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:56.872834 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:49:56.889168 11369 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:56.890398  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.893626  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.894445  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.895920  9555 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:49:56.897645 11413 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:56.888866  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.897475  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.904945  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.815642 11392 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:56.928453  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:56.930331  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:56.861193 11372 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:56.935612 11402 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:56.936833  9629 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:49:56.937785  9629 raft_consensus.cc:606] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:56.941046 11409 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:56.941653  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.941843 11409 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:56.944325 11409 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:56.871537 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:56.902525 11404 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:56.876573  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.965070 11414 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:56.921463  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:56.969923  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:56.977432 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:56.953011 10332 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:49:56.986634 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 68364.7us
I20260808 01:49:56.987550 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:49:56.987262  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 338456us
I20260808 01:49:56.991047  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:56.996099  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:57.005008  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:56.969770  9502 debug-util.cc:398] Leaking SignalData structure 0x7b080035fce0 after lost signal to thread 10393
I20260808 01:49:57.010342 11253 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:57.023654 11253 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:57.024552 11253 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:57.001345 11384 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:56.968631 11372 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:57.049814  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.005430 11395 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:57.051175  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.055292  9632 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "17e220e6a25747939bdf18f26b732407"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:49:57.048331  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.051321 11395 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:57.057050  9632 raft_consensus.cc:606] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:57.070071  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:56.899102 11413 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:57.055929 11373 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:49:57.073408  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 77606.6us
I20260808 01:49:57.076426  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:57.078843  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.082098 11412 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:57.083110 11412 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:57.083366 11386 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:57.083869 11412 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:57.087658  9599 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:57.032531 11366 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:57.071957 11395 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:49:57.075007  9502 debug-util.cc:398] Leaking SignalData structure 0x7b080035fc40 after lost signal to thread 11351
I20260808 01:49:57.103274  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.084395  9554 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
W20260808 01:49:57.111857  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:49:57.008925 11414 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:57.129269  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:57.002461 11402 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:57.139616 11402 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:57.142023  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 54830.8us
I20260808 01:49:57.145388  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:57.153667  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.107363 11371 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:57.120249 11390 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:57.125134 11403 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:57.176292  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:57.152909 10362 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: []
I20260808 01:49:57.191814 11386 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:57.153209 11383 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:57.196096 11366 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:57.197715 11414 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:57.152912 11415 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:57.171290 11378 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:57.287070 11378 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:57.286271 11415 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:57.287806 11386 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:57.289364 11415 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:57.133805 11396 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
W20260808 01:49:57.175275 11375 consensus_meta.cc:287] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.589s	user 0.014s	sys 0.008s
I20260808 01:49:57.175370 11416 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:57.219310  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.219937  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.298216 11375 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:57.298331  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.300233  9631 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:57.097080 11357 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:57.291842 11393 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:57.299110 11416 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:57.292157 11402 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:57.303458 11402 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:57.269367 11372 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:57.270166  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.202963 11371 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:57.306159  9630 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:57.306937  9635 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:57.203629 11390 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:57.308877 11390 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:57.309468 11375 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:57.309655 11390 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:57.301908 11393 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:57.316596 11386 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:57.202991 11377 raft_consensus.cc:993] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:49:57.319805 11377 raft_consensus.cc:1081] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Signalling peer 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:49:57.319937  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.320604 11402 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:57.200416 11420 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:57.314229 11393 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:57.283900  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:57.261034  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.200172 11387 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:57.273468  9600 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:57.282225  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:57.271888  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.217633 11413 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:57.339648  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:49:57.344213  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 146316us
W20260808 01:49:57.359081  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 85827.3us
I20260808 01:49:57.362090  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:57.364037  9476 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: []
I20260808 01:49:57.366927  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.374495  9476 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: []
I20260808 01:49:57.341832 11414 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:57.295452 11401 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:57.392405  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:57.396739  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:57.368624 11382 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:57.355859 11373 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:57.423398  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:57.449316  9479 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: []
I20260808 01:49:57.403595  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:57.404783  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:57.407539  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.417307  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.419431  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.420558  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.451754  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.421703  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:57.417855  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.398638  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:57.430819  9551 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "17e220e6a25747939bdf18f26b732407"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:53208
I20260808 01:49:57.410097  9472 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: []
I20260808 01:49:57.338372 11420 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:57.454071  9551 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:49:57.454880  9551 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:57.421485  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:57.410485 10332 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.568s (SENT)
I20260808 01:49:57.398533 11419 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:57.405927 11414 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:57.372505 11422 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:57.439437  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:57.439992 11371 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:57.418475  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.420614  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:57.418912 11413 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:57.441982  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:57.419967  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:57.347255 11387 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:57.341495 11372 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:49:57.435564  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:49:57.468950 11375 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:57.488054 11419 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:57.511732  9473 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:57.516112  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.516597  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:57.517732  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46088.1us
I20260808 01:49:57.409814 11401 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:57.545101  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:57.525197  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.534243  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.541451  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:57.555567 11389 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:57.515095 11413 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:57.558923  9476 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:57.496078 11377 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:57.563751  9633 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:57.575387  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:57.578961 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 168529us
I20260808 01:49:57.579553  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:57.581102  9481 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:57.593314  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:57.595165  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.596267  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:57.591241  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 91232.9us
I20260808 01:49:57.593510  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.597448  9478 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:57.601289  9601 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:57.600389  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:57.602547  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.546352 11357 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:57.607558  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:57.564635 11420 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:57.593317 11392 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.632s	user 0.013s	sys 0.013s
I20260808 01:49:57.620354  9444 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:57.617921  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:57.575711 11384 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:57.557950 11375 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:57.614279 11426 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:57.540434 11372 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:57.633109  9475 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:57.663849  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:57.598868 11417 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:57.686409  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:57.687472  9478 raft_consensus.cc:2393] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:57.619117 11393 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:57.681424  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:57.714910 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 103262us
W20260808 01:49:57.683467  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 119319us
W20260808 01:49:57.750852  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 134278us
I20260808 01:49:57.656984 11420 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:57.751516 10334 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:57.631740 11378 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:57.757433  9475 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:49:57.753902  9551 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:57.752585  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:57.597399 11377 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:57.763919 10332 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:57.671918 11426 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:57.656579 11392 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:49:57.711476 11385 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.836s	user 0.023s	sys 0.005s
W20260808 01:49:57.763404 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 51413.2us
W20260808 01:49:57.818337 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 54584.9us
I20260808 01:49:57.816136  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:57.832914 11357 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:57.844352  9557 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:57.845180 11392 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:57.775807 11397 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:49:57.847123 11382 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:57.851593 11387 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:57.851756  9551 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:57.768087 11409 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.823s	user 0.015s	sys 0.008s
W20260808 01:49:57.761269  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 103575us
I20260808 01:49:57.820358  9554 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:57.668484 11384 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:57.675984 11416 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:57.860932 11385 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:57.859912 10332 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:57.873021  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:57.874168 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:57.714403 11417 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:57.862030  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:57.877604  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:49:57.885803  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 123849us
I20260808 01:49:57.894845 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:57.832502 11426 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:57.911370 10367 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:57.870200 11372 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:57.888392 11382 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:57.890053 11371 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:49:57.897770 11397 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:57.924062  9601 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:49:57.925714  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 170030us
I20260808 01:49:57.906899 11366 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:57.871922 11409 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:49:57.922190 11253 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.897s	user 0.019s	sys 0.003s
I20260808 01:49:57.934552  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:57.915472 11414 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:57.914211 11419 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:57.929847 11437 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:49:57.922536 11382 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:57.959355  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:57.961110 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:49:57.961530 11375 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:57.973392  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:57.950419 11413 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:57.974270 11402 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:49:57.967270 11390 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.657s	user 0.013s	sys 0.008s
I20260808 01:49:57.985154  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:49:57.991734  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 101801us
I20260808 01:49:57.997036  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:57.957733 11382 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:58.017418 10363 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:57.977308 11434 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:57.950611 11253 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:58.041028  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:58.050273  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:58.052289  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:58.054981  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:58.055358  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:58.059579  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:58.023890 11413 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:58.060858  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:49:58.046756 11395 consensus_meta.cc:287] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.917s	user 0.027s	sys 0.001s
I20260808 01:49:58.069079  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:58.071529  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:58.058091 11412 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.973s	user 0.009s	sys 0.014s
I20260808 01:49:58.073048  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:58.047806 11419 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:58.079521  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:58.080698 11406 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:58.081001 11443 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:58.023425 11390 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:58.078661  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:58.088788  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:58.082049 10362 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:58.099238  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:58.074805 11420 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260808 01:49:58.091598 11404 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.086s	user 0.013s	sys 0.010s
I20260808 01:49:58.083195 11401 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:58.083900 11377 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:58.110230  9477 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 2: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:58.085194 11392 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:58.094770 11384 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:58.127795 11253 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:49:58.124260  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 64112.9us
I20260808 01:49:58.077634 11395 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:58.134236 11392 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:58.147325  9445 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50140 dropped due to backpressure: service queue is full with 50 items
I20260808 01:49:58.086174 11412 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:58.132781  9521 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:58.157595  9521 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:49:58.167532  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 243721us
I20260808 01:49:58.168864  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:58.102240 11417 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:58.176915 11448 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:58.167124 11359 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:58.180073 11395 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:58.181542  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:58.201977  9473 raft_consensus.cc:2393] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 2: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:58.183143  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:58.184176  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:58.184952  9474 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:58.166421 11440 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:58.185876  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:58.147825  9445 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50140 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1741)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1539)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1543)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1742)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1535)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1544)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1548)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1762)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 159)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1536)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1545)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1557)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1534)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1546)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1541)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1540)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1551)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1537)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1542)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1763)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1761)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1550)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1549)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1556)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:38182 (request call id 7)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1743)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1745)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1747)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 156)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 157)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 158)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1749)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1750)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1751)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1753)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1754)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1755)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1756)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1757)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1533)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1758)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1538)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1759)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1760)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1547)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1552)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1553)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1554)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1555)
I20260808 01:49:58.194133  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:58.187345 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:58.205211 11392 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:58.130549 11390 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:58.061753 11434 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:58.210477  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:58.219317  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:58.200930 11377 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:58.224740  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.228890  9475 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:58.200973 11359 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:58.225677  9601 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:58.232247  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.232329  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.119433 11404 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:58.236514  9601 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:58.246425  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.248957  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:58.249943  9629 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:58.220384 11440 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:58.252094  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.237186 11359 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:58.253207  9472 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:58.248160  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.258957  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:58.238119 11392 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:58.260453  9480 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:49:58.262845  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.263352  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:58.268544  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.269438  9472 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:58.268716  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:49:58.270895  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:49:58.270119  9601 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:58.270524  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.276489  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:58.266923 11447 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:58.277345  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.255216 11440 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:58.275828  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:58.284871  9601 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:58.285063  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:58.277000  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:58.278522  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:58.280076  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:58.280616  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:58.281118  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.281951  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1456) took 1049 ms. Trace:
I20260808 01:49:58.282752  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:58.282181  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:58.282289  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:58.288165  9475 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:58.287415  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:58.290359 11445 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:58.279079  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:58.264231 11392 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:58.292115 11392 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:58.295655  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:58.298435  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.299685  9472 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:58.299916  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:58.301355  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:58.299849  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:58.261957 11377 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:58.304543  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.288113  9633 rpcz_store.cc:276] 0808 01:49:57.219022 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:57.219506 (+   484us) service_pool.cc:224] Handling call
0808 01:49:58.267785 (+1048279us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b5400337f54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:49:58.268001 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":607174912}
W20260808 01:49:58.278645 11415 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.982s	user 0.017s	sys 0.008s
I20260808 01:49:58.308072  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.312721  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 1783) took 1464 ms. Trace:
I20260808 01:49:58.313714  9472 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:58.313921 10367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 396) took 1211 ms. Trace:
I20260808 01:49:58.272786 11435 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:58.314038  9557 rpcz_store.cc:276] 0808 01:49:56.846718 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:56.847813 (+  1095us) service_pool.cc:224] Handling call
0808 01:49:56.848369 (+   556us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:57.845093 (+996724us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:58.306203 (+461110us) spinlock_profiling.cc:243] Waited 996 ms on lock 0x7b540002ec54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:49:58.306413 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2289954560}
I20260808 01:49:58.319793 11452 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:58.320295 11441 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:58.314893 10367 rpcz_store.cc:276] 0808 01:49:57.094655 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:57.095324 (+   669us) service_pool.cc:224] Handling call
0808 01:49:57.095872 (+   548us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:57.917511 (+821639us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:58.305679 (+388168us) spinlock_profiling.cc:243] Waited 815 ms on lock 0x7b54004c3454. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:49:58.305895 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1874721536}
I20260808 01:49:58.316156 11445 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:58.321130 11415 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:58.337167  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.330062  9478 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 2: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:58.340245  9474 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:58.352914  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.354408  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.355871  9472 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:58.353945  9477 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:49:58.365028 11432 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:58.361657 10334 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:58.323729 11253 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:58.335237 11450 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.204s
W20260808 01:49:58.553223 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 191784us
I20260808 01:49:58.557669 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:49:58.558969 10366 raft_consensus.cc:2393] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:49:58.388263  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:58.564054 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:58.607532  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.328476 11445 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:58.611955  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:58.624265  9556 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:49:58.624562  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.617758  9629 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:49:58.607595  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1738) took 1170 ms. Trace:
I20260808 01:49:58.629107  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.630196  9474 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:58.453490 11433 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:58.625119 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:58.658154  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.642622  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1529) took 1215 ms. Trace:
I20260808 01:49:58.668253  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.666211  9481 rpcz_store.cc:276] 0808 01:49:57.404103 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:57.404356 (+   253us) service_pool.cc:224] Handling call
0808 01:49:58.617440 (+1213084us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b540009a1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:49:58.617636 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":300513536}
I20260808 01:49:58.671154  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.627089  9473 rpcz_store.cc:276] 0808 01:49:57.437429 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:57.658869 (+221440us) service_pool.cc:224] Handling call
0808 01:49:58.607222 (+948353us) spinlock_profiling.cc:243] Waited 514 ms on lock 0x7b5400403dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:49:58.607481 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1181623680}
I20260808 01:49:58.672536  9481 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:49:58.636109  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:58.679081  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.680022  9475 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
W20260808 01:49:58.683982  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 65667.6us
I20260808 01:49:58.684438  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.685901  9445 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:58.687671  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.689054  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:58.690460  9558 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 2: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:49:58.611557 11417 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:49:58.700258  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.632493 11445 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:58.705108  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.705405  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.604795 11412 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:58.707522  9481 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:58.716178  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.717687  9481 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:58.710893  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.720784  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:58.719462  9474 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:49:58.464311 11435 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:58.366612 11441 consensus_queue.cc:1059] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.344s
I20260808 01:49:58.483683 11382 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:49:58.720443  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 414341us
I20260808 01:49:58.737540  9474 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:38182
I20260808 01:49:58.740069  9474 raft_consensus.cc:606] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:58.740345  9601 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:58.743052  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:58.743829  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.744443  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:58.746528  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:58.747710  9481 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:58.753613  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:58.730875 11412 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:58.633939 11253 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:58.757922  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1745) took 1311 ms. Trace:
I20260808 01:49:58.759691 11436 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:58.760986 11253 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:58.761128 11436 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:58.762100  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 151) took 2053 ms. Trace:
I20260808 01:49:58.757354 11412 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:49:58.771103  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:58.758592  9475 rpcz_store.cc:276] 0808 01:49:57.446973 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.756202 (+1309229us) service_pool.cc:224] Handling call
0808 01:49:58.756868 (+   666us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:58.757247 (+   379us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:58.757303 (+    56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:58.757388 (+    85us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:49:58.757499 (+   111us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:58.757607 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:58.757826 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:49:58.777115  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:38182 (request call id 7) took 1229 ms. Trace:
I20260808 01:49:58.781188  9474 rpcz_store.cc:276] 0808 01:49:57.511427 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.737297 (+1225870us) service_pool.cc:224] Handling call
0808 01:49:58.740978 (+  3681us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:49:58.762614  9480 rpcz_store.cc:276] 0808 01:49:56.685403 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:56.685675 (+   272us) service_pool.cc:224] Handling call
0808 01:49:56.699165 (+ 13490us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:58.262799 (+1563634us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:58.738364 (+475565us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b540038d1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:49:58.738829 (+   465us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3589614976}
I20260808 01:49:58.789075  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1747) took 1341 ms. Trace:
I20260808 01:49:58.790083  9474 rpcz_store.cc:276] 0808 01:49:57.447724 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.787209 (+1339485us) service_pool.cc:224] Handling call
0808 01:49:58.787845 (+   636us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:58.788205 (+   360us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:58.788277 (+    72us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:58.788380 (+   103us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:49:58.788488 (+   108us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:58.788634 (+   146us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:58.788947 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:49:58.787899  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1743) took 1342 ms. Trace:
I20260808 01:49:58.737320 11404 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:58.762800 11372 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:58.731015 11437 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:58.701355 11433 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:58.797945  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 157) took 1304 ms. Trace:
I20260808 01:49:58.794696  9473 rpcz_store.cc:276] 0808 01:49:57.445910 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.745394 (+1299484us) service_pool.cc:224] Handling call
0808 01:49:58.786611 (+ 41217us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:58.787224 (+   613us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:58.787287 (+    63us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:58.787440 (+   153us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:49:58.787525 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:58.787624 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:58.787802 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:49:58.807930  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:58.796403 11385 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:58.804212  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 156) took 1311 ms. Trace:
I20260808 01:49:58.823449  9475 rpcz_store.cc:276] 0808 01:49:57.493794 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.796293 (+1302499us) service_pool.cc:224] Handling call
0808 01:49:58.796998 (+   705us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:58.797322 (+   324us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:58.797383 (+    61us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:58.797471 (+    88us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:49:58.797537 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:58.797640 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:58.797836 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:49:58.825354  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:49:58.793519  9523 consensus_peers.cc:597] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: affdd24b03ae4ee795a90703613305f7. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:49:58.828925  9474 rpcz_store.cc:276] 0808 01:49:57.493089 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.792822 (+1299733us) service_pool.cc:224] Handling call
0808 01:49:58.793547 (+   725us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:58.803467 (+  9920us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:58.803544 (+    77us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:58.803664 (+   120us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:49:58.803763 (+    99us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:58.803930 (+   167us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:58.804115 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:49:58.832248  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:49:58.832417  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 56703.5us
I20260808 01:49:58.721720 11450 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.092s
I20260808 01:49:58.834555 11450 consensus_queue.cc:1059] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:49:58.838181 11451 raft_consensus.cc:3055] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:49:58.843425  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.840297  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 158) took 1346 ms. Trace:
I20260808 01:49:58.838579  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:58.854115  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1750) took 1331 ms. Trace:
I20260808 01:49:58.860917  9480 rpcz_store.cc:276] 0808 01:49:57.494406 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.828967 (+1334561us) service_pool.cc:224] Handling call
0808 01:49:58.839332 (+ 10365us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:58.839760 (+   428us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:58.839802 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:58.839866 (+    64us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:49:58.839943 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:58.840059 (+   116us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:58.840217 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:49:58.861630  9474 rpcz_store.cc:276] 0808 01:49:57.522757 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.832915 (+1310158us) service_pool.cc:224] Handling call
0808 01:49:58.833263 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:58.833589 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:58.833650 (+    61us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:58.833758 (+   108us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:49:58.853725 (+ 19967us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:58.853832 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:58.854010 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:49:58.867615  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1751) took 1294 ms. Trace:
I20260808 01:49:58.854116  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:58.868525  9474 rpcz_store.cc:276] 0808 01:49:57.573854 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.865224 (+1291370us) service_pool.cc:224] Handling call
0808 01:49:58.865950 (+   726us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:58.866294 (+   344us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:58.866375 (+    81us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:58.866486 (+   111us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:49:58.866599 (+   113us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:58.867040 (+   441us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:58.867366 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:49:58.799108 11434 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:58.871930  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1749) took 1354 ms. Trace:
I20260808 01:49:58.817180 11439 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:58.859968  9481 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:58.877820  9629 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:49:58.801616 11435 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:58.872020  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:49:58.856498 11440 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.540s	user 0.012s	sys 0.005s
I20260808 01:49:58.843657 11404 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:49:58.867098 11389 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:58.888958  9629 raft_consensus.cc:606] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Received request to transfer leadership to TS 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:58.843217 11412 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:49:58.885262  9475 rpcz_store.cc:276] 0808 01:49:57.517838 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.829869 (+1312031us) service_pool.cc:224] Handling call
0808 01:49:58.870362 (+ 40493us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:58.871060 (+   698us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:58.871139 (+    79us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:58.871215 (+    76us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:49:58.871267 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:58.871396 (+   129us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:58.871626 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:49:58.869415 11392 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.576s	user 0.013s	sys 0.005s
I20260808 01:49:58.896417  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1755) took 1301 ms. Trace:
I20260808 01:49:58.900760  9481 rpcz_store.cc:276] 0808 01:49:57.595411 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.890426 (+1295015us) service_pool.cc:224] Handling call
0808 01:49:58.891096 (+   670us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:58.891441 (+   345us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:58.891489 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:58.895794 (+  4305us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:49:58.895912 (+   118us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:58.896066 (+   154us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:58.896316 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:49:58.859606 11451 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:58.902912  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 1614) took 1485 ms. Trace:
I20260808 01:49:58.876535 11382 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:58.906544  9637 rpcz_store.cc:276] 0808 01:49:57.412752 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:57.414850 (+  2098us) service_pool.cc:224] Handling call
0808 01:49:58.271294 (+856444us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:58.271612 (+   318us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:58.271661 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:58.271738 (+    77us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:49:58.271786 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:58.271905 (+   119us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:58.897830 (+625925us) spinlock_profiling.cc:243] Waited 856 ms on lock 0x7b5400337f50. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:49:58.898020 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1968427776}
I20260808 01:49:58.908862 11382 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:58.898211 11440 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:58.910591 11383 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:58.911265  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:58.911579 11383 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:58.912503 11440 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:58.845077 11417 consensus_queue.cc:1059] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260808 01:49:58.920029 11417 raft_consensus.cc:3055] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:49:58.920710 11417 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:49:58.912413  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1754) took 1317 ms. Trace:
I20260808 01:49:58.914247  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:58.925752  9635 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:58.924329  9480 rpcz_store.cc:276] 0808 01:49:57.594947 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.888375 (+1293428us) service_pool.cc:224] Handling call
0808 01:49:58.911525 (+ 23150us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:58.911847 (+   322us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:58.911888 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:58.911968 (+    80us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:49:58.912046 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:58.912140 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:58.912321 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:49:58.822646 11455 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.095s
I20260808 01:49:58.933859 10363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 398) took 1538 ms. Trace:
I20260808 01:49:58.934723 10363 rpcz_store.cc:276] 0808 01:49:57.395257 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:57.398778 (+  3521us) service_pool.cc:224] Handling call
0808 01:49:57.558047 (+159269us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:58.063005 (+504958us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:58.933500 (+870495us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b5400785ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:49:58.933743 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":920940800}
I20260808 01:49:58.939417  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260808 01:49:58.829744 11437 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:58.928237 11404 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:49:58.951189  9480 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:58.932426  9479 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:58.897202 11378 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:58.897028 11439 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:58.939893 11417 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:49:58.953034 11443 consensus_meta.cc:287] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.871s	user 0.010s	sys 0.011s
I20260808 01:49:58.894972 11395 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:58.930794 11433 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:58.926503  9635 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:58.955780  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1756) took 1360 ms. Trace:
I20260808 01:49:58.978662  9475 rpcz_store.cc:276] 0808 01:49:57.604526 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.900266 (+1295740us) service_pool.cc:224] Handling call
0808 01:49:58.954845 (+ 54579us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:58.955196 (+   351us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:58.955252 (+    56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:58.955320 (+    68us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:49:58.955392 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:58.955486 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:58.955680 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:49:58.889572 11377 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:58.975798 11308 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:58.975315 11378 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:58.988622 11443 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:58.988761 11377 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:58.989156 11308 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:58.960139 11437 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:49:58.968549 11426 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.052s	user 0.008s	sys 0.013s
I20260808 01:49:58.990828 11378 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:58.962507  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1741) took 1523 ms. Trace:
I20260808 01:49:58.991573  9629 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 01:49:58.964673  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1533) took 1289 ms. Trace:
I20260808 01:49:58.971273  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:58.959396 11440 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:58.904937 11392 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:58.893577 11432 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:58.991585  9479 rpcz_store.cc:276] 0808 01:49:57.444654 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.837910 (+1393256us) service_pool.cc:224] Handling call
0808 01:49:58.962191 (+124281us) spinlock_profiling.cc:243] Waited 70.8 ms on lock 0x7b5400392bd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:49:58.962417 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":162929024}
I20260808 01:49:58.991518 11308 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:58.993911  9480 rpcz_store.cc:276] 0808 01:49:57.675190 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.928866 (+1253676us) service_pool.cc:224] Handling call
0808 01:49:58.950749 (+ 21883us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:58.959260 (+  8511us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:58.964551 (+  5291us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:49:58.959676 11434 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:49:58.995185  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 311502us
I20260808 01:49:58.996467  9481 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
W20260808 01:49:59.004993  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 125338us
I20260808 01:49:58.967094 11436 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:49:59.012795  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 1808) took 1594 ms. Trace:
I20260808 01:49:59.013993  9554 rpcz_store.cc:276] 0808 01:49:57.418670 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:57.422149 (+  3479us) service_pool.cc:224] Handling call
0808 01:49:57.524516 (+102367us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:57.866771 (+342255us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:59.012463 (+1145692us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x7b540029f4d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:49:59.012703 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":674790784}
I20260808 01:49:59.020300  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:59.007836  9635 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:58.999554 11447 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:59.043507  9635 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:49:59.045550  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 359758us
I20260808 01:49:59.068889  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1759) took 1207 ms. Trace:
I20260808 01:49:59.043574  9550 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:49:59.072144  9445 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:58.991918 11452 consensus_queue.cc:1059] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260808 01:49:59.071870  9480 rpcz_store.cc:276] 0808 01:49:57.861490 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:59.006215 (+1144725us) service_pool.cc:224] Handling call
0808 01:49:59.068225 (+ 62010us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:59.068483 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:59.068513 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:59.068569 (+    56us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:49:59.068610 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:59.068676 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:59.068809 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:49:59.044687  9551 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:59.074810  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:59.024667  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:49:59.069171 11359 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.815s	user 0.014s	sys 0.007s
I20260808 01:49:59.076241  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:59.076382  9445 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:58.990979 11426 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:59.075559  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 118115us
I20260808 01:49:59.070746 11436 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:59.105872 11415 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:59.106746  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:59.000954 11440 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:59.107404 11436 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:59.108341 11386 raft_consensus.cc:3055] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:59.108837 11386 raft_consensus.cc:740] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:59.109138 11440 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:59.109830 11386 consensus_queue.cc:260] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:59.101202  9445 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:59.074709 11404 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:59.104765 11426 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:59.107013  9476 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:59.109927 11436 consensus_queue.cc:1059] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:49:59.110877 11389 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:49:59.111029  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:59.111827  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:59.112291 10365 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:59.124373  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:59.126220  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:59.113211 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:49:59.113394  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:49:59.135478 11439 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:59.136600  9371 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50084 (request call id 227) took 1741 ms. Trace:
I20260808 01:49:59.136931  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:59.137262  9371 rpcz_store.cc:276] 0808 01:49:57.395257 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:57.395546 (+   289us) service_pool.cc:224] Handling call
0808 01:49:59.136489 (+1740943us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":2}
I20260808 01:49:59.139374  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:49:59.138295  9635 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:49:59.114418  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1547) took 1173 ms. Trace:
I20260808 01:49:59.114526 11383 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:59.002854 11432 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:49:59.086812  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 62288.4us
I20260808 01:49:59.149724  9524 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:59.151203 11432 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:59.073690 11417 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:59.102249  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1760) took 1213 ms. Trace:
I20260808 01:49:59.154196 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:59.154848  9480 rpcz_store.cc:276] 0808 01:49:57.889258 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:59.078551 (+1189293us) service_pool.cc:224] Handling call
0808 01:49:59.101487 (+ 22936us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:59.101752 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:59.101794 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:59.101859 (+    65us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:49:59.101910 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:59.101983 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:59.102168 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:49:59.004475 11451 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:59.100065 11460 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260808 01:49:59.119635  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:49:59.121934  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 249898us
W20260808 01:49:59.122820  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 100985us
I20260808 01:49:59.115051 11392 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:59.118651 11435 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:59.112473  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:59.081385 11455 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260808 01:49:59.189793 11465 raft_consensus.cc:3055] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:59.190173 11455 raft_consensus.cc:3055] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:49:59.190709 11455 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:59.191215  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1557) took 1045 ms. Trace:
I20260808 01:49:59.113003  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1757) took 1476 ms. Trace:
I20260808 01:49:59.013648 11434 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:59.142700 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:59.143167  9472 rpcz_store.cc:276] 0808 01:49:57.940775 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:59.113618 (+1172843us) service_pool.cc:224] Handling call
0808 01:49:59.113935 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:59.114127 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:59.114149 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:59.114187 (+    38us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:49:59.114222 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:59.114275 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:59.114361 (+    86us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:49:59.191751  9481 rpcz_store.cc:276] 0808 01:49:57.636634 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.904513 (+1267879us) service_pool.cc:224] Handling call
0808 01:49:58.905124 (+   611us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:59.004876 (+ 99752us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:59.112742 (+107866us) spinlock_profiling.cc:243] Waited 91.1 ms on lock 0x7b54004042d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:49:59.112922 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":209561856}
I20260808 01:49:59.192888  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.193457 11434 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:59.193742  9472 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 2: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:59.147301  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:59.070178 11447 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:59.152441  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:49:59.158813 11377 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:59.161783 11408 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:49:59.194993 11408 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:59.195539  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:59.196035 11447 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:59.158176 11439 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:59.146394 11436 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:59.166141 10362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 399) took 1187 ms. Trace:
I20260808 01:49:59.166844 11450 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:59.196817  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:59.197098 11436 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:59.197499  9551 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:49:59.197356 11450 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:59.162667 11456 raft_consensus.cc:3055] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:49:59.198554 11456 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:59.199522 11436 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:59.200187  9552 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:59.117897  9445 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:59.167068 10332 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:59.172911  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:59.093698 11359 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:59.201164  9599 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:59.175937  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:59.178341 11458 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:49:59.208453 10332 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:59.184393  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:59.188145  9478 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1740) took 1750 ms (client timeout 2045 ms). Trace:
I20260808 01:49:59.188915  9479 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:59.190244 11465 raft_consensus.cc:740] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:59.224295 11465 consensus_queue.cc:260] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:59.225859 11465 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:59.199672 11450 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:49:59.214591  9478 rpcz_store.cc:269] 0808 01:49:57.438268 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:57.752175 (+313907us) service_pool.cc:224] Handling call
0808 01:49:59.187871 (+1435696us) spinlock_profiling.cc:243] Waited 511 ms on lock 0x7b540019c8d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:49:59.188048 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1175648128}
I20260808 01:49:59.228295  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.229460  9478 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:59.230325  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.231222  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.191604  9476 rpcz_store.cc:276] 0808 01:49:58.145563 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.270108 (+124545us) service_pool.cc:224] Handling call
0808 01:49:59.190991 (+920883us) spinlock_profiling.cc:243] Waited 827 ms on lock 0x7b540023aad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:49:59.191152 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1901350272}
I20260808 01:49:59.232501  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.210018 11416 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.193594  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:59.234549  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1552) took 1116 ms. Trace:
I20260808 01:49:59.234997  9476 rpcz_store.cc:276] 0808 01:49:58.117914 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:59.232321 (+1114407us) service_pool.cc:224] Handling call
0808 01:49:59.232658 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:59.234188 (+  1530us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:59.234223 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:59.234276 (+    53us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:49:59.234317 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:59.234375 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:59.234492 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1792}
W20260808 01:49:59.201828  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 84040.5us
I20260808 01:49:59.236383  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1554) took 1092 ms. Trace:
I20260808 01:49:59.236928  9473 rpcz_store.cc:276] 0808 01:49:58.143721 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:59.235581 (+1091860us) service_pool.cc:224] Handling call
0808 01:49:59.235892 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:59.236077 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:59.236101 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:59.236139 (+    38us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:49:59.236174 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:59.236226 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:59.236327 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:49:59.238379  9473 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:49:59.170991  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:59.239228 11377 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:49:59.239367  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 68537.6us
I20260808 01:49:59.240293  9601 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:59.241176 11377 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:59.241566  9601 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:59.191913 11409 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:59.242666 11441 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.243896  9601 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:49:59.244908  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 72333.8us
I20260808 01:49:59.246325 11386 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:49:59.211565  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:49:59.212774  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:59.213662  9599 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:49:59.215668 10332 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:59.197109 10362 rpcz_store.cc:276] 0808 01:49:57.978875 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:57.979203 (+   328us) service_pool.cc:224] Handling call
0808 01:49:57.986416 (+  7213us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:58.103964 (+117548us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:59.165849 (+1061885us) spinlock_profiling.cc:243] Waited 76.4 ms on lock 0x7b5400690fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:49:59.166071 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":175715200}
I20260808 01:49:59.209510 11366 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.196817 11439 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:59.216667 11448 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:59.284102  9559 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:59.284512  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:59.217113 11436 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.217732  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:59.218281 11395 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:49:59.218869 11437 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.219563  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1758) took 1446 ms. Trace:
W20260808 01:49:59.209852 11460 consensus_queue.cc:1175] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "259bcb3f0d254e8b9effac3277e654e0" responder_term: 2 status { last_received { term: 2 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 2 index: 2 } }
I20260808 01:49:59.222733  9552 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:49:59.222995  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:59.228410  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:59.233498  9480 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:49:59.236491  9476 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:59.195961 11408 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:59.247535 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:49:59.199654 11456 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:59.248847  9481 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3. Ops: []
I20260808 01:49:59.231470  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.251175 10334 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:59.234409  9477 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:59.204831  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:49:59.120886 11415 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:49:59.257131 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:49:59.265028  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.271471  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1538) took 1426 ms. Trace:
W20260808 01:49:59.273113  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 59495.2us
I20260808 01:49:59.186013 11445 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:49:59.276283 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 60676.2us
I20260808 01:49:59.286731  9475 rpcz_store.cc:276] 0808 01:49:57.773822 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.980790 (+1206968us) service_pool.cc:224] Handling call
0808 01:49:58.998257 (+ 17467us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:59.218114 (+219857us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:59.218151 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:49:59.218197 (+    46us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:49:59.218230 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:59.219318 (+  1088us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b540019b4d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:49:59.219361 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:59.219501 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":505061248}
I20260808 01:49:59.288549  9524 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:59.249075 11441 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.283672 11439 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:59.285974  9556 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:49:59.298885  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1553) took 1156 ms. Trace:
I20260808 01:49:59.372819  9479 rpcz_store.cc:276] 0808 01:49:57.844710 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:59.002100 (+1157390us) service_pool.cc:224] Handling call
0808 01:49:59.188712 (+186612us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:59.215365 (+ 26653us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:59.271134 (+ 55769us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b540019c8d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:49:59.271361 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":334521600}
I20260808 01:49:59.304749 10368 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: []
I20260808 01:49:59.300253  9472 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:59.291929  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1555) took 1147 ms. Trace:
I20260808 01:49:59.378075  9476 rpcz_store.cc:276] 0808 01:49:58.144337 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:59.236013 (+1091676us) service_pool.cc:224] Handling call
0808 01:49:59.236317 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:59.291680 (+ 55363us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:59.291869 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:49:59.376472  9477 rpcz_store.cc:276] 0808 01:49:58.143082 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:59.233853 (+1090771us) service_pool.cc:224] Handling call
0808 01:49:59.234225 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:59.298595 (+ 64370us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:59.298820 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:49:59.308557 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 57472.9us
I20260808 01:49:59.306308  9601 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:59.282291 11451 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:59.279750 11444 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:49:59.301915  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1570) took 1016 ms. Trace:
W20260808 01:49:59.375584  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:49:59.303267  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 90534.7us
I20260808 01:49:59.387503  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:49:59.301420 11366 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.388360  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.372865 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:49:59.383663 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:49:59.381507  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 80357.1us
I20260808 01:49:59.291712 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:49:59.396407  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 107940us
I20260808 01:49:59.397534  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:59.384032  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.394196  9559 rpcz_store.cc:276] 0808 01:49:58.286008 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.311393 (+ 25385us) service_pool.cc:224] Handling call
0808 01:49:58.311801 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:49:59.284868 (+973067us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:59.301675 (+ 16807us) spinlock_profiling.cc:243] Waited 972 ms on lock 0x7b540002d5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:49:59.301854 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2235492224}
I20260808 01:49:59.303309 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:59.399971  9637 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:59.295451 11458 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:59.403877  9559 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:59.402518  9524 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:59.405786  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.406653  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:59.406963  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.408708  9524 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:59.295979 11448 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:59.191604 11455 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:59.379112 11412 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.392248 11408 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:59.412283  9524 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:59.286554 11387 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:59.422549 10366 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3. Ops: []
I20260808 01:49:59.380390 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:59.413966 11456 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:59.430608 11160 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:59.398994 11366 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.433885  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.423581  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.290591 11392 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.109s
I20260808 01:49:59.413539 11394 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.413050 11467 ts_tablet_manager.cc:933] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:59.429256  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.413578 11440 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:59.430136  9524 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:59.458076  9524 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:49:59.436395  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.436549  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.403488 11447 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:49:59.399521 11443 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:59.405260 11458 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:59.404373 11415 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:59.410107 11466 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:59.462124 11466 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:59.462594 11466 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:59.445359  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.464780  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:49:59.448745  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45428.9us
I20260808 01:49:59.450479  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.453751 11377 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.431250 11160 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:59.455410 11392 raft_consensus.cc:3055] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:59.456954  9601 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:59.431798 11387 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:59.460147  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:59.472081  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.475538  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.477382  9600 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:59.478576  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:59.480880  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:59.481155 11378 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:59.485291  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:49:59.486855  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:59.460558 11450 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:59.459812 11447 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:59.464138 11458 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:59.473608 11440 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.462615 11464 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:49:59.503993 11464 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:59.476826 11160 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:59.511686  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:59.521147  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:59.522902 11445 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:59.523993  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:59.487703 11459 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:59.502874 11359 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:59.481514 11372 consensus_queue.cc:1059] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260808 01:49:59.465942 11415 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:59.476984 11392 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:49:59.480274 11387 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:59.491209 11452 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:59.536131  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:59.506434 11383 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:59.490296 11433 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:59.494808 11447 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:49:59.501250  9601 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:49:59.506114 11468 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260808 01:49:59.491833 11450 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:59.478818 11471 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260808 01:49:59.556517 11472 consensus_queue.cc:1059] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260808 01:49:59.558023 11372 raft_consensus.cc:3055] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:59.583823 11372 raft_consensus.cc:740] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:59.585096  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1753) took 1991 ms. Trace:
I20260808 01:49:59.584955 11372 consensus_queue.cc:260] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:59.585570  9474 rpcz_store.cc:276] 0808 01:49:57.594227 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:58.876278 (+1282051us) service_pool.cc:224] Handling call
0808 01:49:58.876980 (+   702us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:49:59.068196 (+191216us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:49:59.068237 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:49:59.079803 (+ 11566us) log.cc:844] Serialized 26 byte log entry
0808 01:49:59.285714 (+205911us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:49:59.285760 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:49:59.285822 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:49:59.584826 (+299004us) raft_consensus.cc:1685] finished
0808 01:49:59.584852 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:49:59.585023 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":30933,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":204,"thread_start_us":322560,"threads_started":2,"wal-append.queue_time_us":306766}
I20260808 01:49:59.575668 11450 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:59.536523  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:59.510589 11464 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:59.558184 11384 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:49:59.562748 11455 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:49:59.564616  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 83761us
W20260808 01:49:59.567413  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66280.2us
I20260808 01:49:59.480585 11394 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260808 01:49:59.546882 11467 tablet_copy_client.cc:323] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.194:36095
I20260808 01:49:59.576730  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:59.558104 11415 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:59.542524 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:49:59.603349 10360 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 2: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:49:59.604609 11390 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:49:59.605705  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:49:59.591722 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 198046us
I20260808 01:49:59.559894 11452 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:59.606817 11463 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.582144 11439 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:59.563654 11385 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:49:59.607523  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:59.568228 11445 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.613193 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:59.613452  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:59.609786  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:59.599272 11372 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:59.558758 11392 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:59.654609  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:49:59.608325 11452 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:49:59.657526  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46749.1us
I20260808 01:49:59.663936  9554 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3. Ops: []
I20260808 01:49:59.667963  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:59.668936 11395 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:49:59.674934  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:59.675937 11404 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260808 01:49:59.682902  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:49:59.498871 11448 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:59.616372 11390 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.583317 11462 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260808 01:49:59.653545 11444 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.653553 11471 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.689426  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:59.677436 11447 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:59.657238 11416 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.611713 11432 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20260808 01:49:59.683934 11412 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:59.615832 11385 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:59.612074 11435 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20260808 01:49:59.700124  9524 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:59.706766 11434 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:49:59.679549 11359 consensus_queue.cc:1059] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
W20260808 01:49:59.698856 11417 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.531s	user 0.005s	sys 0.008s
I20260808 01:49:59.693869 11392 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:59.725208 10367 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:49:59.691305 11308 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260808 01:49:59.736456  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:59.717106 11412 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:59.754086 11452 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:49:59.757601 11393 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:49:59.758123 11393 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:49:59.759264 11452 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:49:59.759013 11393 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:59.760456 11393 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:59.762379 11369 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:59.764597 11382 raft_consensus.cc:3055] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:59.765058 11382 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:49:59.766217 11382 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:59.770462  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:49:59.781764 10332 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:49:59.795861  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 59472.2us
I20260808 01:49:59.798262  9524 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:59.769362 11394 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.796762 11382 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:59.805401 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:49:59.806083 11480 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.813063 11378 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.831219 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:49:59.834465 11461 raft_consensus.cc:993] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: : Instructing follower 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:49:59.836809 11461 raft_consensus.cc:1081] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Signalling peer 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:49:59.844435  9600 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:59.855033  9600 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:49:59.798697 11433 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260808 01:49:59.828287 11437 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:49:59.833088 11439 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:59.809496 11417 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260808 01:49:59.844904 11377 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:49:59.860463 11366 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:49:59.859136 11432 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:49:59.860440 11452 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:49:59.853181 11160 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:59.875296 11437 raft_consensus.cc:697] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:49:59.897377 11377 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:49:59.923816  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 270606us
I20260808 01:49:59.925508 10367 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3. Ops: []
I20260808 01:49:59.927896  9556 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:59.907891 11439 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:49:59.905936 11447 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:49:59.901262 11366 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:49:59.851924 11466 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:59.930315  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:59.932020  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:59.947618  9629 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:59.948648  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:59.854226 11412 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:49:59.950994  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:59.952425  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:59.934860 11385 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:49:59.939325 11369 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:49:59.966063 11369 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:49:59.916530 11433 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:49:59.911525 11452 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:49:59.930099  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:59.933250  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:59.969067  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:59.925305 11378 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:49:59.953831  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:49:59.959479  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 281541us
I20260808 01:49:59.953639  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:49:59.970557  9638 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:49:59.972514  9635 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: []
I20260808 01:49:59.981769  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:49:59.990868  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 192774us
I20260808 01:49:59.991115  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:49:59.991871  9524 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:49:59.968700 11408 consensus_meta.cc:287] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.537s	user 0.003s	sys 0.010s
I20260808 01:49:59.996042  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:00.021370  9634 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
W20260808 01:50:00.017375  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:00.023275  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:00.023340  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:00.024489  9474 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:50:00.009564  9635 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:42840
I20260808 01:50:00.016091  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:00.026140 10334 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:00.026170  9635 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:50:00.027120  9635 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:49:59.971417 11433 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:00.035923  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 44127.1us
I20260808 01:50:00.040632  9524 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:49:59.971794 11452 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:50:00.042079  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 51659.6us
I20260808 01:49:59.976140 11456 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:00.060552  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:00.063747  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:00.064041  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:00.064349  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:49:59.981518 11378 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.043548 11439 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:00.033974 11417 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:00.071149  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:00.049019 11456 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:00.034751 11481 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:00.054257 11408 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:00.083702  9521 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:00.062896 11372 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:00.090397  9521 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:00.037798 11366 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:00.092193 11476 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20260808 01:50:00.028307 11377 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:00.088955  9601 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:00.089277  9631 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:50:00.096024 11433 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.072315 11417 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:00.130010  9635 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:00.136518  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47644.1us
I20260808 01:50:00.137843  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:00.141539 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 330634us
I20260808 01:50:00.048162 11437 consensus_queue.cc:237] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:00.118072 11458 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.611s	user 0.013s	sys 0.000s
I20260808 01:50:00.147712  9521 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:00.105602 11485 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:50:00.106447 11366 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:00.114746 11447 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:00.163177 11466 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:00.163520 11485 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:00.133509 11160 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:00.164613 11466 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:00.139474 11417 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:00.150089 11470 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:50:00.142666 11482 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:00.175720 11377 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:00.173316 11433 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:00.107352 11492 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.124s
I20260808 01:50:00.199565  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:00.200109  9571 tablet_copy_service.cc:140] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Received BeginTabletCopySession request for tablet affdd24b03ae4ee795a90703613305f7 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:60552)
I20260808 01:50:00.201300  9571 tablet_copy_service.cc:161] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Beginning new tablet copy session on tablet affdd24b03ae4ee795a90703613305f7 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:60552: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-affdd24b03ae4ee795a90703613305f7
W20260808 01:50:00.206849 11386 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.943s	user 0.015s	sys 0.002s
I20260808 01:50:00.212528  9554 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:50:00.156873 11458 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.226953  9557 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:00.228778 11390 raft_consensus.cc:3055] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:00.229604 11390 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:00.169687 11456 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.192632 11482 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:00.229648 11440 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:00.266110 11440 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:00.244175 11378 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:00.240163 11456 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:50:00.238606 11451 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.843s	user 0.014s	sys 0.001s
W20260808 01:50:00.248943 11443 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.778s	user 0.015s	sys 0.001s
I20260808 01:50:00.228869 11458 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:00.273427 11443 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:00.247686 11492 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Stepping down as leader of term 1
W20260808 01:50:00.263233 11450 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.674s	user 0.005s	sys 0.013s
W20260808 01:50:00.302884 11383 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.733s	user 0.019s	sys 0.005s
W20260808 01:50:00.301266 11387 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.732s	user 0.014s	sys 0.008s
I20260808 01:50:00.312853  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:00.329015  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 257623us
I20260808 01:50:00.338269  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:00.340191  9475 raft_consensus.cc:3055] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:00.340620  9475 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:00.347579  9632 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
W20260808 01:50:00.325369 11459 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.770s	user 0.008s	sys 0.011s
I20260808 01:50:00.305760 11492 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:00.359781 11480 raft_consensus.cc:3055] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:00.364794  9635 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:00.330529 11387 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:00.381541  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 233931us
I20260808 01:50:00.330197 11383 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:00.382534  9521 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:00.350988 11359 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.015s	sys 0.022s
I20260808 01:50:00.385630  9521 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:00.354069 11464 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.752s	user 0.017s	sys 0.005s
I20260808 01:50:00.390647  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:00.391018 11451 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:00.327894 11450 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:00.384029 11382 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.558s	user 0.012s	sys 0.006s
I20260808 01:50:00.383719 11459 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:00.395610  9636 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 2: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:00.394065 11451 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:00.396256  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:00.417377 11415 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.807s	user 0.010s	sys 0.008s
I20260808 01:50:00.314713 11390 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:00.382359 11480 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:50:00.429020  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:00.466948  9474 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:00.440780  9637 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
W20260808 01:50:00.388876 11359 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.655s	user 0.015s	sys 0.022s
I20260808 01:50:00.391515 11464 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:00.445940 11455 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.846s	user 0.015s	sys 0.004s
W20260808 01:50:00.441429  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 51856.1us
I20260808 01:50:00.464236 11486 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:00.453249 11387 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:00.470649  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 74456.2us
I20260808 01:50:00.468580 11377 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:00.386512 11448 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:00.250953 11482 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:50:00.490043  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:00.493566  9635 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:00.375492 11440 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:00.494601 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 308354us
I20260808 01:50:00.502368  9555 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:00.396044 11492 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.486310  9521 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:00.493264  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 428940us
I20260808 01:50:00.504418  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:50:00.505101  9552 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
W20260808 01:50:00.510814  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:00.511227  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:00.512671  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:00.476101 11451 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:00.466907 11415 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.467496 11417 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:00.524209  9632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 163) took 1547 ms (client timeout 1730 ms). Trace:
I20260808 01:50:00.529528 11470 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:50:00.532145 11434 raft_consensus.cc:3055] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:00.504861 11377 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:00.534011  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:00.534799  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:00.535218  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:00.507581 11503 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:00.517484 11451 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:00.568306  9481 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:00.536691  9524 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:00.483438 11486 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:00.539208  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:00.551002  9632 rpcz_store.cc:269] 0808 01:49:58.976875 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:59.112171 (+135296us) service_pool.cc:224] Handling call
0808 01:50:00.524025 (+1411854us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540008c854. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:00.524142 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2377094912}
I20260808 01:50:00.519311 11359 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:00.519052 11455 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:00.571544  9550 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: []
W20260808 01:50:00.571916 11500 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.239s	user 0.000s	sys 0.001s
I20260808 01:50:00.571838  9524 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:00.567960 11476 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:00.520982 11504 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:00.575258  9521 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:00.585336 11359 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:00.556073 11482 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:00.591502  9560 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:00.595000 11455 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.552431 11492 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:50:00.550472 11460 consensus_queue.cc:1175] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6e7856bed1be43acaa821290d506ee97" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20260808 01:50:00.573680  9556 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:00.555333 11434 raft_consensus.cc:740] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:50:00.571053 11452 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.521s	user 0.011s	sys 0.008s
I20260808 01:50:00.628578  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:00.624042  9629 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:50:00.638661  9635 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:00.657552 11486 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:00.579129 11451 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:00.622241 11464 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:00.664140  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:00.665535  9635 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:00.601254 11459 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:00.609416 11481 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:00.614287 11359 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:00.625676 11383 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:00.622833 11439 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:00.670145  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:00.622560 11377 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:00.657763  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:00.659224 11455 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:00.679056  9599 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:00.665035 11450 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:00.703011 11489 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260808 01:50:00.704295 11489 raft_consensus.cc:3055] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:00.704679 11489 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:00.670586 11432 consensus_queue.cc:1059] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20260808 01:50:00.707882  9481 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:00.708509  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:00.709139  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:00.709764  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:00.710047  9476 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:50:00.710341  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:00.708290  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:00.713338 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:00.714108 10366 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
W20260808 01:50:00.711887  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 187896us
I20260808 01:50:00.666823 11452 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:00.705209 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:00.676522 11470 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260808 01:50:00.587320 11503 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.716998  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:00.723726  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:00.737733  9472 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:00.729312  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:00.731617 11506 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:00.741578  9560 raft_consensus.cc:2393] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:00.694267 11436 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:50:00.625959 11476 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.747582  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:00.748905 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:00.714370 11390 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:00.751999  9472 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:00.684886 11383 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.751926  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:00.756345  9600 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:00.764447  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1660) took 1236 ms. Trace:
I20260808 01:50:00.764976  9637 rpcz_store.cc:276] 0808 01:49:59.527709 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:59.530213 (+  2504us) service_pool.cc:224] Handling call
0808 01:50:00.764212 (+1233999us) spinlock_profiling.cc:243] Waited 836 ms on lock 0x7b5400338954. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:00.764365 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1923921664}
I20260808 01:50:00.766357  9633 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: [1.2-1.2]
I20260808 01:50:00.748169 10361 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:00.718025 11450 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:00.757306  9635 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:00.596433 11480 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:00.781635  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 1843) took 1070 ms. Trace:
I20260808 01:50:00.616925 11504 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.774089 11415 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:00.759881 11481 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:00.712272 11489 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:00.779271 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 98142.4us
I20260808 01:50:00.779596  9478 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:00.767867 11495 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:00.778087 11450 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:00.757388 11455 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260808 01:50:00.807080  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:00.790036  9521 leader_election.cc:336] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.751s (SENT)
I20260808 01:50:00.742388 11416 consensus_queue.cc:1059] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260808 01:50:00.793792  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:00.782225  9552 rpcz_store.cc:276] 0808 01:49:59.711639 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:59.711791 (+   152us) service_pool.cc:224] Handling call
0808 01:49:59.712257 (+   466us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:00.509360 (+797103us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:00.781287 (+271927us) spinlock_profiling.cc:243] Waited 793 ms on lock 0x7b54001aca54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:00.781508 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1822948736}
I20260808 01:50:00.709838 11434 consensus_queue.cc:260] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.816295  9521 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:00.754333 11490 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260808 01:50:00.751420 11506 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:00.781988 11433 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.587s	user 0.013s	sys 0.009s
I20260808 01:50:00.789470 11439 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:00.825117 11439 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:00.800477 11369 consensus_meta.cc:287] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.833s	user 0.007s	sys 0.014s
I20260808 01:50:00.822484 11486 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:50:00.826562 11369 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.827659 11486 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:00.827728  9551 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:00.824354 11506 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:00.816591 11508 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.250s	user 0.000s	sys 0.002s
I20260808 01:50:00.797266 11436 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:00.829823  9550 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:00.799808 11417 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:00.825589 11504 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:00.798898 11495 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:00.828254 11486 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:00.838294 11439 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:00.838904  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:00.839545  9629 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:00.838215 11495 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.840471 11495 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:00.844353  9524 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:00.844902  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:00.845794  9474 raft_consensus.cc:2393] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:00.846113  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:00.846975  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 1831) took 1706 ms. Trace:
I20260808 01:50:00.847302  9637 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:00.847378  9557 rpcz_store.cc:276] 0808 01:49:59.140384 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:59.385227 (+244843us) service_pool.cc:224] Handling call
0808 01:49:59.516954 (+131727us) spinlock_profiling.cc:243] Waited 8.05 ms on lock 0x7b5800003c98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:49:59.517179 (+   225us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:00.227682 (+710503us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:00.846726 (+619044us) spinlock_profiling.cc:243] Waited 709 ms on lock 0x7b54003142d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:00.846911 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1649804800}
I20260808 01:50:00.848424  9560 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:00.851291  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:00.851581  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:00.851933 11479 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:00.852520 11479 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.857077 11476 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:00.798520 11384 consensus_queue.cc:1175] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "259bcb3f0d254e8b9effac3277e654e0" responder_term: 2 status { last_received { term: 2 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 2 index: 5 } }
I20260808 01:50:00.859748  9477 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
W20260808 01:50:00.832957 11456 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.559s	user 0.008s	sys 0.010s
I20260808 01:50:00.820724 11404 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:50:00.861486 11484 consensus_queue.cc:1059] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:50:00.865798 11452 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:00.836057 11460 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:00.819319  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:00.841941 11436 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:00.869549 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:00.876560  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:00.878389  9637 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:50:00.878474 11404 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:00.877419 11484 raft_consensus.cc:3055] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:00.881776 11484 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:00.882133 11404 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.885100 11445 raft_consensus.cc:3055] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:00.885604 11445 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:00.838215 11486 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.889997 11486 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:00.892374 11383 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:00.894502  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:00.872876 11417 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:00.888713 11445 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.895776 11417 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:00.896377 11445 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:00.898391 11417 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:00.901661  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:00.895318  9550 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:00.904122 11450 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:00.904266  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:00.906526 11369 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:00.879278  9524 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:00.911343 11369 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:00.911888  9475 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:00.914557 10367 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:00.915650 11394 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:00.919314 11394 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:00.921947 11394 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:00.879199 11439 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:50:00.888698  9571 tablet_copy_source_session.cc:215] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:50:00.923995 11439 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:00.924698  9372 catalog_manager.cc:5697] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 1 to 2, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 2 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:00.925940  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:00.902411  9557 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:00.916203 11479 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:00.933393  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:00.936717  9521 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:00.940459  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:00.943856  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:00.944651  9635 raft_consensus.cc:2376] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2. Re-sending same reply.
I20260808 01:50:00.945389  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:00.983314  9632 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:00.925637 11439 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.966308  9478 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:00.984843  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:00.986707  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:00.987550  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:00.988193  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:00.991487  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:00.992280  9557 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:00.992956  9521 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:00.911974 11369 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:01.004578 11369 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:00.998924 11433 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:00.999415 11437 consensus_queue.cc:1059] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:50:01.007450  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:01.008340 11437 raft_consensus.cc:3055] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:01.008590  9554 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:50:01.009057 11437 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:01.010916 11437 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:01.002938 11456 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:50:00.949690 11452 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:00.982774 11482 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:50:01.014716 11452 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:00.939245 11436 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:01.017329  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:01.017373  9599 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:01.019382 10334 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:01.023249 11450 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:50:01.026240  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:01.028203 11440 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:01.028796  9630 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:01.032799 11452 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:01.032639 11484 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:01.034267  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:01.035212 11480 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:01.035676  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:01.041054  9479 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0ea307d9adc442539683547d0176a05b"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:38182
I20260808 01:50:01.042006  9479 raft_consensus.cc:606] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:01.034312 11484 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:01.044198 11432 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:01.046274  9560 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
W20260808 01:50:01.048633  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 55975.5us
I20260808 01:50:01.012874 11437 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:01.055629  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:01.064035 11475 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:01.065694 11467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet affdd24b03ae4ee795a90703613305f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:50:01.044957 11432 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:01.051695  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:01.066915  9371 catalog_manager.cc:5697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 1 to 2, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 2 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:01.074090  9478 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:01.047873  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:01.077158 11416 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:01.077976 11416 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:01.078840 11416 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:01.079957 11489 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:01.079960 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:01.081738 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:01.082037 10367 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:01.082466 10366 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 2: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:01.055809 11445 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:01.089079  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:01.067500 11450 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260808 01:50:01.094383  9555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 1834) took 1945 ms (client timeout 2094 ms). Trace:
I20260808 01:50:01.096118  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:01.097273  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:01.072759 11459 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 01:50:01.096923  9555 rpcz_store.cc:269] 0808 01:49:59.141101 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:59.228188 (+ 87087us) service_pool.cc:224] Handling call
0808 01:50:01.085870 (+1857682us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b540008acd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:01.085983 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2787927936}
W20260808 01:50:01.104511  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:01.066485 11359 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:01.078356 11451 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:01.082006 10361 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:01.159523 10367 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:01.163429  9373 catalog_manager.cc:5697] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 1 to 2, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 2 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:50:01.140196  9555 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:50:01.167749 11432 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:01.168277 11507 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:50:01.169335 11432 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:01.171257  9559 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:01.173669 11434 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:01.100018  9630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:50:01.179699  9630 raft_consensus.cc:606] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:01.183498  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1490) took 1249 ms. Trace:
I20260808 01:50:01.184139  9633 rpcz_store.cc:276] 0808 01:49:59.933808 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:59.976640 (+ 42832us) service_pool.cc:224] Handling call
0808 01:50:00.765959 (+789319us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:00.767954 (+  1995us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:01.183257 (+415303us) spinlock_profiling.cc:243] Waited 789 ms on lock 0x7b5400338950. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:01.183425 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1813564800}
W20260808 01:50:01.187588  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 118610us
I20260808 01:50:01.187641  9474 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:50:01.189508 11448 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:01.190027 11448 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:01.141567  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 86918us
I20260808 01:50:01.193099  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:01.194870  9550 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:01.194989 10334 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:01.196413 11409 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:01.087016 11475 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:01.197034 11409 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:01.197706 11409 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:01.198096 11475 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:01.208231  9633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:50:01.208518  9524 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:01.209209  9633 raft_consensus.cc:606] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:01.208518  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:01.082617 11503 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:01.145802  9600 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:01.217984  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 72261.1us
I20260808 01:50:01.218933  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:01.105055  9521 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:01.149434  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:01.221884  9634 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:01.140610 11452 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:50:01.153775  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 60714.9us
I20260808 01:50:01.126736 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:01.147539  9632 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:50:01.223363 10360 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:01.223680  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:01.106848 11492 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:01.224359 11492 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:01.224499 11479 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:01.224838 11492 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:01.224992 11479 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:01.091380 11445 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:01.226132 11383 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:01.226444 11445 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:01.226713 11383 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:01.091883 11443 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:01.227285 11462 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:50:01.227787 11383 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:01.228488 11462 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:50:01.098897 11456 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:01.214075 11377 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:01.229678 11456 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:01.230160 11456 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:01.223068 11452 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:01.230849 11404 raft_consensus.cc:3055] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:01.231268 11404 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:01.232213 11404 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:01.232756 11452 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:01.233523 11404 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:01.235054  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:01.235160 11492 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:01.236202  9472 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 2: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:01.230536 11462 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:01.236828 11462 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:01.237679 11492 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:01.239521  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:01.240247  9632 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:01.242411 11462 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:01.244063  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:01.245119 11510 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:01.245712  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:01.246395  9634 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:01.157529  9475 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:01.247391  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:01.248960 11462 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:01.250370 11463 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:01.225638 11390 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:50:01.227973 11443 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:01.252596 11461 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:50:01.253237 11443 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:01.245878 11510 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:01.256417 11510 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:01.257259 11510 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:01.257890 11510 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:01.251067 11463 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:01.256407  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:01.254436 11438 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:01.260403 11463 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:01.260630 11438 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:01.261638 11438 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:01.262753 11438 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:01.265568  9475 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:01.249068 11479 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:01.268148  9474 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:01.278895 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:01.279677 10360 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:01.280184  9601 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:01.281993 11443 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:01.282753 11443 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:01.283349 11443 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:01.283597  9445 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:01.285393 11390 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:01.286036 11491 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:01.286324 11390 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:01.222993 11439 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260808 01:50:01.222412  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 117567us
I20260808 01:50:01.269631 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:01.291656 10367 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 2: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:01.293095  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:01.271485 11456 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:01.274523  9523 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:01.276079  9600 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:01.109982 11459 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:01.287464 11390 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:50:01.289912 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 65076.2us
I20260808 01:50:01.215279  9599 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:01.308125  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 55872.4us
I20260808 01:50:01.277420 11510 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:01.330586  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:01.320883  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:01.327375  9521 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:01.330385  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:01.101557 11369 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:01.317158 11377 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:50:01.351568  9633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
W20260808 01:50:01.339529  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 124370us
I20260808 01:50:01.288740 11450 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:01.369943  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:01.370450  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:01.370826  9558 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:01.328227 11440 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:01.370525  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:01.319694 11459 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:01.366811 11415 raft_consensus.cc:993] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: : Instructing follower 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:50:01.342203  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:01.287288 11491 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:01.312626 11384 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
I20260808 01:50:01.330070 11445 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:01.517040 11448 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:01.475698  9634 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:01.355556 11455 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:50:01.351651 11372 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:01.466231  9599 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:01.533890  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:01.553059  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:01.383924 11440 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:01.559012  9633 raft_consensus.cc:606] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:01.602622  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:01.377007 11450 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:01.539577  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:01.663837  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 197718us
I20260808 01:50:01.548213  9629 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:50:01.381191 11470 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:50:01.490760 11521 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260808 01:50:01.372903 11486 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.292s
I20260808 01:50:01.662933  9552 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
W20260808 01:50:01.721603 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 43908.8us
I20260808 01:50:01.534544  9558 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 2. (term mismatch)
I20260808 01:50:01.735388  9558 pending_rounds.cc:85] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 1
I20260808 01:50:01.735782  9558 pending_rounds.cc:107] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.2
I20260808 01:50:01.736035  9558 raft_consensus.cc:2917] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Aborting config change with OpId 1.2: Aborted: Op aborted by new leader
I20260808 01:50:01.709877  9475 raft_consensus.cc:2393] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:01.700537  9632 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3. Ops: []
W20260808 01:50:01.536727  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:01.426569 11359 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:01.558058 11369 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:01.514597 11460 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:01.502832 11394 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:01.738380  9633 raft_consensus.cc:614] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:50:01.523267  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:01.738114  9636 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:50:01.736428  9558 raft_consensus.cc:2889] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:01.544047 11451 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:01.544031 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:01.654414 11377 raft_consensus.cc:697] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:01.721279 11523 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:01.545439 11378 raft_consensus.cc:993] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:50:01.773993 11378 raft_consensus.cc:1081] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Signalling peer 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:50:01.745520 11440 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:01.775178  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:01.648726 11463 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:01.753299  9479 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:01.721423 11487 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:01.544128 11416 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:01.663142 11488 raft_consensus.cc:1081] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Signalling peer 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:50:01.676784 11372 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:01.539129 11389 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:01.755468 11451 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:01.771427 11522 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:01.558287 11390 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:01.772166 11377 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:01.473265 11495 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:01.779359  9478 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:50:01.742292 11450 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:01.780248  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:60546 (request call id 14) took 4767 ms. Trace:
I20260808 01:50:01.539778 11456 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:01.756585 11409 consensus_meta.cc:287] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.558s	user 0.008s	sys 0.018s
W20260808 01:50:01.745990 11404 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.512s	user 0.007s	sys 0.006s
I20260808 01:50:01.748338 11455 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:01.800202 11487 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:01.806589 11389 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:01.767617 11460 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:01.782657 11472 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:01.823549  9472 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:01.807488 11482 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:50:01.767884 11394 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:01.822197  9558 rpcz_store.cc:276] 0808 01:49:56.988990 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:56.989276 (+   286us) service_pool.cc:224] Handling call
0808 01:49:57.195845 (+206569us) log.cc:844] Serialized 506 byte log entry
0808 01:49:58.138156 (+942311us) spinlock_profiling.cc:243] Waited 28 ms on lock 0x7b5400285da0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44913f06 00007f5d4490d87a 00007f5d4490caf4 00007f5d448ed28a 00007f5d448f4133 00007f5d448f40b9 00007f5d448f4049 00007f5d448f4011 00007f5d448f31dd 00000000003aa554
0808 01:49:58.244774 (+106618us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b5400285da0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490c522 00007f5d448ed28a 00007f5d448f4133 00007f5d448f40b9 00007f5d448f4049 00007f5d448f4011 00007f5d448f31dd 00000000003aa554 00000000003aa399 00007f5d4188cdb5
0808 01:49:58.320144 (+ 75370us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b5400285da0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490c522 00007f5d448ed28a 00007f5d448f4133 00007f5d448f40b9 00007f5d448f4049 00007f5d448f4011 00007f5d448f31dd 00000000003aa554 00000000003aa399 00007f5d4188cdb5
0808 01:49:58.915876 (+595732us) spinlock_profiling.cc:243] Waited 4.9 ms on lock 0x7b4c0000b980. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d42c5cfc0 00007f5d42c6dfa9 00007f5d42c6e68f 00007f5d42c31e7b 00007f5d42c604dc 00007f5d42c6099e 00007f5d42ef066b 00007f5d448ef761 00007f5d448ed64b 00007f5d448f4133 00007f5d448f40b9
0808 01:49:59.186204 (+270328us) spinlock_profiling.cc:243] Waited 51.2 ms on lock 0x7b4c00248640. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d42c5cfc0 00007f5d42c6dfa9 00007f5d42c6e68f 00007f5d42c31e7b 00007f5d42c604dc 00007f5d42c6099e 00007f5d42ef066b 00007f5d448ef761 00007f5d448ed64b 00007f5d448f4133 00007f5d448f40b9
0808 01:50:01.756428 (+2570224us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50283,"raft.queue_time_us":28621,"raft.run_cpu_time_us":65086,"raft.run_wall_time_us":4058434,"spinlock_wait_cycles":256362496,"thread_start_us":185913,"threads_started":1,"wal-append.queue_time_us":206879,"wal-append.run_cpu_time_us":3182,"wal-append.run_wall_time_us":1051446}
I20260808 01:50:01.817430 11463 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:01.814611 11451 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:01.845695 11503 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:01.947021  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:01.948724  9629 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
W20260808 01:50:01.939266  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 74264.4us
I20260808 01:50:01.955992 11451 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:01.956290 11460 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:01.956434 11524 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.300s	user 0.000s	sys 0.001s
I20260808 01:50:01.854027 11455 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:01.827555 11409 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:01.958977 11394 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:01.865537 11436 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.798s	user 0.008s	sys 0.005s
I20260808 01:50:01.844718 11480 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.102s
I20260808 01:50:01.822821 11495 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:01.939177 11437 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:01.990926  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 215816us
I20260808 01:50:01.999634  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:01.950438 11482 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:01.846726 11522 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.011018  9521 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:01.960285 11389 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:01.993047 11479 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.715s	user 0.008s	sys 0.008s
I20260808 01:50:01.980268  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:02.016518 10367 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:50:02.025457  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:01.862819 11472 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.032629  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:01.980263 11372 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.002787 11445 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:02.063129  9633 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
W20260808 01:50:02.049686 11443 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.765s	user 0.014s	sys 0.015s
I20260808 01:50:02.030027 11486 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:02.053884 11369 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:02.054183  9630 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:02.035912 11497 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:02.076321 11390 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:01.997107 11436 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:02.117599 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:02.007319 11437 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:02.020246 11460 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:50:02.030426 11479 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:01.971715 11503 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.038697 11487 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.031590 11448 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:02.063971 11378 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:02.030773 11491 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:02.028620 11451 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:02.135308 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 133423us
I20260808 01:50:02.037238 11470 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.085s
W20260808 01:50:02.117187 11518 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.693s	user 0.000s	sys 0.009s
I20260808 01:50:02.107030 11416 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.082412 11459 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.139891  9637 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:02.108978 11443 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.147636 11440 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:02.130448 11448 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:02.131821 11527 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.318s	user 0.002s	sys 0.000s
W20260808 01:50:02.155712 11432 consensus_meta.cc:287] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.986s	user 0.015s	sys 0.008s
W20260808 01:50:02.151422 11518 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.002s	sys 0.009s
I20260808 01:50:02.111598 11486 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:02.118178 11369 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.137161 11409 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:02.172617 11459 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.173604 11437 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.175336 11522 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:02.147748 11455 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.147832 11482 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:02.122272 11497 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:02.182878 11436 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:02.185124 11443 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:02.179019 11479 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.136914 11378 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:02.160337 11448 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:02.191915 11495 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:02.192617 11450 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.200852 11450 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.142779 11451 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:02.200991 11503 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.204703 11451 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:02.208621 11452 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:02.211126 11452 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:02.212014 11452 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.198650 11497 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.208709 11486 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:02.217725 11472 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:02.272524  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet affdd24b03ae4ee795a90703613305f7: Aborted: Op aborted by new leader
I20260808 01:50:02.103231  9555 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:53208
I20260808 01:50:02.276295  9555 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:50:02.277382  9555 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:50:02.285307  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:02.286357  9560 raft_consensus.cc:2368] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4.
W20260808 01:50:02.298228  9598 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.720s	user 0.007s	sys 0.010s
W20260808 01:50:02.299050  9598 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.910s	user 0.007s	sys 0.011s
W20260808 01:50:02.301553  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 50fd024764ea49c7a1f16e992e7599f3 (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:50:02.303618  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260808 01:50:02.310524  9472 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:50:02.312875  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:02.315588  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:02.316380  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:02.316681  9478 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:02.317826  9479 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:02.310060  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:02.319257  9475 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:02.322815  9477 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:02.314244  9474 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:02.324702  9629 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:02.326632  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1507) took 1382 ms. Trace:
I20260808 01:50:02.329381  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:02.331930  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:02.332901  9477 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:02.333209  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:02.329437  9637 rpcz_store.cc:276] 0808 01:50:00.943944 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:00.944220 (+   276us) service_pool.cc:224] Handling call
0808 01:50:02.326138 (+1381918us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400339354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:02.326393 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2646406016}
I20260808 01:50:02.335528  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:02.336691  9637 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 4
I20260808 01:50:02.333245  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:02.339859  9636 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:02.345387  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:02.328490  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:02.348587  9634 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:02.330621  9635 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:50:02.364957  9633 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:02.322528  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:50:02.361285 11416 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:02.373049 11416 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.374967 11451 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:02.376042 11451 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.328588  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:02.391494  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:02.398607 11369 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.405089  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:02.410079  9480 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:50:02.411136 11452 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:02.412134 11440 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:02.412642 11547 log.cc:927] Time spent T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
I20260808 01:50:02.419885  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:02.421236  9559 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:02.424290  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 140379us
I20260808 01:50:02.425096  9599 leader_election.cc:400] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4.
I20260808 01:50:02.425575  9599 leader_election.cc:403] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:50:02.426761 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:02.427685 10369 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:02.426353 11390 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:02.428076 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:02.428745 11390 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.429129 10366 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:02.431366 11390 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.441996 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:02.442051 11482 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:02.443975 11416 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.442229 11369 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:02.446913 11440 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:02.447414  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:02.448024 11440 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:02.450170 11448 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.455416  9524 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:02.455513  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:02.456981  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:02.457428 11448 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.458014  9557 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:02.459110 11377 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:02.448627  9554 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:02.459873 11377 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.446230 11369 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.443904 11482 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.457016  9556 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 2: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:02.469846  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:02.474578 11470 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:02.475924 11549 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.249s	user 0.000s	sys 0.001s
I20260808 01:50:02.476243 11467 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:56276 (request call id 307) took 3382 ms. Trace:
I20260808 01:50:02.477285  9629 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:02.493202  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:02.494140 11440 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:02.494369  9551 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:02.475844 11470 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.489073  9555 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.500026 10369 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:02.500689  9636 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:02.504760  9550 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:02.505887  9479 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:02.507114 11491 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:02.507879  9630 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:02.507965 11491 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.509312 11503 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:02.510895 11451 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.511381 11529 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:02.511579  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 85974.3us
I20260808 01:50:02.510948 11443 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:02.512436  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:02.512701  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 187003us
W20260808 01:50:02.512887  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 88268us
I20260808 01:50:02.512976 11443 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:02.515050  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 80790us
I20260808 01:50:02.515662 11389 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260808 01:50:02.517277  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:02.517655 11389 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:02.518198 11389 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:02.518636 11389 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:02.515888  9600 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:02.523716  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:02.524780  9551 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:02.515075  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:02.527757  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:02.528869  9521 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:02.529505  9630 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
W20260808 01:50:02.530812  9523 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.711s	user 0.002s	sys 0.027s
W20260808 01:50:02.531250  9523 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.002s	sys 0.027s
I20260808 01:50:02.531086  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:02.531898  9472 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
W20260808 01:50:02.533075  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 739904us
I20260808 01:50:02.534899  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:02.535862  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47028.8us
I20260808 01:50:02.537559 11434 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:02.538173 11434 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:02.538900  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:02.538594 11436 consensus_queue.cc:1059] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:50:02.535000 11440 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:02.539991  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:02.564275  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 48535.8us
I20260808 01:50:02.586073  9477 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:02.539764  9638 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:50:02.539381  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:02.474632 11462 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:02.589236  9555 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.569790  9599 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:02.500391 11377 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:02.538668 11434 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:02.582373 11440 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:02.576749 11491 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:02.543215 10361 consensus_meta.cc:287] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.749s	user 0.013s	sys 0.013s
I20260808 01:50:02.565007 11522 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:02.555406  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:02.599242  9637 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:02.598376  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:02.545651 11451 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:02.653148 11451 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:02.653807 11451 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:02.585637 11482 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.659121  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:02.660427  9474 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:02.628516 11479 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260808 01:50:02.608409 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:02.609128 11384 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:02.610174 11372 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:02.610653 11448 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:50:02.612190 11432 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:02.512182 11529 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.564769 11486 raft_consensus.cc:3055] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:02.615818 11450 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:02.615885  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 85502.8us
I20260808 01:50:02.617286 11531 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:02.617754 10360 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260808 01:50:02.616750  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 47147.8us
I20260808 01:50:02.594861 11416 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:02.619455 10361 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:02.619248 10334 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:02.623575 11470 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:02.624402  9600 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:02.626885 11369 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:02.651913 11523 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.855s	user 0.019s	sys 0.012s
I20260808 01:50:02.652361  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:02.666448  9524 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:02.667389 11539 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4.
I20260808 01:50:02.476855 11467 rpcz_store.cc:276] 0808 01:49:59.093662 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:59.093880 (+   218us) service_pool.cc:224] Handling call
0808 01:49:59.459344 (+365464us) ts_tablet_manager.cc:934] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
0808 01:50:01.063234 (+1603890us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:50:02.160615 (+1097381us) tablet_metadata.cc:665] Metadata flushed
0808 01:50:02.168456 (+  7841us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:50:02.474190 (+305734us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:50:02.475171 (+   981us) tablet_replica.cc:206] Creating consensus instance
0808 01:50:02.475755 (+   584us) ts_tablet_manager.cc:989] Replica registered.
0808 01:50:02.475927 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":34741,"tablet-copy.queue_time_us":287399,"thread_start_us":109376,"threads_started":2}
I20260808 01:50:02.547276 11409 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:02.671891  9632 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:02.564460 11436 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:02.671092  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 72807.2us
I20260808 01:50:02.603690 11440 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:02.678745  9479 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:02.683550  9557 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
W20260808 01:50:02.710413 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 110658us
I20260808 01:50:02.687196 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:02.676448 11377 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:02.670526 11522 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:02.704972  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:02.627269 11378 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:02.684541 11486 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:02.682179 10362 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:02.605008 11462 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:02.721138  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 120057us
W20260808 01:50:02.722191  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 120311us
I20260808 01:50:02.726770 11369 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:02.726333 11436 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.727311 11463 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:02.727450 11369 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:02.728255 11369 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:02.729313 11467 tablet_copy_client.cc:806] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:50:02.729434 11436 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:02.730232 11467 tablet_copy_client.cc:670] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:50:02.712347 11523 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:02.732812 11488 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:02.733608  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45693us
I20260808 01:50:02.728312 11463 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.735486  9599 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:02.735841  9631 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:50:02.674023 11384 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:02.711030 11470 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:02.734313 11529 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:02.768819  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47250.3us
I20260808 01:50:02.709867 11416 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.713222 11377 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:02.772419 10366 raft_consensus.cc:2368] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4.
I20260808 01:50:02.716657 11522 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:02.709048  9630 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:02.737710 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 331) took 1172 ms. Trace:
I20260808 01:50:02.695257 11450 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.789603 11479 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:02.789858 11541 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:02.784111  9601 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:02.803504  9550 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:50:02.805276  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:02.805927  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:02.806610  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:02.807806 11437 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:02.796641 11452 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:02.808437  9372 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50068 (request call id 236) took 2398 ms. Trace:
I20260808 01:50:02.809141  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:02.809912  9481 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:02.710712 11432 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.808043 11486 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:02.727578 11372 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.778180 11461 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:02.809037 11452 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:02.802736  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:02.792261 11377 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:02.829656 11495 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:02.829950 11550 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:02.810184 11463 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.830873 11543 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:02.796099 11531 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:02.833791  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:02.834558 11378 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:02.778620 11409 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:02.733635 11488 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.840597  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:02.844568  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:02.850935 11482 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:02.850708 11461 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.894397 11482 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.895153 11464 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:02.895465 11432 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:02.895809 11464 raft_consensus.cc:697] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:02.896133 11432 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:50:02.896526 11461 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.897059 11482 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.897742 11550 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.791189 10361 rpcz_store.cc:276] 0808 01:50:01.543536 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:01.543727 (+   191us) service_pool.cc:224] Handling call
0808 01:50:02.710750 (+1167023us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:02.851289 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:02.853695 11448 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:02.855278 11369 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.838300 11452 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:02.860109 11416 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.847790 11462 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.864940  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:02.876060  9480 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:02.936375  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:02.864149 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:02.892421  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:02.851466 11372 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.823239 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:02.911402 11565 raft_consensus.cc:3055] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:02.913484 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:02.916042 11438 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Stepping down as leader of term 2
W20260808 01:50:02.910487  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 114587us
I20260808 01:50:02.928897  9481 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:02.933182 11455 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:02.933017 11448 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.959450  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:02.818569 11541 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:02.934152 11495 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.935894  9472 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:02.885221 11436 raft_consensus.cc:993] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:50:02.974869  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:02.943593  9633 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:02.937338  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 153353us
I20260808 01:50:02.959578 11438 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:02.839816 11479 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.981103  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:02.925511 11437 raft_consensus.cc:697] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:02.897091 11464 consensus_queue.cc:237] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.924206 11451 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:02.876866 11531 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:02.888706 11543 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:02.916671  9637 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0ea307d9adc442539683547d0176a05b"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:42840
I20260808 01:50:02.933686 11557 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:02.963527 11559 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:02.956977  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:02.966993  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:02.975147 11436 raft_consensus.cc:1081] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Signalling peer 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:50:02.936815 11378 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:02.950232  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 87429.5us
I20260808 01:50:02.809517  9372 rpcz_store.cc:276] 0808 01:50:00.374430 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:00.374570 (+   140us) service_pool.cc:224] Handling call
0808 01:50:02.771397 (+2396827us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:50:01.015993 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:50:01.042310 (+ 26317us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:50:01.042372 (+    62us) write_op.cc:435] Acquired schema lock
0808 01:50:01.042398 (+    26us) tablet.cc:662] Decoding operations
0808 01:50:01.043054 (+   656us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:50:01.043174 (+   120us) write_op.cc:641] Partition lock acquired for write op
0808 01:50:01.043224 (+    50us) tablet.cc:685] Acquiring locks for 1 operations
0808 01:50:01.043654 (+   430us) tablet.cc:701] Row locks acquired
0808 01:50:01.043672 (+    18us) write_op.cc:260] PREPARE: finished
0808 01:50:01.043910 (+   238us) write_op.cc:270] Start()
0808 01:50:01.044067 (+   157us) write_op.cc:276] Timestamp: P: 1786153801043859 usec, L: 0
0808 01:50:01.044087 (+    20us) op_driver.cc:348] REPLICATION: starting
0808 01:50:01.094187 (+ 50100us) log.cc:844] Serialized 541 byte log entry
0808 01:50:02.171544 (+1077357us) op_driver.cc:464] REPLICATION: finished
0808 01:50:02.222765 (+ 51221us) write_op.cc:301] APPLY: starting
0808 01:50:02.223228 (+   463us) tablet.cc:1367] starting BulkCheckPresence
0808 01:50:02.223520 (+   292us) tablet.cc:1370] finished BulkCheckPresence
0808 01:50:02.225024 (+  1504us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:50:02.383109 (+158085us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:50:02.383682 (+   573us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 01:50:02.383811 (+   129us) write_op.cc:312] APPLY: finished
0808 01:50:02.434180 (+ 50369us) log.cc:844] Serialized 23 byte log entry
0808 01:50:02.434574 (+   394us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:02.434765 (+   191us) write_op.cc:454] Released schema lock
0808 01:50:02.434858 (+    93us) write_op.cc:341] FINISH: Updating metrics
0808 01:50:02.435666 (+   808us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:02.435711 (+    45us) write_op.cc:454] Released schema lock
0808 01:50:02.586349 (+150638us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:02.589446 (+  3097us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":37753,"apply.run_cpu_time_us":4941,"apply.run_wall_time_us":213602,"num_ops":1,"prepare.queue_time_us":82979,"prepare.run_cpu_time_us":27102,"prepare.run_wall_time_us":305761,"raft.queue_time_us":963450,"raft.run_cpu_time_us":48547,"raft.run_wall_time_us":428109,"replication_time_us":1127194,"spinlock_wait_cycles":207616,"thread_start_us":966179,"threads_started":4,"wal-append.queue_time_us":194537}]]}
W20260808 01:50:03.066816  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 122373us
I20260808 01:50:02.987779 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:03.064284  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:03.068581  9481 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:50:03.066411  9478 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:03.070024  9637 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:50:03.000898 10362 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:03.070524  9637 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:02.956303 11565 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:03.104480  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:03.107183  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:03.000738 11550 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:02.981441 11479 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:03.116492 11550 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:03.117151 11550 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:03.000804 11384 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:03.120639 11369 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:03.120980 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 136300us
I20260808 01:50:03.121498 11369 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:03.122118 11369 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:03.123174 11562 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:03.123682 11562 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:03.124039 11562 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:03.124477 11543 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:03.126135  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:03.127293  9476 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:03.126976 11437 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:03.128064 11566 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:03.128878 11566 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:03.131702 11488 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:03.133097 11378 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.135205 11568 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:03.135967 11432 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:03.063894 11486 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:03.000777 11450 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:03.137696  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 66759.4us
I20260808 01:50:02.983309 11503 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:03.040933 11409 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:03.122177 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:03.136684 11432 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:03.154938 11565 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.156643 11565 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:03.156891 11432 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:03.157706 11559 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:03.136011 11568 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.160945 11559 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:03.183988 11559 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:03.138480  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:03.184499 11559 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:03.184868  9477 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:50:03.184994 11559 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:03.146070  9551 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:03.187812 11550 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:03.143003 11557 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.147149  9472 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:50:02.999809  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:50:03.158651 11452 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.191104  9371 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50064 (request call id 234) took 2475 ms. Trace:
I20260808 01:50:03.148710  9480 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:03.149987  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:03.076220  9630 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:50:03.153786 11451 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:03.154109  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:03.163172  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:03.177564  9373 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55058 (request call id 123) took 2192 ms. Trace:
I20260808 01:50:03.191854 11436 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:03.193424 10334 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:03.142200  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 185304us
I20260808 01:50:03.196022  9633 raft_consensus.cc:2393] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:03.055745 11470 raft_consensus.cc:697] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:03.249126  9551 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
W20260808 01:50:03.196731 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 74760.6us
I20260808 01:50:03.165810  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:03.229853  9632 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:03.175608  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:03.002987 11455 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.195269 11443 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:03.255946 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 62658.1us
I20260808 01:50:03.387051 11560 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:03.402993 11560 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.180579 11438 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.404625 11438 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:03.405815 11560 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:03.355654  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 255499us
I20260808 01:50:03.397277  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1521) took 1373 ms. Trace:
I20260808 01:50:03.354600 11436 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.410706 11550 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:03.416069  9475 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:03.173156 11440 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:03.411072 11456 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:03.305688 11409 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 01:50:03.182667 11434 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.510s	user 0.003s	sys 0.007s
I20260808 01:50:03.150300 11503 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:03.412055  9521 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:03.417254  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1822) took 1002 ms. Trace:
I20260808 01:50:03.283025  9637 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.414779  9631 rpcz_store.cc:276] 0808 01:50:02.024231 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:02.025573 (+  1342us) service_pool.cc:224] Handling call
0808 01:50:02.033346 (+  7773us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:02.772611 (+739265us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:03.396935 (+624324us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x7b54001b44d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:03.397174 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1612949248}
W20260808 01:50:03.387475 11522 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.568s	user 0.008s	sys 0.005s
W20260808 01:50:03.419983  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 93232us
I20260808 01:50:03.148293 11541 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:03.195675  9371 rpcz_store.cc:276] 0808 01:50:00.715470 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:00.715623 (+   153us) service_pool.cc:224] Handling call
0808 01:50:03.190879 (+2475256us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:50:01.387364 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:50:01.490210 (+102846us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:50:01.490264 (+    54us) write_op.cc:435] Acquired schema lock
0808 01:50:01.490297 (+    33us) tablet.cc:662] Decoding operations
0808 01:50:01.490882 (+   585us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:50:01.490964 (+    82us) write_op.cc:641] Partition lock acquired for write op
0808 01:50:01.491007 (+    43us) tablet.cc:685] Acquiring locks for 1 operations
0808 01:50:01.491246 (+   239us) tablet.cc:701] Row locks acquired
0808 01:50:01.491268 (+    22us) write_op.cc:260] PREPARE: finished
0808 01:50:01.491476 (+   208us) write_op.cc:270] Start()
0808 01:50:01.491603 (+   127us) write_op.cc:276] Timestamp: P: 1786153801491396 usec, L: 0
0808 01:50:01.491626 (+    23us) op_driver.cc:348] REPLICATION: starting
0808 01:50:01.492443 (+   817us) log.cc:844] Serialized 541 byte log entry
0808 01:50:02.186004 (+693561us) op_driver.cc:464] REPLICATION: finished
0808 01:50:02.673514 (+487510us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b18002747e0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:50:02.673792 (+   278us) write_op.cc:301] APPLY: starting
0808 01:50:02.673990 (+   198us) tablet.cc:1367] starting BulkCheckPresence
0808 01:50:02.675909 (+  1919us) tablet.cc:1370] finished BulkCheckPresence
0808 01:50:02.677267 (+  1358us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:50:02.677642 (+   375us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:50:02.677850 (+   208us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 01:50:02.677900 (+    50us) write_op.cc:312] APPLY: finished
0808 01:50:02.707680 (+ 29780us) log.cc:844] Serialized 23 byte log entry
0808 01:50:02.741226 (+ 33546us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:02.781974 (+ 40748us) write_op.cc:454] Released schema lock
0808 01:50:02.789115 (+  7141us) write_op.cc:341] FINISH: Updating metrics
0808 01:50:02.824410 (+ 35295us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:02.827263 (+  2853us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":377246,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":39928,"apply.run_cpu_time_us":10684,"apply.run_wall_time_us":141593,"num_ops":1,"prepare.queue_time_us":275587,"prepare.run_cpu_time_us":3934,"prepare.run_wall_time_us":119413,"replication_time_us":694162,"spinlock_wait_cycles":386173184,"thread_start_us":38054,"threads_started":1}]]}
I20260808 01:50:03.383335 11570 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:03.426079  9524 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:03.436960  9374 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50084 (request call id 230) took 1917 ms. Trace:
I20260808 01:50:03.437820 11434 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.439114  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:03.254323  9373 rpcz_store.cc:276] 0808 01:50:00.984705 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:00.984874 (+   169us) service_pool.cc:224] Handling call
0808 01:50:03.177316 (+2192442us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:50:01.492952 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:50:01.493555 (+   603us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:50:01.493595 (+    40us) write_op.cc:435] Acquired schema lock
0808 01:50:01.493608 (+    13us) tablet.cc:662] Decoding operations
0808 01:50:01.494022 (+   414us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:50:01.494089 (+    67us) write_op.cc:641] Partition lock acquired for write op
0808 01:50:01.498815 (+  4726us) tablet.cc:685] Acquiring locks for 1 operations
0808 01:50:01.505174 (+  6359us) tablet.cc:701] Row locks acquired
0808 01:50:01.505202 (+    28us) write_op.cc:260] PREPARE: finished
0808 01:50:01.505405 (+   203us) write_op.cc:270] Start()
0808 01:50:01.505543 (+   138us) write_op.cc:276] Timestamp: P: 1786153801505348 usec, L: 0
0808 01:50:01.505569 (+    26us) op_driver.cc:348] REPLICATION: starting
0808 01:50:01.506348 (+   779us) log.cc:844] Serialized 536 byte log entry
0808 01:50:02.392599 (+886251us) op_driver.cc:464] REPLICATION: finished
0808 01:50:02.436306 (+ 43707us) write_op.cc:301] APPLY: starting
0808 01:50:02.436468 (+   162us) tablet.cc:1367] starting BulkCheckPresence
0808 01:50:02.436690 (+   222us) tablet.cc:1370] finished BulkCheckPresence
0808 01:50:02.436805 (+   115us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:50:02.437192 (+   387us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:50:02.437376 (+   184us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 01:50:02.437435 (+    59us) write_op.cc:312] APPLY: finished
0808 01:50:02.437671 (+   236us) log.cc:844] Serialized 23 byte log entry
0808 01:50:02.437886 (+   215us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:02.437981 (+    95us) write_op.cc:454] Released schema lock
0808 01:50:02.438037 (+    56us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":220634,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":43351,"apply.run_cpu_time_us":2273,"apply.run_wall_time_us":37013,"num_ops":1,"prepare.queue_time_us":317311,"prepare.run_cpu_time_us":3111,"prepare.run_wall_time_us":13750,"replication_time_us":886639,"thread_start_us":110442,"threads_started":1}]]}
I20260808 01:50:03.440413  9523 leader_election.cc:400] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4.
I20260808 01:50:03.440347  9521 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:03.445502  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:03.447019  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:03.448550 11451 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:03.456558  9477 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:50:03.456670  9480 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:03.448964  9556 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:03.439179  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:03.430650  9475 rpcz_store.cc:276] 0808 01:50:02.414732 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:02.423276 (+  8544us) service_pool.cc:224] Handling call
0808 01:50:03.165593 (+742317us) spinlock_profiling.cc:243] Waited 40.6 ms on lock 0x7b5800005498. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:50:03.417046 (+251453us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":93430016}
I20260808 01:50:03.163408 11472 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:03.176510 11568 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:03.167146 11566 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:03.388355 11479 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:03.163949 11487 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.309s
I20260808 01:50:03.431280 11409 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:03.458063  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:03.415648 11452 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:03.482589  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:03.429823 11389 consensus_meta.cc:287] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.911s	user 0.012s	sys 0.004s
I20260808 01:50:03.454751  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:03.483803 11432 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:03.483562 11389 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.484416 11432 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.485509 11451 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:03.485970 11389 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:03.455674  9374 rpcz_store.cc:276] 0808 01:50:01.519439 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:01.519579 (+   140us) service_pool.cc:224] Handling call
0808 01:50:03.432836 (+1913257us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":187506,"reported_tablets":1}
I20260808 01:50:03.477494  9601 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:03.489056  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 48845.2us
I20260808 01:50:03.487564  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:03.172274 11495 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:03.461917  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:03.462119  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:03.462267  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:03.493072  9633 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:03.457702  9472 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:03.463148  9521 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:50:03.452767 11377 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.575s	user 0.012s	sys 0.000s
I20260808 01:50:03.288659 11557 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:03.472306  9637 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:03.475907  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:03.476898  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:03.499225  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:03.481110 11482 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:03.481297  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:03.476572 11523 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:03.483999  9630 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:03.164294 10360 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:03.459430  9474 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:50:03.453552 11443 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:03.493001  9478 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:03.454077 11550 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.486713  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:03.512837  9631 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:03.498319  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 72405.6us
I20260808 01:50:03.486655 11432 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:03.514354  9524 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:03.168290 11369 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.499953  9555 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:03.501107  9558 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:53208
I20260808 01:50:03.520202  9558 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:50:03.520777  9558 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:03.504513  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:03.508906  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 45842.4us
I20260808 01:50:03.521682  9554 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:03.523401  9480 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:03.488121 11409 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:03.490819 11448 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:03.525708  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:03.503403 11482 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:03.524549  9523 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:03.454226 11522 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:03.531224  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:03.528599  9551 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:03.536705  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:03.492461 11568 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:03.518313 11523 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.476516 11472 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.527431  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:03.527079 11581 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:03.467010 11456 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.502418 11377 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.505260 11452 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:03.567073  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:03.504848 11570 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.542395 11482 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:03.494867 11560 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:03.534571  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:03.579931  9634 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:50:03.581466  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:03.547652  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:03.583506  9629 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:03.548527  9599 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:03.507344 11479 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:03.534811  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:03.550523  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:03.510849 11450 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:03.591213  9551 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:03.555148  9557 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:50:03.594949 11557 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:03.595459 11557 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:03.595624  9523 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:03.530424 11443 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.595805  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:03.546396 11575 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:03.558354  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:03.600919  9554 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:03.598790  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:03.563623  9524 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:03.486589 11451 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.537758  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:03.533931  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:03.553843 11464 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:03.605438  9556 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
W20260808 01:50:03.575870 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46403us
I20260808 01:50:03.570883 11452 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:03.573747  9633 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:03.591048  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:03.609781  9559 raft_consensus.cc:2393] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:03.558717  9560 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:03.596796  9550 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:03.570557 11581 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:50:03.615077  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 51534.4us
I20260808 01:50:03.584652  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:03.595855 11557 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:03.563696 11568 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:03.603353 11575 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:03.640174  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:03.640784  9481 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 3
I20260808 01:50:03.604001  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:03.642549  9554 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:50:03.604235  9551 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:03.603111  9634 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:50:03.601153  9552 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:03.597591  9630 raft_consensus.cc:2368] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3.
I20260808 01:50:03.644501  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:03.609957 11440 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:03.624243  9599 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:03.625910  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:03.588168  9558 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:03.615095  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 137698us
I20260808 01:50:03.650162  9599 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:03.650627  9555 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:03.639236  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:03.612406  9557 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 2: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:03.610991 11448 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:03.601714 11531 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:03.611189 11452 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:03.645735  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:03.628134 11450 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:50:03.653707 11497 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:03.621443 11567 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:03.589123 11559 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:03.600569 11566 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260808 01:50:03.650851  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:03.652246  9476 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:50:03.660961  9599 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:03.600539 11560 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.567123 11470 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:03.655069  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:03.674223  9479 raft_consensus.cc:2368] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4.
W20260808 01:50:03.649984 11378 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.019s	sys 0.015s
I20260808 01:50:03.644606 11575 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:03.625160 11578 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:03.671197  9523 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:03.676913  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:03.662022 11455 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:03.678795 11467 tablet_copy_client.cc:538] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:50:03.677597  9478 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:03.687093  9601 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:03.691876  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:03.693045  9480 raft_consensus.cc:2393] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:03.671768 11436 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:03.679769 11550 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:03.677991 11378 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.020s	sys 0.015s
W20260808 01:50:03.679854 11562 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.555s	user 0.007s	sys 0.005s
I20260808 01:50:03.812074  9558 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:03.683652 11580 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
W20260808 01:50:03.804083  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:03.816301  9629 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:03.829409  9632 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4. Ops: []
W20260808 01:50:03.829907  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 169071us
I20260808 01:50:03.831990  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:03.665549 11432 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:03.698014  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:03.834239  9481 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:03.677872 11434 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:03.685041 11570 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:03.688151 11578 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:03.838430  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 149327us
I20260808 01:50:03.839761  9599 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:03.840438  9601 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:03.695891 11495 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:03.822512 11531 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:03.697610 11539 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:03.774350 11448 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:03.683808 11472 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:03.729393 11559 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:03.685477 11479 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:03.848412  9601 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:03.845016 11495 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:03.697377 11450 raft_consensus.cc:697] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:50:03.888587  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:03.937888  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:03.845103 11578 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:03.836724 11522 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:03.773366 11567 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:03.852612 11531 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:03.680202 11497 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:03.841872  9556 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:50:03.845359 11378 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260808 01:50:03.900276  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:03.848790 11523 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:03.845192 11455 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:03.938666  9629 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:50:03.683189 11581 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.894483 11443 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:03.886304 11539 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:03.940825  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:03.983232  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:03.983688  9481 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:50:03.941864  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 271715us
I20260808 01:50:03.942073  9524 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:03.851145 11377 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:03.755219 11583 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:03.837531 11562 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:03.899121  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:03.953230  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:03.954128  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:03.893057 11434 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:03.838364 11432 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:03.960618  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:03.939298 11448 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:03.939070 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:03.948503 11531 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:03.942871 11495 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:04.011929  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:03.988340  9478 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:50:03.949519 11456 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:03.990388  9524 leader_election.cc:400] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4.
I20260808 01:50:03.995148  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:03.961825  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 74804.6us
I20260808 01:50:04.017930  9524 leader_election.cc:403] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260808 01:50:04.001533  9558 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:03.953038 11378 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:04.009541  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:04.001937  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:03.988415 11583 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:03.953006 11436 raft_consensus.cc:2764] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260808 01:50:03.987637 11523 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260808 01:50:04.020435  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:03.986816  9637 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:50:03.988756 11539 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:04.021646  9555 raft_consensus.cc:2393] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:04.022095  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:03.996423 11448 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:04.013885 10362 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
W20260808 01:50:04.029901 11584 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260808 01:50:04.030541  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:04.031188 11573 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:04.031430  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:04.012138 11467 tablet_bootstrap.cc:492] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:50:03.996148 11432 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:03.995759 11587 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:04.034215 11438 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:04.034471  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:04.035159  9630 raft_consensus.cc:2368] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3.
I20260808 01:50:04.160612  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:03.961826 11568 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:04.036679  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:04.142884 11567 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:03.986333 11570 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260808 01:50:04.003059 11434 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:04.144608  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:04.082523  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 96606.2us
I20260808 01:50:04.155891  9481 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:04.255390  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:03.997061 11455 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:03.959791 11450 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:04.033890 11486 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.884s	user 0.013s	sys 0.000s
I20260808 01:50:04.035496  9480 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260808 01:50:04.209396  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 43542.8us
I20260808 01:50:04.032567 11583 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:04.043109 11588 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:04.164067  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:04.160925  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:04.021395 11378 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:04.166725 11566 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:04.230242 11593 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:04.235474 11443 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260808 01:50:04.237988 11557 consensus_meta.cc:287] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.575s	user 0.008s	sys 0.000s
I20260808 01:50:04.298398 11557 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:04.242023 11452 consensus_meta.cc:287] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.542s	user 0.007s	sys 0.002s
I20260808 01:50:04.301194 11452 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:04.302069 11434 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:04.303594 11452 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:04.242864 11587 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:04.304000 11455 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:04.304503 11565 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.147s	user 0.015s	sys 0.000s
I20260808 01:50:04.305881 11587 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:04.306699  9631 raft_consensus.cc:2393] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:04.308257  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:04.308873  9631 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:50:04.309856  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:04.310418  9631 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 3
I20260808 01:50:04.311092  9638 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 5: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:50:04.312263 11575 consensus_meta.cc:287] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.584s	user 0.007s	sys 0.002s
I20260808 01:50:04.312729 11575 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:04.312899 11566 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:04.313357  9638 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260808 01:50:04.086642 11523 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:04.212330 11438 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:04.333408 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:04.334512 11438 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:04.247897  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:04.335407  9559 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:04.336493  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:04.337272  9559 raft_consensus.cc:2368] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3.
I20260808 01:50:04.338162  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:04.248347 11495 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:04.339548 11575 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:04.249738  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 401382us
I20260808 01:50:04.211045  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:04.210768  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:04.219319 10332 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.194:36095 timed out after 1.650s (SENT) [suppressed 23 similar messages]
I20260808 01:50:04.107620 11456 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4.
I20260808 01:50:04.079304 11560 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:04.166602 11573 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:04.276033  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:04.208082 11487 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:04.281128  9550 raft_consensus.cc:2368] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4.
I20260808 01:50:04.282701  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:04.280117  9445 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:04.254035 11543 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:04.254887 11409 consensus_meta.cc:287] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.699s	user 0.000s	sys 0.015s
I20260808 01:50:04.287405  9558 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 2: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:04.290798  9636 raft_consensus.cc:2410] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 01:50:04.289155  9557 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:50:04.297757  9560 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 1.
I20260808 01:50:04.318953 11593 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:04.322986  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:04.323488 11541 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:50:04.324441  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:04.325809 11432 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:04.339107 11497 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:04.341532  9629 raft_consensus.cc:2368] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4.
I20260808 01:50:04.644246  9480 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 1.
I20260808 01:50:04.366356  9635 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
W20260808 01:50:04.645538  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 365522us
I20260808 01:50:04.646984  9551 raft_consensus.cc:2368] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 4. Current term is 5.
I20260808 01:50:04.647687  9555 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:50:04.408027  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 178564us
I20260808 01:50:04.648666  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:04.342185  9559 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 2: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:04.359211  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:04.649350  9630 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:50:04.657027  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 1721) took 1005 ms. Trace:
I20260808 01:50:04.655113  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:04.657621  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:04.658931  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:04.614944 11597 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
W20260808 01:50:04.378031 11503 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.902s	user 0.010s	sys 0.000s
I20260808 01:50:04.664804 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:04.653702 11409 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:04.316030 11588 raft_consensus.cc:697] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:04.662629  9552 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 2: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:04.659113  9551 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:50:04.662034 10334 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:04.668270  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:04.691295  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 415300us
I20260808 01:50:04.452838 11566 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:04.605804 11443 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:04.660257 11560 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:04.713279 11560 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:04.714222  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:04.715167  9478 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:04.716019  9631 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:04.654780 11543 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:04.669734  9481 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:50:04.719424 11389 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:04.720057 11389 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:04.721006 11543 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:04.721580 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:04.671037 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:04.673221 11369 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:04.669196  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:04.723438 10362 raft_consensus.cc:2368] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4.
I20260808 01:50:04.723266 11369 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:04.653481 11482 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.056s	user 0.010s	sys 0.003s
I20260808 01:50:04.727299 11378 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:04.677505  9632 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260808 01:50:04.670463 11578 consensus_meta.cc:287] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.688s	user 0.005s	sys 0.005s
I20260808 01:50:04.730224 11456 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:04.678571  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:04.735145  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:04.328322 11470 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:04.341065 11495 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:04.654109 11438 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:04.656574 11565 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:04.687043 11437 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:04.688050  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:04.689271 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:04.689353 11448 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:04.660789 11487 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:04.690816 10368 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:04.652457 11573 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:04.693789  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:04.691998  9560 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:50:04.696538 11590 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:04.659765 11583 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:04.698410 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:04.699138  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1538) took 1231 ms. Trace:
I20260808 01:50:04.700341  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:04.700527 11593 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:04.701771  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:04.702221 11432 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:04.702693 11557 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:04.704250 11539 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.540s	user 0.009s	sys 0.000s
I20260808 01:50:04.665223 11503 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:04.705417 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:04.707293 11409 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:04.707605 11531 consensus_meta.cc:287] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.677s	user 0.004s	sys 0.007s
I20260808 01:50:04.663549 11486 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260808 01:50:04.711316 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:04.712342 11443 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:04.670878 11575 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:04.656345  9601 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:04.695125  9524 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:04.727078 11369 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:04.727862 11562 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:04.727658 11482 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:04.352918 11523 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:04.728324 11543 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:04.728439  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:04.729101 11567 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:04.659490  9634 rpcz_store.cc:276] 0808 01:50:03.645580 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:03.656478 (+ 10898us) service_pool.cc:224] Handling call
0808 01:50:03.676510 (+ 20032us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:03.697564 (+ 21054us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:03.697599 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:03.796973 (+ 99374us) log.cc:844] Serialized 26 byte log entry
0808 01:50:03.985195 (+188222us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:03.985242 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:03.985313 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:04.332977 (+347664us) raft_consensus.cc:1685] finished
0808 01:50:04.333001 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:04.651253 (+318252us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":37921,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":16371,"thread_start_us":91646,"threads_started":1,"wal-append.queue_time_us":322720}
I20260808 01:50:04.725376 11497 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:04.750736  9555 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:04.933393  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1713) took 1782 ms. Trace:
W20260808 01:50:04.968024  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 311751us
I20260808 01:50:04.770409  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:04.756687  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:04.969652  9559 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:04.977450  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 283737us
W20260808 01:50:04.976958  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 281913us
I20260808 01:50:04.976936  9556 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:04.967511  9636 rpcz_store.cc:276] 0808 01:50:03.151304 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:03.151458 (+   154us) service_pool.cc:224] Handling call
0808 01:50:04.933137 (+1781679us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54000a9bd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:04.933317 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2376864128}
I20260808 01:50:04.975291  9478 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:50:04.976361  9635 rpcz_store.cc:276] 0808 01:50:03.468035 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:03.475655 (+  7620us) service_pool.cc:224] Handling call
0808 01:50:03.536521 (+ 60866us) spinlock_profiling.cc:243] Waited 5.04 ms on lock 0x7b5800231698. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:50:04.698984 (+1162463us) spinlock_profiling.cc:243] Waited 759 ms on lock 0x7b540027e054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:04.699083 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1757771008}
I20260808 01:50:04.978667  9599 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:04.988787  9524 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:04.959035  9523 consensus_peers.cc:597] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3759209338e14437aff5f92a2fff36da. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:50:04.989898  9524 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:04.893252 11464 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.022s	sys 0.045s
W20260808 01:50:05.021680  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 267524us
I20260808 01:50:05.033818  9637 raft_consensus.cc:2368] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4.
I20260808 01:50:05.050215 11495 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:05.057410 11495 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:04.753993 11588 consensus_queue.cc:237] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.057943 11495 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:05.059911 11587 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:05.060163 11503 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:05.060370  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:05.060534 11587 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.061057  9554 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:05.062089  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:04.760450 11378 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:05.062968 11587 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:05.063166 11581 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:05.063427  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:05.063781 11581 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:05.064728  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:05.065420 11587 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:05.051113  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:05.066001  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:04.771070 11389 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:05.051792  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:05.066690 11587 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.066131  9478 raft_consensus.cc:3055] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:05.067590  9478 raft_consensus.cc:740] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:05.068291 11583 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:05.068600  9478 consensus_queue.cc:260] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.069571 10361 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:05.070034  9478 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:04.758373 11568 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:05.070528 11503 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:05.071324 11503 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.071775 11568 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:05.071899  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:05.072589  9475 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 3
I20260808 01:50:05.072825 11568 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.073729 11503 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:05.074033 11568 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:05.067090 11389 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:04.976182 11543 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:05.074769  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:05.074996 11389 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:05.075587  9473 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:05.075045 11543 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:05.076923 11443 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:05.077323 11389 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:05.078132 11389 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:05.021013  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 374035us
I20260808 01:50:05.056509  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:05.083245  9479 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:05.084260 11487 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:05.064826 11581 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.078637 11389 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:04.958410 11560 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:05.023433  9481 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:50:05.037457 10366 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260808 01:50:05.091549  9557 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:50:04.758597 11570 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:04.733366 11578 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:05.016520  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 209107us
I20260808 01:50:04.750625 11452 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:04.758810 11438 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:04.765655 11456 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.096866  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:05.161023  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 1623) took 1144 ms. Trace:
W20260808 01:50:05.104776 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 896415us
I20260808 01:50:05.098940  9551 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:50:05.095871  9560 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:05.169669  9523 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:04.986017  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:05.029675  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:05.175288  9554 rpcz_store.cc:276] 0808 01:50:03.990213 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:03.990386 (+   173us) service_pool.cc:224] Handling call
0808 01:50:05.134310 (+1143924us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400257e54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:05.134503 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2392553856}
I20260808 01:50:05.169838  9475 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:05.028929  9521 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:05.171705  9478 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:05.172719 10334 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:04.758616 11565 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:05.080693 11587 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:04.756548 11437 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:04.987195 11451 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:05.187306 10334 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:04.987493 11539 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:04.893915 11592 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:05.184801  9600 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:05.192711  9555 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:50:05.021792 11450 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:05.209982 11450 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:05.210405 11450 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:04.983412 11595 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:05.196487 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:05.176829 11434 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260808 01:50:05.211575  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:05.212301  9479 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:05.213232  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:05.214051  9476 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 5
I20260808 01:50:05.213981 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:05.215678  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:05.216344  9636 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 1.
I20260808 01:50:05.216988  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:05.217610  9630 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:05.218849 11443 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:05.219609 11451 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:05.220086 11583 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:05.066061 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:04.987783 11531 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.220773 11378 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:05.221338 11378 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:05.222374 11531 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:05.222482 11562 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:05.223093 11562 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:05.223186 11531 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:05.223475 11378 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:05.223842 11531 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:05.224452  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:05.224715  9374 catalog_manager.cc:5697] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 2 to 4, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 4 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:50:05.226092 11531 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:05.226195  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:05.196641 11606 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:05.227030  9631 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:05.227175 11606 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.228055  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:05.229485 11606 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:05.201510 11587 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:05.230105 11437 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:05.230870 11587 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:05.231323 11587 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:05.220634 11451 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.232429 11587 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:05.233873 11587 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:04.900852 11470 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:05.235517 11578 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:05.235853 11607 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260808 01:50:04.984952 11567 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:05.237550 11470 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:05.239164 11567 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:05.239326 11539 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:05.240245  9633 raft_consensus.cc:2368] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3.
I20260808 01:50:05.241173  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:05.241823  9633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1724) took 1397 ms (client timeout 1622 ms). Trace:
I20260808 01:50:05.177310 11570 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.199831  9552 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:05.242591  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:04.987093 11377 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 01:50:05.242257  9633 rpcz_store.cc:269] 0808 01:50:03.844482 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:03.851786 (+  7304us) service_pool.cc:224] Handling call
0808 01:50:05.241613 (+1389827us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b540008cd54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:05.241754 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2880778112}
I20260808 01:50:04.983860 11497 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:05.173007 11438 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:05.022819 11369 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.168s
I20260808 01:50:05.203764  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:05.204267  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:05.205025  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:05.276288  9481 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
W20260808 01:50:05.208133  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:05.208841  9473 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:50:05.209527  9550 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:05.210975  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1733) took 1123 ms. Trace:
I20260808 01:50:05.213310 11560 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:05.219360 11443 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.225446  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:05.242902  9638 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:05.131702 11482 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:04.980257 11590 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:05.218526  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:05.220630  9557 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260808 01:50:05.166880 11464 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.840s	user 0.025s	sys 0.057s
I20260808 01:50:05.161815 11389 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:05.260306  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 75593.4us
I20260808 01:50:05.226552  9374 catalog_manager.cc:5697] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 2 to 4, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 4 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:50:05.048211 11575 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:05.269150  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 290574us
I20260808 01:50:05.281530  9637 rpcz_store.cc:276] 0808 01:50:04.088377 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:04.161781 (+ 73404us) service_pool.cc:224] Handling call
0808 01:50:05.210737 (+1048956us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b540032cb54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:05.210910 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":896015744}
I20260808 01:50:05.196643 11593 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:05.186638 11452 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:05.449345  9560 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:05.433871  9630 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
W20260808 01:50:05.477701  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 448900us
I20260808 01:50:05.223491 11562 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:05.461522  9476 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:05.361022  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:05.497341  9633 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:05.495858  9521 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:05.493559  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:05.499312  9630 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:05.491847  9636 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:05.461870  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 1904) took 1302 ms. Trace:
I20260808 01:50:05.406917  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:05.463105  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 1735) took 1295 ms. Trace:
I20260808 01:50:05.401983 10367 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:50:05.492156  9599 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:05.322455  9478 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:50:05.500283  9552 rpcz_store.cc:276] 0808 01:50:04.159991 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:04.687637 (+527646us) service_pool.cc:224] Handling call
0808 01:50:05.321858 (+634221us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b54001acf54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:05.461786 (+139928us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":513905536}
I20260808 01:50:05.528025  9600 leader_election.cc:400] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4.
I20260808 01:50:05.536633  9600 leader_election.cc:403] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260808 01:50:05.536819  9635 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:05.497014  9374 catalog_manager.cc:5697] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 2 to 3, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 3 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:50:05.516911  9521 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:05.518034  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:50:05.263620 11377 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:05.541667  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 49769.4us
I20260808 01:50:05.543568  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:05.542290  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 50221.5us
I20260808 01:50:05.535475  9638 rpcz_store.cc:276] 0808 01:50:04.167348 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:04.314452 (+147104us) service_pool.cc:224] Handling call
0808 01:50:04.662651 (+348199us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:05.321282 (+658631us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:05.368430 (+ 47148us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b540032d554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:05.463036 (+ 94606us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1333856640}
I20260808 01:50:05.259775 11595 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:05.267762 11470 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:05.269150 11497 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:05.220630 11583 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.267683 11409 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:05.495499 11590 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:05.403453 11448 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:05.548579  9633 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:05.550117  9476 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:05.425786 11610 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:05.584187  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:05.547765  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 450976us
I20260808 01:50:05.563019  9599 leader_election.cc:400] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4.
I20260808 01:50:05.586012  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:05.586344  9599 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:05.560601  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 1903) took 1403 ms. Trace:
I20260808 01:50:05.403007 11608 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:05.259603 11592 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.595590  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:05.405777 11565 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:05.559378 11578 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:05.562340  9632 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 1.
I20260808 01:50:05.588258  9559 rpcz_store.cc:276] 0808 01:50:04.157444 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:05.321212 (+1163768us) service_pool.cc:224] Handling call
0808 01:50:05.559826 (+238614us) spinlock_profiling.cc:243] Waited 35.3 ms on lock 0x7b5800003c98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:50:05.560048 (+   222us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:05.560230 (+   182us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:05.560256 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:05.560296 (+    40us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:05.560328 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:05.560386 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:05.560520 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":81250688}
I20260808 01:50:05.593681  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:05.608409  9480 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:05.542639 11593 raft_consensus.cc:697] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:05.559568 11567 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:05.569410 11434 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:05.362010 11560 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.471752 11456 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:05.546989 11495 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.604401  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:05.284123 11573 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.591203 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:05.613838  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 444212us
I20260808 01:50:05.600256 11595 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.598985  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:05.564990 11531 consensus_queue.cc:1059] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260808 01:50:05.584235 11409 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:05.603762 11470 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.107s
I20260808 01:50:05.642033 11583 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:05.591260 11608 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:05.571827 11541 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:05.644755  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:05.798820  9473 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 3
I20260808 01:50:05.569298 11620 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:05.561054 11377 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:50:05.711264 11543 consensus_meta.cc:287] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.635s	user 0.008s	sys 0.003s
I20260808 01:50:05.707540  9630 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260808 01:50:05.595533 11588 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election won for term 3
W20260808 01:50:05.704689 11523 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.642s	user 0.010s	sys 0.002s
I20260808 01:50:05.624558 11567 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:05.837265  9558 raft_consensus.cc:2368] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3.
I20260808 01:50:05.715137  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:05.838267  9481 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 4
I20260808 01:50:05.728018  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:05.788669  9478 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 5: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:05.701233  9476 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:05.647938  9635 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:50:05.790774  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 244853us
I20260808 01:50:05.700219  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:05.841254  9638 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:50:05.607599 11578 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:05.800345  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1636) took 1492 ms. Trace:
I20260808 01:50:05.607249 11615 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:05.809883  9636 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:05.799399 11608 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:05.681933 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:05.809230  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:05.706732  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:05.861552  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:05.861913  9633 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 5: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:05.609896 11611 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:05.709075 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:05.708590 11452 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:05.856639  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:05.897047  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:05.898216  9636 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:05.839184 11583 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:05.906898 11583 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:05.907390 11583 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:05.907434 11369 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:05.908103 11369 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:05.908666 11573 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:05.910279 11369 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:05.857952  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:05.885572  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:05.625541 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:05.862154 10366 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:05.604117 11610 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:05.811444  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 212583us
I20260808 01:50:05.703541 11593 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:05.846627 11567 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:05.900171  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:05.925197  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:05.925647  9476 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:05.925999  9552 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:50:05.902165 11409 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:05.838935 11543 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:05.904661 11575 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:05.905985 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:05.706199 11560 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:05.897331 11472 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:05.911386  9559 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:50:05.911969  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:05.915400  9475 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:05.860090  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:05.896046  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:05.865744  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:05.864126  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 55008.4us
I20260808 01:50:05.838135 11570 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:05.918480 10332 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.021s	sys 0.033s
I20260808 01:50:05.923794  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:05.866487  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:05.927554  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 82350.6us
I20260808 01:50:05.927647 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:05.928758  9481 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:05.856670 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:06.035348 10332 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.029s	sys 0.041s
W20260808 01:50:06.035808 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 629610us
I20260808 01:50:05.986172  9550 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:50:06.036144  9601 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:05.985050  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:06.039687  9560 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:05.999361  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:06.036962 10332 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:05.863390 11625 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260808 01:50:05.942286 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:06.048885  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:06.049571  9481 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:05.855283 11578 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:06.068087  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:06.069054  9636 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:06.070564  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:05.943424  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:05.849031 11523 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:05.862857 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 227343us
I20260808 01:50:05.817534 11541 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:05.917945 11588 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:06.035068 11472 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:06.042369 11456 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:06.148512  9637 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:06.085016 10362 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 3: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:05.866078 11539 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4.
I20260808 01:50:06.047381  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:06.050889  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:06.038002  9638 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:50:06.072106  9633 raft_consensus.cc:2368] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4.
I20260808 01:50:06.083585 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:06.083923  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:06.085225  9558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 1644) took 1767 ms (client timeout 1700 ms). Trace:
I20260808 01:50:05.834342 11620 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:05.914330 11590 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:06.182219  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 236) took 1427 ms. Trace:
I20260808 01:50:06.046201  9556 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:06.157938  9558 rpcz_store.cc:269] 0808 01:50:04.309773 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:04.654812 (+345039us) service_pool.cc:224] Handling call
0808 01:50:06.076683 (+1421871us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54002a8354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:06.076893 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2710200064}
W20260808 01:50:06.102420  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:06.113401  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:06.151109  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:06.153270  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 108211us
I20260808 01:50:06.189724  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:06.157091  9551 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 5: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:06.156538  9476 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 3
I20260808 01:50:06.088157 10369 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:06.066561 11560 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:05.855763 11615 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:06.195946 11615 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:06.196102  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:06.179059  9633 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260808 01:50:06.179451 11628 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260808 01:50:06.211568  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:06.211953 11452 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:50:06.214022  9475 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:06.214785 11637 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:06.215174  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:06.216004  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:06.216432  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:06.217382  9556 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:06.217537  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:06.218093  9475 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:06.219008  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:06.198019  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:06.220953  9478 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 4
I20260808 01:50:06.182871  9555 rpcz_store.cc:276] 0808 01:50:04.754555 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:05.251203 (+496648us) service_pool.cc:224] Handling call
0808 01:50:05.251571 (+   368us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:05.266336 (+ 14765us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:05.266415 (+    79us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:05.367374 (+100959us) log.cc:844] Serialized 26 byte log entry
0808 01:50:05.916314 (+548940us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:06.033537 (+117223us) log.cc:844] Serialized 14 byte log entry
0808 01:50:06.044265 (+ 10728us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:06.044344 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:06.181916 (+137572us) raft_consensus.cc:1685] finished
0808 01:50:06.181947 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:06.182138 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28586,"prepare.queue_time_us":116844,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":15373,"raft.queue_time_us":30533,"raft.run_cpu_time_us":436,"raft.run_wall_time_us":56417,"spinlock_wait_cycles":69925632,"thread_start_us":465148,"threads_started":1,"wal-append.queue_time_us":749009}
I20260808 01:50:06.091777 11592 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:06.211362  9550 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:06.215351  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:06.092487 11626 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:06.120550 11612 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:06.130471 11472 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:06.187903  9600 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:06.264675 10368 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:06.283480  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:06.280241 11626 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:06.293205 11626 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:06.156462  9635 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "439040d168824496acc076b1c280bde3"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:50:06.297255  9635 raft_consensus.cc:606] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:06.175356  9638 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:06.223032 11565 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:06.304924 11565 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:06.305378 11565 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:06.308156 11583 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:06.309480  9630 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:06.215369 11637 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:06.315037 11637 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:06.315318 11631 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:06.316049 11610 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:06.298082 11612 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:06.298379 11467 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260808 01:50:06.161098 11456 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:06.319676 11592 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:06.320101 11592 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:06.320461 11592 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:06.323328  9635 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:50:06.324015  9635 raft_consensus.cc:606] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:06.324370  9635 raft_consensus.cc:614] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:50:06.326857  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 4c485e8f4266462e8413079adcf8faef (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:50:06.330907 11588 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:06.333719 11450 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.121s	user 0.003s	sys 0.007s
I20260808 01:50:06.333544 11615 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:06.334283 11450 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:06.336391 11620 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:06.337209  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1546) took 1120 ms. Trace:
I20260808 01:50:06.387698  9634 rpcz_store.cc:276] 0808 01:50:05.216938 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:05.432420 (+215482us) service_pool.cc:224] Handling call
0808 01:50:05.463382 (+ 30962us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:05.706523 (+243141us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:05.706553 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:05.706592 (+    39us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:05.706625 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:06.336939 (+630314us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b54001b3fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:06.336993 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:06.337142 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":558521728}
I20260808 01:50:06.269016 11560 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:06.389941 11615 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:06.319103  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 131316us
I20260808 01:50:06.393148  9558 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:06.324338 11568 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.250s	user 0.014s	sys 0.001s
I20260808 01:50:06.366204 11443 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:06.355187  9636 raft_consensus.cc:2410] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:06.387297 11452 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:05.929340 11611 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:06.442070 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 405171us
I20260808 01:50:06.445293 10332 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:06.438735  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:06.447335  9551 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:06.446336  9476 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:06.487743  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:06.487308 11452 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:06.488242  9560 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
W20260808 01:50:06.497550 11587 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.263s	user 0.014s	sys 0.008s
I20260808 01:50:06.506572  9552 raft_consensus.cc:2410] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:50:05.850438  9554 rpcz_store.cc:276] 0808 01:50:04.307430 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:05.193494 (+886064us) service_pool.cc:224] Handling call
0808 01:50:05.228941 (+ 35447us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:05.229218 (+   277us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:50:05.475784 (+246566us) log.cc:844] Serialized 14 byte log entry
0808 01:50:05.799726 (+323942us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:05.799802 (+    76us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:05.799865 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:05.799957 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:05.800267 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":338768,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":65,"raft.queue_time_us":8358,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":24730,"thread_start_us":498860,"threads_started":2,"wal-append.queue_time_us":297360}
I20260808 01:50:06.508770  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:06.509267  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:06.509666  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:06.513059  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:06.521507  9637 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:06.527654  9478 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:06.497707 11565 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:06.539131 11578 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:06.540591 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 460428us
I20260808 01:50:06.510098 11592 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:06.550028 11472 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:06.639071 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 193907us
W20260808 01:50:06.638800  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 602710us
I20260808 01:50:06.642323  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:06.641799  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:06.639815  9551 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:06.637964 11456 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:06.652915  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:06.653908  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 464224us
I20260808 01:50:06.652906  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:06.608641  9473 consensus_meta.cc:287] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.776s	user 0.012s	sys 0.006s
I20260808 01:50:06.655375  9473 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:06.655866  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1853) took 1012 ms. Trace:
I20260808 01:50:06.656569  9473 rpcz_store.cc:276] 0808 01:50:05.643940 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:05.644409 (+   469us) service_pool.cc:224] Handling call
0808 01:50:06.655778 (+1011369us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:06.652343 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:06.514026 11632 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:50:06.557139 11562 consensus_meta.cc:287] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.007s	user 0.012s	sys 0.003s
W20260808 01:50:06.635398 11608 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.734s	user 0.015s	sys 0.003s
I20260808 01:50:06.636924 11467 tablet_bootstrap.cc:492] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260808 01:50:06.660722  9476 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
W20260808 01:50:06.670149 10365 consensus_meta.cc:287] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.620s	user 0.005s	sys 0.009s
I20260808 01:50:06.663672  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:06.664697  9631 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
W20260808 01:50:06.553894 11438 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.248s	user 0.006s	sys 0.010s
I20260808 01:50:06.671962  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:06.673094  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:06.550933 11450 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:06.671500  9444 leader_election.cc:400] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3.
I20260808 01:50:06.675315  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:06.671959 10365 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:06.676151  9560 raft_consensus.cc:2368] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3.
I20260808 01:50:06.675467 11541 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:06.677107  9444 leader_election.cc:403] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:50:06.677898 11620 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3.
I20260808 01:50:06.678052 11568 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:50:06.679210 11543 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:06.679723 11565 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:06.640110 11611 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:06.679526  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:06.681243  9473 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 4: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:06.693262  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:06.660382 11583 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:06.702764  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:06.702526  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:06.707819  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:06.708294  9473 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:38182
I20260808 01:50:06.708676  9476 raft_consensus.cc:2393] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:06.708949  9473 raft_consensus.cc:606] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:06.709919  9476 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:06.711524  9476 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260808 01:50:06.712538  9476 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cacb9622c47c4737beff104125e98b77"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:38182
I20260808 01:50:06.713308  9476 raft_consensus.cc:606] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:06.713651  9476 raft_consensus.cc:614] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:50:06.714046  9473 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260808 01:50:06.714967  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet cacb9622c47c4737beff104125e98b77 (from 6e7856bed1be43acaa821290d506ee97 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:50:06.715790  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 01:50:06.717182  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:06.717919  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
W20260808 01:50:06.720187 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 67924.4us
I20260808 01:50:06.703429 11467 tablet_bootstrap.cc:492] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:50:06.721668  9444 leader_election.cc:400] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4.
W20260808 01:50:06.723194  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 70980us
I20260808 01:50:06.741474  9444 leader_election.cc:403] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:50:06.742203  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:06.744037  9601 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:06.703018 11631 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:06.745893  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:06.746732 11631 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:06.747216 11631 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:06.748831  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:06.751870  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:06.683266 11438 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:06.755770 11389 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4.
I20260808 01:50:06.757824  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:06.759666  9444 leader_election.cc:400] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3.
I20260808 01:50:06.759979  9444 leader_election.cc:403] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:50:06.760996  9444 leader_election.cc:400] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4.
I20260808 01:50:06.761289  9444 leader_election.cc:403] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:50:06.763360  9479 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:06.764351  9478 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:06.765012 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:06.718976 11611 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:06.680397 11608 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:06.718888 11565 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:06.705528 11641 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:50:06.767771  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:06.674333 11562 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:06.724253 11620 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 01:50:06.774729  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 106070us
I20260808 01:50:06.762233 11487 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:06.714488 11637 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:06.776501  9599 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:06.757867 11438 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:06.794268  9477 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:06.771157 11541 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260808 01:50:06.770347 11543 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:06.768641 11611 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:06.800388 10367 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:06.804081  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:06.792800 11467 ts_tablet_manager.cc:1403] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 2.835s	user 0.037s	sys 0.056s
I20260808 01:50:06.775254 11620 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:06.797910 11643 ts_tablet_manager.cc:933] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:06.773921 11389 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:06.773185 11610 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
W20260808 01:50:06.817528  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 65729.6us
I20260808 01:50:06.820637  9601 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:06.803694 11543 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:06.769351 11631 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:06.807917 11611 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:06.776337 11565 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:06.822638  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 61777.8us
I20260808 01:50:06.791921 11639 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3.
I20260808 01:50:06.815021 11620 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:06.826751  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:06.815265 11583 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260808 01:50:06.823570 11377 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.960s	user 0.016s	sys 0.006s
I20260808 01:50:06.834950  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:06.830766 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 56571.4us
W20260808 01:50:06.827831 11497 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.234s	user 0.015s	sys 0.008s
I20260808 01:50:06.822639 11438 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4.
I20260808 01:50:06.849695  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:06.855592  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:06.824721 11543 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:06.848549  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:06.860170  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:06.862751  9638 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:06.829403 11389 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:06.868175 11389 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:06.870721 11389 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:06.871989  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:06.872799  9636 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 1.
I20260808 01:50:06.873824  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:06.874562  9473 raft_consensus.cc:2410] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:06.877249  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:06.847519 10332 connection.cc:256] RPC call timeout handler was delayed by -1.416965131968027s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1799s for a response
I20260808 01:50:06.878110  9476 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:50:06.879122 10332 connection.cc:256] RPC call timeout handler was delayed by -1.342046997975558s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2189s for a response
I20260808 01:50:06.880023  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:06.880070  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:06.880817  9634 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:50:06.881057  9478 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:06.881871  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:06.834990 11472 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:06.828907 11649 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:06.775487 11641 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:06.833781 11583 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:50:06.893949  9444 connection.cc:256] RPC call timeout handler was delayed by -2.0546717749966774s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1849s for a response
I20260808 01:50:06.856429 11578 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:06.903791  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:06.906502 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:06.906549  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:06.908224  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:06.908885  9636 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:06.908924 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:06.909072  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:06.907982  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:06.909875  9473 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 4
I20260808 01:50:06.913347  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:06.914300  9478 raft_consensus.cc:2368] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3.
I20260808 01:50:06.879222 11592 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:06.923708 11409 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:06.932032 11409 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:06.855274 11377 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:06.927749 11562 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:06.926597 11636 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:06.909081 11583 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:06.929793  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:06.953608  9476 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:06.861166 11497 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:06.878072 11650 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:06.895907 11389 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:06.890209 11610 consensus_queue.cc:1059] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
W20260808 01:50:06.932909 11567 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.950s	user 0.024s	sys 0.003s
I20260808 01:50:06.934362 11409 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:06.958410  9473 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:06.965503  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:06.969416  9636 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:50:06.940816 11649 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:50:06.959923 11646 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:06.972666  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:06.970705  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 64160.3us
W20260808 01:50:06.976897  9444 connection.cc:256] RPC call timeout handler was delayed by -2.0249785820196848s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2143s for a response
I20260808 01:50:06.942692 11651 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260808 01:50:06.975778  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 199396us
I20260808 01:50:06.964920 11636 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:06.966528 11608 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:06.980172  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:06.980702  9444 connection.cc:256] RPC call timeout handler was delayed by -2.0186904510192107s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2433s for a response
I20260808 01:50:06.984781  9473 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:06.979185  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:06.985929  9636 raft_consensus.cc:2368] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5.
I20260808 01:50:06.995767 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:06.998375 10332 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:07.003261  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:07.027595  9444 connection.cc:256] RPC call timeout handler was delayed by -1.8960789809934795s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1952s for a response
W20260808 01:50:07.028316  9444 connection.cc:256] RPC call timeout handler was delayed by -1.7342904510151129s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27490000000000003s for a response
W20260808 01:50:07.028829  9444 connection.cc:256] RPC call timeout handler was delayed by -1.7072904510132503s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2779s for a response
I20260808 01:50:07.030505  9476 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:07.031677 11636 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:06.968436 11650 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:07.041857  9444 connection.cc:256] RPC call timeout handler was delayed by -1.4661849880067166s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1928s for a response
I20260808 01:50:07.030520  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:07.043035  9636 raft_consensus.cc:2376] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4. Re-sending same reply.
W20260808 01:50:07.042371  9444 connection.cc:256] RPC call timeout handler was delayed by -1.2833904510189313s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.325s for a response
I20260808 01:50:07.047863 11651 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:50:07.048205 11631 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:07.049618 11645 log.cc:927] Time spent T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
I20260808 01:50:07.030910 11562 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:06.975980 11649 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:06.966962 11389 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:06.968245 11567 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:06.977385 11646 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:07.343406  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 340282us
I20260808 01:50:07.209314  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:07.365077  9601 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:07.365736  9636 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:50:07.343152 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 344879us
W20260808 01:50:07.349954 10368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 435) took 1788 ms (client timeout 2030 ms). Trace:
W20260808 01:50:07.210839  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 181917us
I20260808 01:50:07.347378  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:07.138424  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 209) took 1512 ms. Trace:
I20260808 01:50:07.045286  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:07.343851  9476 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260808 01:50:07.352615  9630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1569) took 1647 ms (client timeout 1660 ms). Trace:
W20260808 01:50:07.399224  9477 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1661) took 1639 ms (client timeout 2106 ms). Trace:
I20260808 01:50:07.353632  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1855) took 1611 ms. Trace:
W20260808 01:50:07.413242  9637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1563) took 1709 ms (client timeout 2106 ms). Trace:
W20260808 01:50:07.417092  9637 rpcz_store.cc:269] 0808 01:50:05.703350 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:05.999021 (+295671us) service_pool.cc:224] Handling call
0808 01:50:07.412912 (+1413891us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54000db454. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:07.413130 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":246564224}
I20260808 01:50:07.418241  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:07.366411  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:07.393128  9480 rpcz_store.cc:276] 0808 01:50:05.626132 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:05.627850 (+  1718us) service_pool.cc:224] Handling call
0808 01:50:05.700689 (+ 72839us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:06.438169 (+737480us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:06.438219 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:06.438285 (+    66us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:06.438318 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:07.138082 (+699764us) spinlock_profiling.cc:243] Waited 577 ms on lock 0x7b54002a6cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:07.138148 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:07.138314 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1326044544}
W20260808 01:50:07.414320  9477 rpcz_store.cc:269] 0808 01:50:05.716759 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:05.727678 (+ 10919us) service_pool.cc:224] Handling call
0808 01:50:07.356209 (+1628531us) spinlock_profiling.cc:243] Waited 954 ms on lock 0x7b540038ccd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:07.356387 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2195193216}
I20260808 01:50:07.355640  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1658) took 2072 ms. Trace:
I20260808 01:50:07.415657  9479 rpcz_store.cc:276] 0808 01:50:05.742624 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:05.742820 (+   196us) service_pool.cc:224] Handling call
0808 01:50:05.813283 (+ 70463us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:06.763829 (+950546us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:07.353227 (+589398us) spinlock_profiling.cc:243] Waited 950 ms on lock 0x7b540021e154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:07.353542 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2184509824}
W20260808 01:50:07.391319 10368 rpcz_store.cc:269] 0808 01:50:05.561598 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:05.562114 (+   516us) service_pool.cc:224] Handling call
0808 01:50:07.349647 (+1787533us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b540046c154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:07.349852 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1099913984}
W20260808 01:50:07.428757  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 62441.9us
I20260808 01:50:07.439491  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:07.433889  9472 rpcz_store.cc:276] 0808 01:50:05.283209 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:05.623373 (+340164us) service_pool.cc:224] Handling call
0808 01:50:05.623723 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:06.607993 (+984270us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:06.608040 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:06.608103 (+    63us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:06.608147 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:07.355256 (+747109us) spinlock_profiling.cc:243] Waited 984 ms on lock 0x7b540019c3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:07.355329 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:07.355547 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2262859008}
W20260808 01:50:07.337738 11611 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.505s	user 0.011s	sys 0.000s
I20260808 01:50:07.033267 11541 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:07.457870  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:07.105997 11657 log.cc:927] Time spent T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
W20260808 01:50:07.414047  9630 rpcz_store.cc:269] 0808 01:50:05.705039 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:05.845965 (+140926us) service_pool.cc:224] Handling call
0808 01:50:07.352265 (+1506300us) spinlock_profiling.cc:243] Waited 393 ms on lock 0x7b540003b6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:07.352513 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":904956800}
I20260808 01:50:07.054141 11637 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:07.061862 11608 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:07.056025 11641 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:07.358338 11497 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:07.106316 11377 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:07.466045  9599 leader_election.cc:400] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3.
I20260808 01:50:07.468832  9601 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:07.465611 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:07.463891  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:07.143791 11613 raft_consensus.cc:3055] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Stepping down as leader of term 2
W20260808 01:50:07.466306  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:07.469602  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 424539us
I20260808 01:50:07.477017  9599 leader_election.cc:403] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260808 01:50:07.477984  9601 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:07.478762  9600 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:07.478961  9601 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:07.419219 11662 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:07.462164  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:07.484177  9632 raft_consensus.cc:2376] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2. Re-sending same reply.
W20260808 01:50:07.470309 11620 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.608s	user 0.013s	sys 0.000s
I20260808 01:50:07.477392 10366 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 5
I20260808 01:50:07.469362 11631 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:07.462584 11578 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:07.473359 11622 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:07.488131  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:07.461848 11661 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:07.104394 11650 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:07.136256 11643 tablet_copy_client.cc:323] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.194:36095
I20260808 01:50:07.467183 11541 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:07.523132 11541 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:07.494891  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:07.527697 11495 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:07.528189  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:07.528818 11655 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:07.529379  9632 raft_consensus.cc:2393] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:50:07.531965  9631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1551) took 2313 ms (client timeout 1577 ms). Trace:
W20260808 01:50:07.532582  9631 rpcz_store.cc:269] 0808 01:50:05.218432 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:05.227800 (+  9368us) service_pool.cc:224] Handling call
0808 01:50:07.531641 (+2303841us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b540003bbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:07.531805 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2682282752}
I20260808 01:50:07.316069 11543 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:07.538475 11639 raft_consensus.cc:3055] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:07.509790 10362 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:07.539681 11543 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:07.496752  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:07.501796 11377 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:07.540870 10365 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260808 01:50:07.499560  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:07.489456 11583 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.523s	user 0.012s	sys 0.000s
W20260808 01:50:07.492015 11409 consensus_meta.cc:287] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.517s	user 0.011s	sys 0.000s
I20260808 01:50:07.510937  9600 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:07.510079 11578 raft_consensus.cc:697] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:07.539036 11639 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:07.553622 11578 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:07.557457  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:50:07.495690 11620 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:07.559628  9558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 1920) took 1386 ms (client timeout 1504 ms). Trace:
W20260808 01:50:07.560151  9558 rpcz_store.cc:269] 0808 01:50:06.173572 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:06.195754 (+ 22182us) service_pool.cc:224] Handling call
0808 01:50:07.559356 (+1363602us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b54001aca54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:07.559550 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":450477440}
I20260808 01:50:07.560565 11620 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:07.459692 11467 raft_consensus.cc:359] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:07.483909 11662 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:07.498168 11631 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:07.491200 11592 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:07.479714 11613 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:07.467682 11611 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:07.526118 11452 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:07.494302 11660 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:07.059904 11649 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:07.577682  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 66834.8us
I20260808 01:50:07.579022  9600 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:07.551017 10366 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:07.564935  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:07.570482  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 104590us
I20260808 01:50:07.476403 11608 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:07.574641  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 75230.1us
I20260808 01:50:07.536376 11612 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:07.583911  9630 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:07.605090 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:07.605692  9521 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:07.570020 11651 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3.
I20260808 01:50:07.608762  9637 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:07.607493 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:07.573810 11662 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:07.609712 10366 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:50:07.592499  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:07.547205 11661 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:07.573356 11665 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:07.573155 11655 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:07.577159 11467 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:50:07.576190 11622 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:07.581740 11592 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:07.581594 11452 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:07.571642 11409 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:07.565732 11583 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:07.624751  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 145886us
I20260808 01:50:07.633992  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 1921) took 1382 ms. Trace:
I20260808 01:50:07.625097  9630 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:50:07.627009  9600 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:07.608189 11648 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:07.615249 11660 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:07.634034 11495 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260808 01:50:07.636196 11660 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:07.634433  9556 rpcz_store.cc:276] 0808 01:50:06.437876 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:06.452929 (+ 15053us) service_pool.cc:224] Handling call
0808 01:50:06.453417 (+   488us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:07.628748 (+1175331us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:07.628779 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:07.628823 (+    44us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:07.628852 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:07.628927 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:07.629046 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1174862,"raft.queue_time_us":1178049}
W20260808 01:50:07.638581  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46215.9us
I20260808 01:50:07.639524  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:07.623484 11620 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:07.621076 11661 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:07.628402 11448 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:07.617297 11434 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260808 01:50:07.647953 11608 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:07.649761 11583 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:07.643816 11665 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:07.642278 11450 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:07.651489  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:07.607297 11639 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:07.665705 11450 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:07.666325  9556 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 3
I20260808 01:50:07.666646 11639 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:07.642036 11660 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:07.666951 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:07.660439  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:07.642714 11646 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:07.645928 11656 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:07.663678  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:07.664321  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:07.664887  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:07.684712  9551 raft_consensus.cc:2393] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:07.664470  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:07.650889 11581 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:07.658954 11389 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:07.670863  9558 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:07.657496  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:07.633514 11651 consensus_queue.cc:1059] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260808 01:50:07.653139 11448 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:07.678277  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:07.658218 11613 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:07.684939  9601 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:07.643505 11567 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:07.691399  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:07.699184 10363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 436) took 2128 ms (client timeout 2042 ms). Trace:
I20260808 01:50:07.701529  9478 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:07.713438  9558 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:07.676481 11456 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:07.683347 11573 raft_consensus.cc:476] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Not starting pre-election -- already a leader
I20260808 01:50:07.718940  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:07.700204 11448 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:07.689812 11673 raft_consensus.cc:3055] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:07.745627  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:07.734333  9472 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260808 01:50:07.737366  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 131743us
I20260808 01:50:07.740253  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:07.748574  9480 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 2: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:07.720132  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:07.645388 11467 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:50:07.741189  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:07.695120 11646 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:07.700677 11389 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:07.694892 11581 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:07.743490  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 58684us
I20260808 01:50:07.744419  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:07.756932  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:07.758029  9524 connection.cc:256] RPC call timeout handler was delayed by -1.3087658539880067s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15610000000000002s for a response
I20260808 01:50:07.704041 11583 consensus_queue.cc:1059] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260808 01:50:07.759927  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:07.762912  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:07.763141  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:07.760191  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:07.739938 11409 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:07.762050 11567 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:07.771445 11675 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260808 01:50:07.752801 11450 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:07.757869  9523 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:07.771742 11668 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:07.775020  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:07.785957 11612 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 01:50:07.705981 10363 rpcz_store.cc:269] 0808 01:50:05.570783 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:05.591492 (+ 20709us) service_pool.cc:224] Handling call
0808 01:50:07.666777 (+2075285us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:50:07.699070 (+ 32293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":31501696}
I20260808 01:50:07.781848  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:07.758594  9524 connection.cc:256] RPC call timeout handler was delayed by -1.3005025229940657s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1502s for a response
I20260808 01:50:07.782378  9558 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 2: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:07.784843  9478 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
W20260808 01:50:07.757721  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 88039.2us
I20260808 01:50:07.737069 11637 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260808 01:50:07.754146 11578 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:07.692812 11655 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:07.745422 11673 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:07.775702  9551 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:07.688868 11592 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:07.739107 11456 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:07.792099  9523 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:07.788096 11634 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:07.780997 11631 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:07.756623 11543 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260808 01:50:07.752161 11622 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260808 01:50:07.760083 11646 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:07.841991  9556 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
W20260808 01:50:07.840165  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 58396.6us
I20260808 01:50:07.792101  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:07.786319 11651 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:07.905830  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:07.907027  9599 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:07.907630  9556 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:07.908116  9601 leader_election.cc:400] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3.
I20260808 01:50:07.908529  9601 leader_election.cc:403] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260808 01:50:07.908599  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:07.909118  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:07.909760  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:07.911379  9633 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:07.911661  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:07.912302  9556 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:07.913213  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:07.909742  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 117723us
I20260808 01:50:07.909916 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:07.913781  9556 raft_consensus.cc:2376] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4. Re-sending same reply.
I20260808 01:50:07.875528  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 1926) took 1364 ms. Trace:
I20260808 01:50:07.910022  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:07.835330 11672 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260808 01:50:07.914748  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:07.933383  9555 rpcz_store.cc:276] 0808 01:50:06.511097 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:06.511257 (+   160us) service_pool.cc:224] Handling call
0808 01:50:06.511766 (+   509us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:07.633515 (+1121749us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:07.633553 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:07.642625 (+  9072us) log.cc:844] Serialized 26 byte log entry
0808 01:50:07.754721 (+112096us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:07.754760 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:07.754839 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:07.875132 (+120293us) raft_consensus.cc:1685] finished
0808 01:50:07.875151 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:07.875435 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1116997,"raft.queue_time_us":1119756,"spinlock_wait_cycles":14240128,"thread_start_us":91028,"threads_started":1,"wal-append.queue_time_us":168860}
W20260808 01:50:07.822193  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47263.1us
I20260808 01:50:07.842211 11581 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:07.915876 11573 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:07.775059 11665 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:07.940668  9521 leader_election.cc:400] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4.
I20260808 01:50:07.917155 11539 raft_consensus.cc:3055] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:07.918356 10367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 357) took 2046 ms. Trace:
I20260808 01:50:07.919471  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:07.919906  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:07.918202  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:07.896051 11676 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
W20260808 01:50:07.903757  9524 connection.cc:256] RPC call timeout handler was delayed by -1.0927658539731055s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1801s for a response
I20260808 01:50:07.933781  9558 raft_consensus.cc:2368] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4.
W20260808 01:50:07.828353 11669 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
I20260808 01:50:07.917203 11467 ts_tablet_manager.cc:1434] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 1.100s	user 0.014s	sys 0.003s
I20260808 01:50:07.916891 11673 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:07.956077 11467 ts_tablet_manager.cc:1467] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Tablet startup took 1147ms
I20260808 01:50:07.956012 11541 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:07.811652 11668 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:07.924289  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:07.925148  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 123306us
I20260808 01:50:07.935036  9560 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:07.794574 11434 raft_consensus.cc:2764] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260808 01:50:07.936316  9630 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 01:50:07.934757 11470 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:07.938623 11679 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:07.939495  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:07.944139  9551 raft_consensus.cc:2368] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3.
I20260808 01:50:07.945531  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:07.792762 11567 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:07.820118 11495 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:07.943329 10367 rpcz_store.cc:276] 0808 01:50:05.854502 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:06.153581 (+299079us) service_pool.cc:224] Handling call
0808 01:50:06.221262 (+ 67681us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:06.808308 (+587046us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:07.900322 (+1092014us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b54003200d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:07.900558 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1323687808}
I20260808 01:50:07.950969  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:07.974885 11587 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:07.945817  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:07.914108 11578 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:07.943109 11539 raft_consensus.cc:740] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:07.972061 11541 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:07.972640 11622 raft_consensus.cc:993] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: : Instructing follower 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:50:07.935204 11651 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:07.967362 11470 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:07.980337  9560 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:07.991325  9634 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:07.993652  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:07.995018  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:08.019716  9630 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:50:07.985394  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 61221.2us
I20260808 01:50:07.996295  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:07.996043 11587 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:07.996515 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:07.995075 11679 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:50:08.004271  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 58485.7us
I20260808 01:50:07.982865 11575 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:07.955904 11456 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:07.954875 11683 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:07.920389 11608 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:08.008064  9571 tablet_copy_service.cc:140] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Received BeginTabletCopySession request for tablet 3759209338e14437aff5f92a2fff36da from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:60552)
I20260808 01:50:08.008296 11523 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:08.009244 11389 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:08.010123 10369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 432) took 2993 ms. Trace:
I20260808 01:50:07.991956 11495 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:07.971508 11650 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:08.022401  9601 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:08.024454  9633 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:07.960954 11637 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:07.896256 11662 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:08.088511  9571 tablet_copy_service.cc:161] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Beginning new tablet copy session on tablet 3759209338e14437aff5f92a2fff36da from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:60552: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-3759209338e14437aff5f92a2fff36da
I20260808 01:50:08.088886 11654 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:08.089663 11654 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:08.090126 11662 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:07.992710 11443 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:07.946734 11581 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260808 01:50:08.090927 11443 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:08.091337 11443 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:08.039072 11679 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:07.996981 11673 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:08.019093  9555 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:07.997552 11578 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:08.039258  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:08.104444  9634 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:08.106230  9634 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:07.946475 11675 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.075s
I20260808 01:50:08.081701  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1570) took 1110 ms. Trace:
I20260808 01:50:08.083565  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:08.086629  9630 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:08.025393 11470 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:50:08.087628  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 65294.9us
I20260808 01:50:08.111271  9631 rpcz_store.cc:276] 0808 01:50:06.977180 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:07.544646 (+567466us) service_pool.cc:224] Handling call
0808 01:50:07.544918 (+   272us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:07.576385 (+ 31467us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:07.576434 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:07.580022 (+  3588us) log.cc:844] Serialized 26 byte log entry
0808 01:50:07.638652 (+ 58630us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:07.727378 (+ 88726us) log.cc:844] Serialized 14 byte log entry
0808 01:50:07.753782 (+ 26404us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:07.753871 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:08.081364 (+327493us) raft_consensus.cc:1685] finished
0808 01:50:08.081399 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:08.081612 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56638,"prepare.queue_time_us":147315,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":1142,"raft.queue_time_us":60257,"raft.run_cpu_time_us":390,"raft.run_wall_time_us":32975,"thread_start_us":94042,"threads_started":2,"wal-append.queue_time_us":135933}
I20260808 01:50:08.041695 11608 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:08.038188 11487 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.214s	user 0.016s	sys 0.023s
W20260808 01:50:08.123772 11487 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.301s	user 0.016s	sys 0.023s
I20260808 01:50:08.124089 11487 raft_consensus.cc:697] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:08.087859 11495 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:08.033480  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:08.092377  9636 raft_consensus.cc:2393] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:08.090256 11654 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:08.084481 10369 rpcz_store.cc:276] 0808 01:50:05.017146 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:05.024308 (+  7162us) service_pool.cc:224] Handling call
0808 01:50:05.024708 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:06.191225 (+1166517us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:08.009810 (+1818585us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400646c54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:08.010023 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2445075200}
I20260808 01:50:08.081171 11389 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:08.035612 11587 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:08.034313 11565 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:08.122026  9521 leader_election.cc:336] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.580s (SENT)
I20260808 01:50:08.157773  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:08.122761  9631 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:08.093452  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:08.125048 11487 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:08.132148 11377 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260808 01:50:08.133317 11667 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3.
I20260808 01:50:08.082556 11650 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:08.041618 10363 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:08.024093 11448 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:08.148267 11639 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:08.094048 11637 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:08.134207 11608 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:08.092253 11683 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:08.147846  9551 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:08.196401  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:08.197374  9551 raft_consensus.cc:2368] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3.
I20260808 01:50:08.100121 11662 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:08.153493 10332 leader_election.cc:400] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5.
I20260808 01:50:08.172896 11456 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:08.152572 11567 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:08.170280  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48292.7us
I20260808 01:50:08.198465  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:08.208564  9554 raft_consensus.cc:2368] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3.
W20260808 01:50:08.179250  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:07.956583 11467 ts_tablet_manager.cc:1469] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Trace:
0808 01:49:59.093662 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:49:59.093880 (+   218us) service_pool.cc:224] Handling call
0808 01:49:59.459344 (+365464us) ts_tablet_manager.cc:934] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
0808 01:50:01.063234 (+1603890us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:50:02.160615 (+1097381us) tablet_metadata.cc:665] Metadata flushed
0808 01:50:02.168456 (+  7841us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:50:02.474190 (+305734us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:50:02.475171 (+   981us) tablet_replica.cc:206] Creating consensus instance
0808 01:50:02.475755 (+   584us) ts_tablet_manager.cc:989] Replica registered.
0808 01:50:02.475927 (+   172us) inbound_call.cc:177] Queueing success response
0808 01:50:03.847627 (+1371700us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0808 01:50:06.821329 (+2973702us) ts_tablet_manager.cc:1435] Starting tablet replica
0808 01:50:07.051180 (+229851us) tablet_replica.cc:261] Starting instrumentation
0808 01:50:07.052041 (+   861us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":34741,"raft.queue_time_us":2177,"raft.run_cpu_time_us":533,"raft.run_wall_time_us":20985,"tablet-copy.queue_time_us":287399,"tablet-download-pool.queue_time_us":329,"tablet-download-pool.run_cpu_time_us":1674,"tablet-download-pool.run_wall_time_us":947445,"thread_start_us":109376,"threads_started":2}
I20260808 01:50:08.158392 11651 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:08.181917 11685 raft_consensus.cc:1081] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Signalling peer 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:50:08.194388 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:08.202268 10332 leader_election.cc:403] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20260808 01:50:08.154263  9550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 1915) took 2106 ms (client timeout 1675 ms). Trace:
I20260808 01:50:08.202916  9473 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:08.180505 11650 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:08.099967 11539 consensus_queue.cc:260] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:08.208545  9521 leader_election.cc:336] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.636s (SENT)
I20260808 01:50:08.236078  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1750) took 1359 ms. Trace:
I20260808 01:50:08.249017  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:08.040855 11575 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:08.208966  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:08.249138  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:08.174360  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:08.212586  9551 raft_consensus.cc:2368] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3.
I20260808 01:50:08.252205  9555 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:08.252607  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:08.213066  9478 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:50:08.212540  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:08.254949  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:08.193034 11637 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:50:08.247040 11682 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
I20260808 01:50:08.221243 11565 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:08.250245  9636 rpcz_store.cc:276] 0808 01:50:06.876584 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:07.479294 (+602710us) service_pool.cc:224] Handling call
0808 01:50:08.235818 (+756524us) spinlock_profiling.cc:243] Waited 552 ms on lock 0x7b5400421dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:08.236003 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1268456192}
I20260808 01:50:08.260437  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:08.261102  9555 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:50:08.262311 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 109015us
W20260808 01:50:08.252642  9521 leader_election.cc:336] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.701s (SENT)
I20260808 01:50:08.266664  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:08.193399 11639 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:08.269584  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 300859us
I20260808 01:50:08.269968  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:08.271179  9478 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:08.271984 11647 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5.
W20260808 01:50:08.249297  9550 rpcz_store.cc:269] 0808 01:50:06.042114 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:06.070196 (+ 28082us) service_pool.cc:224] Handling call
0808 01:50:08.148382 (+2078186us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54003142d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:08.148579 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":289525120}
I20260808 01:50:08.272234  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:08.134315 11668 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:08.274132  9481 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:08.275024  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:08.276890  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:08.278311  9551 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:08.279377  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:08.274147  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:08.280236  9551 raft_consensus.cc:2368] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 4. Current term is 5.
I20260808 01:50:08.280884  9550 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:08.248523 11646 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:08.277554  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:08.285996  9636 raft_consensus.cc:2376] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6. Re-sending same reply.
I20260808 01:50:08.254861 11434 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:08.288756  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:08.289455  9481 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:08.269090 11590 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:08.295715 11685 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:08.221462 11567 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260808 01:50:08.298316  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:08.307490  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:08.221980 11389 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:08.270270 11456 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260808 01:50:08.318425 11567 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:08.319217 11567 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:08.320369 11448 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:08.283793 11626 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260808 01:50:08.324820  9570 tablet_copy_service.cc:342] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-affdd24b03ae4ee795a90703613305f7 received from {username='slave'} at 127.0.0.1:60552
I20260808 01:50:08.325802  9570 tablet_copy_service.cc:434] P 52ccf4dcca1843f7bb29e9172e2d4ab4: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-affdd24b03ae4ee795a90703613305f7 on tablet affdd24b03ae4ee795a90703613305f7 with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:08.326224 11456 raft_consensus.cc:3055] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:08.326921 11456 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:08.328030  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:08.273860 11565 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:08.334806 11660 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.651s	user 0.010s	sys 0.005s
I20260808 01:50:08.331660  9478 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:08.343886  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:08.345093  9551 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:50:08.351590  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:08.357882  9571 tablet_copy_source_session.cc:215] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:50:08.358312  9481 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:50:08.364014  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:08.366998  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:08.373610 11636 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:08.374583 11636 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:08.334924 11456 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:08.376402 11685 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:08.383284 11565 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:08.386761  9555 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
W20260808 01:50:08.387084  9552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 1912) took 2399 ms (client timeout 1693 ms). Trace:
W20260808 01:50:08.432754  9552 rpcz_store.cc:269] 0808 01:50:05.988091 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:06.047065 (+ 58974us) service_pool.cc:224] Handling call
0808 01:50:08.386852 (+2339787us) spinlock_profiling.cc:243] Waited 348 ms on lock 0x7b540002c1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:08.387008 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":801098112}
I20260808 01:50:08.391824  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:08.336385 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:08.347401 11660 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:08.411415 11608 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:08.415505 10360 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 338) took 3245 ms (client timeout 2091 ms). Trace:
I20260808 01:50:08.419924 10368 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260808 01:50:08.420425  9634 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:08.426163  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:08.428279  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:08.437434  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:08.448287  9555 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:50:08.449386  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:08.451597  9638 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 2: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:08.459633  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:08.427407 11654 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:08.445925 10360 rpcz_store.cc:269] 0808 01:50:05.169643 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:05.172785 (+  3142us) service_pool.cc:224] Handling call
0808 01:50:08.415079 (+3242294us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b540037dcd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:08.415348 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3479830400}
I20260808 01:50:08.462594  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:08.452703  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:08.464159  9635 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 5: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:08.464907  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:08.315008 11683 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:08.418414 11679 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:08.466114  9475 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1869) took 2560 ms (client timeout 2408 ms). Trace:
I20260808 01:50:08.470851  9552 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
I20260808 01:50:08.470014  9638 raft_consensus.cc:2393] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 2: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:08.466423  9551 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:08.479578  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:08.480083  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:08.482885  9638 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:08.484308  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:08.488796  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:08.476092  9475 rpcz_store.cc:269] 0808 01:50:05.905619 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:08.465806 (+2560187us) service_pool.cc:211] Skipping call since client already timed out
0808 01:50:08.465993 (+   187us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260808 01:50:08.490535  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:08.490586  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:08.498098  9550 raft_consensus.cc:2393] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 2: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:08.500248  9634 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 1.
I20260808 01:50:08.502878  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:08.468024 11639 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:08.508471  9481 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:08.473507 11654 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:08.513850  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:08.517916  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:08.519431  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:08.523272 11636 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:08.525429 11661 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.863s	user 0.013s	sys 0.000s
I20260808 01:50:08.526970 11611 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:08.529206 11608 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:08.537645 11679 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:08.498081 11560 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:08.538337 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:08.487653 11565 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:08.541582  9521 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:08.546977  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:08.559279  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:08.559999  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:08.561116  9634 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 2: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:08.561964  9475 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:08.563048 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:08.563227  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:08.563853 10369 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 2: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:08.564973  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:08.526096 11661 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:08.553791 11443 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:08.596190  9445 connection.cc:256] RPC call timeout handler was delayed by -1.5453937999845948s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20910000000000004s for a response
I20260808 01:50:08.572371 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:08.596832  9445 connection.cc:256] RPC call timeout handler was delayed by -1.4310937999980524s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2218s for a response
I20260808 01:50:08.599007  9445 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:08.608665  9552 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260808 01:50:08.621071 10334 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:08.620045  9552 pending_rounds.cc:85] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 4
I20260808 01:50:08.597437  9634 raft_consensus.cc:2393] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:08.604549 11448 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:08.616235 11612 raft_consensus.cc:993] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:50:08.629952 11612 raft_consensus.cc:1081] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Signalling peer 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
W20260808 01:50:08.617926  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:08.631551  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:08.632300  9636 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:50:08.633903 11673 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.533s	user 0.005s	sys 0.008s
W20260808 01:50:08.636904 11578 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.515s	user 0.014s	sys 0.001s
I20260808 01:50:08.576082 11573 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:08.622361  9552 pending_rounds.cc:107] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260808 01:50:08.556119 11675 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20260808 01:50:08.618356 11495 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:08.611531 11613 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:08.639727  9552 raft_consensus.cc:2889] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:08.627245 11631 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:08.573009 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:08.560746 11565 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:08.566931 11679 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.072s
W20260808 01:50:08.644611  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 45700.2us
I20260808 01:50:08.625775  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:08.651544 11631 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:08.637538 11578 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:08.624878 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:08.660362  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:08.661630  9636 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:50:08.664157 11660 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:08.672268  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:08.640693 11573 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:08.676110  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:08.675542  9481 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 1.
I20260808 01:50:08.678581  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:08.659137  9445 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:08.679612  9636 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:08.652446 11448 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:08.653801 11611 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:08.683599  9552 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 01:50:08.693982  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:08.707669  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 48664.8us
I20260808 01:50:08.722259  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:08.723922  9479 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:08.682775 10360 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 1.
I20260808 01:50:08.725872  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:08.732470  9478 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:08.732187  9445 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:08.707084 11655 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.000s	sys 0.014s
I20260808 01:50:08.742587  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:08.743700  9479 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:08.744221 11608 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:50:08.745121  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:08.746177  9479 raft_consensus.cc:2368] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3.
I20260808 01:50:08.681354 11637 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:08.759581 11443 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:08.760509 11567 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:08.761832 11495 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:08.762902 11691 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:08.763592 11691 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:08.765571  9637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1573) took 1786 ms (client timeout 1841 ms). Trace:
I20260808 01:50:08.768069  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 1935) took 1113 ms. Trace:
I20260808 01:50:08.769080  9445 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:08.773667  9474 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
W20260808 01:50:08.777117 11650 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.510s	user 0.008s	sys 0.008s
I20260808 01:50:08.777827 11639 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:08.778510 11639 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:08.778951 11639 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:08.779651 11578 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:08.781126 11631 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:08.783407 11567 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:08.777861 11650 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:08.804526  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:08.805404  9479 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:08.768122 11661 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:08.791355 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:08.808215 11636 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:08.768730  9560 rpcz_store.cc:276] 0808 01:50:07.655117 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:07.655808 (+   691us) service_pool.cc:224] Handling call
0808 01:50:08.767824 (+1112016us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b54000de154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:08.767964 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":574355200}
I20260808 01:50:08.809023 11636 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:50:08.739795 11655 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.000s	sys 0.015s
I20260808 01:50:08.792003 11608 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:50:08.766093  9637 rpcz_store.cc:269] 0808 01:50:06.979265 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:07.418011 (+438746us) service_pool.cc:224] Handling call
0808 01:50:08.765341 (+1347330us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b540008c854. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:08.765497 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":436795136}
I20260808 01:50:08.740176  9636 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:50:08.809792 11636 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:08.819905  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:08.827540  9632 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:08.803885 11691 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:08.841442  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:08.849889 11649 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:08.865146 11389 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:08.880307 11650 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:08.829255 11608 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:08.893844  9636 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:08.945624  9636 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260808 01:50:08.878613 11470 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.717s	user 0.010s	sys 0.007s
I20260808 01:50:08.838382 11655 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:08.947788  9637 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:50:08.936631  9479 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
W20260808 01:50:08.948879  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 85855.5us
W20260808 01:50:08.951172  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 1.02931e+06us
I20260808 01:50:08.947794  9636 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: [2.3-2.3]
W20260808 01:50:08.979957  9635 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 197) took 2140 ms (client timeout 1544 ms). Trace:
W20260808 01:50:08.980552  9635 rpcz_store.cc:269] 0808 01:50:06.839198 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:06.862447 (+ 23249us) service_pool.cc:224] Handling call
0808 01:50:08.351397 (+1488950us) spinlock_profiling.cc:243] Waited 3.58 ms on lock 0x7b5800231698. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:50:08.979631 (+628234us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54003fb454. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:08.979845 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":264729856}
I20260808 01:50:08.983760 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:08.985795  9637 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260808 01:50:08.988981 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:08.946875 11575 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:08.991655 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:08.999092  9634 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:50:08.999472 10365 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 2: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:08.947122 11573 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:09.010125  9523 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.007s	sys 0.032s
I20260808 01:50:09.013511 10365 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260808 01:50:09.015079  9523 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.007s	sys 0.033s
I20260808 01:50:08.954643 11470 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:08.995263 11565 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:09.014658  9555 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.000s	sys 0.027s
W20260808 01:50:09.014066  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 495575us
W20260808 01:50:09.075909  9551 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.004s	sys 0.036s
I20260808 01:50:08.999645 11568 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:09.006861 11622 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:09.077325  9551 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.530s	user 0.004s	sys 0.036s
I20260808 01:50:09.004609 11620 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:09.005191 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:08.977504 11650 consensus_queue.cc:1059] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260808 01:50:08.975296 11655 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:09.076350  9555 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.000s	sys 0.039s
I20260808 01:50:09.085285  9630 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 4. Ops: []
I20260808 01:50:09.088040 10363 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:09.075495  9524 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:09.021109  9637 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260808 01:50:09.030304  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.742s	user 0.004s	sys 0.035s
I20260808 01:50:08.999759 11593 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260808 01:50:09.083258 10369 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
W20260808 01:50:09.108978  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.824s	user 0.004s	sys 0.035s
I20260808 01:50:09.084229 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:50:09.109894  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 832970us
I20260808 01:50:09.110096 11581 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:50:09.111800  9444 leader_election.cc:400] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4.
I20260808 01:50:09.112309  9444 leader_election.cc:403] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:50:09.082602 11587 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260808 01:50:09.115685  9444 leader_election.cc:400] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3.
I20260808 01:50:09.121661  9444 leader_election.cc:403] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:50:09.122573 11656 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3.
I20260808 01:50:09.123322 11675 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:09.123723 11615 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:50:09.124531  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:09.125885  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:09.125826 11675 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:09.129112  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:09.130558  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:09.082438 11622 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.082479 11568 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.135314 11593 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:09.135917 11593 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:09.137387 11539 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:09.137095 11593 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:09.137917 11639 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:09.157948 11639 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:09.125972 11615 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:09.140554 11622 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:09.144454 11448 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:09.158862 10361 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:50:09.159344 11448 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:50:09.148676 11613 consensus_meta.cc:287] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.502s	user 0.005s	sys 0.011s
I20260808 01:50:09.158921 11615 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:09.120965 11691 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4.
I20260808 01:50:09.146366 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:09.106220 11575 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:09.153750  9373 catalog_manager.cc:5697] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 1 to 2, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 2 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:50:09.147794  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:09.146209 11470 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:09.148623 11456 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:09.163803 11608 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:09.164072 11694 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260808 01:50:09.164914  9631 raft_consensus.cc:2410] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:50:09.166618 11637 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:09.150219  9599 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:09.167871  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:09.154340  9554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 1934) took 1499 ms (client timeout 1676 ms). Trace:
W20260808 01:50:09.169040  9554 rpcz_store.cc:269] 0808 01:50:07.654706 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:07.654858 (+   152us) service_pool.cc:224] Handling call
0808 01:50:09.154145 (+1499287us) spinlock_profiling.cc:243] Waited 522 ms on lock 0x7b54003264d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:09.154276 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1200576256}
I20260808 01:50:09.133087 11581 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260808 01:50:09.167908 11575 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:09.171044 10368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 441) took 1707 ms. Trace:
I20260808 01:50:09.170828 11575 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:09.084033 11620 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:09.171399 10368 rpcz_store.cc:276] 0808 01:50:07.464159 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:07.464746 (+   587us) service_pool.cc:224] Handling call
0808 01:50:08.419525 (+954779us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:08.446343 (+ 26818us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:09.170825 (+724482us) spinlock_profiling.cc:243] Waited 944 ms on lock 0x7b540037dcd0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:09.170982 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2170994432}
I20260808 01:50:09.172374 11661 raft_consensus.cc:3055] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:09.172829 11661 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:09.173106 11620 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:09.169770  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:09.175117 11686 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:50:09.175572 11575 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:09.162153 11615 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:09.162204 11573 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:09.175721 11686 raft_consensus.cc:697] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:09.179775  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:09.169939  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 1757) took 1257 ms. Trace:
I20260808 01:50:09.179862 11686 consensus_queue.cc:237] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.180684  9630 rpcz_store.cc:276] 0808 01:50:07.912241 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:08.111884 (+199643us) service_pool.cc:224] Handling call
0808 01:50:08.112240 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:09.085839 (+973599us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:09.169699 (+ 83860us) spinlock_profiling.cc:243] Waited 973 ms on lock 0x7b5400338954. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:09.169869 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2237172480}
I20260808 01:50:09.168804  9556 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:09.183032  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:09.169864 11655 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:09.183970  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 195) took 2355 ms. Trace:
I20260808 01:50:09.185012  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:09.185523 11434 raft_consensus.cc:3055] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:09.185920 11434 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:09.186046 11613 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:09.186949 11434 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:09.187366 11656 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:09.184370  9632 rpcz_store.cc:276] 0808 01:50:06.828627 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:07.530443 (+701816us) service_pool.cc:224] Handling call
0808 01:50:08.771130 (+1240687us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:08.843822 (+ 72692us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:09.178338 (+334516us) spinlock_profiling.cc:243] Waited 56.1 ms on lock 0x7b540008c854. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:09.183789 (+  5451us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b540008c850. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:09.183915 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2947725824}
I20260808 01:50:09.188092 11656 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:09.189011 11656 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:09.187712  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:09.191347  9556 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "439040d168824496acc076b1c280bde3"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:53208
I20260808 01:50:09.173820 11661 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:09.191457 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:09.186980  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:09.192740  9550 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
W20260808 01:50:09.193087  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 64274.2us
I20260808 01:50:09.194123  9635 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260808 01:50:09.194550  9635 pending_rounds.cc:85] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 4
I20260808 01:50:09.194896  9444 leader_election.cc:400] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3.
I20260808 01:50:09.024808  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:09.214802 10368 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:09.214946  9444 leader_election.cc:400] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3.
I20260808 01:50:09.215404  9444 leader_election.cc:403] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260808 01:50:09.216329  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 90527.2us
W20260808 01:50:09.216743  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 192044us
I20260808 01:50:09.217190  9524 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:09.194864  9635 pending_rounds.cc:107] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260808 01:50:09.219038  9635 raft_consensus.cc:2889] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:09.223727 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:09.224920 11661 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:09.225449 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:09.222028  9373 catalog_manager.cc:5697] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 1 to 2, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 2 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:09.227162 11573 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:50:09.227770  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:09.227856  9373 catalog_manager.cc:5697] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 2 to 3, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 3 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:09.229096  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:09.229628  9552 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:09.226760 11626 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:09.231005  9599 leader_election.cc:400] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3.
I20260808 01:50:09.220599 11587 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:09.221283 11655 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:09.232585  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:09.233840  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:09.234349  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:09.231016 11626 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:09.158903 11568 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:09.235487 11626 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:09.236083  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:09.236754  9557 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:09.234413  9550 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:09.239307  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:09.078166  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 1942) took 1380 ms. Trace:
I20260808 01:50:09.237730  9374 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55058 (request call id 124) took 4204 ms. Trace:
W20260808 01:50:09.246016  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 49921.7us
I20260808 01:50:09.246833 11583 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:09.260867 11583 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.263398 11583 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:09.246891 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:09.261281  9444 leader_election.cc:400] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 4. Current term is 5.
I20260808 01:50:09.246903  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:09.264809  9444 leader_election.cc:403] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:50:09.266805  9554 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:09.268252  9554 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:50:09.247790 11622 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3.
I20260808 01:50:09.249816 11615 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.250916  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:09.251772 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:09.272348 10368 raft_consensus.cc:2410] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 01:50:09.272994  9557 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:09.253262 11434 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:50:09.253489 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:09.244906  9374 rpcz_store.cc:276] 0808 01:50:05.050785 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:05.051470 (+   685us) service_pool.cc:224] Handling call
0808 01:50:09.237641 (+4186171us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:50:06.714934 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:50:06.715800 (+   866us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:50:06.715900 (+   100us) write_op.cc:435] Acquired schema lock
0808 01:50:06.715926 (+    26us) tablet.cc:662] Decoding operations
0808 01:50:06.716692 (+   766us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:50:06.716789 (+    97us) write_op.cc:641] Partition lock acquired for write op
0808 01:50:06.716824 (+    35us) tablet.cc:685] Acquiring locks for 3 operations
0808 01:50:06.717259 (+   435us) tablet.cc:701] Row locks acquired
0808 01:50:06.717275 (+    16us) write_op.cc:260] PREPARE: finished
0808 01:50:06.717474 (+   199us) write_op.cc:270] Start()
0808 01:50:06.717618 (+   144us) write_op.cc:276] Timestamp: P: 1786153806717403 usec, L: 0
0808 01:50:06.717638 (+    20us) op_driver.cc:348] REPLICATION: starting
0808 01:50:06.775381 (+ 57743us) log.cc:844] Serialized 1302 byte log entry
0808 01:50:07.668866 (+893485us) op_driver.cc:464] REPLICATION: finished
0808 01:50:07.919863 (+250997us) write_op.cc:301] APPLY: starting
0808 01:50:07.919957 (+    94us) tablet.cc:1367] starting BulkCheckPresence
0808 01:50:07.920202 (+   245us) tablet.cc:1370] finished BulkCheckPresence
0808 01:50:07.948686 (+ 28484us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:50:07.949700 (+  1014us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:50:07.950020 (+   320us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0808 01:50:07.950090 (+    70us) write_op.cc:312] APPLY: finished
0808 01:50:07.950451 (+   361us) log.cc:844] Serialized 35 byte log entry
0808 01:50:07.950737 (+   286us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:07.950911 (+   174us) write_op.cc:454] Released schema lock
0808 01:50:07.950967 (+    56us) write_op.cc:341] FINISH: Updating metrics
0808 01:50:07.951558 (+   591us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:07.951635 (+    77us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":70092,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":249520,"apply.run_cpu_time_us":3671,"apply.run_wall_time_us":31586,"num_ops":3,"prepare.queue_time_us":849533,"prepare.run_cpu_time_us":30074,"prepare.run_wall_time_us":149521,"raft.queue_time_us":96038,"raft.run_cpu_time_us":13779,"raft.run_wall_time_us":140333,"replication_time_us":937293,"thread_start_us":515562,"threads_started":4,"wal-append.queue_time_us":113531,"wal-append.run_cpu_time_us":31375,"wal-append.run_wall_time_us":2042665}]]}
I20260808 01:50:09.245441  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:09.284350  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:09.287788  9599 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:09.273375  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:09.244323  9551 rpcz_store.cc:276] 0808 01:50:07.698026 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:08.492448 (+794422us) service_pool.cc:224] Handling call
0808 01:50:08.519758 (+ 27310us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:09.077778 (+558020us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:09.077811 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:09.077863 (+    52us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:09.077895 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:09.077969 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:09.078107 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27541,"raft.queue_time_us":83856,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":219,"thread_start_us":529943,"threads_started":1}
I20260808 01:50:09.285537  9550 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:09.218773  9524 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:09.297832  9477 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
W20260808 01:50:09.299787  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 81129.2us
I20260808 01:50:09.301425  9524 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:09.302883  9556 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:50:09.304013  9556 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:09.181834 11636 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:09.186964 11613 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.275620 11593 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:50:09.283589 11668 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.019s	sys 0.061s
I20260808 01:50:09.306104  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:09.284281 11639 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 4. Current term is 5.
I20260808 01:50:09.320729  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:09.307747  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:09.320739  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:09.307331 11487 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.858s	user 0.021s	sys 0.041s
I20260808 01:50:09.329358  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:09.330616 10366 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:09.303438 11583 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 01:50:09.321887 11668 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.019s	sys 0.062s
W20260808 01:50:09.309120 11651 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.000s	sys 0.017s
I20260808 01:50:09.338596  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:09.309353 11451 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:09.309011 11389 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:09.304911 11649 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:09.341087  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:09.344470 10367 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260808 01:50:09.323428 11611 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:09.347172  9599 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:09.378458  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:09.383939  9557 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 4
I20260808 01:50:09.379705  9638 raft_consensus.cc:2410] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:50:09.318997 11675 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:50:09.333956 11487 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.897s	user 0.021s	sys 0.041s
I20260808 01:50:09.330582 11655 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 01:50:09.342392 11651 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.000s	sys 0.018s
I20260808 01:50:09.388341  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:09.415136  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:09.406531  9557 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
I20260808 01:50:09.416308  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:09.416563  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:09.389972  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:09.417568  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:09.418118  9479 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:50:09.418390  9476 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.780s	user 0.033s	sys 0.045s
W20260808 01:50:09.418824  9476 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.781s	user 0.034s	sys 0.045s
I20260808 01:50:09.419224  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:09.420732  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:09.391487  9475 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260808 01:50:09.391516 11565 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260808 01:50:09.399533 11703 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:09.400064  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:09.309475 11568 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260808 01:50:09.439757  9480 raft_consensus.cc:2368] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4.
W20260808 01:50:09.402400  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:09.340174 11650 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:09.340752 11620 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:09.416605  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:09.327943 11683 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.710s	user 0.007s	sys 0.012s
I20260808 01:50:09.430552  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:09.431602  9472 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.672s	user 0.008s	sys 0.010s
I20260808 01:50:09.434067  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:09.456805  9474 raft_consensus.cc:2376] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5. Re-sending same reply.
I20260808 01:50:09.431420  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 203) took 1109 ms. Trace:
I20260808 01:50:09.434975  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:09.457811  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:09.435030  9630 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:42840
I20260808 01:50:09.457811  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:09.458662  9630 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:50:09.459012  9474 raft_consensus.cc:2410] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:09.459134  9630 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:09.459594  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:09.459899  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:09.460736  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:09.437397  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:09.470525  9636 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:09.472903  9636 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:09.383556 11611 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:09.438751  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:09.473913 11611 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:09.475147  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:09.476792  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:09.476890  9630 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.477982  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:09.479478 11636 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:09.450111 11691 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:09.483242 11691 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.485817 11691 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:09.445261 11487 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:09.486802  9481 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.045s	sys 0.054s
I20260808 01:50:09.486802 11487 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:09.487582  9481 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.045s	sys 0.054s
I20260808 01:50:09.489362 11487 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:09.449474 11703 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.493541  9473 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: [2.3-2.3]
I20260808 01:50:09.495155  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:09.495586 11611 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.496744 11568 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:09.497326 11568 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.499614 11568 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:09.497347  9480 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260808 01:50:09.503752  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:09.504598  9480 raft_consensus.cc:2393] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 2: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:09.507931 10367 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:50:09.507997  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:09.510385  9635 raft_consensus.cc:2393] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:09.335544 11583 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:09.403640 11578 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.050s
I20260808 01:50:09.403718 11389 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:09.530718 11615 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:09.406941 11655 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:09.451897  9632 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:09.452204  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:09.344043 11689 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.100s
I20260808 01:50:09.403471 11539 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:09.458299  9631 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:09.461035  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:09.461488  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:09.457968  9638 rpcz_store.cc:276] 0808 01:50:08.322465 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:08.488492 (+166027us) service_pool.cc:224] Handling call
0808 01:50:09.415400 (+926908us) spinlock_profiling.cc:243] Waited 878 ms on lock 0x7b5400339354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:09.431350 (+ 15950us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2020155648}
W20260808 01:50:09.461898  9472 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s	user 0.008s	sys 0.010s
I20260808 01:50:09.453742 11622 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:09.381745 11668 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260808 01:50:09.539276  9559 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:50:09.464324  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:09.431097 11654 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:09.464073 11687 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:09.469338  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:09.489591  9630 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:09.405941  9556 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.450598 11650 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:09.450250 11620 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:09.514077 11626 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.514390 11583 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:09.478592  9551 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
I20260808 01:50:09.516410 11656 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:09.519472 11703 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:09.528805 11448 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:09.532255  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:09.408610 11595 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.671s	user 0.017s	sys 0.007s
W20260808 01:50:09.466298 11683 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.838s	user 0.007s	sys 0.015s
I20260808 01:50:09.534327  9557 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:09.540346  9637 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:50:09.545750  9559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 255) took 1506 ms (client timeout 1929 ms). Trace:
W20260808 01:50:09.596482  9559 rpcz_store.cc:269] 0808 01:50:08.039617 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:08.363660 (+324043us) service_pool.cc:224] Handling call
0808 01:50:09.545559 (+1181899us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540002e254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:09.545686 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2493656448}
W20260808 01:50:09.597499  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.024s	user 0.013s	sys 0.035s
W20260808 01:50:09.541085 10332 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.110s	user 0.015s	sys 0.015s
W20260808 01:50:09.597905  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.033s	user 0.013s	sys 0.035s
W20260808 01:50:09.541579  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 80185.6us
I20260808 01:50:09.599287  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:09.530928 11389 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.555051  9475 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:09.535538 11655 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:50:09.584808  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 49011.4us
I20260808 01:50:09.536952 11539 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:09.543365 11650 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:09.587563  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 246) took 1897 ms. Trace:
W20260808 01:50:09.599382  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 1.04009e+06us
I20260808 01:50:09.635253  9521 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:09.633914  9555 rpcz_store.cc:276] 0808 01:50:07.690581 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:08.455111 (+764530us) service_pool.cc:224] Handling call
0808 01:50:08.455549 (+   438us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:08.456366 (+   817us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:08.456410 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:08.464274 (+  7864us) log.cc:844] Serialized 26 byte log entry
0808 01:50:09.092948 (+628674us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:09.586639 (+493691us) log.cc:844] Serialized 14 byte log entry
0808 01:50:09.586951 (+   312us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:09.587170 (+   219us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:09.587221 (+    51us) raft_consensus.cc:1685] finished
0808 01:50:09.587241 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:09.587487 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2124,"raft.run_cpu_time_us":1413,"raft.run_wall_time_us":24201,"thread_start_us":1026459,"threads_started":2,"wal-append.queue_time_us":905869}
I20260808 01:50:09.600932  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:09.605189  9554 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 01:50:09.543481 11620 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:09.647171  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:09.648102  9551 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
W20260808 01:50:09.598042 10332 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.167s	user 0.016s	sys 0.016s
I20260808 01:50:09.536039 11615 consensus_queue.cc:1059] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20260808 01:50:09.588889 11703 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:09.605239  9477 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 2: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:09.625849  9560 raft_consensus.cc:2368] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4.
I20260808 01:50:09.626346  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:09.651198  9559 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:09.627118  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:09.627795  9478 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 3. Current term is 4. Ops: []
W20260808 01:50:09.586447 11595 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.851s	user 0.017s	sys 0.007s
I20260808 01:50:09.541882 11451 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260808 01:50:09.569376 11560 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.752s	user 0.010s	sys 0.018s
I20260808 01:50:09.589027 11587 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:09.628639  9631 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
I20260808 01:50:09.586314 11687 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:09.551095  9552 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
W20260808 01:50:09.644663  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:09.633353 11539 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:09.645406  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:09.552412 11637 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.086s
W20260808 01:50:09.650108 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 1.21936e+06us
I20260808 01:50:09.703290  9559 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:09.590416 11613 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:09.658280  9555 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:09.660903  9556 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:09.568107 11661 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.071s
W20260808 01:50:09.673826  9633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1759) took 1761 ms (client timeout 2155 ms). Trace:
W20260808 01:50:09.707993  9633 rpcz_store.cc:269] 0808 01:50:07.912790 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:08.083301 (+170511us) service_pool.cc:224] Handling call
0808 01:50:09.673603 (+1590302us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b5400337f54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:09.673749 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3437792128}
I20260808 01:50:09.708874  9556 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3. Ops: []
W20260808 01:50:09.686813  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:09.657616 11565 raft_consensus.cc:2764] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20260808 01:50:09.738627  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:09.749231 11626 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:09.736909 11613 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:09.751739 11448 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:09.731470 11704 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260808 01:50:09.628285 11696 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 01:50:09.670194 11560 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.853s	user 0.011s	sys 0.022s
I20260808 01:50:09.702723 11675 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:09.765900  9550 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:09.766865 11608 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:09.767292 11608 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:09.768299 11608 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:09.770285 11702 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260808 01:50:09.741068  9559 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:09.741283  9599 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:09.657586 11622 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:09.751443 11613 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:09.785841  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 74784.2us
I20260808 01:50:09.787525 11613 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:09.755066  9629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1550) took 4523 ms (client timeout 1777 ms). Trace:
I20260808 01:50:09.755587  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 222) took 1124 ms. Trace:
I20260808 01:50:09.788851  9476 rpcz_store.cc:276] 0808 01:50:08.631550 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:08.636149 (+  4599us) service_pool.cc:224] Handling call
0808 01:50:08.636550 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:08.636954 (+   404us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:08.636985 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:08.637637 (+   652us) log.cc:844] Serialized 26 byte log entry
0808 01:50:09.419262 (+781625us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:09.419307 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:09.419386 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:09.755338 (+335952us) raft_consensus.cc:1685] finished
0808 01:50:09.755362 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:09.755518 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1211,"raft.queue_time_us":1494,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":4916,"thread_start_us":781042,"threads_started":1,"wal-append.queue_time_us":1073420}
W20260808 01:50:09.790477  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 145209us
I20260808 01:50:09.791791  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:09.739287  9551 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
W20260808 01:50:09.681512 11699 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260808 01:50:09.788763  9629 rpcz_store.cc:269] 0808 01:50:05.218145 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:05.242308 (+ 24163us) service_pool.cc:224] Handling call
0808 01:50:09.711775 (+4469467us) spinlock_profiling.cc:243] Waited 4.45 s on lock 0x7b54003fa054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:09.741062 (+ 29287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10238751104}
I20260808 01:50:09.762365 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:09.665573 11703 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.771593 11656 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:09.772354 11712 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
I20260808 01:50:09.773100 11631 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:09.658706 11639 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:09.763420  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:09.779838  9637 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:09.784890 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:09.784750 11622 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.786865 11668 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:09.787469 11683 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:09.647348 11700 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:09.793764  9599 leader_election.cc:400] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4.
W20260808 01:50:09.795537 11710 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260808 01:50:09.799592  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:09.807135  9524 leader_election.cc:400] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4.
I20260808 01:50:09.832815  9524 leader_election.cc:403] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:50:09.776787 11615 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:09.810581  9601 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.265s	user 0.062s	sys 0.084s
W20260808 01:50:09.834116  9601 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.288s	user 0.063s	sys 0.084s
W20260808 01:50:09.834553  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 1.28904e+06us
I20260808 01:50:09.804378  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:09.823644  9373 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.008s	sys 0.045s
I20260808 01:50:09.801183 11448 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:50:09.826421  9558 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.659s	user 0.031s	sys 0.048s
W20260808 01:50:09.874737  9558 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.708s	user 0.031s	sys 0.048s
I20260808 01:50:09.830358  9599 leader_election.cc:403] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
W20260808 01:50:09.806667 11686 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.027s	sys 0.050s
W20260808 01:50:09.856072  9373 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.009s	sys 0.045s
W20260808 01:50:09.859752 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:09.808200 11611 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:09.893553  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 99995.5us
I20260808 01:50:09.812839 11451 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:09.806344 11636 consensus_queue.cc:1059] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
I20260808 01:50:09.826051 11668 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:09.895305  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 477782us
W20260808 01:50:09.880466 11686 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.028s	sys 0.050s
I20260808 01:50:09.915302  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:09.917892  9637 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260808 01:50:09.920444  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:09.931588  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 124633us
I20260808 01:50:09.932756  9524 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:09.934551  9629 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:09.937130  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:09.940522  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:09.923041  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:09.916318 11661 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:09.826894 11639 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.920547  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:09.942332  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:09.943565  9638 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:09.943578  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:09.944972  9633 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:50:09.892417 10360 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:09.940863  9632 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:50:09.946636  9521 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.906s (SENT) [suppressed 47 similar messages]
I20260808 01:50:09.923555 11668 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:09.814657 11631 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:09.946666  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:09.937919 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 207525us
I20260808 01:50:09.955067  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:09.832111 11683 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.893952 11615 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:09.969105  9524 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:09.966424  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:09.925005 11655 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.829703 11687 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:09.947563 11651 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:09.952565 11700 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:09.923632 11674 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260808 01:50:09.936300 11709 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4.
I20260808 01:50:09.990856  9550 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:50:09.975369  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:09.979574  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:09.979955  9629 raft_consensus.cc:2376] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3. Re-sending same reply.
I20260808 01:50:09.981812  9632 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:09.993239  9633 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:09.993270  9631 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:09.967885 11451 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:10.009357  9635 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:09.973904 11482 raft_consensus.cc:3055] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:09.972707 11613 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:10.012835  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:09.989050  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:09.973431 11626 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:09.966535 11611 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
W20260808 01:50:10.029434  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 82793.6us
I20260808 01:50:10.031090  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:09.970710 11691 consensus_queue.cc:1059] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260808 01:50:09.973665 11448 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20260808 01:50:10.033294  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:09.995996 11651 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:09.982982 11722 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:09.973650 11456 consensus_queue.cc:1059] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20260808 01:50:09.995484 11700 raft_consensus.cc:697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:10.017040 11482 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:10.007731 11593 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:10.057605  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:10.063465  9521 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:10.043658 11693 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260808 01:50:10.064898 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 94619.3us
I20260808 01:50:10.094355  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:50:10.054087 11613 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:10.103252  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:10.079151 11568 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:10.155777 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 71621.7us
I20260808 01:50:10.156900 10332 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:10.158567 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:10.089942 11703 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:10.103343 11622 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:10.140388 11448 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:10.086141 11593 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:10.098889 11649 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
W20260808 01:50:10.204319  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:10.154623 11451 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260808 01:50:10.169032 11708 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.338s	user 0.001s	sys 0.001s
I20260808 01:50:10.224177  9631 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
W20260808 01:50:10.191011  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 143586us
W20260808 01:50:10.233784  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 178023us
I20260808 01:50:10.234959  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:10.236613  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:10.166198 11573 consensus_queue.cc:1059] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
I20260808 01:50:10.217868 11622 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:10.155153 11709 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.071s
I20260808 01:50:10.250722  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:10.231513 11711 log.cc:927] Time spent T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.292s	user 0.002s	sys 0.000s
I20260808 01:50:10.167287 11675 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20260808 01:50:10.173535 11456 consensus_queue.cc:1059] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20260808 01:50:10.162087 11568 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:10.246272 11656 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:10.230589 11643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3759209338e14437aff5f92a2fff36da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:50:10.222991 11639 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:10.281229  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 125800us
W20260808 01:50:10.279474 11729 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260808 01:50:10.256240 11722 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:10.245254 11631 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:10.281952  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:10.279258 11568 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:10.221686 11482 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:10.288709  9550 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:10.290508  9601 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:10.294384  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:10.285559  9521 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:10.294574  9601 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:10.219316 11700 consensus_queue.cc:237] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:10.276072 11683 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:10.277571 11741 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:10.298638  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1885) took 1606 ms. Trace:
W20260808 01:50:10.288683 11721 log.cc:927] Time spent T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.265s	user 0.002s	sys 0.000s
I20260808 01:50:10.303086  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1893) took 1586 ms. Trace:
I20260808 01:50:10.313341  9472 rpcz_store.cc:276] 0808 01:50:08.692181 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:08.692827 (+   646us) service_pool.cc:224] Handling call
0808 01:50:08.693262 (+   435us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:08.697426 (+  4164us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:08.697475 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:08.713626 (+ 16151us) log.cc:844] Serialized 26 byte log entry
0808 01:50:09.537548 (+823922us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:09.537590 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:09.537655 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:10.298383 (+760728us) raft_consensus.cc:1685] finished
0808 01:50:10.298412 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:10.298572 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":778307,"threads_started":1,"wal-append.queue_time_us":1023793}
I20260808 01:50:10.319505  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:10.320868  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:10.321481  9472 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:10.267724 11613 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:10.321926  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:10.306319  9599 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:10.323289 11744 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:50:10.323846 11731 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
I20260808 01:50:10.299098  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:10.295660 11719 log.cc:927] Time spent T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.280s	user 0.000s	sys 0.002s
I20260808 01:50:10.322710  9477 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:10.329694  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:10.330650  9477 raft_consensus.cc:3055] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:10.331089  9477 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:10.323706 11434 raft_consensus.cc:3055] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:10.311039  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:10.308012  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:10.317174  9481 rpcz_store.cc:276] 0808 01:50:08.716317 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:08.779408 (+ 63091us) service_pool.cc:224] Handling call
0808 01:50:08.779903 (+   495us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:08.803570 (+ 23667us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:08.803607 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:08.809668 (+  6061us) log.cc:844] Serialized 26 byte log entry
0808 01:50:09.487961 (+678293us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:09.487999 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:09.488069 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:10.302742 (+814673us) raft_consensus.cc:1685] finished
0808 01:50:10.302782 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:10.302993 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23104,"raft.queue_time_us":23985,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":1837,"thread_start_us":677788,"threads_started":1,"wal-append.queue_time_us":1005111}
I20260808 01:50:10.267220 11622 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:10.244750 11651 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:10.326624  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:10.337101  9480 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:10.326612  9521 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:10.340502  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:10.348737  9521 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:10.331993  9477 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:10.289740 11451 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260808 01:50:10.317096 11741 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:10.338980 11691 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:10.334537 11703 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260808 01:50:10.366298 11703 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:10.357357  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:10.367871  9481 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 6
I20260808 01:50:10.296018 11739 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
W20260808 01:50:10.360164 11620 consensus_meta.cc:287] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.690s	user 0.009s	sys 0.002s
I20260808 01:50:10.359184 11675 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:50:10.360137 11740 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20260808 01:50:10.361217 11748 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:10.293591 11573 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260808 01:50:10.361613 11622 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:10.290225 11639 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260808 01:50:10.326190 11722 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:10.352895  9477 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:10.328006 11631 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:50:10.323338 11709 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:10.336642 11434 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:10.356822 11482 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:10.367149  9480 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:10.375608 11691 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:10.358458  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 1945) took 1193 ms. Trace:
I20260808 01:50:10.422411  9558 rpcz_store.cc:276] 0808 01:50:09.165342 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:09.165494 (+   152us) service_pool.cc:224] Handling call
0808 01:50:09.165853 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:09.166106 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:09.166129 (+    23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:09.166604 (+   475us) log.cc:844] Serialized 26 byte log entry
0808 01:50:09.875463 (+708859us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:09.875532 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:09.875625 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:10.358220 (+482595us) raft_consensus.cc:1685] finished
0808 01:50:10.358238 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:10.358394 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":708324,"threads_started":1,"wal-append.queue_time_us":776610}
W20260808 01:50:10.356885 11725 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260808 01:50:10.298226 11656 consensus_queue.cc:1059] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20260808 01:50:10.375918  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:10.370850 11703 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:10.371438 11683 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:10.371881 11689 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:10.368048 11741 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:10.372807 11451 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Stepping down as leader of term 2
W20260808 01:50:10.413542  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 107961us
W20260808 01:50:10.415144 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:10.368746  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:10.393523 11539 consensus_meta.cc:287] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.656s	user 0.013s	sys 0.000s
I20260808 01:50:10.419999 11568 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:10.375783 11742 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:10.391795 11583 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.803s	user 0.013s	sys 0.000s
I20260808 01:50:10.392289 11608 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:10.392339 11573 raft_consensus.cc:3055] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:10.428716 11683 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:50:10.420403  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 84828us
I20260808 01:50:10.445194  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:10.445372  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:10.384501 11620 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:10.443778  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 95227.4us
I20260808 01:50:10.455341  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:10.454599  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:10.454598  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:10.461913  9521 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:10.428448 11631 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:50:10.428594  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 131985us
I20260808 01:50:10.460008  9634 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
I20260808 01:50:10.462219  9629 raft_consensus.cc:2393] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:10.457072  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:10.442160 11451 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:10.452692 11568 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:10.442233 11722 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 4 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:10.466243  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:10.466259  9521 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:10.463104  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:10.461680 11683 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:10.457221 11573 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:10.474943  9477 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:10.482177  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:10.483767  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:10.484858  9477 raft_consensus.cc:2393] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:10.386878 11749 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:10.458186 11583 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:10.480652  9481 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:10.453300 11539 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:10.470331 11568 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:10.458827 11742 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:10.442304 11689 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:10.489476  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:11.189481  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1905) took 1116 ms. Trace:
I20260808 01:50:11.189893  9481 rpcz_store.cc:276] 0808 01:50:10.073013 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:10.489077 (+416064us) service_pool.cc:224] Handling call
0808 01:50:11.189403 (+700326us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:11.191210  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:11.192575  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:11.193234  9481 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:50:11.193825  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 237) took 1020 ms. Trace:
I20260808 01:50:10.474633 11756 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:11.194583 11756 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:11.197139 11756 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:10.465708 11748 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260808 01:50:10.502898 11539 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:10.503587 11583 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:10.488018 11749 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:10.514909 11620 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:10.461351 11434 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:50:10.519218 11451 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:10.520761 11703 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:10.493661 11709 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:10.522670 11689 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:10.525115 11650 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.860s	user 0.014s	sys 0.000s
I20260808 01:50:11.198632 11749 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:50:10.508139 11730 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260808 01:50:10.529603 11683 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:10.530777 11661 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.583s	user 0.008s	sys 0.007s
I20260808 01:50:11.201692 11661 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4.
I20260808 01:50:10.522857 11631 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:10.514461 11742 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:11.203081 11683 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:11.204043  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 1971) took 1696 ms. Trace:
I20260808 01:50:11.204452  9560 rpcz_store.cc:276] 0808 01:50:09.507676 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:09.712450 (+204774us) service_pool.cc:224] Handling call
0808 01:50:09.753457 (+ 41007us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:09.789854 (+ 36397us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:09.789906 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:09.790479 (+   573us) log.cc:844] Serialized 26 byte log entry
0808 01:50:10.035269 (+244790us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:10.035328 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:10.035421 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:10.626530 (+591109us) spinlock_profiling.cc:243] Waited 9.25 ms on lock 0x7b48000c03d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:50:11.203715 (+577185us) raft_consensus.cc:1685] finished
0808 01:50:11.203741 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:11.203966 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38411,"raft.queue_time_us":36041,"raft.run_cpu_time_us":5564,"raft.run_wall_time_us":836864,"spinlock_wait_cycles":21265280,"thread_start_us":244326,"threads_started":1,"wal-append.queue_time_us":582437}
I20260808 01:50:10.487888 11722 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:10.540891 11675 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:10.552454 11744 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:10.584136 11754 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:50:10.586040 11668 consensus_meta.cc:287] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.618s	user 0.009s	sys 0.006s
I20260808 01:50:11.198906  9481 rpcz_store.cc:276] 0808 01:50:10.173829 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.192265 (+1018436us) service_pool.cc:224] Handling call
0808 01:50:11.193760 (+  1495us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:11.207628 11668 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:11.208511  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:11.200210 11650 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:11.209985 11668 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:10.625659 11757 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:11.210824 11757 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:11.211409 11650 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:11.213171 11757 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:10.628611 11758 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:10.634848 11760 consensus_queue.cc:1059] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260808 01:50:10.651300 11626 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.619s	user 0.012s	sys 0.005s
I20260808 01:50:10.662490 11568 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:11.216360 11568 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:10.675091 11639 raft_consensus.cc:3055] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:11.217077 11639 raft_consensus.cc:740] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:11.218048 11639 consensus_queue.cc:260] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:11.219110 11639 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:10.702152 11741 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:10.690441 11643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:56276 (request call id 360) took 4775 ms. Trace:
I20260808 01:50:11.223064 11643 rpcz_store.cc:276] 0808 01:50:05.914622 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:06.196239 (+281617us) service_pool.cc:224] Handling call
0808 01:50:06.820937 (+624698us) ts_tablet_manager.cc:934] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
0808 01:50:09.956518 (+3135581us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:50:10.506106 (+549588us) tablet_metadata.cc:665] Metadata flushed
0808 01:50:10.506240 (+   134us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:50:10.689018 (+182778us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:50:10.689613 (+   595us) tablet_replica.cc:206] Creating consensus instance
0808 01:50:10.690167 (+   554us) ts_tablet_manager.cc:989] Replica registered.
0808 01:50:10.690273 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":156843,"tablet-copy.queue_time_us":286485,"thread_start_us":210051,"threads_started":2}
I20260808 01:50:11.224591 11741 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:11.224888 11643 tablet_copy_client.cc:806] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:50:10.775537 11691 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:11.225675 11643 tablet_copy_client.cc:670] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:50:11.225569 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:10.782027 11389 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:11.226207 11758 raft_consensus.cc:3055] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:10.828655 11647 raft_consensus.cc:3055] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:11.226092 11691 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:11.226850 11647 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:11.227453  9631 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:11.225504 11626 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:50:10.819264 10360 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 446) took 1855 ms (client timeout 2042 ms). Trace:
I20260808 01:50:10.953648 11487 raft_consensus.cc:3055] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
W20260808 01:50:10.947751 11613 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.003s	sys 0.000s
I20260808 01:50:11.227885 11647 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:11.228539 11487 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:10.998109 11448 raft_consensus.cc:3055] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:11.229336 11647 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:10.956136 11593 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:11.229629 11448 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:11.230194  9556 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:10.889313 11615 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:11.001955 11693 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:11.230707 11448 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:11.229540 11487 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:11.231031 11389 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:11.232098 11448 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:11.007268  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:10.999776  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:11.233848 11389 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:11.234375  9480 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:11.231065 11693 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:11.236747  9638 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:11.001474 11654 raft_consensus.cc:3055] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:11.049293  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:11.052234 11587 raft_consensus.cc:3055] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:11.238896 11654 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:11.239219 11587 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:11.240065 11654 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:11.240372 11587 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:11.241277 11654 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:11.241727 11587 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:11.004216 11578 raft_consensus.cc:3055] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:11.242856 11578 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:11.031996 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:11.243387  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:11.044423  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 1970) took 1542 ms. Trace:
I20260808 01:50:11.246879  9554 rpcz_store.cc:276] 0808 01:50:09.502078 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:09.707750 (+205672us) service_pool.cc:224] Handling call
0808 01:50:09.752899 (+ 45149us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:09.893893 (+140994us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:09.893948 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:09.913189 (+ 19241us) log.cc:844] Serialized 26 byte log entry
0808 01:50:10.104823 (+191634us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:10.104880 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:11.043926 (+939046us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b5400382a54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:11.043987 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:11.044047 (+    60us) raft_consensus.cc:1685] finished
0808 01:50:11.044060 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:11.044346 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":323075328,"thread_start_us":145338,"threads_started":1,"wal-append.queue_time_us":289789}
I20260808 01:50:11.249267 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:11.228427 10360 rpcz_store.cc:269] 0808 01:50:08.963818 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:08.975041 (+ 11223us) service_pool.cc:224] Handling call
0808 01:50:09.146264 (+171223us) spinlock_profiling.cc:243] Waited 7.74 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d4185593c 00007f5d418558a6 000000000038d209 00007f5d49914940 00007f5d499dced4 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:50:10.818933 (+1672669us) spinlock_profiling.cc:243] Waited 672 ms on lock 0x7b5400690354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:10.819144 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1563947008}
I20260808 01:50:11.256953 11593 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:11.257735 11593 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:11.261380  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:11.261765  9600 leader_election.cc:336] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Timed out: RequestConsensusVote RPC to 127.7.165.196:33455 timed out after 2.042s (SENT)
I20260808 01:50:11.262210  9600 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:11.263051 11568 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:11.072398  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1898) took 1011 ms. Trace:
W20260808 01:50:11.081128  9598 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.015s	sys 0.027s
W20260808 01:50:11.263865  9598 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.016s	sys 0.028s
I20260808 01:50:11.263753  9477 rpcz_store.cc:276] 0808 01:50:10.060584 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:10.483428 (+422844us) service_pool.cc:224] Handling call
0808 01:50:11.072320 (+588892us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:11.265040  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:11.264902  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:11.265549  9477 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
W20260808 01:50:11.075110  9599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.027s	sys 0.043s
W20260808 01:50:11.266963  9599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.807s	user 0.027s	sys 0.043s
I20260808 01:50:11.266881  9635 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:50:11.267623  9635 raft_consensus.cc:606] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:11.267978  9635 raft_consensus.cc:614] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:50:11.110342  9633 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:11.268956  9633 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:11.269829  9633 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:11.270911  9633 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 6
I20260808 01:50:11.271037 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:11.271986 11744 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 01:50:11.088346  9601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.030s	sys 0.029s
I20260808 01:50:11.120662 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:11.272445 11744 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:11.272944 11744 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:11.273550  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 103b913d6c014692b6dea1780af2aa20 (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:50:11.126667 11655 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:11.146754 11622 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:11.275336 11622 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:11.276253 11622 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:11.277432 11622 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:11.155104 11611 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:11.160434 11573 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:11.162962  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.682s	user 0.009s	sys 0.015s
W20260808 01:50:11.291926  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.819s	user 0.009s	sys 0.016s
W20260808 01:50:11.292346  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 819616us
I20260808 01:50:11.293164  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:11.165997 11700 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:11.169740  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1701) took 1741 ms. Trace:
I20260808 01:50:11.188205  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1710) took 1738 ms. Trace:
I20260808 01:50:11.306257  9474 rpcz_store.cc:276] 0808 01:50:09.428017 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:09.460817 (+ 32800us) service_pool.cc:224] Handling call
0808 01:50:11.169482 (+1708665us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b5400404cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:11.169664 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3398510976}
I20260808 01:50:11.199664 11434 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:11.209564  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1912) took 1050 ms. Trace:
I20260808 01:50:11.308101  9474 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:38182
I20260808 01:50:11.308872  9474 raft_consensus.cc:606] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:11.308672  9481 rpcz_store.cc:276] 0808 01:50:10.158851 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.208093 (+1049242us) service_pool.cc:224] Handling call
0808 01:50:11.209481 (+  1388us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:11.309517  9474 raft_consensus.cc:614] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:50:11.213346 11749 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:10.710551  9472 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
W20260808 01:50:11.311475  9472 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1902) took 1240 ms (client timeout 1651 ms). Trace:
I20260808 01:50:10.781710  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:11.311903  9472 rpcz_store.cc:269] 0808 01:50:10.071362 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:10.320618 (+249256us) service_pool.cc:224] Handling call
0808 01:50:11.311403 (+990785us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:11.226701 11758 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:50:11.228665 11613 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.856s	user 0.004s	sys 0.000s
I20260808 01:50:10.973223  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:11.314316  9475 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:11.306435  9478 rpcz_store.cc:276] 0808 01:50:09.449859 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:09.810744 (+360885us) service_pool.cc:224] Handling call
0808 01:50:09.873072 (+ 62328us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:09.873564 (+   492us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:09.873607 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:09.912848 (+ 39241us) log.cc:844] Serialized 26 byte log entry
0808 01:50:10.032666 (+119818us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:10.043030 (+ 10364us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:10.043161 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:11.186733 (+1143572us) spinlock_profiling.cc:243] Waited 25 ms on lock 0x7b24000ef228. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41713594 00007f5d4171169b 00007f5d418cd6d9 00007f5d418cd365 00007f5d418cc8c0 00007f5d4455a638 00007f5d4455ad98 00007f5d44532588 00007f5d4452d95a 00007f5d4452cda6
0808 01:50:11.187729 (+   996us) raft_consensus.cc:1685] finished
0808 01:50:11.187754 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:11.188109 (+   355us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1160,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":22628,"spinlock_wait_cycles":57401088,"thread_start_us":81119,"threads_started":1,"wal-append.queue_time_us":361293}
I20260808 01:50:11.232175 11487 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:11.268386 11754 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:11.119843 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:11.349169  9550 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:50:11.349671 10366 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:11.152994  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:11.272671  9601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.792s	user 0.031s	sys 0.029s
I20260808 01:50:10.480202  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1704) took 1044 ms. Trace:
W20260808 01:50:11.350873  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:11.352586  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:11.351445  9473 rpcz_store.cc:276] 0808 01:50:09.436002 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:09.510573 (+ 74571us) service_pool.cc:224] Handling call
0808 01:50:09.539322 (+ 28749us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:09.544685 (+  5363us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:09.544715 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:09.568579 (+ 23864us) log.cc:844] Serialized 26 byte log entry
0808 01:50:09.983409 (+414830us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:09.983458 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:09.984190 (+   732us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:10.479871 (+495681us) raft_consensus.cc:1685] finished
0808 01:50:10.479896 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:10.480097 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8030,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":34518,"thread_start_us":349811,"threads_started":1,"wal-append.queue_time_us":555649}
I20260808 01:50:11.280256 10334 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:11.353555 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 73417.1us
W20260808 01:50:11.070662  9479 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1698) took 1658 ms (client timeout 2018 ms). Trace:
W20260808 01:50:11.144060 10332 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.979s	user 0.009s	sys 0.013s
W20260808 01:50:11.355163 10332 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.191s	user 0.009s	sys 0.014s
W20260808 01:50:11.355657 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 1.19711e+06us
I20260808 01:50:11.290594 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:11.356999 10365 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
W20260808 01:50:11.354247  9479 rpcz_store.cc:269] 0808 01:50:09.411984 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:09.420404 (+  8420us) service_pool.cc:224] Handling call
0808 01:50:11.070441 (+1650037us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b540021c854. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:11.070588 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3789203328}
W20260808 01:50:11.153748  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:11.298125  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:11.359688  9558 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:11.299926  9557 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:11.301576  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:11.302876  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:11.362316  9555 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:11.303853  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:11.308805  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 60851us
I20260808 01:50:11.365113  9477 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:11.365674  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:11.367131  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:11.368009  9524 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:11.312762  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 531263us
I20260808 01:50:11.369473  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:11.320446  9472 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:50:11.370649  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 233) took 1856 ms. Trace:
I20260808 01:50:11.371042  9472 rpcz_store.cc:276] 0808 01:50:09.513934 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.319786 (+1805852us) service_pool.cc:224] Handling call
0808 01:50:11.320120 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:11.370362 (+ 50242us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:11.370592 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:11.373332  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:11.320453  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:11.375306  9554 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:11.273554 11568 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:11.380246 10332 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:11.382339 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:11.326630  9521 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:11.327343  9633 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:11.385653  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 234) took 1417 ms. Trace:
I20260808 01:50:11.340297  9559 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 01:50:11.387735  9444 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:11.351707  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 888659us
I20260808 01:50:11.389370  9601 leader_election.cc:400] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3.
I20260808 01:50:11.389743  9601 leader_election.cc:403] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260808 01:50:11.387940  9474 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:11.390571  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 235) took 1419 ms. Trace:
I20260808 01:50:11.390998  9474 rpcz_store.cc:276] 0808 01:50:09.971173 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.387439 (+1416266us) service_pool.cc:224] Handling call
0808 01:50:11.387760 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:11.390282 (+  2522us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:11.390496 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:11.392030  9480 raft_consensus.cc:2410] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:50:11.392325  9474 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:11.392529  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1897) took 1332 ms. Trace:
I20260808 01:50:11.386215  9472 rpcz_store.cc:276] 0808 01:50:09.968511 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.384697 (+1416186us) service_pool.cc:224] Handling call
0808 01:50:11.385063 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:11.385253 (+   190us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:11.385279 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:11.385321 (+    42us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:11.385353 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:11.385415 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:11.385580 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:11.392947  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1714) took 1418 ms. Trace:
I20260808 01:50:11.290565 11611 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:11.397796  9474 rpcz_store.cc:276] 0808 01:50:09.974916 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.391853 (+1416937us) service_pool.cc:224] Handling call
0808 01:50:11.392160 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:11.392724 (+   564us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:11.392897 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:11.396387 11620 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:11.373234 11668 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:11.323493 11741 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:11.327495 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:11.421756  9474 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
W20260808 01:50:11.441009  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 51798.6us
I20260808 01:50:11.474170  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1895) took 1414 ms. Trace:
I20260808 01:50:11.477131  9474 rpcz_store.cc:276] 0808 01:50:10.059343 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.421146 (+1361803us) service_pool.cc:224] Handling call
0808 01:50:11.421527 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:11.473700 (+ 52173us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:11.473973 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:11.479044  9474 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:11.480139  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1903) took 1408 ms. Trace:
I20260808 01:50:11.482882  9474 rpcz_store.cc:276] 0808 01:50:10.072118 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.478195 (+1406077us) service_pool.cc:224] Handling call
0808 01:50:11.478600 (+   405us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:11.479793 (+  1193us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:11.480050 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:11.392867  9480 rpcz_store.cc:276] 0808 01:50:10.060247 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:10.999551 (+939304us) service_pool.cc:224] Handling call
0808 01:50:11.392466 (+392915us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:11.492455  9475 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:11.493522  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1716) took 1515 ms. Trace:
I20260808 01:50:11.493958  9475 rpcz_store.cc:276] 0808 01:50:09.977841 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:10.481866 (+504025us) service_pool.cc:224] Handling call
0808 01:50:11.493418 (+1011552us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:11.432646  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 1977) took 1444 ms. Trace:
I20260808 01:50:11.499996  9477 raft_consensus.cc:2410] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:50:11.436162  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 236) took 1456 ms. Trace:
I20260808 01:50:11.500955  9478 rpcz_store.cc:276] 0808 01:50:09.979342 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.397250 (+1417908us) service_pool.cc:224] Handling call
0808 01:50:11.435605 (+ 38355us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:11.435804 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:11.435827 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:11.435903 (+    76us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:11.435932 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:11.435986 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:11.436097 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:11.422348  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 20a4f92418934341a6ac4927db61dc87 (from 6e7856bed1be43acaa821290d506ee97 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:50:11.433741  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1613) took 1376 ms. Trace:
I20260808 01:50:11.505559  9638 rpcz_store.cc:276] 0808 01:50:10.057167 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:10.057312 (+   145us) service_pool.cc:224] Handling call
0808 01:50:11.433544 (+1376232us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540003acd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:11.433675 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2598570752}
I20260808 01:50:11.429527  9476 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:11.499759  9556 rpcz_store.cc:276] 0808 01:50:09.988459 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:09.988672 (+   213us) service_pool.cc:224] Handling call
0808 01:50:10.026813 (+ 38141us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:11.230628 (+1203815us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:11.432322 (+201694us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540015bb54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:11.432570 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2767081856}
W20260808 01:50:11.413149 11649 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 6 is higher.
I20260808 01:50:11.371649 11675 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 01:50:11.522084  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46665.4us
I20260808 01:50:11.533016  9601 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:11.386808 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:11.413355 11784 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:11.397037 11726 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:50:11.525923  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1900) took 1464 ms. Trace:
I20260808 01:50:11.526947  9554 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:11.492507  9480 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:50:11.397037 11683 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:11.386255 11773 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:11.545033  9472 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:11.414363 11650 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:11.567499  9477 rpcz_store.cc:276] 0808 01:50:10.061276 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.266432 (+1205156us) service_pool.cc:224] Handling call
0808 01:50:11.525812 (+259380us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:11.550144  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:11.574201  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1904) took 1501 ms. Trace:
I20260808 01:50:11.585765  9480 rpcz_store.cc:276] 0808 01:50:10.072569 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.491798 (+1419229us) service_pool.cc:224] Handling call
0808 01:50:11.492157 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:11.573846 (+ 81689us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:11.574119 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:11.397470 11762 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:11.420033 11703 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:11.429340 11611 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:11.313761 11700 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:11.437220 11639 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260808 01:50:11.681919 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:11.694633  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 144595us
W20260808 01:50:11.688406  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 300719us
I20260808 01:50:11.604522 10366 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:11.704578  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:11.577024  9555 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:11.705791  9600 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:11.522184 11565 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3.
I20260808 01:50:11.528888 11709 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:11.530913 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:11.398571 11727 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.190s
I20260808 01:50:11.416659 11568 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:11.541525 11675 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:11.751062 11675 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:50:11.654923 10368 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.004s	sys 0.027s
W20260808 01:50:11.756198 10368 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.004s	sys 0.028s
I20260808 01:50:11.435190 11668 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:11.692701  9478 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:50:11.760327  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1911) took 1612 ms. Trace:
I20260808 01:50:11.760842  9478 rpcz_store.cc:276] 0808 01:50:10.148053 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.539928 (+1391875us) service_pool.cc:224] Handling call
0808 01:50:11.692244 (+152316us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:11.759918 (+ 67674us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:11.760242 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:11.385548 11754 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:11.570979 11757 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:11.567257 11784 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:11.390468 11578 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:11.767555  9479 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:38182
I20260808 01:50:11.768704  9479 raft_consensus.cc:606] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:11.769277  9479 raft_consensus.cc:614] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:50:11.428190 11620 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:11.570659 11691 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:11.640405 11650 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:11.747320  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1717) took 1768 ms. Trace:
I20260808 01:50:11.815086  9472 rpcz_store.cc:276] 0808 01:50:09.978718 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.394593 (+1415875us) service_pool.cc:224] Handling call
0808 01:50:11.394983 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:11.570495 (+175512us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:11.746873 (+176378us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b54004047d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:11.747222 (+   349us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":344568704}
I20260808 01:50:11.817176  9472 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:11.583163 11683 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:11.713073 11539 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:11.837143  9478 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:50:11.838142  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1916) took 1599 ms. Trace:
W20260808 01:50:11.750327  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 44946.4us
I20260808 01:50:11.714859 11709 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:11.851029  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:11.719374 11726 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.123s
I20260808 01:50:11.827260 11784 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:11.872172  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 90465c2f160641a7901ab0815e723ce1 (from 6e7856bed1be43acaa821290d506ee97 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
W20260808 01:50:11.868927  9524 leader_election.cc:336] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 2.018s (SENT)
I20260808 01:50:11.845697 11565 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:11.555709 11758 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:11.891902 10332 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.060s	sys 0.123s
W20260808 01:50:11.893204 10332 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.508s	user 0.060s	sys 0.123s
W20260808 01:50:11.895041 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 512771us
I20260808 01:50:11.854125 11739 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:11.857719 11709 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:11.870740 11539 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:11.873147 11568 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:11.883764 11693 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.642s	user 0.010s	sys 0.009s
I20260808 01:50:11.830029 11691 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:11.902393 11593 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:11.838582  9478 rpcz_store.cc:276] 0808 01:50:10.238317 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.771635 (+1533318us) service_pool.cc:224] Handling call
0808 01:50:11.836677 (+ 65042us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:11.837786 (+  1109us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:11.838076 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:11.824977  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 238) took 1438 ms. Trace:
I20260808 01:50:11.961090  9472 rpcz_store.cc:276] 0808 01:50:10.386954 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.816467 (+1429513us) service_pool.cc:224] Handling call
0808 01:50:11.816913 (+   446us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:11.817678 (+   765us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:11.824876 (+  7198us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:11.763496 11686 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.684s	user 0.014s	sys 0.028s
W20260808 01:50:11.933698  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 82815.4us
I20260808 01:50:11.911289 11758 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:11.977854  9638 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
W20260808 01:50:11.940464  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 613882us
I20260808 01:50:11.899276 10332 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:11.813251 11661 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.025s	sys 0.019s
W20260808 01:50:11.971771 11686 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.892s	user 0.014s	sys 0.029s
I20260808 01:50:11.964586  9472 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260808 01:50:11.979162  9638 raft_consensus.cc:606] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:11.983439 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 84263.1us
I20260808 01:50:11.984925 10332 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:11.963090 11389 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 01:50:12.000382 10334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.015s	sys 0.009s
I20260808 01:50:11.962579 11749 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:12.014834 10334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.015s	sys 0.009s
W20260808 01:50:12.017871 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 623855us
W20260808 01:50:11.984187 11661 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.707s	user 0.027s	sys 0.020s
I20260808 01:50:11.975440 11794 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:11.972378 11539 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:11.909935 11739 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:12.026046 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:12.037587 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:12.009346 11389 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:12.000846 11722 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.792s	user 0.012s	sys 0.014s
I20260808 01:50:11.960960 11693 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:12.004971 11727 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260808 01:50:12.034801  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 330302us
W20260808 01:50:12.048478  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:12.049190  9474 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.010s	sys 0.014s
I20260808 01:50:12.036616 11593 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:50:12.065405  9474 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.010s	sys 0.014s
W20260808 01:50:12.051689  9520 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.016s	sys 0.008s
I20260808 01:50:12.059710  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:12.034150 11755 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260808 01:50:12.067644  9520 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.016s	sys 0.008s
I20260808 01:50:12.036345 11795 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:12.052021 11722 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.843s	user 0.013s	sys 0.014s
W20260808 01:50:12.085865  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 380132us
I20260808 01:50:12.086079  9638 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:50:12.086794  9638 raft_consensus.cc:606] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:12.072273  9601 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:12.066627  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1910) took 1926 ms. Trace:
I20260808 01:50:12.051971 11796 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:12.051456 11389 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:12.075294  9554 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:12.087994 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:12.092630  9638 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:12.091182  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:12.096700  9634 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:12.103067  9601 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:12.058539 11727 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:12.088795  9474 rpcz_store.cc:276] 0808 01:50:10.139903 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.511745 (+1371842us) service_pool.cc:224] Handling call
0808 01:50:11.512183 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:12.066076 (+553893us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:12.066124 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:12.066195 (+    71us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:12.066244 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:12.066346 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:12.066541 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18748,"raft.queue_time_us":70646,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":98897,"thread_start_us":553301,"threads_started":1}
I20260808 01:50:12.063949 11700 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:12.084885 11789 log.cc:927] Time spent T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.265s	user 0.000s	sys 0.002s
I20260808 01:50:12.071019 11749 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:12.114717  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 01:50:12.115082  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:12.142815 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:12.126348  9559 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260808 01:50:12.121515  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:12.084300 11668 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:12.156775 10367 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:12.089087 11639 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:12.150592  9635 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 3. Ops: []
I20260808 01:50:12.159401  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:12.166472 10368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 456) took 1050 ms. Trace:
I20260808 01:50:12.160459  9600 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:12.161993  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:12.163407  9634 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:12.164050  9637 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:12.166723  9632 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:12.149573  9631 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:12.037488 11794 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:12.193034  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:12.193962  9554 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
W20260808 01:50:12.096698 11748 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.009s	sys 0.007s
W20260808 01:50:12.197619 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 160493us
I20260808 01:50:12.200929 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:12.169648 10368 rpcz_store.cc:276] 0808 01:50:11.116542 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.118343 (+  1801us) service_pool.cc:224] Handling call
0808 01:50:11.118761 (+   418us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:11.119030 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:50:11.756872 (+637842us) log.cc:844] Serialized 14 byte log entry
0808 01:50:12.155752 (+398880us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:12.155840 (+    88us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:12.155908 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:12.156017 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:12.166369 (+ 10352us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":841013,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"thread_start_us":1028459,"threads_started":2}
I20260808 01:50:12.112656 11650 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:12.207235  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 46877.3us
I20260808 01:50:12.113517 11661 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:12.114846 11568 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:12.111892 11683 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:12.116076 11727 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:50:12.207262 11748 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.009s	sys 0.007s
I20260808 01:50:12.212109  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:12.218469 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:12.219650  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:12.194749 11788 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.317s	user 0.000s	sys 0.002s
I20260808 01:50:12.220600  9550 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 4
I20260808 01:50:12.159328 11583 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:12.164875 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:12.160866 11647 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260808 01:50:12.224622 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:12.226169  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:12.229028  9601 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:12.161595 11757 raft_consensus.cc:697] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:12.247825 11757 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:12.166138 11703 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:12.237596 11686 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:12.192852 11754 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:12.112836 11798 consensus_queue.cc:1059] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.137s
I20260808 01:50:12.273250  9550 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:12.097751 11795 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:12.253830 11568 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:12.161155 11539 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:12.213202 11668 consensus_queue.cc:1059] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260808 01:50:12.259950 11691 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:12.260926 11781 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.227s	user 0.000s	sys 0.001s
W20260808 01:50:12.261754 11787 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
W20260808 01:50:12.263367 11786 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.219s	user 0.001s	sys 0.000s
I20260808 01:50:12.195544 11639 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:12.256744 10364 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:12.267803 11807 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:12.268455 11810 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:12.272133 11818 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:12.257683  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:12.273905  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:12.216126 11802 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:12.278919  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:12.280471  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1720) took 1018 ms. Trace:
I20260808 01:50:12.281347  9638 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:12.281144  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:12.283206  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:12.115731 11796 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:12.293659  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:12.274369 11593 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:12.296586  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:12.302083 11703 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:12.252875 11683 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260808 01:50:12.304423 11755 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:12.253825 11801 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:12.352707  9550 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:12.284796 11804 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:12.354640  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:12.355698  9554 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:12.355284  9472 rpcz_store.cc:276] 0808 01:50:11.262242 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.985037 (+722795us) service_pool.cc:224] Handling call
0808 01:50:12.279931 (+294894us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:12.280127 (+   196us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:12.280153 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:12.280205 (+    52us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:12.280238 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:12.280300 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:12.280407 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:12.364408 11815 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:50:12.365355 11815 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:12.364800 11807 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:12.364413 11686 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:12.367568 11686 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:12.368073 11686 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:12.360829 11818 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:12.369369 11817 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:12.366595 11615 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:12.370296 11817 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:12.371197 11615 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:12.375617 11803 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260808 01:50:12.351617 11802 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:12.380111 11802 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:12.373880  9559 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
W20260808 01:50:12.382393  9633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1611) took 2352 ms (client timeout 1861 ms). Trace:
W20260808 01:50:12.383076  9633 rpcz_store.cc:269] 0808 01:50:10.029845 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:10.030796 (+   951us) service_pool.cc:224] Handling call
0808 01:50:11.110187 (+1079391us) spinlock_profiling.cc:243] Waited 4.82 ms on lock 0x7b54003cfda0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490b1a0 00007f5d449971f8 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:12.382137 (+1271950us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b540032d054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:12.382287 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":44408448}
I20260808 01:50:12.292465 11748 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:12.389261  9599 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:12.394553  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:12.356844  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:12.398924  9523 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:12.401909  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:12.324883 11739 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:12.361009 11790 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.199s	user 0.001s	sys 0.001s
I20260808 01:50:12.342978 11722 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:12.407680  9638 raft_consensus.cc:2368] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3.
I20260808 01:50:12.360659 11755 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:12.416040  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:12.422572  9630 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:12.416802  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:12.428068  9631 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 01:50:12.429392  9637 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:12.343370 11821 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:12.357630 11810 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:12.431788 11722 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:12.432981 11749 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:12.400949 11801 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:12.507200  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 100098us
I20260808 01:50:12.496392 11803 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260808 01:50:12.514225  9599 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:12.518309 11808 consensus_queue.cc:1059] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.131s
I20260808 01:50:12.477427 11815 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:12.497279 11821 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:12.408054 11748 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:12.558561 11821 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:12.559502 11821 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:12.390111 11804 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:12.563622 11794 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:12.567468 11801 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:12.570076  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1711) took 3119 ms. Trace:
I20260808 01:50:12.590148  9550 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:12.580977 11796 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:12.592454 11796 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:12.593036 11796 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:12.593360  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:12.593542 11796 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:50:12.594069  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:12.598472  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:12.598475  9558 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:12.583423  9481 rpcz_store.cc:276] 0808 01:50:09.450539 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.313770 (+1863231us) service_pool.cc:224] Handling call
0808 01:50:11.396680 (+ 82910us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:11.643460 (+246780us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:11.643490 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:11.643537 (+    47us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:11.643577 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:12.569700 (+926123us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b54003926d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:12.569769 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:12.569978 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":566976384}
I20260808 01:50:12.581512  9554 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:12.606074  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:12.606976  9554 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:12.612010  9551 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260808 01:50:12.616986  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:12.620110  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:12.596689 11818 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:12.627244  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:12.628475  9550 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:12.612083 11808 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.018s
W20260808 01:50:12.631839 11744 consensus_meta.cc:287] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.358s	user 0.018s	sys 0.012s
I20260808 01:50:12.631975 11808 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:50:12.632598 11744 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:12.632671  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:12.640411  9550 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:12.641711 11810 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:12.560075 11821 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:12.623864 11762 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:12.668253 11792 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:12.667351 11808 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:12.692106  9554 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:12.578665 11802 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:12.672043 11817 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:12.693298 11808 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:12.700033  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:12.713702 10334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.502s	user 0.041s	sys 0.064s
I20260808 01:50:12.702479 10363 raft_consensus.cc:2368] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 4. Current term is 5.
I20260808 01:50:12.721016  9550 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
W20260808 01:50:12.722203 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 509934us
I20260808 01:50:12.746902 11748 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:12.704257 11808 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:12.697808 11792 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:12.769933 11792 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:12.749207 11795 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:12.698733 11762 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:12.788911 11805 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:12.800452 11805 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:12.933035  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 533961us
W20260808 01:50:12.848228  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.045s	sys 0.059s
W20260808 01:50:12.848937  9443 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.002s	sys 0.028s
W20260808 01:50:12.865692  9479 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.013s	sys 0.104s
W20260808 01:50:12.868185  9476 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1901) took 2805 ms (client timeout 2307 ms). Trace:
I20260808 01:50:12.870875  9478 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260808 01:50:12.871218  9520 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.007s	sys 0.005s
I20260808 01:50:12.722801  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:12.716176  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:12.934562  9476 rpcz_store.cc:269] 0808 01:50:10.062159 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.048970 (+986811us) service_pool.cc:224] Handling call
0808 01:50:12.867026 (+1818056us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54004051d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:12.867553 (+   527us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":437840256}
I20260808 01:50:12.939771  9554 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
W20260808 01:50:12.872360  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.006s	sys 0.049s
I20260808 01:50:12.940956 11827 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:12.941661  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:12.780742 11744 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:12.912629  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 434464us
W20260808 01:50:12.903225 10332 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.010s	sys 0.005s
I20260808 01:50:12.920688 11792 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:12.947520 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:12.947971  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1721) took 1685 ms. Trace:
I20260808 01:50:12.921941 11798 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:12.906353  9560 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.010s	sys 0.005s
I20260808 01:50:12.914089 11821 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:12.936182  9443 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.002s	sys 0.028s
I20260808 01:50:12.937520 11824 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:12.941285  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.724s	user 0.006s	sys 0.050s
I20260808 01:50:12.900538 11749 raft_consensus.cc:697] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:12.946110 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:12.944808  9479 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.748s	user 0.014s	sys 0.105s
W20260808 01:50:12.946486  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.663s	user 0.046s	sys 0.060s
I20260808 01:50:12.918426  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:12.953796 10332 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.010s	sys 0.006s
I20260808 01:50:12.948498 11792 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:12.955472  9520 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.008s	sys 0.005s
I20260808 01:50:12.959355 10360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 363) took 1648 ms. Trace:
I20260808 01:50:12.959650  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:12.948558  9478 rpcz_store.cc:276] 0808 01:50:11.262547 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:12.355288 (+1092741us) service_pool.cc:224] Handling call
0808 01:50:12.869301 (+514013us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:12.934665 (+ 65364us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:12.947579 (+ 12914us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b54004051d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:12.947809 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1173921536}
W20260808 01:50:12.960668  9560 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.014s	sys 0.005s
I20260808 01:50:12.966056 10360 rpcz_store.cc:276] 0808 01:50:11.311772 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.314368 (+  2596us) service_pool.cc:224] Handling call
0808 01:50:12.946789 (+1632421us) spinlock_profiling.cc:243] Waited 23.6 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:50:12.959185 (+ 12396us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":54328576}
I20260808 01:50:12.971208 11795 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:12.972677  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:12.971254  9559 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:12.976712 11748 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:12.977178 11821 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:12.977721 11748 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:12.978284 11821 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:12.981266 11748 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:12.981431 11821 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:12.982303  9474 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:12.983309  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:12.984401 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:12.985198  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:12.985579  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:12.986618  9476 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:12.987648  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:12.988611 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:12.989117 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:12.989876 11792 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:12.991719 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:12.993062 10367 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
W20260808 01:50:12.993559 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 792669us
I20260808 01:50:12.972446 11795 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:12.995203 11758 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.973s	user 0.008s	sys 0.010s
I20260808 01:50:12.995290  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:12.957742 11802 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:12.997035 11795 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:12.997254  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:12.997515  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:12.984712  9559 raft_consensus.cc:2410] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 01:50:12.954376 11798 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:12.942188 11827 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:13.001116  9480 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
W20260808 01:50:13.000983  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 486710us
I20260808 01:50:13.015671 11762 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:13.016644 11762 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:12.978317 11805 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:13.003084  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 524316us
I20260808 01:50:12.995297 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:13.008002  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:12.998196  9554 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:12.973426 11824 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:13.017035  9599 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:13.012796 11792 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:13.010540  9521 leader_election.cc:336] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.106s (SENT)
I20260808 01:50:13.036258  9481 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:50:13.044303 10360 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:13.045048  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 239) took 2632 ms. Trace:
W20260808 01:50:13.046399  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:13.044375 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:13.047824 11744 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:50:13.048098  9599 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:13.049103 11842 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:13.045501  9479 rpcz_store.cc:276] 0808 01:50:10.413201 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.848726 (+1435525us) service_pool.cc:224] Handling call
0808 01:50:11.878878 (+ 30152us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:11.880925 (+  2047us) raft_consensus.cc:1534] Early marking committed up to index 5
0808 01:50:12.195353 (+314428us) log.cc:844] Serialized 14 byte log entry
0808 01:50:13.043139 (+847786us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:13.044531 (+  1392us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:13.044639 (+   108us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:13.044751 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:13.044967 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":378549,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":75586,"thread_start_us":1130219,"threads_started":3,"wal-append.queue_time_us":589167}
I20260808 01:50:13.053160 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:13.053785  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:13.041740  9521 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:13.034009 11805 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:13.057826  9480 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 5
I20260808 01:50:13.057778 10364 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:50:13.062594 11824 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:13.063163 10332 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:13.065348 11840 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:13.059275 10368 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:13.074354 11805 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:13.076476  9523 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:13.003171 11749 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:13.084944 11795 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260808 01:50:13.085377  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 74996.6us
I20260808 01:50:13.087214  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:13.088521 11795 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:13.089232 11795 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:13.089782 11842 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:13.083242 11744 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:50:13.078320 11805 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:13.090739 11795 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:13.053057 11762 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:13.101377  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:13.091418 11744 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:13.067675 11840 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:13.109812  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:13.110836  9478 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:13.110282 11744 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:13.111835 11762 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:13.112931  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:13.076556 11808 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:13.080577 11792 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:13.103768 11846 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:13.114660  9523 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:13.123284  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:13.116550  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:13.074402 11796 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:13.091388 11842 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:13.124421  9554 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 01:50:13.172602  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:13.179641  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:13.137609  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:13.191705 11649 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:13.136703  9479 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:13.157763  9480 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:13.099527 11827 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:13.203495  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:13.203176 11823 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260808 01:50:13.205847  9477 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:13.206996  9560 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:13.210219  9478 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:13.214951  9476 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:13.216068  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:13.196498 11846 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:13.216640 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:13.216952 11846 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:13.215103  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:13.160301 11835 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:13.212195 11649 raft_consensus.cc:697] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:13.136965 11838 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.087s	user 0.000s	sys 0.002s
W20260808 01:50:13.219475 11784 consensus_meta.cc:287] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.219s	user 0.008s	sys 0.018s
I20260808 01:50:13.216694 11792 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:13.228339 11784 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:13.230780  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:13.231462  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:13.232260  9478 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:13.232354 11784 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:13.232836  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:13.230690  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 116805us
I20260808 01:50:13.233814  9474 raft_consensus.cc:2410] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:13.232381  9552 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:13.227741 11798 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:13.228684  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:13.166546 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 49776.7us
I20260808 01:50:13.235484  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:13.237205 11808 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:13.242471 11808 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:13.238154  9477 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:13.242259  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:13.229156  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:13.243831  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:13.244200  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 128142us
I20260808 01:50:13.243124 11808 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:13.251263 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:13.253046 11840 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:13.253961  9550 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:50:13.253378 10368 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:13.240764 11835 raft_consensus.cc:697] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:13.251905 11808 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:13.273087  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:13.276820  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:13.278035  9635 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:13.281585 11649 consensus_queue.cc:237] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:12.385917 11693 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:13.293466  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:13.295080 11834 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:13.295992 11834 raft_consensus.cc:697] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:13.297619  9476 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:13.261687 11784 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:13.306397  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:13.308589  9521 leader_election.cc:400] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3.
I20260808 01:50:13.309156  9521 leader_election.cc:403] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260808 01:50:13.310465 11835 consensus_queue.cc:237] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:13.312422  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 242) took 1287 ms. Trace:
I20260808 01:50:13.312896  9472 rpcz_store.cc:276] 0808 01:50:12.025338 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:12.363122 (+337784us) service_pool.cc:224] Handling call
0808 01:50:12.373473 (+ 10351us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:12.390807 (+ 17334us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:12.390849 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:12.400554 (+  9705us) log.cc:844] Serialized 26 byte log entry
0808 01:50:13.014428 (+613874us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:13.014615 (+   187us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:13.014742 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:13.311952 (+297210us) raft_consensus.cc:1685] finished
0808 01:50:13.311974 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:13.312218 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":157,"raft.run_cpu_time_us":541,"raft.run_wall_time_us":46486,"thread_start_us":490055,"threads_started":1,"wal-append.queue_time_us":603735}
I20260808 01:50:13.315423 11796 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:13.310575  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 63429.1us
I20260808 01:50:13.304525  9637 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:13.275071  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:13.297888 11834 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:13.325100  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:13.325831  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:13.327310  9479 raft_consensus.cc:2410] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:50:13.330232  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:13.331049  9479 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:13.331089  9476 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:13.305181 11784 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:13.336622 11823 consensus_queue.cc:1059] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260808 01:50:13.339112  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 93632.6us
W20260808 01:50:13.342715  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 90448.3us
I20260808 01:50:13.342867 11840 raft_consensus.cc:2764] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260808 01:50:13.344419  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:13.354887 11802 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260808 01:50:13.345274 11784 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:13.368562  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:13.344950 11754 raft_consensus.cc:3055] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:13.376029  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:13.374091  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:13.380227  9472 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:13.378826 11754 raft_consensus.cc:740] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:13.363762 11771 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.042s
I20260808 01:50:13.411116 11815 raft_consensus.cc:3055] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Stepping down as leader of term 2
W20260808 01:50:13.416352 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 43891.9us
I20260808 01:50:13.416633  9472 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:13.393301 11802 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:13.419847 11760 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3.
W20260808 01:50:13.421928  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 64945.3us
I20260808 01:50:13.423720 11754 consensus_queue.cc:260] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:13.434083 11802 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:13.438159 11802 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:13.439527 11840 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:50:13.450081 11693 raft_consensus.cc:3055] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:13.450799 11693 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:13.452142 11693 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:13.453903 11693 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:13.441881 11840 consensus_queue.cc:1059] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260808 01:50:13.419076 11815 raft_consensus.cc:740] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:13.479388  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:13.481357 11749 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:13.487500 11749 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:13.465287 11808 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:13.490558 11749 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:13.502955 11760 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260808 01:50:13.509874 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 293571us
I20260808 01:50:13.521371  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:13.486646 11744 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
W20260808 01:50:13.535079  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 453252us
I20260808 01:50:13.573890  9474 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:13.574607  9523 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:13.576088  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:13.583032  9472 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:13.583032  9476 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
W20260808 01:50:13.587848 11858 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260808 01:50:13.589783 11754 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:12.298166 11611 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:13.613763  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 105682us
I20260808 01:50:13.613162  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:13.621692  9474 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:13.636294  9476 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260808 01:50:13.645124  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 372160us
I20260808 01:50:13.599052 11815 consensus_queue.cc:260] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:13.684319 11835 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:13.694169 11611 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:13.695621 11835 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:13.696187 11834 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260808 01:50:13.697718 10390 debug-util.cc:398] Leaking SignalData structure 0x7b08002eba60 after lost signal to thread 7834
I20260808 01:50:13.698814  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:13.698316 11757 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:13.699846  9476 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 4: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:13.700094 11757 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:13.702884  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 368363us
I20260808 01:50:13.703384 11835 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:12.402179 11668 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 1.314s
I20260808 01:50:13.706296 11846 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:13.707940 11757 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:13.709374 11846 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:13.643424 11798 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:13.808876  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:13.646436 11758 raft_consensus.cc:2764] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260808 01:50:13.710129 11686 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.247s	user 0.010s	sys 0.011s
I20260808 01:50:13.822813  9445 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:13.757043  9524 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:13.816087 11856 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.190s	user 0.000s	sys 0.000s
I20260808 01:50:13.870366  9476 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:13.703290 11674 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:13.815543 11859 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
W20260808 01:50:13.888278  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 65545.9us
I20260808 01:50:13.889724  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:13.900918 11857 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
W20260808 01:50:13.850385  9444 connection.cc:256] RPC call timeout handler was delayed by -1.354841549997218s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1912s for a response
W20260808 01:50:13.907799  9444 connection.cc:256] RPC call timeout handler was delayed by -1.3494415500026662s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1918s for a response
W20260808 01:50:13.908366  9444 connection.cc:256] RPC call timeout handler was delayed by -1.2540415500116069s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20240000000000002s for a response
I20260808 01:50:13.809414 11611 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:13.711259 11808 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:13.909839  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2002) took 1606 ms. Trace:
W20260808 01:50:13.923478  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:13.926179 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 302492us
I20260808 01:50:13.814635 11784 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:13.930583  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:13.932016  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:13.919492 10390 debug-util.cc:398] Leaking SignalData structure 0x7b08002ec140 after lost signal to thread 9412
I20260808 01:50:13.916812 11758 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:13.983417  9637 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 5: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:13.917286  9559 rpcz_store.cc:276] 0808 01:50:12.303387 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:13.011432 (+708045us) service_pool.cc:224] Handling call
0808 01:50:13.067807 (+ 56375us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:13.068330 (+   523us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:13.379786 (+311456us) log.cc:844] Serialized 14 byte log entry
0808 01:50:13.909425 (+529639us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:13.909496 (+    71us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:13.909540 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:13.909616 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:13.909772 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50398,"prepare.queue_time_us":510425,"prepare.run_cpu_time_us":2625,"prepare.run_wall_time_us":16489,"thread_start_us":692532,"threads_started":2}
I20260808 01:50:13.814230 11809 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.108s
I20260808 01:50:13.910482 11649 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:14.004489  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 76838.3us
W20260808 01:50:14.005537  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 247418us
I20260808 01:50:14.007434  9472 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:13.905006 11686 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.009290 11674 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:14.012084 11674 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:14.013646 11802 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.575s	user 0.008s	sys 0.003s
W20260808 01:50:13.993547 10390 debug-util.cc:398] Leaking SignalData structure 0x7b08002ec200 after lost signal to thread 9426
W20260808 01:50:14.014750 11675 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 2.069s	user 0.003s	sys 0.025s
I20260808 01:50:14.015408 11675 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:14.016739 10390 debug-util.cc:398] Leaking SignalData structure 0x7b08003d8800 after lost signal to thread 9506
W20260808 01:50:14.017176 10390 debug-util.cc:398] Leaking SignalData structure 0x7b08003d8820 after lost signal to thread 9507
I20260808 01:50:14.017076  9633 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:14.018277 11675 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:14.023907 11693 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.569s	user 0.014s	sys 0.000s
I20260808 01:50:14.001025 11758 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.058887 11758 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:13.815045 11862 consensus_queue.cc:1059] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.113s
I20260808 01:50:14.060179 11862 raft_consensus.cc:3055] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:14.060628 11862 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:14.014243 11802 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:13.976979 11840 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:13.976744 11821 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:14.031687 11784 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:14.061726 11840 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:13.980194 11798 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.062276 11840 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:14.035825  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:14.036013  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.063230 11802 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:14.036960 11649 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.044596  9474 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:14.051285 11686 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:14.056077  9558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 1691) took 1756 ms (client timeout 2024 ms). Trace:
W20260808 01:50:14.034252 11709 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 2.076s	user 0.010s	sys 0.010s
I20260808 01:50:14.069974  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:14.068933 11749 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.568s	user 0.010s	sys 0.003s
W20260808 01:50:14.073364  9558 rpcz_store.cc:269] 0808 01:50:12.296007 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:12.296208 (+   201us) service_pool.cc:224] Handling call
0808 01:50:14.052163 (+1755955us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b54003147d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:14.052433 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":565204352}
I20260808 01:50:14.061803 11821 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:14.082059 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:14.083748  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:13.919561 11877 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:14.031431 11875 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:14.089954 11862 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.080960  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:14.095453  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.096061 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:14.080245 11796 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:14.097133 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:14.097339 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:14.098114 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:14.092068 11749 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.099185 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:14.091235 10390 debug-util.cc:398] Leaking SignalData structure 0x7b08003d9040 after lost signal to thread 9585
I20260808 01:50:14.077292 11757 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:14.088209 11821 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:14.104274  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.105943 10365 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
W20260808 01:50:14.110131  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:14.102422  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:14.111929  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:14.112829  9474 raft_consensus.cc:2376] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6. Re-sending same reply.
I20260808 01:50:14.100224 11875 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:14.116609  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 108133us
I20260808 01:50:14.119130 10369 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:14.119300  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.119932  9558 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
W20260808 01:50:14.121083  9600 connection.cc:256] RPC call timeout handler was delayed by -1.0376698460022453s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19920000000000002s for a response
I20260808 01:50:14.121276  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.121932  9558 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
I20260808 01:50:14.122242 11674 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:14.122781  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:14.082446 11709 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:14.128664  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:14.130551 11709 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:14.130903 11808 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:14.131150 11668 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:14.131511  9554 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:14.132045 11668 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.131655 11808 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.135246 11668 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:14.136307  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.118616  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:14.136682 11808 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:14.140964 11693 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:14.142457  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:14.143463 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:14.107023 11796 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:14.149755  9558 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:14.150110  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.151059 11693 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:14.151634 10390 debug-util.cc:398] Leaking SignalData structure 0x7b08003af760 after lost signal to thread 9663
I20260808 01:50:14.150888  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.152894  9550 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:14.155093  9632 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:14.157275  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:14.149490 11877 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.170851 11741 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:14.123267 11784 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260808 01:50:14.175719  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:14.158390 11709 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 01:50:14.177315  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 99402.4us
I20260808 01:50:14.154795 11611 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:14.179960  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:14.187670  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:14.182658  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:14.192265  9559 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:14.177799 11674 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.194164  9633 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260808 01:50:14.196502 11798 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:14.197824 11611 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:14.168670 11757 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.200560 11611 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:14.181313 11709 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:14.202335 11757 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:14.203011 11709 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:14.203603 11434 consensus_meta.cc:287] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 2.889s	user 0.010s	sys 0.010s
I20260808 01:50:14.204681 11578 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:14.205574 11862 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:14.207342  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:14.215674  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:14.216452  9637 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:14.212244  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.221314 11693 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:14.221650 11881 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260808 01:50:14.192034 11754 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:14.222154 11754 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:14.222544 11754 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:14.224370  9550 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:14.228050  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:14.228610  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:14.229346  9632 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:50:14.230211  9631 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:50:14.223516 11434 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:14.232481  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.238987 11693 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:14.239710 11693 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:14.240242 11693 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:14.240913  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:14.241653  9478 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:14.241343 11754 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:14.244462 11754 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:12.321547 11631 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:14.246963  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:14.247254 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62543.7us
I20260808 01:50:14.248329 11798 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260808 01:50:12.355026 11622 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.062s	user 0.013s	sys 0.001s
W20260808 01:50:14.252735 11626 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.022s	user 0.011s	sys 0.030s
I20260808 01:50:14.253651 11824 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:14.254400 11824 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:14.246237 11631 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:14.256376 11631 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:14.257627 11631 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:14.260568 11749 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:14.261943  9524 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:14.262876  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 49106.8us
I20260808 01:50:14.234211  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.263325 11649 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:14.263744  9524 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:14.264575  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:14.264953 11885 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:14.265282  9558 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:14.265451 11885 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:14.267688 11885 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:14.268034 11434 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:14.268731 11434 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:14.269892 11741 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:14.274452  9633 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:14.275434 11620 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:14.269232 11434 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:14.277125  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 1993) took 2136 ms. Trace:
I20260808 01:50:14.277591  9552 rpcz_store.cc:276] 0808 01:50:12.140461 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:12.140612 (+   151us) service_pool.cc:224] Handling call
0808 01:50:12.140982 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:13.236583 (+1095601us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:14.276730 (+1040147us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400325ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:14.277025 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2508974848}
I20260808 01:50:14.257997 11824 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:14.219401 11611 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:14.279649 10332 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:14.288318 11649 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:14.289409 10332 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:14.192853 11884 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:14.290220 11884 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:14.290267 11864 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:14.290917 11884 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:14.286664  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.289609 11649 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:14.223275 11796 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:14.293160  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46357.7us
W20260808 01:50:14.313351  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 48478.1us
I20260808 01:50:14.314524  9524 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:14.241343 11840 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:14.317813  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:14.306618  9637 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:14.321209 11840 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:14.272820 11626 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.045s	user 0.013s	sys 0.031s
I20260808 01:50:14.323061  9476 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:14.294579 11649 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:14.317823 11796 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:14.325775  9524 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:14.318805  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:14.326920  9552 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:14.341742 11760 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:14.342401 11760 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:14.348206  9550 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
W20260808 01:50:14.348111 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 58825.5us
I20260808 01:50:14.350057  9599 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:14.353981  9629 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 01:50:14.275450 11741 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:14.367614  9478 raft_consensus.cc:2410] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260808 01:50:14.370975 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:14.371870 11842 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:14.365899 11693 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:14.368016 11758 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:14.372562 11842 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.381991  9558 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:14.350986 11760 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:14.389077  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:14.386487  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:14.386639  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:14.392709  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 112834us
W20260808 01:50:14.393127  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 74462.5us
I20260808 01:50:14.394039  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:14.352382 11875 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:14.338140 11821 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.399138  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.399796 11689 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:14.400104  9558 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:14.400482 11689 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.401878  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:14.382531 11758 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:14.403370  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.403620 11758 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:14.404035 11816 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:14.404577  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:14.404783 11867 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:14.409552  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.387897 11709 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:14.411474  9554 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:14.388716 11578 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:14.388458 11840 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:14.412429  9550 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:14.390059 11626 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:14.414382  9558 raft_consensus.cc:2410] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 4 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:50:14.404604 11821 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:14.397238 11877 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:14.407521 11809 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election won for term 5
W20260808 01:50:14.446358  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 52419.4us
I20260808 01:50:14.446604  9559 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:14.413964 11835 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 01:50:14.457471 11448 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 3.203s	user 0.007s	sys 0.016s
I20260808 01:50:14.427323 11773 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:14.454299  9472 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 01:50:14.457931 11709 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:14.450752  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:14.459030  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:14.439601 11689 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:14.460047 11565 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
W20260808 01:50:14.467548 11587 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 3.225s	user 0.017s	sys 0.011s
I20260808 01:50:14.467965 11639 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:14.474062  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.473085 11691 raft_consensus.cc:3055] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:14.473244  9601 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:14.478892 11748 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:14.481658  9472 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:14.439656 11578 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:14.413511 11867 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:14.488168 11693 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:14.415727 11650 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.076s
I20260808 01:50:14.489028  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:14.514474 11885 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.515249  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:14.489835 11741 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:14.490543 11877 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:14.519061 11741 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:14.514532 11578 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:14.519573 11877 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:14.519630 11741 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:14.520735 11877 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:14.491094 11654 consensus_meta.cc:287] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 3.237s	user 0.018s	sys 0.004s
W20260808 01:50:14.493836  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 667534us
I20260808 01:50:14.523901  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:14.447968 11626 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.458567 11809 raft_consensus.cc:697] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:14.495174  9554 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:14.495275 11434 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.427829 11686 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260808 01:50:14.499419  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.459473 11835 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:14.500383 11691 raft_consensus.cc:740] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:14.504136 11748 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.505679 11817 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:14.505847  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 48200.1us
I20260808 01:50:14.506397 11565 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260808 01:50:14.511340  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:14.529824  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:14.543596 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 105812us
W20260808 01:50:14.544240 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:14.512373 11864 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:14.540278  9521 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:14.551185  9558 raft_consensus.cc:2410] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:14.551110  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:14.552891  9550 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
W20260808 01:50:14.553200  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:14.553785  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:14.536859 11773 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:14.536322 11639 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:50:14.543538 11835 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:14.560667  9552 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:14.563241  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.553969 11654 raft_consensus.cc:2764] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260808 01:50:14.527860 11758 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.554205 11565 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:14.567533  9601 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:14.570978 11797 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:14.542281 11817 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.538970 11760 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.565384 11693 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:14.554366 11834 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260808 01:50:14.565917 11686 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:14.579774  9556 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 6. Ops: []
I20260808 01:50:14.564973 11639 raft_consensus.cc:697] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:14.572418 11565 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:14.586102  9554 raft_consensus.cc:2410] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:14.629503  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 1990) took 2538 ms. Trace:
I20260808 01:50:14.540731 11884 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:14.580504 11693 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:14.631158 11693 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:14.577735 11796 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:14.635116 11884 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:14.626646 11639 consensus_queue.cc:237] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.638715 11674 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:14.625305 11578 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:14.640126  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2010) took 1188 ms. Trace:
W20260808 01:50:14.627776 11487 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 3.255s	user 0.014s	sys 0.010s
I20260808 01:50:14.628970 11565 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.629920  9556 rpcz_store.cc:276] 0808 01:50:12.090864 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:12.091002 (+   138us) service_pool.cc:224] Handling call
0808 01:50:12.111883 (+ 20881us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:14.627779 (+2515896us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:14.629258 (+  1479us) spinlock_profiling.cc:243] Waited 2.46 s on lock 0x7b5400217fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:14.629446 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5657145600}
I20260808 01:50:14.585923 11587 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:14.635262 11796 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:14.640625  9552 rpcz_store.cc:276] 0808 01:50:13.451944 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:14.563889 (+1111945us) service_pool.cc:224] Handling call
0808 01:50:14.575425 (+ 11536us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:14.623991 (+ 48566us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:14.624045 (+    54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:14.639796 (+ 15751us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:14.639838 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:14.639909 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:14.640032 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:14.627501 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:14.579999 11691 consensus_queue.cc:260] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:14.656800  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 92835.9us
W20260808 01:50:14.660080  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:14.620472 11864 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:14.572149 11875 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260808 01:50:14.693470  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.633672 11686 raft_consensus.cc:697] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:14.656148 11796 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:14.669049  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.656108 11631 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:14.692946 11703 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:14.661298 11848 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:14.716569  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:14.705264 11691 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:14.719637  9552 raft_consensus.cc:2376] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4. Re-sending same reply.
I20260808 01:50:14.708950 11758 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:14.710357 11784 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:14.706327 11795 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:50:14.722309 10362 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 461) took 3411 ms (client timeout 1992 ms). Trace:
W20260808 01:50:14.728367  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:14.661237 11587 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:50:14.721364 11565 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:50:14.726475 11903 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260808 01:50:14.743085  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.746054  9552 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
I20260808 01:50:14.744585 11784 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:14.697003 11864 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:14.725320 11845 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:14.738775 10362 rpcz_store.cc:269] 0808 01:50:11.305716 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.312234 (+  6518us) service_pool.cc:224] Handling call
0808 01:50:14.586241 (+3274007us) spinlock_profiling.cc:243] Waited 49.2 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:50:14.716888 (+130647us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":113106816}
I20260808 01:50:14.743000  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.725142 11794 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:14.723409 11885 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:14.705809 11578 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260808 01:50:14.752645  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:14.752916  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:14.760169  9552 raft_consensus.cc:2393] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:14.722465 11631 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:14.632987 11809 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:14.746090 11898 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.199s	user 0.000s	sys 0.002s
I20260808 01:50:14.722954 11848 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.770884 11879 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:14.771617 11879 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:14.775048  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:50:14.776283 11587 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:14.776702  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:14.776322 10390 debug-util.cc:398] Leaking SignalData structure 0x7b08003701a0 after lost signal to thread 10393
W20260808 01:50:14.777643 10390 debug-util.cc:398] Leaking SignalData structure 0x7b0800370340 after lost signal to thread 10394
I20260808 01:50:14.777799  9550 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:14.735116 11703 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.756227 11845 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.777225  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:50:14.796058 11703 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:14.797340  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:14.798055 11874 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.702s	user 0.001s	sys 0.001s
I20260808 01:50:14.798614  9477 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:14.798839  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:14.774367 11879 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:14.784109 10363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 361) took 3537 ms (client timeout 2075 ms). Trace:
I20260808 01:50:14.784807 11748 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:14.831849  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.832250  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:14.833595  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.761569 11794 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.789371 11784 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:14.834787  9633 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:14.790853 11875 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:14.835776  9550 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
W20260808 01:50:14.830031 10363 rpcz_store.cc:269] 0808 01:50:11.246502 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.250505 (+  4003us) service_pool.cc:224] Handling call
0808 01:50:14.783653 (+3533148us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400646c54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:14.783922 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2583642368}
I20260808 01:50:14.791993 11722 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:14.792191 11864 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:14.836858  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:14.837602 11864 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:50:14.803190 11893 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:14.804191 11741 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.840598 11893 raft_consensus.cc:697] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 4 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:14.843196 11879 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:14.806778  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.807145 11451 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:14.811522 11626 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:14.811142 11835 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:14.817440  9560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 2005) took 2512 ms (client timeout 3259 ms). Trace:
I20260808 01:50:14.837589  9633 raft_consensus.cc:2376] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6. Re-sending same reply.
I20260808 01:50:14.847879  9552 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:14.824821  9477 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:14.825775  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:14.826205  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:14.826215 11565 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:14.829265 11573 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:14.829391 11593 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:50:14.785943  9472 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:14.830545 11815 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:14.836809  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.843998 11879 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:14.856460  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:14.847955  9560 rpcz_store.cc:269] 0808 01:50:12.304564 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:12.997070 (+692506us) service_pool.cc:224] Handling call
0808 01:50:14.817100 (+1820030us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b540008a7d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:14.817306 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":473852160}
I20260808 01:50:14.849586 11639 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:14.779235 11693 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:14.864859  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:14.869768  9552 raft_consensus.cc:2393] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:14.845192 11631 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:14.860652 11565 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:14.867297  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:14.868150  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:14.866451  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:14.870369  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:14.879215  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:14.879973  9635 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:14.880300  9637 raft_consensus.cc:2368] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5.
I20260808 01:50:14.881387  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:14.886337  9552 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:14.886777  9560 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:14.886819 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 70481.6us
I20260808 01:50:14.892570  9632 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 01:50:14.892908  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:14.888428 11722 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:14.894795  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2001) took 2564 ms. Trace:
I20260808 01:50:14.895195 11722 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:14.894533  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:14.895354  9551 rpcz_store.cc:276] 0808 01:50:12.297906 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:12.298183 (+   277us) service_pool.cc:224] Handling call
0808 01:50:12.352011 (+ 53828us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:12.612551 (+260540us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:14.847438 (+2234887us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b540015b154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:14.862009 (+ 14571us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":565080832}
I20260808 01:50:14.896030  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:14.896942  9634 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:14.897411  9551 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:14.898888  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:14.900148  9637 raft_consensus.cc:2410] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:14.897339  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:14.901695  9472 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 7: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:14.903359  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:14.895844 11722 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:14.904337 11852 raft_consensus.cc:3055] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:14.904891 11852 raft_consensus.cc:740] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:14.883055 11741 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:14.910691 11835 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:14.920231  9560 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260808 01:50:14.883688 11631 raft_consensus.cc:3055] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:14.926796 11631 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:14.946025 11631 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:14.949410  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 265) took 1022 ms. Trace:
I20260808 01:50:14.847823 11686 consensus_queue.cc:237] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:14.981428  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:14.868762 11639 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:14.952811 10332 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:14.979563  9560 rpcz_store.cc:276] 0808 01:50:13.929008 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:14.909181 (+980173us) service_pool.cc:224] Handling call
0808 01:50:14.919826 (+ 10645us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:14.921191 (+  1365us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:14.948985 (+ 27794us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:14.843586 11893 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:14.906339 11852 consensus_queue.cc:260] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:14.978010  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 111684us
I20260808 01:50:15.068594  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2009) took 1617 ms. Trace:
I20260808 01:50:15.088333  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:15.081552  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:15.058964  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:15.093457 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:15.017614  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:15.050359  9631 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260808 01:50:15.010250  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.074872  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1924) took 1655 ms. Trace:
W20260808 01:50:15.146637 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 258380us
I20260808 01:50:15.137878  9635 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:15.140125  9559 rpcz_store.cc:276] 0808 01:50:13.451450 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:14.561988 (+1110538us) service_pool.cc:224] Handling call
0808 01:50:14.572733 (+ 10745us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:14.585053 (+ 12320us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:14.585091 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:14.623558 (+ 38467us) log.cc:844] Serialized 52 byte log entry
0808 01:50:14.847019 (+223461us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:14.847070 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:14.847156 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:15.068110 (+220954us) raft_consensus.cc:1685] finished
0808 01:50:15.068144 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:15.068409 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4951,"raft.queue_time_us":3713,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":21374,"thread_start_us":197116,"threads_started":1,"wal-append.queue_time_us":335846}
I20260808 01:50:15.145552  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:15.149830  9479 rpcz_store.cc:276] 0808 01:50:13.435630 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:13.437114 (+  1484us) service_pool.cc:224] Handling call
0808 01:50:13.437738 (+   624us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:13.808141 (+370403us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:13.808175 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:13.808224 (+    49us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:13.808261 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.074528 (+1266267us) spinlock_profiling.cc:243] Waited 370 ms on lock 0x7b5400399c54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:15.074589 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:15.074773 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":851008768}
I20260808 01:50:15.087826  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:15.153232  9629 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:15.032829  9557 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:15.156154  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:15.068614 11880 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.972s	user 0.000s	sys 0.008s
W20260808 01:50:15.014801 11904 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.066s	user 0.000s	sys 0.002s
W20260808 01:50:15.158812  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:15.159050  9560 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:15.029964 11631 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:50:15.063483 11389 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 2.922s	user 0.013s	sys 0.006s
I20260808 01:50:15.090556  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:15.096688 11643 tablet_copy_client.cc:538] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:50:15.142782 11639 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:15.067900 11434 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.091594 11816 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:15.014377 11801 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:15.020486 11565 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:15.070986 11573 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:15.016598 11804 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.016404 11693 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.010951 11487 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:50:15.162914  9444 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.574s (SENT) [suppressed 11 similar messages]
I20260808 01:50:15.167896  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:15.067018 11722 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.170037 11845 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.163813 11691 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:15.170964 11852 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:15.085222 11817 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.171185  9634 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:15.080143 11796 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.172741  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:15.220062  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2011) took 1767 ms. Trace:
I20260808 01:50:15.171032  9559 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 01:50:15.174917 11389 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.181074  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 268) took 1169 ms. Trace:
I20260808 01:50:15.224107  9560 rpcz_store.cc:276] 0808 01:50:14.011756 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.140395 (+1128639us) service_pool.cc:224] Handling call
0808 01:50:15.155240 (+ 14845us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.168231 (+ 12991us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.180975 (+ 12744us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:15.220615  9550 rpcz_store.cc:276] 0808 01:50:13.452240 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:14.869419 (+1417179us) service_pool.cc:224] Handling call
0808 01:50:14.889733 (+ 20314us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.005741 (+116008us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:50:15.057934 (+ 52193us) log.cc:844] Serialized 14 byte log entry
0808 01:50:15.218855 (+160921us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:15.219658 (+   803us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:15.219717 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.219812 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:15.219982 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1547,"prepare.queue_time_us":53084,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":85,"thread_start_us":136066,"threads_started":1}
W20260808 01:50:15.169059 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:15.186901  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:15.189618 11739 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260808 01:50:15.189934  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:15.161115 11814 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:15.164572 11912 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
W20260808 01:50:15.193602 11700 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.896s	user 0.011s	sys 0.018s
I20260808 01:50:15.207109  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:15.214758  9629 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:15.217599  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:15.219414  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 56528.5us
I20260808 01:50:15.236828  9521 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:15.238436  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:15.221697  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1704) took 1209 ms. Trace:
I20260808 01:50:15.226863  9636 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 1.
I20260808 01:50:15.229542 11691 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:15.232503  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.045120 11879 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:15.247299  9480 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 5: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:15.239989  9559 rpcz_store.cc:276] 0808 01:50:14.012709 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.164421 (+1151712us) service_pool.cc:224] Handling call
0808 01:50:15.170778 (+  6357us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:15.221294 (+ 50516us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.221621 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:15.255632  9560 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260808 01:50:15.258296 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 95984.3us
I20260808 01:50:15.258458  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1705) took 1245 ms. Trace:
I20260808 01:50:15.259327  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.260432  9472 raft_consensus.cc:2393] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 5: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:15.233566 10361 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:15.211647 11565 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260808 01:50:15.211094 11573 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.260048  9560 rpcz_store.cc:276] 0808 01:50:14.013068 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.234348 (+1221280us) service_pool.cc:224] Handling call
0808 01:50:15.254118 (+ 19770us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.257400 (+  3282us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.258390 (+   990us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:15.259295  9550 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:15.264302  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1706) took 1250 ms. Trace:
I20260808 01:50:15.264755  9550 rpcz_store.cc:276] 0808 01:50:14.013445 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.247805 (+1234360us) service_pool.cc:224] Handling call
0808 01:50:15.259069 (+ 11264us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.264009 (+  4940us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.264235 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:15.266206 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:15.267769  9559 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:15.268081 10332 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:15.268630  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1707) took 1254 ms. Trace:
I20260808 01:50:15.253131 11817 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:15.268952  9559 rpcz_store.cc:276] 0808 01:50:14.013866 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.248808 (+1234942us) service_pool.cc:224] Handling call
0808 01:50:15.267532 (+ 18724us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.268365 (+   833us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.268574 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:15.197942 11801 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.273748  9560 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:15.224095 11816 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:15.280082  9521 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:15.281009 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:15.283203  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1708) took 1263 ms. Trace:
I20260808 01:50:15.283711  9560 rpcz_store.cc:276] 0808 01:50:14.014205 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.263704 (+1249499us) service_pool.cc:224] Handling call
0808 01:50:15.273613 (+  9909us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.276832 (+  3219us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.277178 (+   346us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:15.284279  9550 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 01:50:15.284880  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2015) took 1151 ms. Trace:
I20260808 01:50:15.285166  9550 rpcz_store.cc:276] 0808 01:50:14.133607 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.265442 (+1131835us) service_pool.cc:224] Handling call
0808 01:50:15.274260 (+  8818us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.284641 (+ 10381us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.284839 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:15.269764 11879 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:15.247211 11814 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.258491 11804 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.022s
I20260808 01:50:15.275131 11568 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:15.289487 11727 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 3.041s	user 0.025s	sys 0.000s
I20260808 01:50:15.275539 11647 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:15.293135  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.303916  9479 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:15.290448 11722 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.287529  9521 leader_election.cc:400] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5.
I20260808 01:50:15.305240  9521 leader_election.cc:403] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260808 01:50:15.288054  9559 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 01:50:15.294540  9556 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:15.302997  9560 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260808 01:50:15.292779 11912 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:15.288225 11817 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:15.306828  9550 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:50:15.307056  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2016) took 1173 ms. Trace:
I20260808 01:50:15.315420  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2017) took 1181 ms. Trace:
I20260808 01:50:15.305572 11647 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:15.317636  9559 rpcz_store.cc:276] 0808 01:50:14.133924 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.272474 (+1138550us) service_pool.cc:224] Handling call
0808 01:50:15.287258 (+ 14784us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:15.306819 (+ 19561us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.307002 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:12.378496 11615 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.321158  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2021) took 1185 ms. Trace:
I20260808 01:50:15.322149  9550 rpcz_store.cc:276] 0808 01:50:14.135641 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.285753 (+1150112us) service_pool.cc:224] Handling call
0808 01:50:15.295260 (+  9507us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.317028 (+ 21768us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.321098 (+  4070us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11520}
I20260808 01:50:15.317777  9560 rpcz_store.cc:276] 0808 01:50:14.134196 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.284537 (+1150341us) service_pool.cc:224] Handling call
0808 01:50:15.295234 (+ 10697us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.314381 (+ 19147us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.315327 (+   946us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1920}
I20260808 01:50:15.325592  9521 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:15.327116 11434 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:15.326658  9521 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:15.316732 11912 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:15.316751 11804 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:15.323236 11647 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:15.335489  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 1988) took 3236 ms. Trace:
I20260808 01:50:15.328900 11722 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:15.324400 11700 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 3.027s	user 0.012s	sys 0.020s
I20260808 01:50:15.332295  9550 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:15.338603 11912 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:15.342150  9555 rpcz_store.cc:276] 0808 01:50:12.068667 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:12.068953 (+   286us) service_pool.cc:224] Handling call
0808 01:50:12.069448 (+   495us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:12.251941 (+182493us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:12.251981 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:14.521111 (+2269130us) spinlock_profiling.cc:243] Waited 2.81 ms on lock 0x7b2c000f57d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:50:14.521739 (+   628us) log.cc:844] Serialized 26 byte log entry
0808 01:50:14.580853 (+ 59114us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:14.697574 (+116721us) log.cc:844] Serialized 14 byte log entry
0808 01:50:14.697873 (+   299us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:14.706196 (+  8323us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:14.948389 (+242193us) spinlock_profiling.cc:243] Waited 2.59 ms on lock 0x7b180005d100. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:50:15.299411 (+351022us) raft_consensus.cc:1685] finished
0808 01:50:15.299439 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:15.304819 (+  5380us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":148854,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":8431,"raft.queue_time_us":197749,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":3547,"spinlock_wait_cycles":12426368,"thread_start_us":281704,"threads_started":3,"wal-append.queue_time_us":184235}
I20260808 01:50:15.347100 11615 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260808 01:50:15.348421  9555 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:15.349053  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2025) took 1162 ms. Trace:
I20260808 01:50:15.350250  9555 rpcz_store.cc:276] 0808 01:50:14.186442 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.347679 (+1161237us) service_pool.cc:224] Handling call
0808 01:50:15.348024 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.348795 (+   771us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.348997 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:15.350631  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 269) took 1185 ms. Trace:
I20260808 01:50:15.351855  9555 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260808 01:50:15.352054 11807 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:15.352494  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2026) took 1129 ms. Trace:
I20260808 01:50:15.352849  9555 rpcz_store.cc:276] 0808 01:50:14.225812 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.351170 (+1125358us) service_pool.cc:224] Handling call
0808 01:50:15.351540 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.352234 (+   694us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.352442 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:15.339187  9521 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:15.351475  9550 rpcz_store.cc:276] 0808 01:50:14.165406 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.323177 (+1157771us) service_pool.cc:224] Handling call
0808 01:50:15.332154 (+  8977us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.350150 (+ 17996us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.350547 (+   397us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:15.339522  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2022) took 1175 ms. Trace:
I20260808 01:50:15.354627  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 2032) took 1071 ms. Trace:
I20260808 01:50:15.355000  9559 rpcz_store.cc:276] 0808 01:50:14.164426 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.320046 (+1155620us) service_pool.cc:224] Handling call
0808 01:50:15.334529 (+ 14483us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.339183 (+  4654us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:15.339213 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:15.339263 (+    50us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:15.339298 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.339359 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:15.339467 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:15.327620 11434 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:15.337225  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.344045  9560 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260808 01:50:15.327461 11389 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.358469  9479 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:15.349715 11700 raft_consensus.cc:3055] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:15.338224 11817 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:15.310869 11727 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:15.365231 11727 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:15.341085 11901 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.669s	user 0.000s	sys 0.008s
I20260808 01:50:15.368152 11649 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.361444 11700 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:15.369670 11649 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:15.371567  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1737) took 2139 ms. Trace:
I20260808 01:50:15.371985  9480 rpcz_store.cc:276] 0808 01:50:13.231773 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:13.237826 (+  6053us) service_pool.cc:224] Handling call
0808 01:50:15.371303 (+2133477us) spinlock_profiling.cc:243] Waited 2 s on lock 0x7b54000212d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:15.371465 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4589019520}
I20260808 01:50:15.373453  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.374101  9480 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:15.375557 10364 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:15.375679 11487 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:15.372715  9550 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:15.377012  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 271) took 1025 ms. Trace:
I20260808 01:50:15.377341  9550 rpcz_store.cc:276] 0808 01:50:14.351865 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.360840 (+1008975us) service_pool.cc:224] Handling call
0808 01:50:15.372530 (+ 11690us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.376727 (+  4197us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.376949 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:15.377888 10364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 367) took 2143 ms. Trace:
I20260808 01:50:15.378698  9550 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:15.379565  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1939) took 1669 ms. Trace:
I20260808 01:50:15.380545 11794 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.383329  9559 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:15.384111  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 272) took 1032 ms. Trace:
I20260808 01:50:15.388273  9559 rpcz_store.cc:276] 0808 01:50:14.352173 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.363306 (+1011133us) service_pool.cc:224] Handling call
0808 01:50:15.382726 (+ 19420us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.383837 (+  1111us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.384044 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:15.359081  9560 pending_rounds.cc:85] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 5
I20260808 01:50:15.384927  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.392248  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:15.385535 11912 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:15.379933  9478 rpcz_store.cc:276] 0808 01:50:13.823380 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:14.372112 (+548732us) service_pool.cc:224] Handling call
0808 01:50:14.377315 (+  5203us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:14.391095 (+ 13780us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:14.391165 (+    70us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:14.413210 (+ 22045us) log.cc:844] Serialized 26 byte log entry
0808 01:50:14.544255 (+131045us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:14.570178 (+ 25923us) log.cc:844] Serialized 14 byte log entry
0808 01:50:14.575272 (+  5094us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:14.575350 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:15.335962 (+760612us) spinlock_profiling.cc:243] Waited 6.24 ms on lock 0x7b24000ef198. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41713594 00007f5d4171169b 00007f5d418cd6d9 00007f5d418cd365 00007f5d418cc9ad 00007f5d445162fe 00007f5d4452da7c 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411
0808 01:50:15.379260 (+ 43298us) raft_consensus.cc:1685] finished
0808 01:50:15.379366 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:15.379504 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12839,"prepare.queue_time_us":30356,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":4648,"raft.queue_time_us":5071,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":14970,"spinlock_wait_cycles":17935488,"thread_start_us":70629,"threads_started":1,"wal-append.queue_time_us":125983}
I20260808 01:50:15.378288 10364 rpcz_store.cc:276] 0808 01:50:13.234345 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:13.240412 (+  6067us) service_pool.cc:224] Handling call
0808 01:50:13.243259 (+  2847us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.376118 (+2132859us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.377616 (+  1498us) spinlock_profiling.cc:243] Waited 2.13 s on lock 0x7b540056fed4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:15.377797 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4903295744}
I20260808 01:50:15.355263  9554 rpcz_store.cc:276] 0808 01:50:14.285650 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:14.543348 (+257698us) service_pool.cc:224] Handling call
0808 01:50:15.354341 (+810993us) spinlock_profiling.cc:243] Waited 21.8 ms on lock 0x7b5400313dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:15.354544 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50143616}
I20260808 01:50:15.400125  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:15.400478  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:15.401434  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.359668 11434 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:15.358144 11796 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.407068  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.407835  9479 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:15.408812  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.395632  9524 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:15.398224 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 126456us
I20260808 01:50:15.387732  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:15.403088  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.391258  9560 pending_rounds.cc:107] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260808 01:50:15.411842  9480 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:15.411993  9560 raft_consensus.cc:2889] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:15.362417 11573 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.341167 11917 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260808 01:50:15.395962 11794 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:15.358846 11722 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.396317 11807 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:15.415071  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 270) took 1234 ms. Trace:
I20260808 01:50:15.414072  9521 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:15.418895  9524 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:15.420819  9560 rpcz_store.cc:276] 0808 01:50:14.180267 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.329175 (+1148908us) service_pool.cc:224] Handling call
0808 01:50:15.339981 (+ 10806us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.414765 (+ 74784us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.415007 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:15.422150  9524 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:15.416743 11796 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:15.416872 11817 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:15.370210 11700 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.435341 11918 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:15.436200 11918 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:15.441421 11760 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.420100 11691 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:15.443565 11722 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.424170 11573 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:15.444403 11573 raft_consensus.cc:697] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:15.423730 11615 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:15.431069 11796 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:15.449062 11796 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:15.423715 11626 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:15.427364 11807 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:15.457026  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2029) took 1200 ms. Trace:
I20260808 01:50:15.447990 11817 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:15.462108  9480 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:15.465428 11573 consensus_queue.cc:237] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.466161  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.466886  9480 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:15.467677  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:15.459110 11434 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.466836  9374 catalog_manager.cc:5697] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 1 to 3, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 3 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:50:15.470665 11434 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.461552  9555 rpcz_store.cc:276] 0808 01:50:14.256878 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.353850 (+1096972us) service_pool.cc:224] Handling call
0808 01:50:15.379954 (+ 26104us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.380196 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:15.380832 (+   636us) log.cc:844] Serialized 14 byte log entry
0808 01:50:15.456656 (+ 75824us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:15.456709 (+    53us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:15.456760 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.456835 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:15.456967 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":203,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":37,"raft.queue_time_us":111,"thread_start_us":75355,"threads_started":1}
I20260808 01:50:15.472419 11912 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:15.472823 10361 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:15.473775 11918 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:15.485309 11434 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:15.483440 11593 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:15.488636 11593 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:15.490262 11804 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5.
I20260808 01:50:15.453684 11639 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.464804 11807 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:15.495507 11639 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.496250 11643 tablet_bootstrap.cc:492] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:50:15.478824 11817 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:15.484381 11852 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:15.500406 11647 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:15.499696 11434 raft_consensus.cc:697] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:15.501484 11794 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:15.518956 11434 consensus_queue.cc:237] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:15.519855  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1913) took 5349 ms. Trace:
I20260808 01:50:15.520282  9475 rpcz_store.cc:276] 0808 01:50:10.170128 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.541022 (+1370894us) service_pool.cc:224] Handling call
0808 01:50:11.541498 (+   476us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:14.572025 (+3030527us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:14.576195 (+  4170us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:14.576237 (+    42us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:14.584004 (+  7767us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.008656 (+424652us) spinlock_profiling.cc:243] Waited 2.3 ms on lock 0x7b4800211158. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:50:15.519422 (+510766us) spinlock_profiling.cc:243] Waited 3 s on lock 0x7b540019d2d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:15.519479 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:15.519654 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2926,"raft.queue_time_us":10274,"raft.run_cpu_time_us":7094,"raft.run_wall_time_us":436676,"spinlock_wait_cycles":6899865728}
I20260808 01:50:15.522038  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.500605 11852 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:15.508922 11796 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.512589  9373 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50064 (request call id 235) took 7000 ms. Trace:
W20260808 01:50:15.515379  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:15.513863  9374 catalog_manager.cc:5697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 2 to 4, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 4 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:50:15.526796  9480 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:15.531400  9524 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:15.532775  9475 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:15.535972  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.536648  9475 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:15.536793  9524 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:15.539486 11794 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:15.539971 11794 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:15.540298 11794 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:15.539602  9374 catalog_manager.cc:5697] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 1 to 3, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 3 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:50:15.541458  9374 catalog_manager.cc:5697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:50:15.525461 11639 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:15.547008 11568 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:15.547943 11639 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:15.551431 10390 debug-util.cc:398] Leaking SignalData structure 0x7b08003f3020 after lost signal to thread 11725
W20260808 01:50:15.551937 10390 debug-util.cc:398] Leaking SignalData structure 0x7b08003f3080 after lost signal to thread 11730
W20260808 01:50:15.552225 10390 debug-util.cc:398] Leaking SignalData structure 0x7b08003f30a0 after lost signal to thread 11733
W20260808 01:50:15.552497 10390 debug-util.cc:398] Leaking SignalData structure 0x7b08003f30c0 after lost signal to thread 11736
W20260808 01:50:15.552767 10390 debug-util.cc:398] Leaking SignalData structure 0x7b08003f3100 after lost signal to thread 11740
I20260808 01:50:15.554351  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.554443  9524 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:15.555160  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:15.538138 11797 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:15.559051 11771 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:15.559638  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.559741 11771 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.560362  9475 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:15.563483 11626 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:15.564076 11626 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:15.564563 11626 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:15.565904 11804 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:15.566546 11804 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.573117 11797 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:15.574617 11797 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.578158 11842 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:15.578540 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 469) took 1488 ms. Trace:
I20260808 01:50:15.578999 10361 rpcz_store.cc:276] 0808 01:50:14.090420 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:14.096654 (+  6234us) service_pool.cc:224] Handling call
0808 01:50:15.578325 (+1481671us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54004c3454. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:15.578470 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2583316864}
I20260808 01:50:15.579425 11647 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:15.580463  9481 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 243) took 2594 ms (client timeout 1640 ms). Trace:
W20260808 01:50:15.580878  9481 rpcz_store.cc:269] 0808 01:50:12.986102 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:12.987070 (+   968us) service_pool.cc:224] Handling call
0808 01:50:15.580271 (+2593201us) spinlock_profiling.cc:243] Waited 47.7 ms on lock 0x7b54002a6cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:15.580375 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":109737216}
I20260808 01:50:15.583115  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:15.589145  9475 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:15.594640  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:15.595736  9629 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:15.599080 10332 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:15.545646  9374 catalog_manager.cc:5697] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:50:15.605041  9636 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:15.528895  9373 rpcz_store.cc:276] 0808 01:50:08.511794 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:08.512303 (+   509us) service_pool.cc:224] Handling call
0808 01:50:15.512257 (+6999954us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:50:10.050306 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:50:10.219548 (+169242us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:50:10.222410 (+  2862us) write_op.cc:435] Acquired schema lock
0808 01:50:10.225257 (+  2847us) tablet.cc:662] Decoding operations
0808 01:50:10.343089 (+117832us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:50:10.362121 (+ 19032us) write_op.cc:641] Partition lock acquired for write op
0808 01:50:10.363662 (+  1541us) tablet.cc:685] Acquiring locks for 3 operations
0808 01:50:10.425868 (+ 62206us) tablet.cc:701] Row locks acquired
0808 01:50:10.425906 (+    38us) write_op.cc:260] PREPARE: finished
0808 01:50:10.449265 (+ 23359us) write_op.cc:270] Start()
0808 01:50:10.452686 (+  3421us) write_op.cc:276] Timestamp: P: 1786153810442943 usec, L: 0
0808 01:50:10.455129 (+  2443us) op_driver.cc:348] REPLICATION: starting
0808 01:50:10.493708 (+ 38579us) log.cc:844] Serialized 1317 byte log entry
0808 01:50:10.807388 (+313680us) op_driver.cc:464] REPLICATION: finished
0808 01:50:10.810497 (+  3109us) write_op.cc:301] APPLY: starting
0808 01:50:10.810594 (+    97us) tablet.cc:1367] starting BulkCheckPresence
0808 01:50:10.810794 (+   200us) tablet.cc:1370] finished BulkCheckPresence
0808 01:50:10.810971 (+   177us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:50:10.811583 (+   612us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:50:10.811792 (+   209us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0808 01:50:10.811855 (+    63us) write_op.cc:312] APPLY: finished
0808 01:50:10.812165 (+   310us) log.cc:844] Serialized 35 byte log entry
0808 01:50:10.812372 (+   207us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:10.812501 (+   129us) write_op.cc:454] Released schema lock
0808 01:50:10.812553 (+    52us) write_op.cc:341] FINISH: Updating metrics
0808 01:50:10.815455 (+  2902us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:10.815479 (+    24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":623464,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":2822,"apply.run_cpu_time_us":2443,"apply.run_wall_time_us":4956,"num_ops":3,"prepare.queue_time_us":785919,"prepare.run_cpu_time_us":22964,"prepare.run_wall_time_us":699019,"raft.queue_time_us":36490,"raft.run_cpu_time_us":1092,"raft.run_wall_time_us":5689,"replication_time_us":349510,"thread_start_us":880802,"threads_started":4,"wal-append.queue_time_us":272707,"wal-append.run_cpu_time_us":19575,"wal-append.run_wall_time_us":1051150}]]}
W20260808 01:50:15.611290  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:15.614766  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:15.610598 11647 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:15.616061 11647 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:15.616540 11647 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:15.605973 11771 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.617816  9601 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:15.606179 11804 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.618793 11754 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:15.619230 11754 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:15.619659 11754 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:15.619663  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:15.620631  9638 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:15.615655  9634 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:15.611675 11842 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:15.623234 11842 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.625816 11842 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:15.626761  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:15.628598  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.628962 11817 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.629859  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:15.631284 11754 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.633204 11794 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:50:15.634416 11754 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:15.629341  9480 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 1.
I20260808 01:50:15.637739  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.638501  9480 raft_consensus.cc:2368] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5.
I20260808 01:50:15.638387  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:15.640187  9634 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:15.628598  9601 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:15.611735 11760 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:15.642962  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:15.648626  9629 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
W20260808 01:50:15.646916  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:15.647660 11842 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:15.655000  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.654511 11842 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:15.651002  9631 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:15.660208  9480 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:15.662859 11760 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:15.663518 11920 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260808 01:50:15.656778 11842 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:15.665344  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:50:15.665446  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:15.667625  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:15.666126  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:15.668682  9637 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 4
I20260808 01:50:15.664937  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:15.666575  9631 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260808 01:50:15.668141  9521 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:15.672403  9521 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:15.674129  9629 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:15.675101 11573 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:15.675550 11573 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:15.675915 11573 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:15.679265  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1724) took 2622 ms. Trace:
I20260808 01:50:15.679740  9473 rpcz_store.cc:276] 0808 01:50:12.983026 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:12.983301 (+   275us) service_pool.cc:224] Handling call
0808 01:50:12.983866 (+   565us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:12.990398 (+  6532us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:50:13.042058 (+ 51660us) log.cc:844] Serialized 14 byte log entry
0808 01:50:13.262164 (+220106us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:13.262230 (+    66us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:13.262281 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.583209 (+2320928us) spinlock_profiling.cc:243] Waited 6.02 ms on lock 0x7b54002c0454. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:15.583259 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:15.605029 (+ 21770us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":115303,"prepare.run_cpu_time_us":115,"prepare.run_wall_time_us":3130,"raft.queue_time_us":1392,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":5797,"spinlock_wait_cycles":13839104,"thread_start_us":227230,"threads_started":2,"wal-append.queue_time_us":343862,"wal-append.run_cpu_time_us":3364,"wal-append.run_wall_time_us":1187927}
I20260808 01:50:15.681949  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.657115 11689 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:15.683148 11689 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.684739 11893 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:15.685320 11893 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.685977 11689 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.688894 11893 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.688941 11796 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.691516  9629 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:50:15.697997  9480 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:15.702943 11807 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:15.704430 11807 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:15.705988 11807 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.709275  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.682591  9473 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:15.712965  9480 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:15.722066 11796 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:15.723124 11796 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:15.745718 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:15.726078 11796 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.767104  9480 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:15.768456  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.744779 11804 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:15.774444 11804 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.777034 11801 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.780167  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.781775  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.765545 11817 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.785669 11639 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:15.786867 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:15.791586 11814 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:15.792989  9477 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1927) took 2270 ms (client timeout 2102 ms). Trace:
I20260808 01:50:15.796989  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.806727  9478 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:50:15.801934  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:15.794648  9637 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:15.803620  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.782585  9475 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 5: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:15.803969  9524 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:50:15.805013  9477 rpcz_store.cc:269] 0808 01:50:13.523122 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:13.523322 (+   200us) service_pool.cc:224] Handling call
0808 01:50:15.792688 (+2269366us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54002dbed4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:15.792845 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2801134464}
I20260808 01:50:15.800159 11852 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:15.732410 11434 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:15.811535  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:15.813004  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.813019  9524 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:15.817744  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.816212  9472 raft_consensus.cc:2376] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6. Re-sending same reply.
W20260808 01:50:15.817890  9476 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1934) took 2109 ms (client timeout 1539 ms). Trace:
I20260808 01:50:15.819900  9475 raft_consensus.cc:2410] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260808 01:50:15.820186  9476 rpcz_store.cc:269] 0808 01:50:13.708116 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:14.027154 (+319038us) service_pool.cc:224] Handling call
0808 01:50:15.817661 (+1790507us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b540021e154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:15.817812 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":597266304}
I20260808 01:50:15.777139 11804 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.823361  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1950) took 1021 ms. Trace:
I20260808 01:50:15.823808  9475 rpcz_store.cc:276] 0808 01:50:14.801489 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.817458 (+1015969us) service_pool.cc:224] Handling call
0808 01:50:15.823286 (+  5828us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:15.824924  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.825425  9475 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:15.819818 11867 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.826808  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:15.801640 11816 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.827653  9558 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:15.800143 11760 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:15.827661  9524 leader_election.cc:400] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5.
I20260808 01:50:15.828176  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:15.828754  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:15.841506  9524 leader_election.cc:403] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260808 01:50:15.825637  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:15.829671  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.827176  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.835372 11742 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:15.806465 11626 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.839768  9473 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:15.841102  9475 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:15.842386  9480 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:15.832149  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:15.841791  9555 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 2.
I20260808 01:50:15.843223  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.822897 11814 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:15.848769  9476 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:15.848074  9472 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:15.853314  9478 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:15.854616 10334 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:15.855926 11816 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:15.856752 11816 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:15.859735  9551 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:15.861366  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.862318  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.829527 11861 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:50:15.863003  9475 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:15.829373 11434 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:15.849514 11568 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:15.866226  9552 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:15.865898 11434 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.805846 11573 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.864238 11867 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:15.878221  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.877609  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.880368 11568 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:15.881250 11925 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260808 01:50:15.849764 11794 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.888463  9477 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:15.892129 10332 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:15.860555 11742 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:15.894862  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.898766  9480 raft_consensus.cc:2410] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:15.871145 11861 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:15.875671 11804 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:15.911267  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.858062 11816 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:15.906723 11861 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:15.915400  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.916291  9473 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:15.875437 11814 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:15.908490  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 248) took 1974 ms. Trace:
I20260808 01:50:15.908829 11631 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:50:15.882059 11568 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.894438 11744 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260808 01:50:15.923276  9474 rpcz_store.cc:276] 0808 01:50:13.934036 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:14.415638 (+481602us) service_pool.cc:224] Handling call
0808 01:50:14.421526 (+  5888us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:14.516632 (+ 95106us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:50:14.549442 (+ 32810us) log.cc:844] Serialized 14 byte log entry
0808 01:50:14.690359 (+140917us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:14.690523 (+   164us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:14.690708 (+   185us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.907695 (+1216987us) spinlock_profiling.cc:243] Waited 3.79 ms on lock 0x7b1800268de0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4185ac56 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229
0808 01:50:15.908235 (+   540us) spinlock_profiling.cc:243] Waited 81.5 ms on lock 0x7b5400399754. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:15.908279 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:15.908422 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7481,"prepare.queue_time_us":28438,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":55,"raft.queue_time_us":29588,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":126,"spinlock_wait_cycles":196252672,"thread_start_us":111177,"threads_started":1,"wal-append.queue_time_us":191774,"wal-append.run_cpu_time_us":1517,"wal-append.run_wall_time_us":1002165}
W20260808 01:50:15.920684  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 95652.2us
I20260808 01:50:15.935304  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.921551 10367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 467) took 1828 ms. Trace:
I20260808 01:50:15.936165  9473 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:15.923574  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.925218  9601 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:15.931191  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.937136  9480 raft_consensus.cc:2376] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3. Re-sending same reply.
I20260808 01:50:15.932328  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.936393 10367 rpcz_store.cc:276] 0808 01:50:14.089549 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:14.095128 (+  5579us) service_pool.cc:224] Handling call
0808 01:50:14.095545 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.295668 (+1200123us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:15.295714 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:15.296521 (+   807us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:15.296586 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.917939 (+621353us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540041f0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:15.917989 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:15.918121 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2759469824}
W20260808 01:50:15.933377  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:15.936192  9445 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:15.940737  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:15.943046  9477 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:15.938200  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.944320  9476 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:15.945111  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.945712  9476 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:15.922241  9475 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:15.947933 11742 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:15.949743  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:15.950929  9557 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 1690) took 3658 ms (client timeout 1912 ms). Trace:
I20260808 01:50:15.951001  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:15.951465  9557 rpcz_store.cc:269] 0808 01:50:12.293034 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:12.293248 (+   214us) service_pool.cc:224] Handling call
0808 01:50:15.914484 (+3621236us) spinlock_profiling.cc:243] Waited 2.68 s on lock 0x7b540029f4d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:15.950830 (+ 36346us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6161002752}
I20260808 01:50:15.951930  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1952) took 1149 ms. Trace:
I20260808 01:50:15.952119 11593 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.952342  9479 rpcz_store.cc:276] 0808 01:50:14.802126 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.917562 (+1115436us) service_pool.cc:224] Handling call
0808 01:50:15.951865 (+ 34303us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:15.952828 11852 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:50:15.953245 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:15.954303 11726 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260808 01:50:15.955209  9554 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:50:15.955029 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:15.955719 11814 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.956350  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:15.956854 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:15.957515  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:15.958263  9554 raft_consensus.cc:2393] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:15.958999  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:15.922335 11754 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:15.959664  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:15.960644 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:15.968360  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:15.969296  9552 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:15.941340  9555 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:15.964619  9556 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:15.965579  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.965920  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:15.971695  9475 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:15.972380  9472 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:15.965636  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:15.967128 10368 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:15.988517  9474 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:15.989267  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1945) took 1840 ms. Trace:
I20260808 01:50:15.989342 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:15.990051 10368 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:15.961680 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:15.991194  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 51448.6us
I20260808 01:50:15.982060 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:15.989660  9474 rpcz_store.cc:276] 0808 01:50:14.149187 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.987920 (+1838733us) service_pool.cc:224] Handling call
0808 01:50:15.988332 (+   412us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.989006 (+   674us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.989185 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:15.993005  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 53600us
W20260808 01:50:15.993842  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 57718.5us
W20260808 01:50:15.994714  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 59602.3us
I20260808 01:50:15.995087  9475 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:15.996011  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 253) took 1728 ms. Trace:
I20260808 01:50:15.974501  9478 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:15.975334  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:15.996613  9475 rpcz_store.cc:276] 0808 01:50:14.268074 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.994573 (+1726499us) service_pool.cc:224] Handling call
0808 01:50:15.994922 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:15.995674 (+   752us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:15.995936 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:15.979642 11626 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:15.979421 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:15.980484 11754 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.981257 10366 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:50:15.938828 11815 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:15.981632 11613 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260808 01:50:15.962647 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:15.952703  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:15.979421  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:15.983089  9477 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260808 01:50:16.017167  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1944) took 1868 ms. Trace:
I20260808 01:50:16.017566  9477 rpcz_store.cc:276] 0808 01:50:14.148965 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.982342 (+1833377us) service_pool.cc:224] Handling call
0808 01:50:15.982769 (+   427us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:16.016875 (+ 34106us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:16.017099 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:15.983435 10360 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:50:15.958177 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:15.986503 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:15.913622 11842 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:15.987723  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:15.997526  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 1946) took 1197 ms. Trace:
I20260808 01:50:16.029421  9478 rpcz_store.cc:276] 0808 01:50:14.800514 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.915148 (+1114634us) service_pool.cc:224] Handling call
0808 01:50:15.997431 (+ 82283us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":137856}
I20260808 01:50:15.999476  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:16.030664  9481 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:15.999881  9550 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260808 01:50:16.005251 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 80284.1us
I20260808 01:50:16.007272  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:16.008776  9475 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260808 01:50:16.033608  9475 pending_rounds.cc:85] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 1
I20260808 01:50:16.033931  9475 pending_rounds.cc:107] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260808 01:50:16.034264  9475 raft_consensus.cc:2889] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:16.034477  9472 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:16.009822  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:16.011565  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:16.018563  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:16.020874  9557 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:16.022265  9552 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:16.022807  9480 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:16.027330 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:16.035073  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1948) took 1234 ms. Trace:
I20260808 01:50:16.008893  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:16.035638  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:16.035663  9481 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260808 01:50:16.008316 10363 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:16.039860  9477 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:50:16.040525  9633 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4. Ops: []
W20260808 01:50:16.040717 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 48870us
I20260808 01:50:16.010917 11815 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:16.068362  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:16.068818  9478 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 01:50:16.069296  9476 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 01:50:16.069572  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1957) took 1249 ms. Trace:
I20260808 01:50:16.069933  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1958) took 1249 ms. Trace:
I20260808 01:50:16.070786  9560 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:16.070513  9481 rpcz_store.cc:276] 0808 01:50:14.820643 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.031404 (+1210761us) service_pool.cc:224] Handling call
0808 01:50:16.035425 (+  4021us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:16.069662 (+ 34237us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:16.069875 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:16.071772  9558 raft_consensus.cc:2393] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:16.069957  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1960) took 1243 ms. Trace:
W20260808 01:50:16.065440  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:16.073066  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 252) took 1857 ms. Trace:
I20260808 01:50:16.072369  9475 rpcz_store.cc:276] 0808 01:50:14.801029 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.003325 (+1202296us) service_pool.cc:224] Handling call
0808 01:50:16.003774 (+   449us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:16.034770 (+ 30996us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:16.035011 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:16.071723  9554 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:16.020880 11639 raft_consensus.cc:2764] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260808 01:50:16.072278  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1949) took 1271 ms. Trace:
I20260808 01:50:16.095082  9445 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:16.095279  9477 rpcz_store.cc:276] 0808 01:50:14.801198 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.018434 (+1217236us) service_pool.cc:224] Handling call
0808 01:50:16.039502 (+ 21068us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:16.072025 (+ 32523us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:16.072222 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:16.069937  9478 rpcz_store.cc:276] 0808 01:50:14.820236 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.030232 (+1209996us) service_pool.cc:224] Handling call
0808 01:50:16.068586 (+ 38354us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:16.069294 (+   708us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:16.069512 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:16.045706  9473 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:16.097313  9473 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:16.091722  9480 rpcz_store.cc:276] 0808 01:50:14.215807 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.993593 (+1777786us) service_pool.cc:224] Handling call
0808 01:50:16.022577 (+ 28984us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:16.072817 (+ 50240us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:16.073010 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:16.097873  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1742) took 1114 ms. Trace:
I20260808 01:50:16.098294  9473 rpcz_store.cc:276] 0808 01:50:14.983243 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.096922 (+1113679us) service_pool.cc:224] Handling call
0808 01:50:16.097183 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:16.097681 (+   498us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:16.097831 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:16.092403  9476 rpcz_store.cc:276] 0808 01:50:14.826782 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.068761 (+1241979us) service_pool.cc:224] Handling call
0808 01:50:16.069079 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:16.069719 (+   640us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:16.069901 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:16.043910 11809 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260808 01:50:16.092409  9475 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:16.108675  9559 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
W20260808 01:50:16.114748  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 107563us
I20260808 01:50:16.115594  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:16.031394 11631 raft_consensus.cc:697] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:50:16.120313  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 204995us
I20260808 01:50:16.125509  9601 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:16.127676 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:16.128634  9473 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:16.130013  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1744) took 1146 ms. Trace:
I20260808 01:50:16.130525  9480 rpcz_store.cc:276] 0808 01:50:14.983826 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.098309 (+1114483us) service_pool.cc:224] Handling call
0808 01:50:16.098715 (+   406us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:16.129532 (+ 30817us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:16.129577 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:16.129643 (+    66us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:16.129694 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:16.129778 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:16.129932 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:16.035640 10361 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:50:16.137869  9637 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:16.121688  9474 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260808 01:50:16.140347  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1961) took 1313 ms. Trace:
W20260808 01:50:16.140636  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:16.140813  9474 rpcz_store.cc:276] 0808 01:50:14.827431 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.071349 (+1243918us) service_pool.cc:224] Handling call
0808 01:50:16.121434 (+ 50085us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:16.139978 (+ 18544us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:16.140265 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:16.122438  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1741) took 1139 ms. Trace:
I20260808 01:50:16.142522  9475 rpcz_store.cc:276] 0808 01:50:14.982959 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.091874 (+1108915us) service_pool.cc:224] Handling call
0808 01:50:16.092223 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:16.113779 (+ 21556us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:16.122361 (+  8582us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:16.039449 11852 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:16.043290 11689 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:16.139021  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1745) took 1154 ms. Trace:
I20260808 01:50:16.138121  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:16.015738 11691 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.108s
I20260808 01:50:16.153736  9473 rpcz_store.cc:276] 0808 01:50:14.985123 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.098892 (+1113769us) service_pool.cc:224] Handling call
0808 01:50:16.128389 (+ 29497us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:16.129223 (+   834us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:16.138907 (+  9684us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:16.158874  9475 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:16.154649  9524 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:16.160959  9524 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:16.163175 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:16.164052 10366 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:16.150942  9480 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260808 01:50:16.039649 11840 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.107s
I20260808 01:50:16.042518 11893 consensus_queue.cc:1059] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.097s
I20260808 01:50:16.127324 11817 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:16.152118 11794 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election won for term 3
W20260808 01:50:16.168164  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 131037us
I20260808 01:50:16.169759  9474 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:50:16.201152  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1763) took 1115 ms. Trace:
I20260808 01:50:16.201596  9474 rpcz_store.cc:276] 0808 01:50:15.085318 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.142230 (+1056912us) service_pool.cc:224] Handling call
0808 01:50:16.169234 (+ 27004us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:16.200833 (+ 31599us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:16.201081 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:16.120548 11726 consensus_queue.cc:1059] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260808 01:50:16.022943 11867 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:16.170725  9524 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:16.136829 11615 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:16.178223  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:16.096768 11744 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.076s
I20260808 01:50:16.166332  9480 pending_rounds.cc:85] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 1
I20260808 01:50:16.212643  9480 pending_rounds.cc:107] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260808 01:50:16.213004  9480 raft_consensus.cc:2889] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:16.196848  9475 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 01:50:16.176327 11807 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:16.117663 11649 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20260808 01:50:16.123152 11834 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20260808 01:50:16.167660 11797 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:16.163581 11801 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:16.209151  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:16.198257  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 72842.2us
I20260808 01:50:16.216704 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:16.238131  9601 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:16.225167  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1760) took 1141 ms. Trace:
I20260808 01:50:16.158041 11852 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:16.165061 11689 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:16.201624 11647 consensus_meta.cc:287] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.585s	user 0.011s	sys 0.002s
I20260808 01:50:16.238153  9524 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:50:16.238889  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 143909us
I20260808 01:50:16.239614  9480 rpcz_store.cc:276] 0808 01:50:15.083897 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.131834 (+1047937us) service_pool.cc:224] Handling call
0808 01:50:16.132249 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:16.224750 (+ 92501us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:16.225079 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:16.238010 10366 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:16.230971  9634 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:16.236241  9475 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260808 01:50:16.207501 11573 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:16.293620  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 60224.6us
I20260808 01:50:16.214247 11626 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:16.214262 11794 raft_consensus.cc:697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:16.210449 11613 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:16.319042 11760 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:16.297663  9475 pending_rounds.cc:85] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 2
I20260808 01:50:16.341907  9475 pending_rounds.cc:107] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260808 01:50:16.304677 11647 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:16.320343  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:16.320528  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:16.321205 11848 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:16.321452  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:16.322204  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:16.322513 11754 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:16.300329 11801 raft_consensus.cc:697] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:16.323944  9634 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:16.324517 11539 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:16.325840 11803 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.209s
W20260808 01:50:16.333963 11755 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 3.847s	user 0.021s	sys 0.002s
W20260808 01:50:16.335247 11937 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.242s	user 0.001s	sys 0.000s
I20260808 01:50:16.314895 11864 raft_consensus.cc:3055] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:16.337772  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:16.335883  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:16.387311  9629 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:16.388916  9629 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 01:50:16.202836 11817 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:16.219519 11807 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:16.320340 11935 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260808 01:50:16.342357  9475 raft_consensus.cc:2889] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:16.348660  9632 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:16.293152 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:16.213553 11867 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:16.230757 11797 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:16.356632  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 338189us
I20260808 01:50:16.359843 10366 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:16.376916 11613 raft_consensus.cc:697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:16.376696  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:16.304941 11451 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:16.202836 11815 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:16.305213 11626 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:16.335842 11939 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:16.382839  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:16.318998 11583 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.263s
I20260808 01:50:16.385324  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:16.334559 11631 consensus_queue.cc:237] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:16.242462 11615 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:16.395368  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:16.300225 11912 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20260808 01:50:16.345259 11760 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:16.295092 11861 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:16.461681  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 166089us
I20260808 01:50:16.460124  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:16.470114  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:16.477762  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:16.316928 11936 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.142s
I20260808 01:50:16.317809 11943 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:16.386605 11864 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:16.449724 10364 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:16.448159 11807 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:16.460193 11539 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:16.448377 11848 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260808 01:50:16.517525 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 445569us
I20260808 01:50:16.514083  9630 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 4: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:16.460654 11842 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:16.517814  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 442462us
W20260808 01:50:16.595197  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 357205us
I20260808 01:50:16.596101  9601 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:16.551241 11689 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:16.478837 10363 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:16.483628 11797 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:16.450532 11809 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
W20260808 01:50:16.545122 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 470582us
I20260808 01:50:16.548285  9634 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:16.410833 11755 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:16.556064  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:16.513427 11939 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:16.535745 11626 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:16.578116 11861 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:16.550391 11760 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:16.637672 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:16.547470 11941 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:16.590453 11700 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:50:16.602030  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 362199us
I20260808 01:50:16.626782 11742 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election won for term 5
W20260808 01:50:16.539676  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 240679us
I20260808 01:50:16.668959  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:50:16.614379 10368 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.624s	user 0.013s	sys 0.000s
I20260808 01:50:16.624773  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:16.633996 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:16.582948 11943 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:16.604445  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:16.661154  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:16.671865 10332 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:16.677870 10332 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:16.625231 11815 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:16.633097 11939 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:16.632797 11647 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:16.674620  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 49956.9us
I20260808 01:50:16.670080 10368 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:16.681833  9480 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:50:16.676394  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:16.691129  9475 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:16.675331  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:16.692253  9478 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:16.676180  9599 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:16.579037 11615 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:16.576311 11794 consensus_queue.cc:237] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:16.583232 11801 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:16.640828 11941 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:16.629359 11593 consensus_queue.cc:1059] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260808 01:50:16.696306 11755 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:16.701653  9599 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:16.715317 11755 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:16.627036 11940 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.089s
I20260808 01:50:16.704177  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:16.631207 11816 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260808 01:50:16.635233 11848 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260808 01:50:16.703105 10367 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:50:16.637202 11744 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260808 01:50:16.685099 11649 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:16.685398 11754 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:16.679829 11739 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:16.732322  9478 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:16.733368  9475 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:16.715445 11945 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:16.685716 11742 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:16.685073 11726 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:16.679639 11936 consensus_queue.cc:1059] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
W20260808 01:50:16.734988  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 253781us
I20260808 01:50:16.763303  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:16.715796 11583 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 01:50:16.748807  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 44753.1us
I20260808 01:50:16.698426 11451 raft_consensus.cc:476] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Not starting pre-election -- already a leader
I20260808 01:50:16.685595 11539 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.070s
W20260808 01:50:16.790433  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:16.785175  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:16.747680 11649 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:16.806886  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:16.760636 11946 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.383s	user 0.002s	sys 0.000s
I20260808 01:50:16.757663 11739 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:16.749840 11852 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:16.755025 11949 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:16.689107 11864 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:16.786856 11938 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.322s	user 0.002s	sys 0.000s
I20260808 01:50:16.822490 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:16.788522 11809 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:16.790306 11583 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:16.674471 11613 consensus_queue.cc:237] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:16.753520 11689 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20260808 01:50:16.790694 11945 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:16.794548 11755 raft_consensus.cc:697] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:16.816980 11867 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:16.828370  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 116843us
I20260808 01:50:16.849732  9557 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 01:50:16.859526 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:16.781366 11754 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:16.812563 11649 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:16.781198 11726 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:16.817310 11631 raft_consensus.cc:3055] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:17.183588 11631 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:17.184108 11754 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:17.184754 11631 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:17.186168 11631 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:16.806346 11817 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:17.155004  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:16.872658  9560 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 4. Current term is 5. Ops: []
W20260808 01:50:17.187479 11969 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.186s	user 0.000s	sys 0.000s
I20260808 01:50:17.187700 11434 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:17.188337 11434 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:17.188683  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:16.823130 11739 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:16.853111  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:17.190106  9556 raft_consensus.cc:2410] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:17.191428  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:17.191635 11809 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:17.192440 11809 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:17.193248  9556 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260808 01:50:17.195442 11964 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.285s	user 0.000s	sys 0.001s
I20260808 01:50:17.191299 11848 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:17.196471 11848 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:17.197881 11726 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:16.895599  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:16.903256  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:50:16.828516  9524 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:16.824452 11451 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:16.831933 11795 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:17.199810 11451 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:17.203356  9560 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:16.952978  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1959) took 2126 ms. Trace:
I20260808 01:50:17.204991  9479 rpcz_store.cc:276] 0808 01:50:14.826099 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.043968 (+1217869us) service_pool.cc:224] Handling call
0808 01:50:16.113977 (+ 70009us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:16.179890 (+ 65913us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:16.179937 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:16.180627 (+   690us) log.cc:844] Serialized 26 byte log entry
0808 01:50:16.184689 (+  4062us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:16.412950 (+228261us) log.cc:844] Serialized 14 byte log entry
0808 01:50:16.413190 (+   240us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:16.413279 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:16.952618 (+539339us) raft_consensus.cc:1685] finished
0808 01:50:16.952652 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:16.952879 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":515100,"raft.run_cpu_time_us":451,"raft.run_wall_time_us":40912,"thread_start_us":264721,"threads_started":3,"wal-append.queue_time_us":60773}
I20260808 01:50:16.865816  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:17.182885 11963 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:17.187130 11615 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:17.198652  9551 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
W20260808 01:50:16.864682 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:16.930111 11966 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
I20260808 01:50:16.861538 11539 consensus_queue.cc:1243] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20260808 01:50:17.209044 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:16.896209 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:16.934279 11917 raft_consensus.cc:3055] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:16.931838 11683 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:16.884688 11583 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:17.223805  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 358054us
I20260808 01:50:16.851873 11955 raft_consensus.cc:3055] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:16.860921 11936 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:16.913407 11647 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:17.226222  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 419482us
I20260808 01:50:16.872877 11691 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:17.200834 11795 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:17.273660 11795 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:17.199906 11434 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:17.275085 11804 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.361s	user 0.015s	sys 0.000s
I20260808 01:50:17.275733 11804 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5.
I20260808 01:50:17.276381 11943 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:17.248970 11683 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:17.280090 11683 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:17.281584 11683 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:17.284070 11959 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:50:17.285881 11967 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:50:17.286886 11803 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:50:17.287853 11967 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:50:17.247201  9480 raft_consensus.cc:3055] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:17.290248  9480 raft_consensus.cc:740] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:17.214648 11867 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:16.897519 11756 consensus_queue.cc:1059] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.523s
I20260808 01:50:17.244163  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:17.255914  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 427471us
I20260808 01:50:16.852775 11568 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.386s
I20260808 01:50:17.231995 11953 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:17.260138  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1963) took 1657 ms. Trace:
I20260808 01:50:16.849931 11949 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:17.258496  9444 leader_election.cc:336] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.574s (SENT)
I20260808 01:50:17.271508  9479 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:50:17.289574 11803 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:17.318365  9473 rpcz_store.cc:276] 0808 01:50:15.602530 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.156663 (+554133us) service_pool.cc:224] Handling call
0808 01:50:16.190544 (+ 33881us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:16.632569 (+442025us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:16.633093 (+   524us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:16.637829 (+  4736us) log.cc:844] Serialized 26 byte log entry
0808 01:50:16.757312 (+119483us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:16.757373 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:16.765093 (+  7720us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:17.259903 (+494810us) raft_consensus.cc:1685] finished
0808 01:50:17.259927 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:17.260072 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34760,"raft.queue_time_us":389319,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":5793,"thread_start_us":465848,"threads_started":2,"wal-append.queue_time_us":171100}
I20260808 01:50:17.260337 11848 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:17.285979 11959 consensus_queue.cc:1059] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:17.242164 11917 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:17.291281  9480 consensus_queue.cc:260] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:17.304235 11977 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.266s	user 0.000s	sys 0.000s
I20260808 01:50:16.897205 11742 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:17.304723 11756 consensus_queue.cc:1059] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:50:17.305299 11451 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:17.259899 11973 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:17.225072 11963 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:17.307137 11802 raft_consensus.cc:3055] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:17.232402 11615 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:17.257319 11936 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:17.308162 11649 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:17.296940  9555 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:17.256474 11955 raft_consensus.cc:740] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:17.322309  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 63840.3us
I20260808 01:50:17.170038 11945 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:50:17.299908 11797 consensus_meta.cc:287] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.667s	user 0.009s	sys 0.004s
I20260808 01:50:17.301054  9524 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:17.350528  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:17.299544 11700 consensus_meta.cc:287] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.633s	user 0.003s	sys 0.008s
I20260808 01:50:17.334885  9478 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:17.466741  9634 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:17.462165  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:17.275126 11809 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:17.413854  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 255) took 2513 ms. Trace:
I20260808 01:50:17.424899 11973 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:17.467816  9632 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
W20260808 01:50:17.432544 10363 consensus_meta.cc:287] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.797s	user 0.012s	sys 0.000s
I20260808 01:50:17.469924  9472 rpcz_store.cc:276] 0808 01:50:14.900664 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.073331 (+1172667us) service_pool.cc:224] Handling call
0808 01:50:16.073665 (+   334us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:16.232633 (+158968us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:16.232681 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:16.340763 (+108082us) log.cc:844] Serialized 26 byte log entry
0808 01:50:16.597876 (+257113us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:16.597919 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:16.597998 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:17.413550 (+815552us) raft_consensus.cc:1685] finished
0808 01:50:17.413580 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:17.413771 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19832,"raft.queue_time_us":278010,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":29070,"thread_start_us":320766,"threads_started":2,"wal-append.queue_time_us":338527}
I20260808 01:50:17.417599 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:16.860749 11941 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:17.334661 11795 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:17.323225 11803 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:17.426128  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 142494us
I20260808 01:50:17.296018 11691 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:17.294979 11647 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:17.433100  9480 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:17.312441 11867 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:17.358498 11953 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:17.350528 11971 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:17.421406 11959 raft_consensus.cc:3055] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Stepping down as leader of term 3
W20260808 01:50:17.436362 11815 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.746s	user 0.006s	sys 0.006s
W20260808 01:50:17.477109  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:17.506631  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 205674us
I20260808 01:50:17.533795  9524 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:17.429363 11802 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:17.463369 11939 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.728s	user 0.011s	sys 0.000s
I20260808 01:50:17.514086  9632 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:17.515280  9636 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
W20260808 01:50:17.508855 11626 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.868s	user 0.006s	sys 0.006s
I20260808 01:50:17.439002 11936 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:17.330555 11755 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:17.535347 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:17.538277  9478 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:50:17.539945 11803 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:17.438712 11451 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:17.462836 11726 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:17.468955 11739 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:17.592941 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:17.592984  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:17.758373  9558 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
I20260808 01:50:17.591872  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 1793) took 1575 ms. Trace:
I20260808 01:50:17.760663  9552 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260808 01:50:17.623565 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:17.552263  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 1738) took 1577 ms. Trace:
I20260808 01:50:17.509795 11861 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:17.540048  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:17.473181 11973 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:17.764053  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:17.645318  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:17.509959 10363 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:17.763847  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:17.763742  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:17.546191  9559 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:50:17.766886  9555 rpcz_store.cc:276] 0808 01:50:15.974507 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.975006 (+   499us) service_pool.cc:224] Handling call
0808 01:50:17.551956 (+1576950us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b540016a3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:17.552159 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2941797760}
I20260808 01:50:17.771827  9635 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:17.757323 10365 raft_consensus.cc:2376] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3. Re-sending same reply.
I20260808 01:50:17.760056  9633 rpcz_store.cc:276] 0808 01:50:16.016506 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.017067 (+   561us) service_pool.cc:224] Handling call
0808 01:50:16.039405 (+ 22338us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:16.073602 (+ 34197us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:17.591437 (+1517835us) spinlock_profiling.cc:243] Waited 871 us on lock 0x7b54004218d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:17.591738 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2004224}
I20260808 01:50:17.644217 10362 raft_consensus.cc:2410] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:50:17.788671  9638 raft_consensus.cc:2410] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260808 01:50:17.547961 11965 log.cc:927] Time spent T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.288s	user 0.000s	sys 0.001s
I20260808 01:50:17.780786  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:17.750315 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:17.472566 11797 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:17.527419 11756 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.367s
I20260808 01:50:17.524240 11917 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:18.173108 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:17.846163  9480 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:50:17.797276  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.177484  9632 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:17.800239  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:17.923094  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:17.954387  9444 leader_election.cc:336] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.702s (SENT)
I20260808 01:50:17.893699  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1813) took 1408 ms. Trace:
I20260808 01:50:17.955118  9552 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260808 01:50:17.915350 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:17.528210 11972 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election won for term 4
W20260808 01:50:17.922817  9551 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s	user 0.025s	sys 0.058s
I20260808 01:50:17.750626 11815 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:17.642483 11962 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:18.180159  9638 rpcz_store.cc:276] 0808 01:50:16.485680 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:17.591840 (+1106160us) service_pool.cc:224] Handling call
0808 01:50:17.893596 (+301756us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1536896}
I20260808 01:50:18.180912  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1746) took 1353 ms. Trace:
I20260808 01:50:18.019022  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:17.534238 11941 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:18.182056  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:18.193828 10365 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:50:18.177052 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:18.177593  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 643819us
I20260808 01:50:18.195358  9524 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:17.920271  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:18.177121 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:18.196892 10369 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:18.198189 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:18.199788 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:17.537477 11807 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.940s	user 0.011s	sys 0.002s
W20260808 01:50:18.186009  9551 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.896s	user 0.038s	sys 0.074s
W20260808 01:50:18.167390 11986 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
I20260808 01:50:18.189143  9552 rpcz_store.cc:276] 0808 01:50:16.827364 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:17.836202 (+1008838us) service_pool.cc:224] Handling call
0808 01:50:17.954715 (+118513us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:18.180514 (+225799us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:18.180834 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:18.191408  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 57844.2us
I20260808 01:50:17.924607  9523 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:17.594610 11823 raft_consensus.cc:3055] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:17.530247 11959 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:18.202677 10369 raft_consensus.cc:2393] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:18.189635  9633 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:50:17.783527 11814 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.740s	user 0.016s	sys 0.024s
I20260808 01:50:17.533908 11795 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:17.842995 10363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 388) took 1814 ms (client timeout 2200 ms). Trace:
I20260808 01:50:18.206137  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:17.925364 11568 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:17.994189 11804 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:17.543139 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:18.194377 11942 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260808 01:50:17.768891 11451 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:18.218638  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:18.195899 11683 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.914s	user 0.008s	sys 0.006s
I20260808 01:50:17.533408 11971 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:17.994082 11949 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:18.209832 11649 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:18.270525 11941 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:18.175565 11955 consensus_queue.cc:260] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:17.995623 11817 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:18.224596 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:18.001087 11982 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:18.226548  9636 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:18.228178  9552 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260808 01:50:18.212361  9524 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:18.230902  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 306399us
I20260808 01:50:18.201535 11803 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:17.774103 11626 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.234498  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:17.785274 11647 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:18.221060 11814 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.181s	user 0.025s	sys 0.044s
I20260808 01:50:17.790727 11861 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.240844  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:18.209326 11823 raft_consensus.cc:740] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:18.292981  9556 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 01:50:18.230909 10361 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:18.297313 11823 consensus_queue.cc:260] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:18.299116 11797 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:18.299594 11993 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:18.299865 11797 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:18.300235 11993 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.303143 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 394) took 2316 ms. Trace:
I20260808 01:50:18.303524 11973 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.303613 10361 rpcz_store.cc:276] 0808 01:50:15.992767 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.206397 (+213630us) service_pool.cc:224] Handling call
0808 01:50:17.750120 (+1543723us) spinlock_profiling.cc:243] Waited 7.15 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d4185589e 000000000038d209 00007f5d49914940 00007f5d499dced4 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:50:18.303062 (+552942us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16444160}
I20260808 01:50:18.305601 10361 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260808 01:50:18.305943 11973 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:18.306830 11993 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:18.307193  9558 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 01:50:18.308544 11823 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:50:18.309773 11583 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.047s	user 0.006s	sys 0.003s
I20260808 01:50:18.221928 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:17.788034 11939 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:18.225860 10363 rpcz_store.cc:269] 0808 01:50:15.984864 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.197246 (+212382us) service_pool.cc:224] Handling call
0808 01:50:17.799590 (+1602344us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:18.206175 11962 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:18.023705 11802 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.213562 11807 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.222370 11917 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:18.238819 11568 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:18.257107 11959 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.240978 11700 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:50:18.176322 11760 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.486s	user 0.012s	sys 0.011s
I20260808 01:50:18.221868 11801 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:18.279718 10365 raft_consensus.cc:2376] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3. Re-sending same reply.
I20260808 01:50:18.225890 11691 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:18.281140  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 279) took 1453 ms. Trace:
I20260808 01:50:18.283926  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.326762  9552 rpcz_store.cc:276] 0808 01:50:16.828072 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.207125 (+1379053us) service_pool.cc:224] Handling call
0808 01:50:18.227825 (+ 20700us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:18.279143 (+ 51318us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:18.281069 (+  1926us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:18.206137 11815 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:18.272533 11739 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:18.295534 11983 log.cc:927] Time spent T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
I20260808 01:50:18.329406 10363 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:18.296301  9550 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:18.330236 10363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 384) took 1080 ms. Trace:
I20260808 01:50:18.330513 11815 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:18.325466 11801 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:18.296633 11626 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:18.284722 10364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 370) took 2349 ms. Trace:
I20260808 01:50:18.273564 11615 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:18.099305 11953 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.300429 11797 raft_consensus.cc:697] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:18.314973  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:18.311691 10369 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:18.366845  9638 raft_consensus.cc:2410] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:18.317408  9633 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:18.286163 11963 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:18.238057 11804 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:18.319505  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 365153us
W20260808 01:50:18.312947 11945 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.802s	user 0.007s	sys 0.006s
I20260808 01:50:18.306311 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 383) took 1087 ms. Trace:
I20260808 01:50:18.310192 11583 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.317835 11939 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:18.283558 11982 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.330638 10363 rpcz_store.cc:276] 0808 01:50:17.250108 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.321320 (+1071212us) service_pool.cc:224] Handling call
0808 01:50:18.329183 (+  7863us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:18.329914 (+   731us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:18.330171 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:18.423074  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 258) took 3379 ms. Trace:
I20260808 01:50:18.423519  9481 rpcz_store.cc:276] 0808 01:50:15.044167 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.127268 (+1083101us) service_pool.cc:224] Handling call
0808 01:50:16.156814 (+ 29546us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:16.378255 (+221441us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:16.378291 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:16.378340 (+    49us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:16.378378 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:18.422789 (+2044411us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b5400093dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:18.422837 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:18.423007 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":508691200}
W20260808 01:50:18.336618  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:18.424729  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:18.425222 11982 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:18.336956  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1971) took 2445 ms. Trace:
I20260808 01:50:18.339154  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:18.419763 10361 rpcz_store.cc:276] 0808 01:50:17.219095 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.304932 (+1085837us) service_pool.cc:224] Handling call
0808 01:50:18.305315 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:18.306028 (+   713us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:18.306250 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:18.427484  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:18.425884  9474 rpcz_store.cc:276] 0808 01:50:15.891976 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.202496 (+310520us) service_pool.cc:224] Handling call
0808 01:50:16.225099 (+ 22603us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:16.225471 (+   372us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:16.225515 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:16.306281 (+ 80766us) log.cc:844] Serialized 26 byte log entry
0808 01:50:16.641379 (+335098us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:16.790888 (+149509us) log.cc:844] Serialized 14 byte log entry
0808 01:50:16.861030 (+ 70142us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:16.861127 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:18.336685 (+1475558us) raft_consensus.cc:1685] finished
0808 01:50:18.336710 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:18.336882 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":66716,"prepare.queue_time_us":887719,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":56,"raft.queue_time_us":68198,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":30154,"thread_start_us":342915,"threads_started":2,"wal-append.queue_time_us":899242}
I20260808 01:50:18.428938  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:18.426107 11804 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:18.331276 10360 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:18.333323  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:18.367749  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 87738.1us
W20260808 01:50:18.431013  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 97823.4us
I20260808 01:50:18.431831  9600 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:18.432219  9523 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:18.433101  9600 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:18.434227 11993 raft_consensus.cc:2764] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260808 01:50:18.415872  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:18.416141  9444 leader_election.cc:336] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.812s (SENT)
I20260808 01:50:18.434460 11982 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:18.435079  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:18.436461  9444 leader_election.cc:336] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.868s (SENT)
I20260808 01:50:18.437161  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:18.368170  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.337369 11944 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:18.333791 11801 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:18.417472  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2048) took 1591 ms. Trace:
I20260808 01:50:18.335970 11959 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:18.449096  9555 rpcz_store.cc:276] 0808 01:50:16.826603 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:17.793297 (+966694us) service_pool.cc:224] Handling call
0808 01:50:17.793766 (+   469us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:17.794101 (+   335us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:18.280419 (+486318us) log.cc:844] Serialized 14 byte log entry
0808 01:50:18.416878 (+136459us) log.cc:844] Serialized 14 byte log entry
0808 01:50:18.417092 (+   214us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:18.417140 (+    48us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:18.417191 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:18.417272 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:18.417406 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":199540,"raft.queue_time_us":213,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":201,"spinlock_wait_cycles":54659968,"thread_start_us":402429,"threads_started":2,"wal-append.queue_time_us":135838}
I20260808 01:50:18.451453  9555 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:18.452224  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2051) took 1604 ms. Trace:
I20260808 01:50:18.452567  9555 rpcz_store.cc:276] 0808 01:50:16.847746 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.450914 (+1603168us) service_pool.cc:224] Handling call
0808 01:50:18.451263 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:18.451857 (+   594us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:18.452153 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:18.348979 10364 rpcz_store.cc:276] 0808 01:50:15.933888 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:15.958760 (+ 24872us) service_pool.cc:224] Handling call
0808 01:50:15.959112 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:16.594022 (+634910us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:18.239268 (+1645246us) spinlock_profiling.cc:243] Waited 426 ms on lock 0x7b5400594054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:18.282959 (+ 43691us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":979918592}
I20260808 01:50:18.325640 11760 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.447427 11944 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:18.455946  9560 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260808 01:50:18.456720 11760 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:18.335088 11807 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:18.334038 11451 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:18.456782 11944 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.349793 11802 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:18.458271 11944 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:50:18.336097  9601 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:42824 dropped due to backpressure: service queue is full with 50 items
I20260808 01:50:18.429821  9632 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:18.435191 11993 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:18.436030 11939 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 01:50:18.438421  9444 leader_election.cc:336] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.877s (SENT)
I20260808 01:50:18.439031  9633 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:18.472848 10366 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260808 01:50:18.473331 10366 pending_rounds.cc:85] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting all ops after (but not including) 4
I20260808 01:50:18.473606 10366 pending_rounds.cc:107] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260808 01:50:18.473848 10366 raft_consensus.cc:2889] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:50:18.416676  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 204402us
I20260808 01:50:18.475062 11971 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:18.475546 10364 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:18.477098 11797 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.478044  9524 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:18.479538 11795 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:18.481633 11583 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:18.481863 11804 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
W20260808 01:50:18.482162 10362 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 398) took 1649 ms (client timeout 1630 ms). Trace:
I20260808 01:50:18.482389 11982 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:18.482888 11451 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:50:18.408102 11815 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:50:18.427541 11994 log.cc:927] Time spent T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260808 01:50:18.419324 11945 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.443816 11995 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:18.442220 11756 raft_consensus.cc:2764] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260808 01:50:18.493968  9633 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:50:18.459635  9601 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:42824 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1662)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1829)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1673)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1686)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1680)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1687)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1832)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1678)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1688)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1830)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1689)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1838)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1841)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1663)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1826)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1843)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1836)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1837)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1839)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1840)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1844)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1668)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1845)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1683)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1834)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1684)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1685)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 234)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 235)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 238)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 239)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1670)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1671)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1672)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1674)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1676)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1679)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1681)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1682)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 1835)
I20260808 01:50:18.484169  9524 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:18.434355  9523 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:18.479657  9444 leader_election.cc:336] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.924s (SENT)
I20260808 01:50:18.494129 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:18.497776  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:18.490962  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2052) took 1594 ms. Trace:
W20260808 01:50:18.502364  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 68084.5us
I20260808 01:50:18.499972  9555 rpcz_store.cc:276] 0808 01:50:16.859962 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.453768 (+1593806us) service_pool.cc:224] Handling call
0808 01:50:18.454102 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:18.454308 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:18.454333 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:18.454375 (+    42us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:18.454404 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:18.454458 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:18.454643 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:18.509163 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:18.509567 10369 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:50:18.510001 10364 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:18.484649 10362 rpcz_store.cc:269] 0808 01:50:16.833215 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.895808 (+ 62593us) service_pool.cc:224] Handling call
0808 01:50:18.481884 (+1586076us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b5400594cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:18.482069 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":731153152}
I20260808 01:50:18.504894  9523 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:18.524477  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:18.525303  9633 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:50:18.525996  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:18.527155  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:18.526158 10364 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:18.529704  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:18.530037 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:18.530668  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:18.534036 11941 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:18.529152 11939 consensus_queue.cc:1059] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260808 01:50:18.502306 11795 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:18.535965 10369 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 01:50:18.501617 11760 raft_consensus.cc:3055] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:18.523514 11998 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:18.516587 11804 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:50:18.527710 11982 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 01:50:18.533078 11936 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.710s	user 0.013s	sys 0.000s
I20260808 01:50:18.526427 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:18.494616 11807 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260808 01:50:18.527782 11942 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:50:18.553264  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.498942 11815 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:18.565287  9633 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:18.539273 11691 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:18.532164 11949 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:18.531726 11972 raft_consensus.cc:697] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:18.549386  9634 raft_consensus.cc:2410] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:50:18.566128  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:18.535302 12000 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:18.566706  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:18.552747 11982 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:18.560601 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:18.568863  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0017617779958528s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3285s for a response
I20260808 01:50:18.540766 11953 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:18.571079 11691 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:18.561460 11631 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.375s	user 0.012s	sys 0.005s
I20260808 01:50:18.547025 11987 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:18.573290  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:18.596385  9475 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:18.573570  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:18.572207 11949 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:18.547667 11613 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:18.599344 11949 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:18.599720 11613 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.587594 11982 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:18.575098  9480 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:18.607910 12001 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
I20260808 01:50:18.610006 11942 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:18.545857 11760 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:18.610667 11942 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.611768 11760 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.613318 11942 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:18.613593 11760 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:18.532364 11995 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:18.615119 11995 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:18.619226 11995 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:18.620817 11987 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.577466  9472 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:18.599359  9474 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 6: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:18.582957  9477 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260808 01:50:18.624297  9475 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:18.578810  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:18.568338  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1662) took 1051 ms. Trace:
I20260808 01:50:18.567317 10361 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:18.586556  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:18.590874 11754 raft_consensus.cc:3055] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:18.591051 10363 raft_consensus.cc:2393] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 2: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:18.577092 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:18.577349 11804 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:18.652374 11804 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.591566 11953 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:18.657243 11801 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.657819 11953 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:18.592065 11815 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:18.658535  9552 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260808 01:50:18.562917 11936 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.659408 10364 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:50:18.592265  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.661386  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:18.661336 12002 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:50:18.587045 12000 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.662760  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.663386  9638 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:18.663463 11972 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:50:18.664311  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.664497 12000 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:18.665480  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.593772  9636 raft_consensus.cc:2410] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260808 01:50:18.667179 11988 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.419s	user 0.000s	sys 0.003s
I20260808 01:50:18.668335  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1828) took 1125 ms. Trace:
I20260808 01:50:18.668740  9636 rpcz_store.cc:276] 0808 01:50:17.542973 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.283684 (+740711us) service_pool.cc:224] Handling call
0808 01:50:18.668104 (+384420us) spinlock_profiling.cc:243] Waited 266 ms on lock 0x7b540025b7d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:18.668266 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":612571904}
I20260808 01:50:18.596729 11691 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:18.671054  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2044) took 1848 ms. Trace:
I20260808 01:50:18.671486  9551 rpcz_store.cc:276] 0808 01:50:16.823123 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:17.221320 (+398197us) service_pool.cc:224] Handling call
0808 01:50:17.245139 (+ 23819us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:17.288832 (+ 43693us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:17.288867 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:17.289459 (+   592us) log.cc:844] Serialized 26 byte log entry
0808 01:50:18.201970 (+912511us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:18.202019 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:18.202095 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:18.666499 (+464404us) spinlock_profiling.cc:243] Waited 29.5 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:18.670703 (+  4204us) raft_consensus.cc:1685] finished
0808 01:50:18.670728 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:18.670971 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31312,"raft.queue_time_us":31471,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":121,"spinlock_wait_cycles":67793792,"thread_start_us":911986,"threads_started":1,"wal-append.queue_time_us":942066}
I20260808 01:50:18.673641  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:18.673924 11949 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:18.676319 11949 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:18.670208  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:18.679060  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:18.679952  9523 leader_election.cc:336] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Timed out: RequestConsensusVote RPC to 127.7.165.196:33455 timed out after 1.580s (SENT)
I20260808 01:50:18.680174  9445 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:18.680559  9523 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:18.682693  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.683993  9638 raft_consensus.cc:2410] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260808 01:50:18.626545  9633 rpcz_store.cc:276] 0808 01:50:17.516685 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.524184 (+1007499us) service_pool.cc:224] Handling call
0808 01:50:18.568169 (+ 43985us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:18.626307  9481 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:18.634780  9480 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
W20260808 01:50:18.642043  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:18.687619  9444 leader_election.cc:336] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.021s (SENT)
I20260808 01:50:18.688067  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:18.649294  9477 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:18.688860  9477 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:18.637581 11754 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:18.661782  9635 raft_consensus.cc:2393] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:18.670164  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:18.558493 11941 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:18.575444 11795 raft_consensus.cc:697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:18.691183  9474 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:18.602504 11613 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:18.641536  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:18.686959  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.693261  9633 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:18.689678  9445 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:18.691834  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.705646  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.708161  9637 raft_consensus.cc:2410] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260808 01:50:18.711588  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:18.712529  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 111010us
I20260808 01:50:18.712311  9551 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:18.714538  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.661427 11936 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:18.718473  9638 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:18.715004  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 175168us
I20260808 01:50:18.689625  9477 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.716553  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.706986 11939 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:18.717501 11982 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:18.707367 12006 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:18.720690 11754 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.686430 11949 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:18.722470  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:18.689141 12000 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:18.722888  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:18.730106 10367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 395) took 2716 ms. Trace:
W20260808 01:50:18.730827 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:18.731304  9445 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:18.725646  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:18.730553 10367 rpcz_store.cc:276] 0808 01:50:16.013456 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.379584 (+366128us) service_pool.cc:224] Handling call
0808 01:50:16.690523 (+310939us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:16.737430 (+ 46907us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:18.729868 (+1992438us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b5400499650. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:18.730041 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":714095488}
I20260808 01:50:18.727391  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:18.732038  9477 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:18.726511  9551 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:50:18.732442  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1822) took 1191 ms. Trace:
I20260808 01:50:18.727494  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.734187  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.734892  9636 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:18.732834  9481 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:18.735967  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:18.736764  9445 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:18.734998  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1663) took 1218 ms. Trace:
I20260808 01:50:18.722373 11754 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:18.725456 10361 raft_consensus.cc:2410] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:50:18.733345  9632 rpcz_store.cc:276] 0808 01:50:17.540968 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.725358 (+1184390us) service_pool.cc:224] Handling call
0808 01:50:18.732379 (+  7021us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:18.744478  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:18.741326  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:18.738765  9638 rpcz_store.cc:276] 0808 01:50:17.516965 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.727200 (+1210235us) service_pool.cc:224] Handling call
0808 01:50:18.734929 (+  7729us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:18.740645 11939 consensus_queue.cc:1059] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260808 01:50:18.723330 11982 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:18.749580 11643 tablet_bootstrap.cc:492] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260808 01:50:18.750140  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.726745 11949 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.750926 11953 consensus_queue.cc:1059] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:18.731119 12007 raft_consensus.cc:2764] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260808 01:50:18.750947 11942 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:50:18.754387  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.752882 11936 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:18.763190  9601 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:18.758399  9636 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:18.743628 12006 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 01:50:18.765390  9444 leader_election.cc:336] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.181s (SENT)
I20260808 01:50:18.772019  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:18.752748 11939 raft_consensus.cc:3055] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:18.773594 11842 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:18.774012 11939 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:18.774255 11842 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:18.775223 11842 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:18.775516 11939 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.777575 11939 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:18.777628  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.760263  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.779587 11949 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:18.779140 11936 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:18.773353  9601 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:18.785050 11987 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:18.738440 12009 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:50:18.812445  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47077.1us
I20260808 01:50:18.744751 11980 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:18.832170 11980 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:18.832669 11980 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:18.780762 11942 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:18.836216  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:18.771092 11643 tablet_bootstrap.cc:492] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:50:18.822168  9633 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:18.841035  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.824275  9638 raft_consensus.cc:2410] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260808 01:50:18.841920  9444 leader_election.cc:336] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.233s (SENT)
I20260808 01:50:18.842532  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:18.841738  9631 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
W20260808 01:50:18.843649  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:18.825423  9477 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
W20260808 01:50:18.827749  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 54473.6us
I20260808 01:50:18.845630  9477 pending_rounds.cc:85] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 1
I20260808 01:50:18.845988  9477 pending_rounds.cc:107] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260808 01:50:18.846352  9477 raft_consensus.cc:2889] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:18.815255  9551 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 01:50:18.851526  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2054) took 1940 ms. Trace:
I20260808 01:50:18.851943  9551 rpcz_store.cc:276] 0808 01:50:16.911020 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.757746 (+1846726us) service_pool.cc:224] Handling call
0808 01:50:18.812097 (+ 54351us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:18.851175 (+ 39078us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:18.851429 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:18.767024 11945 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:18.767961 12006 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:18.855293  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:18.855460 12006 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:18.854252  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:18.857044  9481 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:18.853428 11643 ts_tablet_manager.cc:1403] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 3.357s	user 0.048s	sys 0.034s
I20260808 01:50:18.812896 11953 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:18.864461 12009 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:18.827492 11941 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.836738 12012 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:18.844978 11995 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:18.856267  9473 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:18.853820  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:18.864492  9631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1816) took 1509 ms (client timeout 1861 ms). Trace:
I20260808 01:50:18.865854  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:18.855638 11970 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.327s	user 0.002s	sys 0.003s
I20260808 01:50:18.879316  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.883359  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:18.884163  9629 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:50:18.884876  9629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1818) took 1450 ms (client timeout 1752 ms). Trace:
W20260808 01:50:18.885380  9629 rpcz_store.cc:269] 0808 01:50:17.434198 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.838338 (+1404140us) service_pool.cc:224] Handling call
0808 01:50:18.884778 (+ 46440us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:18.883245  9631 rpcz_store.cc:269] 0808 01:50:17.334558 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:17.544373 (+209815us) service_pool.cc:224] Handling call
0808 01:50:18.840856 (+1296483us) spinlock_profiling.cc:243] Waited 2.35 ms on lock 0x7b5800231698. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:50:18.843406 (+  2550us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5401472}
I20260808 01:50:18.839210 11691 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.892083 11936 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:18.893872  9479 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260808 01:50:18.892656  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.895735  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.896585 11643 raft_consensus.cc:359] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.897303  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.898648  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:18.900143  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.901512  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.902144  9638 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
W20260808 01:50:18.902374  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:18.904363 11691 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:18.904260  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:18.840480 11942 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:18.828620 12007 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
I20260808 01:50:18.767397 11795 consensus_queue.cc:237] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:18.918886  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:18.919809  9551 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:18.921391  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:18.923216  9481 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:18.924422  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:18.925017  9481 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:18.925712  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1668) took 1398 ms. Trace:
I20260808 01:50:18.926324  9631 rpcz_store.cc:276] 0808 01:50:17.527257 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.891141 (+1363884us) service_pool.cc:224] Handling call
0808 01:50:18.925615 (+ 34474us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:18.899164 11643 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:50:18.907620  9477 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:50:18.910383  9521 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:18.911299 11726 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:18.886335 11995 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:18.888839 12012 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:18.907001  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:18.936311  9629 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:18.918632 11801 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:18.938385 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 89619.9us
I20260808 01:50:18.941913  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 234) took 1427 ms. Trace:
I20260808 01:50:18.978350  9629 rpcz_store.cc:276] 0808 01:50:17.514837 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.940631 (+1425794us) service_pool.cc:224] Handling call
0808 01:50:18.941156 (+   525us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:18.941396 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:18.941433 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:18.941495 (+    62us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:18.941541 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:18.941617 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:18.941833 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:18.981047  9629 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260808 01:50:18.982043  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 238) took 1466 ms. Trace:
I20260808 01:50:18.944039  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:18.942819 11840 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
W20260808 01:50:18.951289 12011 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
W20260808 01:50:18.951848 11976 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 1.485s	user 0.006s	sys 0.000s
W20260808 01:50:18.953399 12016 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.116s	user 0.001s	sys 0.000s
I20260808 01:50:18.927462  9638 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:50:18.959183 11861 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:18.943820  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:18.992559  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48824.6us
W20260808 01:50:18.975260  9478 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 265) took 2277 ms (client timeout 2151 ms). Trace:
I20260808 01:50:18.995426 11861 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:50:18.998153  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:18.993780  9478 rpcz_store.cc:269] 0808 01:50:16.697641 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.762866 (+ 65225us) service_pool.cc:224] Handling call
0808 01:50:18.975024 (+2212158us) spinlock_profiling.cc:243] Waited 551 ms on lock 0x7b540038d1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:18.975172 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1266165632}
I20260808 01:50:18.982542  9629 rpcz_store.cc:276] 0808 01:50:17.516061 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.980203 (+1464142us) service_pool.cc:224] Handling call
0808 01:50:18.980673 (+   470us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:18.981654 (+   981us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:18.981962 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:19.009763  9629 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260808 01:50:19.012459  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 239) took 1496 ms. Trace:
I20260808 01:50:18.942569 11842 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:19.015440 12012 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:19.019104  9632 raft_consensus.cc:2410] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260808 01:50:18.960096 11755 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.011s	sys 0.010s
I20260808 01:50:18.937983 11972 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:18.935189 11995 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:19.020785  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:19.022017  9478 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:50:19.037331  9476 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:18.975859 11942 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:19.017642 11643 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
W20260808 01:50:19.415328  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:19.555835  9475 raft_consensus.cc:2393] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 5: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:19.134043  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2053) took 2229 ms. Trace:
W20260808 01:50:19.236696  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:19.082410  9445 leader_election.cc:336] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Timed out: RequestConsensusVote RPC to 127.7.165.196:33455 timed out after 1.788s (SENT)
W20260808 01:50:19.557952  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:19.557968  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 475611us
W20260808 01:50:19.559291  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:19.077800  9444 leader_election.cc:336] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.544s (SENT)
I20260808 01:50:19.212985  9551 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:19.556640  9555 rpcz_store.cc:276] 0808 01:50:16.905044 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.506988 (+1601944us) service_pool.cc:224] Handling call
0808 01:50:18.507502 (+   514us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:18.526936 (+ 19434us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:18.526977 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:18.531662 (+  4685us) log.cc:844] Serialized 26 byte log entry
0808 01:50:18.677674 (+146012us) spinlock_profiling.cc:243] Waited 1.37 ms on lock 0x7b180038a640. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:50:18.916864 (+239190us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:18.916910 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:18.917251 (+   341us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:19.133595 (+216344us) raft_consensus.cc:1685] finished
0808 01:50:19.133628 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:19.133953 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34654,"raft.queue_time_us":25534,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":10663,"spinlock_wait_cycles":3158400,"thread_start_us":238931,"threads_started":1,"wal-append.queue_time_us":400778}
I20260808 01:50:19.561659  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:19.561774  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:19.562523  9551 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 5: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:19.308209  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:19.098613  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:19.042855  9629 rpcz_store.cc:276] 0808 01:50:17.516345 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:19.007887 (+1491542us) service_pool.cc:224] Handling call
0808 01:50:19.009273 (+  1386us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:19.011961 (+  2688us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.012364 (+   403us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:19.540558  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2050) took 2693 ms. Trace:
I20260808 01:50:19.554796  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:19.223663  9477 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260808 01:50:19.564391  9552 rpcz_store.cc:276] 0808 01:50:16.847092 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.331348 (+1484256us) service_pool.cc:224] Handling call
0808 01:50:18.331769 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:18.658995 (+327226us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.540214 (+881219us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b54003147d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:19.540478 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":750909824}
I20260808 01:50:19.564954  9477 pending_rounds.cc:85] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 2
I20260808 01:50:19.565425  9477 pending_rounds.cc:107] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260808 01:50:19.559937  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:19.565855  9477 raft_consensus.cc:2889] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:50:19.568225 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:19.568795 10332 connection.cc:256] RPC call timeout handler was delayed by -1.6725880459998734s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2291s for a response
W20260808 01:50:19.569375 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:19.569855 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:19.570127 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:19.570449 10332 connection.cc:256] RPC call timeout handler was delayed by -1.0648260750167537s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1996s for a response
I20260808 01:50:19.570995 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:19.577594  9629 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:19.580142  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1670) took 2041 ms. Trace:
I20260808 01:50:19.580442  9629 rpcz_store.cc:276] 0808 01:50:17.538989 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:19.576823 (+2037834us) service_pool.cc:224] Handling call
0808 01:50:19.577259 (+   436us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:19.579909 (+  2650us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.580091 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:19.551477 12006 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.695s	user 0.002s	sys 0.012s
W20260808 01:50:19.295292 11998 consensus_meta.cc:287] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.728s	user 0.001s	sys 0.014s
W20260808 01:50:19.318037 11939 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.540s	user 0.013s	sys 0.005s
I20260808 01:50:19.592027 11939 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:19.592746 11867 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:19.593410 11867 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:19.593956 11867 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:19.594460  9634 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1824) took 2052 ms (client timeout 1574 ms). Trace:
W20260808 01:50:19.595064  9634 rpcz_store.cc:269] 0808 01:50:17.541621 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:17.792088 (+250467us) service_pool.cc:224] Handling call
0808 01:50:19.594116 (+1802028us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b5400338954. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:19.594313 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":848635264}
I20260808 01:50:19.595582  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:19.596791  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:19.597138  9634 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:50:19.597947  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1672) took 1822 ms. Trace:
I20260808 01:50:19.587796  9473 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:50:19.555619 11972 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:19.603034  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:19.603150 11972 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:19.603804  9374 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55058 (request call id 125) took 8349 ms. Trace:
I20260808 01:50:19.604714  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1742) took 2785 ms. Trace:
I20260808 01:50:19.605324  9557 rpcz_store.cc:276] 0808 01:50:16.819732 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.819946 (+   214us) service_pool.cc:224] Handling call
0808 01:50:16.849450 (+ 29504us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:17.175485 (+326035us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.604408 (+2428923us) spinlock_profiling.cc:243] Waited 55.1 ms on lock 0x7b54002853a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490b1a0 00007f5d44996de2 00007f5d449931b6 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:50:19.604651 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":126650752}
I20260808 01:50:19.606547  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1826) took 2064 ms. Trace:
I20260808 01:50:19.607018  9632 rpcz_store.cc:276] 0808 01:50:17.542294 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.743815 (+1201521us) service_pool.cc:224] Handling call
0808 01:50:19.606308 (+862493us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b540032cb54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:19.606483 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":628846336}
I20260808 01:50:19.607707 11939 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:19.608242 11939 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:19.608606  9632 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:19.136528 11861 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:19.609884  9554 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:19.609958 11864 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:19.615990 11757 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:19.624823 11757 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:19.604221  9374 rpcz_store.cc:276] 0808 01:50:11.254218 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:11.254361 (+   143us) service_pool.cc:224] Handling call
0808 01:50:19.603735 (+8349374us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:50:15.854813 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:50:15.855425 (+   612us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:50:15.855489 (+    64us) write_op.cc:435] Acquired schema lock
0808 01:50:15.855514 (+    25us) tablet.cc:662] Decoding operations
0808 01:50:15.856337 (+   823us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:50:15.856554 (+   217us) write_op.cc:641] Partition lock acquired for write op
0808 01:50:15.856578 (+    24us) tablet.cc:685] Acquiring locks for 5 operations
0808 01:50:15.857373 (+   795us) tablet.cc:701] Row locks acquired
0808 01:50:15.857404 (+    31us) write_op.cc:260] PREPARE: finished
0808 01:50:15.857648 (+   244us) write_op.cc:270] Start()
0808 01:50:15.857793 (+   145us) write_op.cc:276] Timestamp: P: 1786153815857562 usec, L: 0
0808 01:50:15.857824 (+    31us) op_driver.cc:348] REPLICATION: starting
0808 01:50:15.858558 (+   734us) log.cc:844] Serialized 2069 byte log entry
0808 01:50:17.193238 (+1334680us) op_driver.cc:464] REPLICATION: finished
0808 01:50:17.913376 (+720138us) write_op.cc:301] APPLY: starting
0808 01:50:17.913542 (+   166us) tablet.cc:1367] starting BulkCheckPresence
0808 01:50:17.922076 (+  8534us) tablet.cc:1370] finished BulkCheckPresence
0808 01:50:17.922335 (+   259us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:50:18.075185 (+152850us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:50:18.075576 (+   391us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0808 01:50:18.075648 (+    72us) write_op.cc:312] APPLY: finished
0808 01:50:18.076018 (+   370us) log.cc:844] Serialized 47 byte log entry
0808 01:50:18.287606 (+211588us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:18.291890 (+  4284us) write_op.cc:454] Released schema lock
0808 01:50:18.319993 (+ 28103us) write_op.cc:341] FINISH: Updating metrics
0808 01:50:18.333701 (+ 13708us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:18.333732 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4067161,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":451963,"apply.run_cpu_time_us":26939,"apply.run_wall_time_us":536150,"num_ops":5,"prepare.queue_time_us":147199,"prepare.run_cpu_time_us":13620,"prepare.run_wall_time_us":147671,"raft.queue_time_us":693793,"raft.run_cpu_time_us":14084,"raft.run_wall_time_us":329520,"replication_time_us":1145915,"spinlock_wait_cycles":46848,"thread_start_us":717545,"threads_started":5,"wal-append.queue_time_us":379558,"wal-append.run_cpu_time_us":16295,"wal-append.run_wall_time_us":2746782}]]}
I20260808 01:50:19.625803 11962 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:19.617092  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1674) took 1779 ms. Trace:
I20260808 01:50:19.627344 11945 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:50:19.627871  9632 rpcz_store.cc:276] 0808 01:50:17.837360 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:19.607873 (+1770513us) service_pool.cc:224] Handling call
0808 01:50:19.608236 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:19.616789 (+  8553us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.617025 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:19.630182 11989 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.376s	user 0.000s	sys 0.008s
I20260808 01:50:19.630656  9632 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:19.626561 11962 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:19.632318 11757 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:19.632411 11962 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:19.633031 11990 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.373s	user 0.006s	sys 0.002s
I20260808 01:50:19.633363 11962 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:19.634450 11757 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:19.634686 11842 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:19.635178 11757 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:19.619279  9551 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:19.640612  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 281) took 2422 ms. Trace:
I20260808 01:50:19.641188  9551 rpcz_store.cc:276] 0808 01:50:17.217900 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:19.578824 (+2360924us) service_pool.cc:224] Handling call
0808 01:50:19.619086 (+ 40262us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:19.640241 (+ 21155us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.640520 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:19.556106 11755 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.292s	user 0.024s	sys 0.037s
I20260808 01:50:19.619868  9552 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260808 01:50:19.643759  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 280) took 2720 ms. Trace:
I20260808 01:50:19.644464  9552 rpcz_store.cc:276] 0808 01:50:16.923601 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:19.577163 (+2653562us) service_pool.cc:224] Handling call
0808 01:50:19.619627 (+ 42464us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:19.643379 (+ 23752us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.643685 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:19.598379  9634 rpcz_store.cc:276] 0808 01:50:17.776031 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:19.596310 (+1820279us) service_pool.cc:224] Handling call
0808 01:50:19.596720 (+   410us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:19.597631 (+   911us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.597880 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:19.637801  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1676) took 1584 ms. Trace:
I20260808 01:50:19.621414  9555 raft_consensus.cc:2410] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 01:50:19.639284  9557 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:19.647802  9632 rpcz_store.cc:276] 0808 01:50:18.053171 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:19.629630 (+1576459us) service_pool.cc:224] Handling call
0808 01:50:19.630238 (+   608us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:19.631525 (+  1287us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.637719 (+  6194us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:19.623442 12005 raft_consensus.cc:3055] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:19.653038  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2055) took 2400 ms. Trace:
I20260808 01:50:19.653633  9557 rpcz_store.cc:276] 0808 01:50:17.252678 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:19.606494 (+2353816us) service_pool.cc:224] Handling call
0808 01:50:19.639027 (+ 32533us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:19.650497 (+ 11470us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.652954 (+  2457us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:19.656044  9552 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 01:50:19.640139 11998 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:19.617865 11972 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:19.618187 11939 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:19.664870  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2057) took 2340 ms. Trace:
I20260808 01:50:19.640542 11942 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:19.666656  9634 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 01:50:19.669236  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1750) took 2371 ms. Trace:
I20260808 01:50:19.669236  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1679) took 1379 ms. Trace:
I20260808 01:50:19.669821  9552 rpcz_store.cc:276] 0808 01:50:17.297400 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:19.645737 (+2348337us) service_pool.cc:224] Handling call
0808 01:50:19.655779 (+ 10042us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:19.668964 (+ 13185us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.669172 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:19.670109  9634 rpcz_store.cc:276] 0808 01:50:18.289778 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:19.647181 (+1357403us) service_pool.cc:224] Handling call
0808 01:50:19.665130 (+ 17949us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:19.668999 (+  3869us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.669172 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:19.668277  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1681) took 1351 ms. Trace:
I20260808 01:50:19.671578  9632 rpcz_store.cc:276] 0808 01:50:18.317267 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:19.650543 (+1333276us) service_pool.cc:224] Handling call
0808 01:50:19.667431 (+ 16888us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:19.667806 (+   375us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:50:19.667859 (+    53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:19.667927 (+    68us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:19.668001 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.668083 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:19.668214 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":745}
I20260808 01:50:19.675318 11757 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:19.676877  9634 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260808 01:50:19.679116  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1682) took 1361 ms. Trace:
I20260808 01:50:19.678341  9632 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:19.680651  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 1835) took 1351 ms. Trace:
I20260808 01:50:19.681120  9632 rpcz_store.cc:276] 0808 01:50:18.329196 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:19.672709 (+1343513us) service_pool.cc:224] Handling call
0808 01:50:19.678146 (+  5437us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:19.680364 (+  2218us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.680590 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:19.667290  9557 rpcz_store.cc:276] 0808 01:50:17.317622 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:19.654857 (+2337235us) service_pool.cc:224] Handling call
0808 01:50:19.657041 (+  2184us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:19.657243 (+   202us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:19.657271 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:19.657320 (+    49us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:19.657354 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.657408 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:19.657534 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:19.679574  9634 rpcz_store.cc:276] 0808 01:50:18.317579 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:19.671109 (+1353530us) service_pool.cc:224] Handling call
0808 01:50:19.676530 (+  5421us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:19.678798 (+  2268us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.679053 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:19.687820 10360 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 387) took 2364 ms (client timeout 1788 ms). Trace:
I20260808 01:50:19.689879 10365 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260808 01:50:19.675407 11939 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:19.690430 10365 pending_rounds.cc:85] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting all ops after (but not including) 4
I20260808 01:50:19.703235 10365 pending_rounds.cc:107] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260808 01:50:19.640758 12006 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:19.703744 10365 raft_consensus.cc:2889] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:19.651660 12005 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:50:19.672861 11980 consensus_meta.cc:287] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.840s	user 0.012s	sys 0.005s
W20260808 01:50:19.688529 10360 rpcz_store.cc:269] 0808 01:50:17.323606 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:17.542812 (+219206us) service_pool.cc:224] Handling call
0808 01:50:19.687517 (+2144705us) spinlock_profiling.cc:243] Waited 93.8 ms on lock 0x7b540046c154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:19.687709 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":215774080}
I20260808 01:50:19.749475 10365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 479) took 1254 ms. Trace:
I20260808 01:50:19.751673 10365 rpcz_store.cc:276] 0808 01:50:18.495241 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.495415 (+   174us) service_pool.cc:224] Handling call
0808 01:50:19.677165 (+1181750us) spinlock_profiling.cc:243] Waited 6.31 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:50:19.677423 (+   258us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:19.749150 (+ 71727us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.749396 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14512512}
I20260808 01:50:19.757462 11834 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:19.758162 11834 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:19.711783  9559 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
W20260808 01:50:19.732646  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:19.689630 11972 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:19.703605 11755 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260808 01:50:19.714401 11939 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:19.762575  9635 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1667) took 2235 ms (client timeout 1641 ms). Trace:
I20260808 01:50:19.776157 12006 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:19.770476 11568 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:19.751245 11980 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:19.765868 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:19.793269 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:19.780921  9635 rpcz_store.cc:269] 0808 01:50:17.526976 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:17.799849 (+272873us) service_pool.cc:224] Handling call
0808 01:50:19.762271 (+1962422us) spinlock_profiling.cc:243] Waited 482 ms on lock 0x7b54001b3fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:19.762489 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1108158848}
I20260808 01:50:19.794153 10368 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:19.795411  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:19.822060  9474 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:19.797169 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:19.823730 10365 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 6: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:19.798271  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2043) took 2975 ms. Trace:
I20260808 01:50:19.824451  9556 rpcz_store.cc:276] 0808 01:50:16.822840 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:17.194300 (+371460us) service_pool.cc:224] Handling call
0808 01:50:17.194641 (+   341us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:18.293701 (+1099060us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.797906 (+1504205us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b540015b154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:19.798195 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2524730112}
I20260808 01:50:19.800036 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:19.816020  9638 raft_consensus.cc:606] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:19.826268  9638 raft_consensus.cc:614] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:50:19.832038  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:19.832991  9472 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:19.842597  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:19.843593  9474 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 5: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:19.844229  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:19.844156  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:19.845400  9475 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:19.846168  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:19.848153  9476 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:19.847103  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:19.779529 12005 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:19.857602  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:19.831398 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:19.861155  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:19.862340  9473 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260808 01:50:19.866211  9473 pending_rounds.cc:85] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 2
I20260808 01:50:19.866545  9473 pending_rounds.cc:107] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260808 01:50:19.866768  9477 raft_consensus.cc:2410] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:19.866871  9473 raft_consensus.cc:2889] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:19.862533  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:19.867998  9481 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
W20260808 01:50:19.868561  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 790777us
I20260808 01:50:19.864061  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:19.841105 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:19.883705  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:19.889173  9478 raft_consensus.cc:2393] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:19.890580  9474 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260808 01:50:19.900583  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:19.865186 11864 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:19.914398  9474 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260808 01:50:19.916585  9476 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
W20260808 01:50:19.927752 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 356791us
I20260808 01:50:19.903257 11942 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:19.924705  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:19.932152  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 372471us
I20260808 01:50:19.934798  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:19.935240 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:19.939909  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:19.940152 10369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 481) took 1439 ms. Trace:
I20260808 01:50:19.940757 10369 rpcz_store.cc:276] 0808 01:50:18.500802 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.510467 (+  9665us) service_pool.cc:224] Handling call
0808 01:50:18.528792 (+ 18325us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:18.538077 (+  9285us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.939888 (+1401811us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b5400785ad0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:19.940070 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26808576}
I20260808 01:50:19.943815  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:19.913705 11864 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:19.933696 12028 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election won for term 5
W20260808 01:50:19.947000 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 335746us
I20260808 01:50:19.950908  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:19.946846  9631 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.770s	user 0.015s	sys 0.008s
W20260808 01:50:19.957155  9631 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.781s	user 0.015s	sys 0.008s
I20260808 01:50:19.962885 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:19.965193  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:19.967986  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:19.969099  9472 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:19.969332  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:19.973646  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:19.982944  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 235) took 2467 ms. Trace:
I20260808 01:50:19.983896  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:19.946471 11939 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:19.983738  9631 rpcz_store.cc:276] 0808 01:50:17.515168 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.942308 (+1427140us) service_pool.cc:224] Handling call
0808 01:50:18.978365 (+ 36057us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:18.978609 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:50:19.176269 (+197660us) log.cc:844] Serialized 14 byte log entry
0808 01:50:19.982379 (+806110us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:19.982462 (+    83us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:19.982534 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:19.982637 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:19.982854 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23442,"prepare.queue_time_us":771640,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":54,"thread_start_us":939943,"threads_started":2}
I20260808 01:50:19.937966 11795 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260808 01:50:19.994416 12012 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260808 01:50:19.998306  9473 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:19.938459 11942 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:19.998263  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:20.007819  9481 raft_consensus.cc:2410] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260808 01:50:20.004477  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:20.009439  9477 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:20.004892 12032 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:19.975586 12028 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:19.945832 11757 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260808 01:50:20.006151  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:20.036983  9630 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:19.976320 11998 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:20.021076 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 64542us
I20260808 01:50:19.950397 11936 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
W20260808 01:50:20.058725  9630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1817) took 2617 ms (client timeout 3285 ms). Trace:
W20260808 01:50:20.062925  9630 rpcz_store.cc:269] 0808 01:50:17.420015 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:17.599067 (+179052us) service_pool.cc:224] Handling call
0808 01:50:20.005917 (+2406850us) spinlock_profiling.cc:243] Waited 2.4 ms on lock 0x7b5800231698. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:50:20.037537 (+ 31620us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5525888}
W20260808 01:50:20.065960  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:20.067595 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 104125us
I20260808 01:50:20.040689 11643 ts_tablet_manager.cc:1434] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 1.177s	user 0.043s	sys 0.049s
W20260808 01:50:20.060271  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 76483.1us
I20260808 01:50:20.067109 11842 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:20.073910 10332 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:20.078154  9477 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:20.051601 12029 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:20.079772  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:20.081439  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:20.081907  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:20.086318  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 117100us
I20260808 01:50:20.086966  9475 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:50:20.087733  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:20.051805 12034 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:20.092280  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:20.092681  9473 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:20.093006  9475 raft_consensus.cc:2376] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6. Re-sending same reply.
W20260808 01:50:20.070685 11643 ts_tablet_manager.cc:1467] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a: Tablet startup took 1207ms
I20260808 01:50:20.072288 11861 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260808 01:50:20.051652 11795 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260808 01:50:20.099700  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:20.100546  9552 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 5: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:20.088186 11842 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:20.102978  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:20.062597 11998 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:20.091018 12033 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:20.070474 11995 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:20.091336 12034 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:20.113070 12028 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:20.092505 12029 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:20.112495  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:20.070365 12036 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260808 01:50:20.096575 12032 consensus_queue.cc:1059] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260808 01:50:20.130417 12032 raft_consensus.cc:3055] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:20.131040 12032 raft_consensus.cc:740] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:20.132223 12032 consensus_queue.cc:260] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:20.135257  9637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1669) took 2607 ms (client timeout 1786 ms). Trace:
W20260808 01:50:20.135766  9637 rpcz_store.cc:269] 0808 01:50:17.527502 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.240480 (+712978us) service_pool.cc:224] Handling call
0808 01:50:20.134983 (+1894503us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x7b5400421dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:20.135160 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":960934656}
I20260808 01:50:20.138114 12032 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:20.100078 12051 consensus_queue.cc:1059] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
W20260808 01:50:20.111009  9444 leader_election.cc:336] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.575s (SENT)
I20260808 01:50:20.097653 11757 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:20.153837  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:20.174535 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 122056us
W20260808 01:50:20.182240  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 71193us
W20260808 01:50:20.184744  9524 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.661s (SENT) [suppressed 57 similar messages]
W20260808 01:50:20.142663 11962 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.509s	user 0.007s	sys 0.004s
I20260808 01:50:20.197320 10334 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:20.205955  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 121435us
I20260808 01:50:20.219899  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:20.134441 11834 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:20.123486 12034 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:20.144747 11842 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:20.160692 11757 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:20.197726 11939 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:20.240074  9481 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:20.195588 12033 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:50:20.243772  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:20.203187 11864 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:20.247663  9552 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
W20260808 01:50:20.250131  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 65279us
I20260808 01:50:20.229482 12035 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:20.252411  9444 leader_election.cc:336] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.647s (SENT)
I20260808 01:50:20.253248  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:20.257226 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 183378us
I20260808 01:50:20.238538 11834 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:20.248476 11842 raft_consensus.cc:3055] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:20.241714 11757 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:50:20.264858 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 67635.6us
I20260808 01:50:20.262717  9599 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:20.268445  9444 leader_election.cc:336] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.734s (SENT)
I20260808 01:50:20.240978 12049 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260808 01:50:20.243294 11939 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:20.232081 11962 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:20.283106  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 1755) took 1526 ms. Trace:
I20260808 01:50:20.280686  9556 raft_consensus.cc:1217] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Deduplicated request from leader. Original: 1.1->[3.2-3.2]   Dedup: 3.2->[]
I20260808 01:50:20.277451  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:20.288677 12005 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:20.285262  9557 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260808 01:50:20.326718  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:20.276798  9550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 2045) took 3453 ms (client timeout 1885 ms). Trace:
I20260808 01:50:20.329227  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:20.338009  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:20.339577  9555 raft_consensus.cc:2393] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:20.338918  9550 rpcz_store.cc:269] 0808 01:50:16.823398 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:17.211364 (+387966us) service_pool.cc:224] Handling call
0808 01:50:20.276478 (+3065114us) spinlock_profiling.cc:243] Waited 996 ms on lock 0x7b540008acd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:20.276695 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2290019072}
I20260808 01:50:20.339876  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:20.340603  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:20.341168  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:20.341464  9555 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 8: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:20.264276 11842 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:20.330937  9554 rpcz_store.cc:276] 0808 01:50:18.756412 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.983819 (+227407us) service_pool.cc:224] Handling call
0808 01:50:20.282880 (+1299061us) spinlock_profiling.cc:243] Waited 44.6 ms on lock 0x7b54001ac554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:20.283024 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":102602112}
I20260808 01:50:20.240834 11861 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.025s
W20260808 01:50:20.343400 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 74804.8us
W20260808 01:50:20.346083  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 77670.7us
W20260808 01:50:20.347748  9444 leader_election.cc:336] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.752s (SENT)
I20260808 01:50:20.278890 12029 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:20.283435 12047 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.137s	user 0.000s	sys 0.000s
I20260808 01:50:20.243741 12045 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.027s
I20260808 01:50:20.331202  9556 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 01:50:20.284351  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:20.342002  9550 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:20.285945 11998 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:20.398859 11803 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:20.399461 11803 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:20.399622 11756 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:20.399940 11803 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:20.400409 11756 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:20.403146 12061 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260808 01:50:20.404464  9633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1839) took 2074 ms (client timeout 2181 ms). Trace:
W20260808 01:50:20.404969  9633 rpcz_store.cc:269] 0808 01:50:18.330515 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.853443 (+522928us) service_pool.cc:224] Handling call
0808 01:50:20.404218 (+1550775us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540032d054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:20.404391 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2554163968}
I20260808 01:50:20.406965  9629 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:20.407126 11867 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:20.404520 11842 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:20.409843 11842 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:20.378808 12000 raft_consensus.cc:3055] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:20.412950 12000 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:20.414086 11691 raft_consensus.cc:3055] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:20.414700 11691 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:20.415771 11691 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:20.417356 11691 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:50:20.417920 11756 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:20.379902  9554 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:50:20.264429 12028 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:20.267937 12058 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:50:20.295097 12041 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:20.390767 11995 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:20.391178 12036 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:20.285079 11795 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.094s
I20260808 01:50:20.394182  9552 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:20.395110 12060 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:50:20.395579 12049 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:20.258874 12035 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:20.408846 11939 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:20.342911  9555 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260808 01:50:20.483475  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2049) took 3644 ms. Trace:
W20260808 01:50:20.389400  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 126824us
I20260808 01:50:20.424867  9559 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260808 01:50:20.428709  9554 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260808 01:50:20.485185  9599 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:20.357103  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 139737us
I20260808 01:50:20.413929 12000 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:20.330399 11942 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:20.502444  9599 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:20.344267 11864 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:20.342897 11941 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:20.514441 12035 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:20.479571  9552 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:50:20.522291 11980 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:20.430085 12051 consensus_queue.cc:1059] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20260808 01:50:20.402602 11962 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:20.524274  9476 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:20.525332 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:20.483939  9560 rpcz_store.cc:276] 0808 01:50:16.839635 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.216795 (+1377160us) service_pool.cc:224] Handling call
0808 01:50:18.238370 (+ 21575us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:18.456431 (+218061us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:20.483252 (+2026821us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b5400257e54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:20.483421 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":499742976}
I20260808 01:50:20.479519 12058 raft_consensus.cc:697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:50:20.507272  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:20.526513 10367 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 5: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:20.527951 12058 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:20.531328  9554 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:20.531760  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 1979) took 1962 ms. Trace:
I20260808 01:50:20.532358  9479 rpcz_store.cc:276] 0808 01:50:18.569144 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.569280 (+   136us) service_pool.cc:224] Handling call
0808 01:50:18.569694 (+   414us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:18.894383 (+324689us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:20.531358 (+1636975us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x7b54001013d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:20.531665 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":744923520}
I20260808 01:50:20.533197 11795 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:20.481150 12036 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:20.533896 11795 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:20.511127 12000 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:20.536051 12036 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:20.537161  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1671) took 2775 ms. Trace:
I20260808 01:50:20.482626 12049 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:20.537464  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:20.538437  9474 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:20.537707  9629 rpcz_store.cc:276] 0808 01:50:17.761365 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:19.591003 (+1829638us) service_pool.cc:224] Handling call
0808 01:50:20.406737 (+815734us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:20.407600 (+   863us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:20.536913 (+129313us) spinlock_profiling.cc:243] Waited 771 ms on lock 0x7b540032d050. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:20.537079 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1772222208}
W20260808 01:50:20.513631 12024 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.771s	user 0.001s	sys 0.001s
I20260808 01:50:20.471130 12041 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:20.503734 11864 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:20.542253  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2046) took 3717 ms. Trace:
I20260808 01:50:20.544929  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:20.549067  9558 rpcz_store.cc:276] 0808 01:50:16.825356 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:17.759264 (+933908us) service_pool.cc:224] Handling call
0808 01:50:17.800769 (+ 41505us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:18.307633 (+506864us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:20.541856 (+2234223us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x7b5400325fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:20.542162 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1163457792}
I20260808 01:50:20.550405  9476 raft_consensus.cc:2410] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:20.557929  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:20.558996  9476 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:20.418670 12034 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:20.537501 11795 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:20.575542 12041 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:20.577314 12041 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:20.577728 12049 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:20.580279 12034 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:20.552049 11864 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:50:20.097288 11643 ts_tablet_manager.cc:1469] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a: Trace:
0808 01:50:05.914622 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:06.196239 (+281617us) service_pool.cc:224] Handling call
0808 01:50:06.820937 (+624698us) ts_tablet_manager.cc:934] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
0808 01:50:09.956518 (+3135581us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:50:10.506106 (+549588us) tablet_metadata.cc:665] Metadata flushed
0808 01:50:10.506240 (+   134us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:50:10.689018 (+182778us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:50:10.689613 (+   595us) tablet_replica.cc:206] Creating consensus instance
0808 01:50:10.690167 (+   554us) ts_tablet_manager.cc:989] Replica registered.
0808 01:50:10.690273 (+   106us) inbound_call.cc:177] Queueing success response
0808 01:50:15.495668 (+4805395us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0808 01:50:18.863347 (+3367679us) ts_tablet_manager.cc:1435] Starting tablet replica
0808 01:50:18.873179 (+  9832us) tablet_replica.cc:261] Starting instrumentation
0808 01:50:18.893164 (+ 19985us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":156843,"raft.queue_time_us":88163,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":8629,"tablet-copy.queue_time_us":286485,"tablet-download-pool.queue_time_us":987,"tablet-download-pool.run_cpu_time_us":5560,"tablet-download-pool.run_wall_time_us":3291743,"thread_start_us":602666,"threads_started":3}
I20260808 01:50:20.588166 11955 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:20.589377 11757 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:20.593171  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:20.593964  9476 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:20.594807  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 242453us
W20260808 01:50:20.597358  9444 leader_election.cc:336] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.788s (SENT)
I20260808 01:50:20.597779  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:20.599786  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 112376us
I20260808 01:50:20.601096  9601 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:20.603399  9444 leader_election.cc:336] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.861s (SENT)
W20260808 01:50:20.604749  9444 leader_election.cc:336] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.881s (SENT)
I20260808 01:50:20.605170  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:20.606918  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 104590us
I20260808 01:50:20.622200  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:20.624936  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:20.837929  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:20.677305 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:20.725401  9474 raft_consensus.cc:2410] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:50:20.840829 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:20.826921  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34802 (request call id 25) took 2256 ms. Trace:
I20260808 01:50:20.839051  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:20.827438 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 200257us
W20260808 01:50:20.828534  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:20.842253  9479 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
W20260808 01:50:20.828464  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 194278us
W20260808 01:50:20.829174  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 220885us
I20260808 01:50:20.830835 10332 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:20.831506  9632 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:50:20.832726 12034 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:20.833025 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:20.833174 12000 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:20.829082  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:20.829778 10366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 399) took 3931 ms. Trace:
W20260808 01:50:20.833961 10368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 396) took 4012 ms (client timeout 1580 ms). Trace:
W20260808 01:50:20.834060  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 229315us
W20260808 01:50:20.834769  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:20.853685  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:20.835417 10369 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260808 01:50:20.854245  9444 leader_election.cc:336] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.941s (SENT)
I20260808 01:50:20.836284 10363 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:20.836508 12038 raft_consensus.cc:3055] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:50:20.836723  9476 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:20.836956 11795 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:20.864807  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:20.841658  9638 rpcz_store.cc:276] 0808 01:50:18.584202 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.915938 (+331736us) service_pool.cc:224] Handling call
0808 01:50:20.617705 (+1701767us) spinlock_profiling.cc:243] Waited 825 ms on lock 0x7b54003f9b54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4498dc33 00007f5d4998d41a 00007f5d42ee1934 00007f5d42ee18c2 00007f5d42ee17f1 00007f5d42ee176c 00007f5d42ee0a22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:20.826771 (+209066us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1898212608}
I20260808 01:50:20.844379 10365 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEARNER]: Advancing to term 4
I20260808 01:50:20.842506 10334 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:20.833501 12006 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:20.844828 11867 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:20.864267  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:20.851500  9474 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:50:20.878124  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:20.607007 12061 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:50:20.889407  9560 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 5: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:20.855291 10366 rpcz_store.cc:276] 0808 01:50:16.897977 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:17.212202 (+314225us) service_pool.cc:224] Handling call
0808 01:50:17.212570 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:18.474333 (+1261763us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:20.615773 (+2141440us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54005550d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:20.829711 (+213938us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2897246080}
I20260808 01:50:20.856429  9550 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260808 01:50:20.853508 10368 rpcz_store.cc:269] 0808 01:50:16.821955 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:16.822138 (+   183us) service_pool.cc:224] Handling call
0808 01:50:20.833765 (+4011627us) spinlock_profiling.cc:243] Waited 2.93 s on lock 0x7b5400553054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:20.833883 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6743065344}
W20260808 01:50:20.843251 12005 consensus_meta.cc:287] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.502s	user 0.014s	sys 0.004s
I20260808 01:50:20.859807  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:20.861182 11980 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:20.900207 11980 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:20.901527 11980 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:20.834467 12035 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:20.865556  9554 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:20.859593 11961 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:20.837864 11842 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:20.833849 11941 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:20.916666 12071 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:20.916673  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:20.917186 12067 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:20.917296 12071 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:20.917779 12067 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:20.854964  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:20.918370  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:20.919221  9554 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:20.919273  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 65061.2us
I20260808 01:50:20.919235 11941 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:20.919997 12067 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:20.920235  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:20.921418  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:20.922057  9554 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:20.922324 12071 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:20.925174 12006 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:20.925464  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:20.893332  9475 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:50:20.861011  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:20.885416  9571 tablet_copy_service.cc:342] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-3759209338e14437aff5f92a2fff36da received from {username='slave'} at 127.0.0.1:60552
I20260808 01:50:20.895542 12036 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:20.927033  9571 tablet_copy_service.cc:434] P 52ccf4dcca1843f7bb29e9172e2d4ab4: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-3759209338e14437aff5f92a2fff36da on tablet 3759209338e14437aff5f92a2fff36da with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:20.927211 12036 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:20.895808  9478 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260808 01:50:20.927983  9478 pending_rounds.cc:85] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 5
I20260808 01:50:20.928360  9478 pending_rounds.cc:107] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260808 01:50:20.928779  9478 raft_consensus.cc:2889] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:20.928395 12036 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:20.929466  9554 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 01:50:20.896314 12072 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:20.898649 12028 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:20.930757  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1751) took 2365 ms. Trace:
I20260808 01:50:20.931569  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:20.927764 11864 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:20.935161  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:20.878077  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:20.894623 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52212.2us
W20260808 01:50:20.906128  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52913.2us
I20260808 01:50:20.862030 12038 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:20.904742  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 2c617cb141a34007b6c1b8af0d911923 (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:50:20.931190  9557 rpcz_store.cc:276] 0808 01:50:18.565186 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:19.682391 (+1117205us) service_pool.cc:224] Handling call
0808 01:50:20.285006 (+602615us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:20.337436 (+ 52430us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:20.930507 (+593071us) spinlock_profiling.cc:243] Waited 602 ms on lock 0x7b5400217ad0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:20.930665 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1384893696}
I20260808 01:50:20.908015 11756 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:20.909015 12052 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:20.865208 11939 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:20.859103 12034 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:20.910241  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:20.944845  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:20.911273  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:20.911451  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:20.952242  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:20.911880 12051 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:20.912113  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:20.913029  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:20.959159  9629 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:20.959336 10334 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:20.913380 12044 log.cc:927] Time spent T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.909s	user 0.000s	sys 0.003s
I20260808 01:50:20.914062  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:20.886730 11795 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:20.885471 11867 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:20.915033  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:20.961570  9631 raft_consensus.cc:2410] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:20.915599  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:20.917665  9635 raft_consensus.cc:2376] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6. Re-sending same reply.
I20260808 01:50:20.834496 12068 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:20.925897 12006 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:20.930352 12028 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:20.971788  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:20.930301 12072 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:20.899487 11757 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:20.865607 11942 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:20.977169  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1747) took 3717 ms. Trace:
W20260808 01:50:20.936231  9444 leader_election.cc:336] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.843s (SENT)
I20260808 01:50:20.892270  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:20.939433  9601 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:20.940547  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:20.977833  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:20.939045  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:20.862147 12000 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:20.979969  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 43754.6us
I20260808 01:50:20.981119  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:20.979931  9555 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:20.981976  9632 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:20.963295 10334 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:20.965672  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:20.984057  9629 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:50:20.984790  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 45480.9us
W20260808 01:50:20.981120  9444 leader_election.cc:336] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.886s (SENT)
I20260808 01:50:20.987316  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:20.987510  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:20.939795 12049 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:20.970736  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:20.976759 12028 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:20.977636  9551 rpcz_store.cc:276] 0808 01:50:17.259729 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:19.649234 (+2389505us) service_pool.cc:224] Handling call
0808 01:50:19.667721 (+ 18487us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:19.697405 (+ 29684us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:19.697439 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:19.713064 (+ 15625us) log.cc:844] Serialized 26 byte log entry
0808 01:50:20.015074 (+302010us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:20.015133 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:20.015240 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:20.912757 (+897517us) spinlock_profiling.cc:243] Waited 3.17 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:20.976856 (+ 64099us) raft_consensus.cc:1685] finished
0808 01:50:20.976877 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:20.977097 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":170,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":12362,"spinlock_wait_cycles":7294208,"thread_start_us":257272,"threads_started":1,"wal-append.queue_time_us":248008}
I20260808 01:50:20.969190  9600 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:20.970506  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:20.989010  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:20.984994  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:20.989907  9477 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:50:20.986182  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:20.997903  9444 leader_election.cc:336] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.886s (SENT)
I20260808 01:50:20.991120  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:20.991781  9552 raft_consensus.cc:2410] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:50:20.992206  9556 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:50:20.994204  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:20.998919  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:20.998245  9637 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:20.964222 12046 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260808 01:50:21.000610  9473 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:21.001238  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:20.970920 12060 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:21.001734  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:21.002027 11941 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:21.002154 12060 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:21.002393  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.002745 12060 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:20.972085 11939 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:20.999099  9634 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
W20260808 01:50:21.004160  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 157079us
I20260808 01:50:21.004426  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.005072 11939 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:21.005441  9634 raft_consensus.cc:2410] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:21.006042  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:21.006485  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.007364  9634 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:20.964744 12068 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:21.007721 12068 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:21.008302 12034 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:21.008445  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:21.009122  9478 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:21.010042  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:21.010782 11962 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:21.011449 11962 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:21.011907 11962 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:20.964562 12052 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:21.013127  9635 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:50:20.978000 11945 consensus_queue.cc:1059] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260808 01:50:21.015369  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:20.972820 12051 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:21.017506 11795 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:21.018234  9636 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:20.963050 11961 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:20.977711 11568 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.076s
I20260808 01:50:21.019563  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:21.053746  9558 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260808 01:50:21.069816  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.030161  9638 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:21.082594  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:21.091882  9633 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 4: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:50:21.090879  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 93035.9us
I20260808 01:50:21.081527 10365 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEARNER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260808 01:50:21.080297  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:21.100935  9477 raft_consensus.cc:2410] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:21.082001  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:21.093549  9444 leader_election.cc:336] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.100s (SENT)
I20260808 01:50:21.054090  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.008461  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.115913  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:21.117229  9633 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:21.047654  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:21.092705  9635 raft_consensus.cc:2410] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:50:21.118585  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.008316  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.176406  9473 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 5: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:50:21.154847  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 148853us
W20260808 01:50:21.143127  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 174027us
W20260808 01:50:21.153345  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59816.2us
I20260808 01:50:21.177527  9633 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:21.010637  9478 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:21.182125  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:21.186465  9633 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:21.189353  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.014405 12052 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:21.053851 12042 raft_consensus.cc:3055] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Stepping down as leader of term 4
W20260808 01:50:21.196403  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 54498.7us
I20260808 01:50:21.198738  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:21.199852  9477 raft_consensus.cc:2393] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:21.201658  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:21.202847  9635 raft_consensus.cc:2410] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260808 01:50:20.988682 12070 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:21.191345  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:21.214138 12052 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:50:21.214569  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:21.215202  9524 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:21.216715 11998 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:21.217499 11998 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:50:21.217808 12046 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:50:21.219126 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:21.219415 12046 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
W20260808 01:50:21.223444  9636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1837) took 2893 ms (client timeout 2139 ms). Trace:
W20260808 01:50:21.223949  9636 rpcz_store.cc:269] 0808 01:50:18.329891 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:18.777171 (+447280us) service_pool.cc:224] Handling call
0808 01:50:21.223270 (+2446099us) spinlock_profiling.cc:243] Waited 2.24 s on lock 0x7b54003a0054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:21.223367 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5150790528}
I20260808 01:50:21.225274  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:21.225925  9521 leader_election.cc:336] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.548s (SENT)
I20260808 01:50:21.226537  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:21.227164  9521 leader_election.cc:336] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.630s (SENT)
I20260808 01:50:21.227412  9636 raft_consensus.cc:2410] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260808 01:50:21.227607  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:21.228284  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:21.178267  9481 raft_consensus.cc:2376] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5. Re-sending same reply.
I20260808 01:50:21.193670  9633 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20260808 01:50:21.230126  9633 pending_rounds.cc:85] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 1
I20260808 01:50:21.230437  9633 pending_rounds.cc:107] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260808 01:50:21.168677  9632 raft_consensus.cc:2393] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:21.230729  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:21.231602  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:21.232343  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:21.232851  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.233230  9481 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:21.030117 11565 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:21.235368  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:21.235080 11565 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:20.995460 11864 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:21.239578  9632 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:21.204922 12051 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:21.095499 11756 raft_consensus.cc:3055] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:21.047248 11757 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:21.046952 12069 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:21.177023 12063 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:50:21.268713  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:21.070886 12006 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:21.066282 12038 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:21.206199 12075 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.360s	user 0.000s	sys 0.001s
I20260808 01:50:21.030270 12049 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:21.205261  9631 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260808 01:50:21.277387  9631 pending_rounds.cc:85] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 2
I20260808 01:50:21.277711  9631 pending_rounds.cc:107] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260808 01:50:21.277958  9631 raft_consensus.cc:2889] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:21.208581  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:21.208388 12067 consensus_queue.cc:1059] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 01:50:21.228709  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 48707.1us
I20260808 01:50:21.196422 12000 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:21.230775  9633 raft_consensus.cc:2889] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:21.234477  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:21.287958  9481 raft_consensus.cc:2410] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:21.241617  9636 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:21.245707  9524 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:21.180888  9444 leader_election.cc:336] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.101s (SENT)
I20260808 01:50:21.252389 10369 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:50:21.254776 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:21.047276 12028 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 01:50:21.250133  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45886.3us
I20260808 01:50:20.988933 11942 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:21.268271  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:21.180094 12035 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:21.178869 11939 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:21.315414  9476 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 01:50:21.275902  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:21.276733 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:21.317013 10366 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:21.318089  9478 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 5: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:21.280732  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.229426 11998 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:21.284055  9477 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:38182
I20260808 01:50:21.320737  9477 raft_consensus.cc:606] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:21.321437  9477 raft_consensus.cc:614] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:50:21.285609  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.241060 12042 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:21.286396  9634 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:21.290489  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:21.330793  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.331118  9481 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:21.271003 12052 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260808 01:50:21.265619 11941 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:21.262866 11756 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:21.275532 11803 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.875s	user 0.007s	sys 0.016s
W20260808 01:50:21.298310  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52674.9us
I20260808 01:50:21.179641 12041 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.099s
I20260808 01:50:21.267138 11757 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:21.290066 11802 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.023s
I20260808 01:50:21.306509 12006 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20260808 01:50:21.314415 10361 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:50:21.318750  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.323788 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:21.324972  9635 raft_consensus.cc:2410] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:21.315186 11939 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:21.292091  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:21.294903 11539 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260808 01:50:21.345872 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:21.396613 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 50855.3us
I20260808 01:50:21.311412 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:21.307068 12000 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 01:50:21.351241  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:21.320842 12069 raft_consensus.cc:3055] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:21.364389  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2079) took 1165 ms. Trace:
I20260808 01:50:21.407925  9559 rpcz_store.cc:276] 0808 01:50:20.198628 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:20.485405 (+286777us) service_pool.cc:224] Handling call
0808 01:50:20.485751 (+   346us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:20.486194 (+   443us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:20.486223 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:20.499971 (+ 13748us) log.cc:844] Serialized 26 byte log entry
0808 01:50:20.545226 (+ 45255us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:20.545291 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:20.545392 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:21.364068 (+818676us) raft_consensus.cc:1685] finished
0808 01:50:21.364103 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:21.364301 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":139,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":27786,"thread_start_us":40620,"threads_started":1,"wal-append.queue_time_us":115409}
I20260808 01:50:21.413440  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:21.414398  9477 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:21.367551  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:21.373653  9631 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 3: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:21.349357 12028 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:21.338193  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:21.381664  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:21.329291 11691 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:50:21.393870  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 213000us
I20260808 01:50:21.394039  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.397746 10332 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:21.440224 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:21.399147  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 437504us
I20260808 01:50:21.351043 11757 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:21.304638 12051 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260808 01:50:21.418256  9473 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:21.305281 11961 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:21.366024  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:21.448005  9478 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:21.361099 12070 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:21.351171 12035 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:21.424043  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:21.401536 12066 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:21.428826  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:21.402122 12005 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:21.438321  9635 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:21.451992  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:21.453506  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:21.454599  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:21.465747  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:21.461946  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:21.466595  9635 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:21.467547  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:21.468775  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.469897  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:21.455650  9481 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:50:21.473814  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 73451us
W20260808 01:50:21.477909 12030 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.005s	sys 0.014s
I20260808 01:50:21.350569 11803 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:21.411366 12000 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:21.477254 12035 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:21.440263  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.499470  9633 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:21.500976  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.467572  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.464610  9476 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:21.421877 10360 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:21.483956  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:21.405095 11565 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:21.407693 12069 raft_consensus.cc:740] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:21.484428  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:21.435279 12067 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
I20260808 01:50:21.488309 10364 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260808 01:50:21.395479 12052 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
W20260808 01:50:21.514189 12058 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.014s	sys 0.007s
I20260808 01:50:21.552014  9637 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 8: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:21.343799 11941 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:21.433488 11867 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:21.493319  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:21.503448  9631 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:50:21.553870  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.446790 11802 consensus_queue.cc:1059] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.083s
I20260808 01:50:21.555225  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.504335  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.557207  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.558084  9635 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:50:21.505674  9476 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 5. Current term is 6. Ops: [4.3-5.4]
I20260808 01:50:21.560310  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:21.561324  9476 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
W20260808 01:50:21.561734  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 219768us
I20260808 01:50:21.502595  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:21.405967 12061 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20260808 01:50:21.414186 11568 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.076s
I20260808 01:50:21.453516 11995 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:21.571501  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:21.573297  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:21.574887  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 215220us
I20260808 01:50:21.575569  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:21.575987  9521 leader_election.cc:336] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.780s (SENT)
I20260808 01:50:21.466462 12028 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:21.434634 12042 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:21.422612 11942 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:21.415750 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:21.540063  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:21.484539 12063 raft_consensus.cc:3055] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Stepping down as leader of term 5
W20260808 01:50:21.488322 12030 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.512s	user 0.005s	sys 0.014s
I20260808 01:50:21.554718 12006 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:21.493862 12000 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:21.566021  9634 raft_consensus.cc:2376] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4. Re-sending same reply.
I20260808 01:50:21.522579  9481 raft_consensus.cc:2410] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:21.484586 11972 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.068s
I20260808 01:50:21.449512 11691 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:21.529562 11980 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:21.577242  9521 leader_election.cc:336] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.786s (SENT)
I20260808 01:50:21.565351 11945 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:21.609160  9521 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:21.455173 12070 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:21.554634 12058 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.014s	sys 0.007s
I20260808 01:50:21.546520 12005 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:21.612624 12070 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:21.591450  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:21.613729 12070 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:21.613991  9473 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
W20260808 01:50:21.596765 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 156605us
I20260808 01:50:21.599212  9480 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:50:21.616163  9480 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:21.619122  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:21.620075  9481 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:21.564666  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.601577  9524 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:21.555255 12036 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:21.622588  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:21.599972  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 140705us
I20260808 01:50:21.552752 12012 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.051s
I20260808 01:50:21.626572  9476 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:50:21.584792 11987 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:21.488770 11756 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:21.633255 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:21.636816  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:21.638285  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:21.638427  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:21.593197 12063 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:21.639330 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:21.642375  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:21.642752  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 168135us
I20260808 01:50:21.605397 12006 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:21.656445 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:21.660243 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:21.661839 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:21.615923 12049 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:21.578320 12083 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.070s
I20260808 01:50:21.664448  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:21.633891 12042 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:21.592753 11867 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260808 01:50:21.635317 12069 consensus_queue.cc:260] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:21.679234 12069 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:21.679334 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:21.667816  9480 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260808 01:50:21.602972 12051 consensus_queue.cc:1059] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260808 01:50:21.665513 12006 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:21.680177 10363 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:21.681421 12099 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:21.681672 11756 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:21.681933 12099 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:21.639410 12082 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260808 01:50:21.682794 12012 consensus_queue.cc:1059] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:50:21.683377 11803 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:21.639807 12066 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:21.682411 12099 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:50:21.707931  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 43576.2us
I20260808 01:50:21.622453 11945 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:21.713389  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:21.714205  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 91209.7us
I20260808 01:50:21.623705 12030 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.045s
I20260808 01:50:21.629980 11980 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:21.635795 12041 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
I20260808 01:50:21.630292 12058 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260808 01:50:21.685657 11565 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 01:50:21.737030  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 127918us
W20260808 01:50:21.719485  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:21.704649 12084 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
I20260808 01:50:21.755681  9636 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:21.777902  9636 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:21.778215 10366 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:50:21.637036 11987 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:21.629702 12092 consensus_queue.cc:1059] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.079s
I20260808 01:50:21.782763 11987 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:21.761641  9480 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:21.681423 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:21.672467 11649 raft_consensus.cc:3055] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:21.785027 11649 raft_consensus.cc:740] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:21.785461  9480 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:21.786062 11649 consensus_queue.cc:260] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:21.710372 12049 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:21.774326  9521 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:21.787508 12049 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:21.787173 11649 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:21.884706  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:21.874737 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:21.836057  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 360b0cf59e7749058ca972bb487cbd10 (from 6e7856bed1be43acaa821290d506ee97 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
W20260808 01:50:21.816589 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 154851us
I20260808 01:50:21.710664 12036 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260808 01:50:21.852825  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 139545us
I20260808 01:50:21.884689 10368 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
W20260808 01:50:21.823885 11842 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.886s	user 0.012s	sys 0.000s
I20260808 01:50:21.722651 11939 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:21.715215 12061 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:21.768754 11565 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:21.788677 12034 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:21.904309 10368 pending_rounds.cc:85] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting all ops after (but not including) 4
I20260808 01:50:21.915679 10368 pending_rounds.cc:107] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260808 01:50:21.904593  9601 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:21.897614  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 181220us
W20260808 01:50:21.907877  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:21.867545 11945 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:21.814939 12068 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.807s	user 0.012s	sys 0.002s
I20260808 01:50:21.895493 11864 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:21.916061 10368 raft_consensus.cc:2889] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:21.813102 12100 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:21.927600 10365 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260808 01:50:21.834208 11867 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:21.833921 11980 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:21.883350 12103 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:21.883178 11987 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:21.834153 12105 raft_consensus.cc:3055] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Stepping down as leader of term 4
W20260808 01:50:21.959865  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 55338.2us
W20260808 01:50:21.825661 12097 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
W20260808 01:50:21.823446 12090 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260808 01:50:21.900784 12049 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:21.918191 12092 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20260808 01:50:21.918002 11565 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:21.907655 11568 raft_consensus.cc:3055] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:21.903018 12112 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:21.958753 12030 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260808 01:50:21.929976  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:21.918386 11842 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 01:50:22.033020  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:21.932335 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:21.933590 12107 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260808 01:50:21.933840 12000 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:21.937243 11757 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:21.913973 11941 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:21.915939 12061 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:21.949676 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:21.867278 11972 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.065s
I20260808 01:50:21.954310 12119 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.043s
I20260808 01:50:22.059625  9630 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:21.988037  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1815) took 1080 ms. Trace:
I20260808 01:50:21.961563  9601 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:21.950946 11962 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.938s	user 0.012s	sys 0.005s
W20260808 01:50:21.983188  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.004s	sys 0.006s
I20260808 01:50:22.067160  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:21.948335 11867 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:21.953224 12028 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:22.018950  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 244749us
I20260808 01:50:22.071023  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:22.020689 12080 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260808 01:50:22.072305  9521 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:21.996878 12012 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:21.958827 12105 raft_consensus.cc:740] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:21.958529 12100 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:21.927601 12068 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:21.795205 12063 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:21.989166 12094 raft_consensus.cc:3055] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:22.052101  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:21.987947 11980 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
I20260808 01:50:21.959352 12103 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:22.031412 12085 log.cc:927] Time spent T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.263s	user 0.000s	sys 0.002s
I20260808 01:50:22.056636  9523 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:22.061014  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 99548.1us
I20260808 01:50:22.060635  9475 rpcz_store.cc:276] 0808 01:50:20.936194 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:20.941662 (+  5468us) service_pool.cc:224] Handling call
0808 01:50:20.942079 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:20.942478 (+   399us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:20.942504 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:20.960600 (+ 18096us) log.cc:844] Serialized 26 byte log entry
0808 01:50:21.451485 (+490885us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:21.451526 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:21.451652 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:21.987761 (+536109us) raft_consensus.cc:1685] finished
0808 01:50:21.987783 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:21.987970 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":101,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":16505,"thread_start_us":428728,"threads_started":1,"wal-append.queue_time_us":927901}
I20260808 01:50:21.958719 11803 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.082s
I20260808 01:50:22.002995 10360 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
I20260808 01:50:22.090030  9632 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
W20260808 01:50:22.067302  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.005s	sys 0.007s
I20260808 01:50:22.090560 10366 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260808 01:50:22.091269  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 639366us
I20260808 01:50:22.072701 11962 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:22.071170 11867 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:22.074029  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:22.077018  9634 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:22.049942 12093 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:22.003432 12006 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:22.120471  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0351418460195418s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2804s for a response
I20260808 01:50:22.083401  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:22.086038 10332 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:22.121439  9629 raft_consensus.cc:2368] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 4. Current term is 5.
I20260808 01:50:22.068372 11802 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:21.922490  9524 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:50:22.091290  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:22.095573  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:22.090764  9632 raft_consensus.cc:606] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:22.124312  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:22.124400  9632 raft_consensus.cc:614] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:50:22.021246 11568 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:22.099098  9445 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:22.118382  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 44474.4us
W20260808 01:50:22.131544  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 209148us
I20260808 01:50:22.059134 12061 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:22.127058 10332 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:22.128846  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:22.130698  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47355.4us
I20260808 01:50:22.133327  9521 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:22.133688  9559 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:22.134536  9601 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:22.079360 12012 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:22.155704 10369 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 4: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:22.156064  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:22.156224  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:22.136505 12105 consensus_queue.cc:260] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:22.137892 11941 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:22.140389 12006 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:22.140974  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:22.123778 11802 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:22.124327 12045 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:22.150820 12102 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:22.151422 12103 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:22.086225 12094 raft_consensus.cc:740] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:22.157464  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:22.074224 12046 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20260808 01:50:22.095034 12095 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:22.164811  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:22.166574  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:22.167821  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:22.193511  9559 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:22.168260  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1816) took 1181 ms. Trace:
I20260808 01:50:22.131284 12028 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:22.194720  9474 rpcz_store.cc:276] 0808 01:50:20.987035 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:20.987168 (+   133us) service_pool.cc:224] Handling call
0808 01:50:20.987539 (+   371us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:20.993232 (+  5693us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:20.993261 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:21.019687 (+ 26426us) log.cc:844] Serialized 26 byte log entry
0808 01:50:21.492276 (+472589us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:21.492415 (+   139us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:22.153445 (+661030us) spinlock_profiling.cc:243] Waited 5.44 ms on lock 0x7b540021dc54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:22.153497 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:22.153545 (+    48us) raft_consensus.cc:1685] finished
0808 01:50:22.153560 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:22.168163 (+ 14603us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12515072,"thread_start_us":398694,"threads_started":1,"wal-append.queue_time_us":545216}
I20260808 01:50:22.197227  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:22.170233  9637 raft_consensus.cc:2410] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:22.173642  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:22.177093  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:22.170640  9521 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:22.182515  9524 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:22.184445  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:22.188151  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:22.199649  9632 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:22.200774  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 278) took 1355 ms. Trace:
I20260808 01:50:22.128425 12083 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260808 01:50:22.198756  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:22.199955  9555 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:22.154809 12070 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:22.046578 12112 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:22.240439 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:50:22.241017  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 50fd024764ea49c7a1f16e992e7599f3 (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
W20260808 01:50:22.190469 12060 consensus_meta.cc:287] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.187s	user 0.014s	sys 0.001s
I20260808 01:50:22.172156 11945 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:22.173031 11802 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:22.249595  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 67177.1us
I20260808 01:50:22.244917  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:22.248663  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 126375us
I20260808 01:50:22.154616 12092 consensus_queue.cc:1059] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.095s
I20260808 01:50:22.192646 11941 raft_consensus.cc:697] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 4 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:22.200778  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:22.160288 12113 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:22.201141  9551 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:50:22.259929  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:22.160085 12012 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:22.185678 11842 consensus_queue.cc:1059] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.102s
I20260808 01:50:22.201241  9479 rpcz_store.cc:276] 0808 01:50:20.845984 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:20.846127 (+   143us) service_pool.cc:224] Handling call
0808 01:50:20.861056 (+ 14929us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:20.862423 (+  1367us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:20.862471 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:20.881161 (+ 18690us) log.cc:844] Serialized 26 byte log entry
0808 01:50:21.398756 (+517595us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:21.398820 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:21.398934 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:22.200527 (+801593us) raft_consensus.cc:1685] finished
0808 01:50:22.200557 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:22.200708 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":953,"raft.queue_time_us":2986,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":10760,"thread_start_us":482230,"threads_started":1,"wal-append.queue_time_us":963863}
I20260808 01:50:22.168835 11980 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.096s
W20260808 01:50:22.270498  9444 leader_election.cc:336] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.127s (SENT)
I20260808 01:50:22.293846  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:22.201359  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:22.270840  9524 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:22.295264  9632 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
W20260808 01:50:22.295557  9444 leader_election.cc:336] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.139s (SENT)
I20260808 01:50:22.296191  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:22.296309  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:22.296873  9524 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:22.241500 12105 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:22.172328 11803 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.087s
I20260808 01:50:22.271581  9637 raft_consensus.cc:2410] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:50:22.193171 12095 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:22.205782 12046 consensus_queue.cc:1243] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20260808 01:50:22.300442  9444 leader_election.cc:336] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.188s (SENT)
I20260808 01:50:22.254990 12125 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:22.278779 12113 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:22.201543 12069 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.075s
I20260808 01:50:22.258074 12045 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:22.271409 11945 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:22.262028  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 90653.3us
I20260808 01:50:22.245607 12083 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:22.264117 12119 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:22.261806 12030 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:22.241685 12028 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:22.329561 11945 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:22.336090 12125 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:22.285357  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:22.123699 12093 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:22.301905  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:22.361930 12061 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:22.305702 12095 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:22.329600  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:22.268263 12060 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:22.305222 12092 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:22.332187  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:22.331323  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:22.306121 11939 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:22.304382 12126 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:22.316809 12113 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:22.301836  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:22.334182 12083 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:22.324091 11757 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:22.300506  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:22.331877 12000 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:22.255231 12100 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:22.357687 12072 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:22.358007 12028 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:22.361454  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:22.364315  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 79127.2us
I20260808 01:50:22.364464  9632 raft_consensus.cc:2410] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260808 01:50:22.304549  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:22.388974  9478 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:50:22.389731  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 89357us
W20260808 01:50:22.390506  9444 leader_election.cc:336] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.420s (SENT)
I20260808 01:50:22.390988  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:22.373324  9479 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:22.392767  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:22.349028 12006 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:22.395624  9474 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 8: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:22.396795  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:22.397768  9474 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:22.393563 11803 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:22.394204 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 412) took 1554 ms. Trace:
I20260808 01:50:22.402180 10367 raft_consensus.cc:2393] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 6: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:22.352074 12119 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:22.381654  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:22.382229 12068 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:22.384773  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 52740.7us
I20260808 01:50:22.365428 12069 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:50:22.403537  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:22.404832  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 103069us
W20260808 01:50:22.408242  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:22.389953 12099 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.633s	user 0.012s	sys 0.003s
I20260808 01:50:22.380750 11962 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:22.410259  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:22.415813  9478 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:22.414127  9524 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:22.423102  9521 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:22.396196 12103 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:22.396660 12028 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:22.349963 11941 consensus_queue.cc:237] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:22.394646 12092 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:22.426499 10334 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:22.409798 12126 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:22.395304 12123 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:22.410828 12119 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:22.409489 12000 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:22.432818  9478 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:38182
I20260808 01:50:22.421321 12030 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.015s
I20260808 01:50:22.440910  9478 raft_consensus.cc:606] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:22.487619  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:22.489380  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:22.406201 11803 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:22.473510  9474 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:22.406286 12006 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:22.530259  9601 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:22.531422  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 43930.3us
I20260808 01:50:22.424686 12124 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:50:22.455123 12129 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.195s	user 0.002s	sys 0.000s
W20260808 01:50:22.507746  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:22.531085  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:22.531334 12139 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:22.437204 12120 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:22.493528 11649 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:22.535916  9478 raft_consensus.cc:614] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:50:22.555564  9474 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:22.542722  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:22.488997 12130 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260808 01:50:22.425628 12099 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:22.562897 12099 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:22.564294 12137 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
W20260808 01:50:22.566036 12115 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.201s	user 0.000s	sys 0.001s
W20260808 01:50:22.566895  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:22.548496 11980 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:22.567663 10364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 494) took 1016 ms. Trace:
I20260808 01:50:22.568048 10364 rpcz_store.cc:276] 0808 01:50:21.551109 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:21.564695 (+ 13586us) service_pool.cc:224] Handling call
0808 01:50:21.565104 (+   409us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:21.577632 (+ 12528us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:21.577662 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:21.587873 (+ 10211us) log.cc:844] Serialized 52 byte log entry
0808 01:50:21.892920 (+305047us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:21.892965 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:21.893046 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:22.567400 (+674354us) raft_consensus.cc:1685] finished
0808 01:50:22.567426 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:22.567597 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1458,"raft.queue_time_us":22926,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":18898,"thread_start_us":244964,"threads_started":1,"wal-append.queue_time_us":688815}
I20260808 01:50:22.570115 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:22.570967  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:22.571626  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1718) took 1358 ms. Trace:
I20260808 01:50:22.572113  9636 rpcz_store.cc:276] 0808 01:50:21.212961 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:21.779518 (+566557us) service_pool.cc:224] Handling call
0808 01:50:21.780044 (+   526us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:21.869187 (+ 89143us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:21.869219 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:21.885700 (+ 16481us) log.cc:844] Serialized 26 byte log entry
0808 01:50:22.050428 (+164728us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:22.050469 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:22.050545 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:22.571383 (+520838us) raft_consensus.cc:1685] finished
0808 01:50:22.571407 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:22.571554 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":121803,"raft.queue_time_us":119547,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":7624,"thread_start_us":123370,"threads_started":1,"wal-append.queue_time_us":210704}
I20260808 01:50:22.575268  9636 pending_rounds.cc:85] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 2
I20260808 01:50:22.575572  9636 pending_rounds.cc:107] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260808 01:50:22.528818 12123 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260808 01:50:22.575897  9636 raft_consensus.cc:2889] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:22.576669 12123 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:22.577211 12123 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:22.578601 11649 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:22.583254 11649 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:22.584911 11802 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:22.585650 12095 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:22.587404 12060 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:22.588606 12095 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:22.589697 12012 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:22.593400 12012 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:22.595733  9478 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:50:22.477718 12102 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election won for term 4
W20260808 01:50:22.549908  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 126839us
W20260808 01:50:22.599166 12109 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.250s	user 0.001s	sys 0.001s
I20260808 01:50:22.599416 12102 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:22.600641  9521 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:22.553951  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:22.488974 12028 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:22.602192  9521 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:22.602702 11803 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:22.602273 12028 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:22.603580  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:22.603850 11945 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:22.604427 12093 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:22.604858 12028 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:22.605240 12093 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:22.605559 12028 raft_consensus.cc:2764] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260808 01:50:22.488672 12103 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:22.553443  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:22.606411 12124 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:22.559031 11867 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:22.606730 12132 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260808 01:50:22.606987 12120 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:22.607681 12120 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:22.608383 12120 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:22.608501 11867 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:22.560206 12006 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:22.609123 12120 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:50:22.609576 12038 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:50:22.542269  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 118623us
W20260808 01:50:22.558107 12138 log.cc:927] Time spent T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260808 01:50:22.600667 12102 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:22.611543  9524 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:22.612205 11795 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:22.612670 11795 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:22.613582 11795 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:22.614651 12095 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:22.615402 10365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 492) took 1222 ms. Trace:
I20260808 01:50:22.615793 10365 rpcz_store.cc:276] 0808 01:50:21.393276 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:21.406606 (+ 13330us) service_pool.cc:224] Handling call
0808 01:50:21.873003 (+466397us) spinlock_profiling.cc:243] Waited 9.1 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:50:21.873311 (+   308us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:21.928230 (+ 54919us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:22.615128 (+686898us) spinlock_profiling.cc:243] Waited 54 ms on lock 0x7b5400594cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:22.615325 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":145137408}
I20260808 01:50:22.493604 12069 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:22.620573  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:22.600879  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1832) took 1326 ms. Trace:
I20260808 01:50:22.624243  9477 rpcz_store.cc:276] 0808 01:50:21.274846 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:21.447322 (+172476us) service_pool.cc:224] Handling call
0808 01:50:21.488065 (+ 40743us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:21.536472 (+ 48407us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:21.536541 (+    69us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:21.538097 (+  1556us) log.cc:844] Serialized 26 byte log entry
0808 01:50:21.735483 (+197386us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:21.735561 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:21.735674 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:22.600575 (+864901us) raft_consensus.cc:1685] finished
0808 01:50:22.600606 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:22.600785 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":114236,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":16186,"thread_start_us":218666,"threads_started":2,"wal-append.queue_time_us":751765}
I20260808 01:50:22.627133 10364 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 3: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:50:22.628157  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:22.493767 12068 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:22.406184 10361 rpcz_store.cc:276] 0808 01:50:20.839731 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:20.848141 (+  8410us) service_pool.cc:224] Handling call
0808 01:50:21.206274 (+358133us) spinlock_profiling.cc:243] Waited 4.98 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:50:21.206574 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:21.373486 (+166912us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:22.379935 (+1006449us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54005122d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:22.392553 (+ 12618us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":258271872}
W20260808 01:50:22.654663  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet c99678e59ec845b28293f811d76f24e8 (from 6e7856bed1be43acaa821290d506ee97 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:50:22.645473  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:22.649693  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 107064us
I20260808 01:50:22.658182  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:22.659264 12028 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:22.659965 12028 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:22.660517 12028 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:22.659209 12068 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:22.640451 12069 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:22.669936 12069 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:22.655589  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:50:22.672520 11802 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:22.630784 11945 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:22.673733 11962 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:50:22.675010 10360 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:22.549796 12112 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:22.672777  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:50:22.632508 11980 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:22.677055  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:50:22.677567 11980 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:22.678838 12068 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:22.666860  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:22.680328 12093 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:22.681901  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:22.685576  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:22.439463 12126 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:22.646607 12042 consensus_meta.cc:287] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.925s	user 0.011s	sys 0.003s
I20260808 01:50:22.703233  9631 raft_consensus.cc:2410] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260808 01:50:22.693955 10366 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.021s	sys 0.012s
W20260808 01:50:22.704314 10366 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.021s	sys 0.012s
W20260808 01:50:22.694614 11565 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.655s	user 0.013s	sys 0.000s
I20260808 01:50:22.590794 12123 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:22.617466 12061 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:22.641270 12092 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:22.699441 12045 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:50:22.699919 12049 consensus_meta.cc:287] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.680s	user 0.012s	sys 0.001s
I20260808 01:50:22.655436 11795 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:22.619421 12139 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:22.701050 12006 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:22.734736 12049 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:22.739233 12123 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:22.724699 11757 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:22.717698 11565 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:22.761723 11757 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:22.762284  9630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1853) took 1850 ms (client timeout 1861 ms). Trace:
W20260808 01:50:22.762789  9630 rpcz_store.cc:269] 0808 01:50:20.911390 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:20.913674 (+  2284us) service_pool.cc:224] Handling call
0808 01:50:22.762092 (+1848418us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b540027e554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:22.762222 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2526714240}
I20260808 01:50:22.764163  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:22.764930  9630 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 8: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:22.712929  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2018) took 1144 ms. Trace:
I20260808 01:50:22.766439  9475 rpcz_store.cc:276] 0808 01:50:21.568334 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:22.117356 (+549022us) service_pool.cc:224] Handling call
0808 01:50:22.122872 (+  5516us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:22.123139 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:50:22.289578 (+166439us) log.cc:844] Serialized 14 byte log entry
0808 01:50:22.712438 (+422860us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:22.712526 (+    88us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:22.712594 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:22.712694 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:22.712861 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":137674,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":5847,"raft.queue_time_us":24546,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":5773,"thread_start_us":558377,"threads_started":3,"wal-append.queue_time_us":123524}
I20260808 01:50:22.713526  9479 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:22.688957 12093 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:22.693415 12030 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:22.708987 11867 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:22.606746 12103 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:22.757534 10368 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260808 01:50:22.678133 12028 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:22.761003 11939 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:50:22.761732 12136 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.266s	user 0.000s	sys 0.004s
W20260808 01:50:22.817883  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 191023us
W20260808 01:50:22.818874  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 246879us
W20260808 01:50:22.833168  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:22.844790  9477 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:22.839219  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:22.844050  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:22.847536  9630 raft_consensus.cc:2410] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:22.811659 12148 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:22.838785 11842 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:22.870035  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:22.871938  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:22.873934  9475 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:22.843971 12159 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:22.873581  9630 raft_consensus.cc:2376] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5. Re-sending same reply.
I20260808 01:50:22.811257 12006 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20260808 01:50:22.894224  9475 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:22.895982  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:22.896696  9475 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:22.811959 11867 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:22.899712  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 192525us
I20260808 01:50:22.901358  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:22.903681  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:22.905732  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:22.909179  9477 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:22.809185 12030 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:22.847766 12145 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.094s
I20260808 01:50:22.879060 12049 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:22.921871  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:22.902818 12149 log.cc:927] Time spent T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260808 01:50:22.928321 10367 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260808 01:50:22.919895 12083 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.528s	user 0.008s	sys 0.004s
I20260808 01:50:22.959738  9475 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:22.961261  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:22.915338 12123 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:22.880585 11842 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:22.963222  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 57714.3us
I20260808 01:50:22.962348  9475 raft_consensus.cc:2393] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 6: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:22.964219  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:22.922760 11941 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:22.969223  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:22.897488 12068 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
W20260808 01:50:22.979470  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:22.981498  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:22.955474 11539 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260808 01:50:22.983968 12006 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:22.984902 12139 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:22.985884 12103 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:22.930364 11939 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:22.959435 12093 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:23.018875 12103 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:23.015285  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:23.020468  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:22.889345 12159 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:23.021256  9475 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 5
I20260808 01:50:22.876055 12148 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:22.967233 12126 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:23.019555 12093 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:22.968251 12123 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:23.027987 12164 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260808 01:50:23.028954 12105 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.723s	user 0.006s	sys 0.006s
I20260808 01:50:23.029157  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:23.016491 12006 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:23.030386 12156 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
I20260808 01:50:22.975646 11941 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:23.022341  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 287) took 1152 ms. Trace:
I20260808 01:50:23.058650  9478 rpcz_store.cc:276] 0808 01:50:21.869482 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:22.596965 (+727483us) service_pool.cc:224] Handling call
0808 01:50:22.597453 (+   488us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:22.597751 (+   298us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:22.597796 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:22.599142 (+  1346us) log.cc:844] Serialized 26 byte log entry
0808 01:50:22.651293 (+ 52151us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:22.651329 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:22.651413 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:23.021962 (+370549us) raft_consensus.cc:1685] finished
0808 01:50:23.022000 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:23.022241 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":51544,"threads_started":1,"wal-append.queue_time_us":206547}
I20260808 01:50:23.059894  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:23.060527  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:23.061233  9478 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:23.062340  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:23.029814 12146 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260808 01:50:23.039748 12036 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 01:50:23.042804  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:23.062665  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:22.963222 12083 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:23.048059 12119 consensus_meta.cc:287] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.556s	user 0.009s	sys 0.003s
I20260808 01:50:23.021152 12072 raft_consensus.cc:3055] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:23.068825  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:23.064594 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:23.069139  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:23.061192 12126 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:23.106422  9475 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:23.111613  9478 raft_consensus.cc:2410] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260808 01:50:23.112722  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 43726.5us
I20260808 01:50:23.063644 12006 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:23.065659 12103 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:23.114710  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:23.115149  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:23.123538  9475 raft_consensus.cc:2410] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:50:23.123628  9478 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:23.124996  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:23.128383  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 289) took 1029 ms. Trace:
I20260808 01:50:23.129004  9475 rpcz_store.cc:276] 0808 01:50:22.099345 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:23.126357 (+1027012us) service_pool.cc:224] Handling call
0808 01:50:23.127721 (+  1364us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:23.127972 (+   251us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:23.128028 (+    56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:23.128096 (+    68us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:23.128142 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:23.128216 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:23.128321 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:23.132081  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2021) took 1004 ms. Trace:
I20260808 01:50:23.132758  9475 rpcz_store.cc:276] 0808 01:50:22.127661 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:23.130797 (+1003136us) service_pool.cc:224] Handling call
0808 01:50:23.131231 (+   434us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:23.131462 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:23.131500 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:23.131557 (+    57us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:23.131733 (+   176us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:23.131832 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:23.131997 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:23.116742  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 54141.1us
I20260808 01:50:23.140046  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:23.142196  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:23.087114  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:23.061160 12028 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:23.143594  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:23.144038  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:23.063226 11941 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:23.105134 12105 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:23.112361 10362 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260808 01:50:23.064633 12145 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20260808 01:50:23.112043 12072 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:23.162829 12165 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
W20260808 01:50:23.162976 12161 log.cc:927] Time spent T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260808 01:50:23.117858 12124 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:23.119545 12169 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:23.213605  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1721) took 1954 ms. Trace:
I20260808 01:50:23.235190  9636 rpcz_store.cc:276] 0808 01:50:21.258996 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:22.574128 (+1315132us) service_pool.cc:224] Handling call
0808 01:50:22.574486 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:22.576631 (+  2145us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:22.576668 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:22.577368 (+   700us) log.cc:844] Serialized 26 byte log entry
0808 01:50:22.819773 (+242405us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:22.819831 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:22.819941 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:23.213359 (+393418us) raft_consensus.cc:1685] finished
0808 01:50:23.213380 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:23.213533 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":241676,"threads_started":1,"wal-append.queue_time_us":311196}
I20260808 01:50:23.237212  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:23.146284 11939 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:50:23.067763 12143 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.260s	user 0.003s	sys 0.000s
I20260808 01:50:23.164510 12103 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 01:50:23.259738 12168 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260808 01:50:23.107179 12119 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:23.278626  9636 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:50:23.289026 12154 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
W20260808 01:50:23.291103 12158 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.304s	user 0.001s	sys 0.001s
I20260808 01:50:23.292976 12148 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:23.234946 12169 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:23.293922 12169 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:23.298476  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:23.308687  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:23.277905 12113 consensus_meta.cc:287] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.886s	user 0.014s	sys 0.001s
I20260808 01:50:23.164808 12105 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:23.263221 10362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 392) took 2047 ms. Trace:
I20260808 01:50:23.467479 11842 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:23.259363 12072 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:23.484242 12000 consensus_meta.cc:287] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.006s	user 0.013s	sys 0.000s
I20260808 01:50:23.485400 12093 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:23.462791 12113 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:23.377281 12095 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.013s	sys 0.040s
W20260808 01:50:23.514058 12095 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.014s	sys 0.041s
I20260808 01:50:23.511162 12000 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:23.463065 10362 rpcz_store.cc:276] 0808 01:50:21.215553 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:21.220399 (+  4846us) service_pool.cc:224] Handling call
0808 01:50:21.220805 (+   406us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:23.048025 (+1827220us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b540064b2a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490b1a0 00007f5d449955a0 00007f5d44996611 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c
0808 01:50:23.164938 (+116913us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:23.255970 (+ 91032us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":415426816}
I20260808 01:50:23.545855 12159 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:23.577818 12125 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.204s	user 0.016s	sys 0.000s
I20260808 01:50:23.606495 12125 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:23.617551 12072 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:23.635883 12105 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:23.619916 11795 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.872s	user 0.011s	sys 0.003s
W20260808 01:50:23.669270  9638 consensus_meta.cc:287] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.629s	user 0.009s	sys 0.003s
I20260808 01:50:23.670249  9638 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:23.902581  9478 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:23.906225  9475 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
W20260808 01:50:23.913491  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:23.913965  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1844) took 1289 ms. Trace:
W20260808 01:50:23.991223  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:23.991755  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:23.991125  9477 rpcz_store.cc:276] 0808 01:50:22.623950 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:22.645022 (+ 21072us) service_pool.cc:224] Handling call
0808 01:50:23.913326 (+1268304us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b540019b9d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:23.913668 (+   342us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":429613824}
W20260808 01:50:23.992480  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:23.914347  9599 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.659s	user 0.010s	sys 0.019s
W20260808 01:50:23.994757  9599 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.011s	sys 0.020s
W20260808 01:50:23.995453  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 740542us
W20260808 01:50:23.914781  9631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1712) took 2703 ms (client timeout 2992 ms). Trace:
W20260808 01:50:23.999302  9631 rpcz_store.cc:269] 0808 01:50:21.211155 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:21.590978 (+379823us) service_pool.cc:224] Handling call
0808 01:50:23.913904 (+2322926us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540032d554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:23.914382 (+   478us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2504294272}
I20260808 01:50:24.000156  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 288) took 2129 ms. Trace:
I20260808 01:50:24.000720  9478 rpcz_store.cc:276] 0808 01:50:21.870289 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:23.124755 (+1254466us) service_pool.cc:224] Handling call
0808 01:50:23.901809 (+777054us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:23.983248 (+ 81439us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.000048 (+ 16800us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:23.916656  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1719) took 2703 ms. Trace:
I20260808 01:50:23.918797  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1709) took 2708 ms. Trace:
I20260808 01:50:24.004448  9634 rpcz_store.cc:276] 0808 01:50:21.210293 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:21.656400 (+446107us) service_pool.cc:224] Handling call
0808 01:50:21.683437 (+ 27037us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:22.118105 (+434668us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:23.917911 (+1799806us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b54001b44d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:23.918506 (+   595us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":898754432}
I20260808 01:50:24.003213  9632 rpcz_store.cc:276] 0808 01:50:21.213271 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:22.402167 (+1188896us) service_pool.cc:224] Handling call
0808 01:50:22.417809 (+ 15642us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:22.418240 (+   431us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:22.418272 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:22.418803 (+   531us) log.cc:844] Serialized 26 byte log entry
0808 01:50:22.635216 (+216413us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:22.635280 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:22.635382 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:23.915755 (+1280373us) raft_consensus.cc:1685] finished
0808 01:50:23.915841 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:23.916314 (+   473us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2147,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":2522,"thread_start_us":216029,"threads_started":1,"wal-append.queue_time_us":293633}
W20260808 01:50:23.708087 12006 consensus_meta.cc:287] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.562s	user 0.009s	sys 0.005s
W20260808 01:50:23.923161  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:24.007135 12123 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:24.007530 12006 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:23.939647  9481 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:24.010793  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2028) took 1823 ms. Trace:
I20260808 01:50:24.011037 11691 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:24.011231  9481 rpcz_store.cc:276] 0808 01:50:22.187604 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:23.938115 (+1750511us) service_pool.cc:224] Handling call
0808 01:50:23.938775 (+   660us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:24.010405 (+ 71630us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.010705 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:24.013356  9481 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:24.014149  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 292) took 1592 ms. Trace:
W20260808 01:50:23.746588 12126 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.620s	user 0.014s	sys 0.000s
I20260808 01:50:23.683060 12177 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:23.922559  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:23.987730 10369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 488) took 2678 ms (client timeout 1789 ms). Trace:
I20260808 01:50:23.989952 10368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 418) took 1941 ms. Trace:
W20260808 01:50:23.998526  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:23.990346  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:24.014627  9481 rpcz_store.cc:276] 0808 01:50:22.421515 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.012461 (+1590946us) service_pool.cc:224] Handling call
0808 01:50:24.013074 (+   613us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:24.013837 (+   763us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.014081 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:23.936920  9476 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 284) took 2369 ms (client timeout 2007 ms). Trace:
W20260808 01:50:23.924029  9629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1862) took 3002 ms (client timeout 1881 ms). Trace:
I20260808 01:50:23.994531  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 290) took 1748 ms. Trace:
I20260808 01:50:23.957469 11649 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:24.017979  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2026) took 1831 ms. Trace:
W20260808 01:50:23.931725  9473 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2012) took 2466 ms (client timeout 2224 ms). Trace:
I20260808 01:50:24.035296  9479 rpcz_store.cc:276] 0808 01:50:22.246300 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:23.992626 (+1746326us) service_pool.cc:224] Handling call
0808 01:50:23.993331 (+   705us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:23.993837 (+   506us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:23.993869 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:23.993965 (+    96us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:23.994024 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:23.994123 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:23.994297 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:24.011246 11757 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:23.931676  9636 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 01:50:24.042045 12145 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:24.033270 10369 rpcz_store.cc:269] 0808 01:50:21.308887 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:21.309105 (+   218us) service_pool.cc:224] Handling call
0808 01:50:21.407655 (+ 98550us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:50:21.874536 (+466881us) spinlock_profiling.cc:243] Waited 8.17 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d4185584c 00007f5d4185540b 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:50:23.986998 (+2112462us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b5400449b54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:23.987378 (+   380us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":613104000}
I20260808 01:50:24.044975 12124 raft_consensus.cc:1027] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: attempt to promote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260808 01:50:24.043005  9632 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:24.038283  9475 rpcz_store.cc:276] 0808 01:50:22.186943 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:23.135661 (+948718us) service_pool.cc:224] Handling call
0808 01:50:23.905541 (+769880us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:23.983652 (+ 78111us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.017882 (+ 34230us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:24.033830 10368 rpcz_store.cc:276] 0808 01:50:22.048373 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:22.061430 (+ 13057us) service_pool.cc:224] Handling call
0808 01:50:22.061858 (+   428us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:22.809414 (+747556us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:23.988928 (+1179514us) spinlock_profiling.cc:243] Waited 675 ms on lock 0x7b54005550d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:23.989617 (+   689us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1552798976}
I20260808 01:50:24.051517 12126 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:50:24.034802  9476 rpcz_store.cc:269] 0808 01:50:21.567627 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:21.572950 (+  5323us) service_pool.cc:224] Handling call
0808 01:50:23.936346 (+2363396us) spinlock_profiling.cc:243] Waited 52.3 ms on lock 0x7b540005fd54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:23.936665 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":120256768}
I20260808 01:50:24.010913 11565 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:24.060016 12114 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:24.060175 11691 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:24.061154 12114 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.061388 11691 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.063298 12145 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.067970  9475 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260808 01:50:24.062309 12177 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:24.069003  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1838) took 1446 ms. Trace:
I20260808 01:50:24.069550  9475 rpcz_store.cc:276] 0808 01:50:22.622982 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.066768 (+1443786us) service_pool.cc:224] Handling call
0808 01:50:24.067506 (+   738us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:24.068632 (+  1126us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.068901 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:24.030697 12123 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:24.035175  9629 rpcz_store.cc:269] 0808 01:50:20.921310 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:20.984709 (+ 63399us) service_pool.cc:224] Handling call
0808 01:50:23.923364 (+2938655us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540008cd54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:23.923764 (+   400us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2582027648}
W20260808 01:50:24.039378  9473 rpcz_store.cc:269] 0808 01:50:21.465434 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:21.483603 (+ 18169us) service_pool.cc:224] Handling call
0808 01:50:23.931210 (+2447607us) spinlock_profiling.cc:243] Waited 97 ms on lock 0x7b54002bb1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:23.931458 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":223058176}
I20260808 01:50:24.057727  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2019) took 2331 ms. Trace:
I20260808 01:50:24.071461 12123 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:24.080849  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1726) took 2740 ms. Trace:
I20260808 01:50:24.099149  9632 rpcz_store.cc:276] 0808 01:50:21.319460 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.006109 (+2686649us) service_pool.cc:224] Handling call
0808 01:50:24.019502 (+ 13393us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:24.056063 (+ 36561us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.059477 (+  3414us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:24.066318 11941 consensus_meta.cc:287] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.839s	user 0.009s	sys 0.006s
I20260808 01:50:24.038174 11649 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:24.098183  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:24.105849 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:24.107321 10369 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:24.085268  9474 rpcz_store.cc:276] 0808 01:50:21.580961 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:22.541269 (+960308us) service_pool.cc:224] Handling call
0808 01:50:22.541640 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:22.555979 (+ 14339us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:23.911099 (+1355120us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b540021cd54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:23.911764 (+   665us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":31565568}
W20260808 01:50:24.111753  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 872053us
I20260808 01:50:24.113500  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.114583  9551 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:24.115865  9558 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 1. Current term is 2. Ops: []
I20260808 01:50:24.116063 11649 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:24.116655  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:24.117838 11649 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:24.118053  9555 raft_consensus.cc:2410] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260808 01:50:24.118578 12180 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:24.118876 11941 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.119529 12180 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.120296  9552 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260808 01:50:24.121876 11941 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:24.084508 12145 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:24.103142 12181 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.053s
I20260808 01:50:24.124464 12124 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:50:24.125887  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1722) took 2852 ms. Trace:
I20260808 01:50:24.126384  9636 rpcz_store.cc:276] 0808 01:50:21.273313 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:23.310131 (+2036818us) service_pool.cc:224] Handling call
0808 01:50:23.310522 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:24.072412 (+761890us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.125756 (+ 53344us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:24.123600  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 64271.2us
I20260808 01:50:24.115898  9475 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 4 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260808 01:50:24.144107 12159 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:24.144431  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.145125 12159 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.114579 11691 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:24.147213 12118 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260808 01:50:24.155151 12072 consensus_meta.cc:287] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.526s	user 0.013s	sys 0.003s
I20260808 01:50:24.176667  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1839) took 1553 ms. Trace:
I20260808 01:50:24.181675  9475 rpcz_store.cc:276] 0808 01:50:22.623152 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.080835 (+1457683us) service_pool.cc:224] Handling call
0808 01:50:24.115299 (+ 34464us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:24.142669 (+ 27370us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.176575 (+ 33906us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:24.161141 12083 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:24.163141 12139 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:24.164745 11867 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:24.146533 12145 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:24.167646 10360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 421) took 1953 ms. Trace:
W20260808 01:50:24.168411  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:24.171496 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:24.173502 10362 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260808 01:50:24.173753  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:24.138621 11757 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:24.174708 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:24.174942  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.159332  9560 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 01:50:24.122416 12180 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:24.178406  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.192900  9559 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:24.199098 12145 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:24.199681 11757 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.201202 12119 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:24.199155  9635 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:24.203826 11955 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:24.204754 11955 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.207393  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 298) took 2160 ms. Trace:
I20260808 01:50:24.198464  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1840) took 1575 ms. Trace:
I20260808 01:50:24.185962 10360 rpcz_store.cc:276] 0808 01:50:22.214697 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:22.609493 (+394796us) service_pool.cc:224] Handling call
0808 01:50:22.651332 (+ 41839us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:22.675499 (+ 24167us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.167057 (+1491558us) spinlock_profiling.cc:243] Waited 30.5 ms on lock 0x7b5400681d50. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:24.167382 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":70159744}
I20260808 01:50:24.208740 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:24.210947  9476 rpcz_store.cc:276] 0808 01:50:22.623320 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.109748 (+1486428us) service_pool.cc:224] Handling call
0808 01:50:24.181028 (+ 71280us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:24.197932 (+ 16904us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:24.197963 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:24.198145 (+   182us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:24.198189 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.198258 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:24.198401 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:24.203601  9552 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:24.214210  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1716) took 2996 ms. Trace:
W20260808 01:50:24.223546  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:24.198252 12177 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:24.207871  9556 rpcz_store.cc:276] 0808 01:50:22.047291 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:22.047465 (+   174us) service_pool.cc:224] Handling call
0808 01:50:22.065803 (+ 18338us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:22.066264 (+   461us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:22.066294 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:22.066915 (+   621us) log.cc:844] Serialized 26 byte log entry
0808 01:50:23.961818 (+1894903us) spinlock_profiling.cc:243] Waited 7.25 ms on lock 0x7b1800478b40. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:50:24.127245 (+165427us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:24.127368 (+   123us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.127548 (+   180us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:24.204639 (+ 77091us) spinlock_profiling.cc:243] Waited 3.93 ms on lock 0x7b5400291920. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490919e 00007f5d44913d10 00007f5d44913cb8 00007f5d44913c38 00007f5d44913bf3 00007f5d44912e7c 00007f5d4a837996 00007f5d4a832148 00007f5d4494fbab 00007f5d449526a0
0808 01:50:24.206912 (+  2273us) raft_consensus.cc:1685] finished
0808 01:50:24.206932 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:24.207162 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3134,"raft.queue_time_us":2299,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":7864,"spinlock_wait_cycles":25711744,"thread_start_us":72697,"threads_started":1,"wal-append.queue_time_us":2026254}
I20260808 01:50:24.219707  9635 rpcz_store.cc:276] 0808 01:50:21.212330 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:21.657590 (+445260us) service_pool.cc:224] Handling call
0808 01:50:21.657975 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:24.204357 (+2546382us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.206416 (+  2059us) spinlock_profiling.cc:243] Waited 2.53 s on lock 0x7b54003f9b54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:24.208570 (+  2154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5828338944}
I20260808 01:50:24.233286  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 253) took 2730 ms. Trace:
I20260808 01:50:24.225515  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.236858  9636 rpcz_store.cc:276] 0808 01:50:21.496973 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.127445 (+2630472us) service_pool.cc:224] Handling call
0808 01:50:24.128019 (+   574us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:24.221235 (+ 93216us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:24.221284 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:24.226496 (+  5212us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:24.226563 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.226663 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:24.226869 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8823,"raft.queue_time_us":32696}
W20260808 01:50:24.239176  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 182032us
I20260808 01:50:24.208364 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:24.255851 11987 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:24.256287 12195 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:24.256870 11565 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:24.256747 11987 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.257085 12195 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.257727 11565 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:24.256389  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:24.258745 11568 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.248644 12177 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:24.260670 12177 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:24.262327  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:24.263005 12012 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:24.263216  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.264170 12104 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:24.263926 12012 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:24.264825  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 49174.3us
I20260808 01:50:24.266954  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:24.268455  9552 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:24.269759 12042 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:24.264748 12104 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:24.270743 12042 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:24.271400  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2010) took 2831 ms. Trace:
I20260808 01:50:24.272408 12177 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:24.272922  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.273839  9556 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
I20260808 01:50:24.271899  9472 rpcz_store.cc:276] 0808 01:50:21.440149 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:21.773720 (+333571us) service_pool.cc:224] Handling call
0808 01:50:21.774055 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:21.776140 (+  2085us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:21.776254 (+   114us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:21.779861 (+  3607us) log.cc:844] Serialized 26 byte log entry
0808 01:50:23.908135 (+2128274us) spinlock_profiling.cc:243] Waited 8.76 ms on lock 0x7b18002a33a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:50:24.131240 (+223105us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:24.131289 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.131423 (+   134us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:24.270763 (+139340us) raft_consensus.cc:1685] finished
0808 01:50:24.270786 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:24.271123 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1519,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":42743,"spinlock_wait_cycles":20154624,"thread_start_us":137626,"threads_started":1,"wal-append.queue_time_us":2396631}
I20260808 01:50:24.274767 12083 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:50:24.275365 10360 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260808 01:50:24.276264 12119 raft_consensus.cc:3055] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:24.276774 12119 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:24.277885  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:24.247169 12145 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:24.279040 11809 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:24.279021 12104 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.279994 11809 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:24.280562 12104 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:24.282328  9556 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:24.282807  9552 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260808 01:50:24.283174  9552 pending_rounds.cc:85] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 3
I20260808 01:50:24.283496  9552 pending_rounds.cc:107] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260808 01:50:24.283684  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:24.283813  9552 raft_consensus.cc:2889] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:24.285070 10361 raft_consensus.cc:2410] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:50:24.285885  9472 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:24.286747  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2041) took 1562 ms. Trace:
I20260808 01:50:24.287137  9472 rpcz_store.cc:276] 0808 01:50:22.724372 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.285126 (+1560754us) service_pool.cc:224] Handling call
0808 01:50:24.285585 (+   459us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:24.286440 (+   855us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.286651 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:24.287699  9635 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:50:24.288523  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 254) took 2107 ms. Trace:
I20260808 01:50:24.288883  9635 rpcz_store.cc:276] 0808 01:50:22.181282 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.246374 (+2065092us) service_pool.cc:224] Handling call
0808 01:50:24.287263 (+ 40889us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:24.288172 (+   909us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.288434 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:24.275034 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:24.278146  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:24.283854  9554 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260808 01:50:24.298993  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 252) took 2802 ms. Trace:
I20260808 01:50:24.313228  9632 rpcz_store.cc:276] 0808 01:50:21.496642 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.139384 (+2642742us) service_pool.cc:224] Handling call
0808 01:50:24.226003 (+ 86619us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:24.298535 (+ 72532us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:24.298563 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:24.298627 (+    64us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:24.298663 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.298763 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:24.298894 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:24.301654 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:24.290477  9635 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:24.293280  9472 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:24.312371 10368 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:50:24.300886 12189 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260808 01:50:24.297819  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.326607  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.324996  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.299474  9523 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:24.324276  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 256) took 2139 ms. Trace:
I20260808 01:50:24.332031  9635 rpcz_store.cc:276] 0808 01:50:22.181848 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.289790 (+2107942us) service_pool.cc:224] Handling call
0808 01:50:24.290230 (+   440us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:24.321107 (+ 30877us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.321398 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:24.332790  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 79516.1us
I20260808 01:50:24.325536  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2042) took 1593 ms. Trace:
I20260808 01:50:24.333884  9472 rpcz_store.cc:276] 0808 01:50:22.728498 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.288010 (+1559512us) service_pool.cc:224] Handling call
0808 01:50:24.291583 (+  3573us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:24.321504 (+ 29921us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.321737 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:24.342077  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:24.343458 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:24.317481 11757 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:24.291054 12191 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:24.361131  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1725) took 3058 ms. Trace:
I20260808 01:50:24.328702 12145 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:24.374143 12070 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:24.375000 12070 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.375523 11987 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:24.376343 11987 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:24.376912 11987 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:24.377548 12070 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:24.378822  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 61113.4us
W20260808 01:50:24.379719 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 117694us
W20260808 01:50:24.390329  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 91045.5us
I20260808 01:50:24.371376  9551 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260808 01:50:24.426035  9558 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 5: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:24.380581  9445 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:24.429711  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 72746.9us
W20260808 01:50:24.433931  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 155971us
I20260808 01:50:24.434965  9634 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:50:24.364620  9631 rpcz_store.cc:276] 0808 01:50:21.302831 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.001292 (+2698461us) service_pool.cc:224] Handling call
0808 01:50:24.001943 (+   651us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:24.043280 (+ 41337us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:24.043314 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:24.044197 (+   883us) log.cc:844] Serialized 52 byte log entry
0808 01:50:24.137223 (+ 93026us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:24.323728 (+186505us) log.cc:844] Serialized 14 byte log entry
0808 01:50:24.360432 (+ 36704us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.360612 (+   180us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:24.360723 (+   111us) raft_consensus.cc:1685] finished
0808 01:50:24.360750 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:24.360981 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40917,"raft.queue_time_us":42165,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":335,"thread_start_us":228701,"threads_started":3,"wal-append.queue_time_us":160905}
I20260808 01:50:24.437902  9560 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:24.278175 12119 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:24.440830 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:24.442102  9550 raft_consensus.cc:2410] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:50:24.443508  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.366989 11757 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:50:24.457969  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:24.378104 11987 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:24.485713  9638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1706) took 3177 ms (client timeout 1780 ms). Trace:
I20260808 01:50:24.485956  9552 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:24.486155  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:24.486338  9638 rpcz_store.cc:269] 0808 01:50:21.204039 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:21.208299 (+  4260us) service_pool.cc:224] Handling call
0808 01:50:24.380470 (+3172171us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b540008d254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:24.380722 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4039763328}
I20260808 01:50:24.487198 11809 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:24.439220 11955 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:24.488932  9555 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260808 01:50:24.423758 11941 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
W20260808 01:50:24.496418  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 51452.4us
I20260808 01:50:24.502156 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:24.498942  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:24.504456  9631 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:24.494459 12180 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:24.518428  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:24.452023 12191 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:24.493582 12094 consensus_queue.cc:260] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:24.520293  9557 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: []
W20260808 01:50:24.523170  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 142714us
I20260808 01:50:24.537269  9552 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260808 01:50:24.543708  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:24.565100  9559 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:24.567742  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:24.568212  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.568918  9559 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:24.569242  9552 raft_consensus.cc:2410] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:50:24.569963  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.571071  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:24.572605  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:24.573964  9552 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 5: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:24.573009  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:24.574535  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:24.571553  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:24.494084 12105 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:24.564183  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:24.573854  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:24.575788  9550 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:24.587806  9560 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:24.575448  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 68353.6us
I20260808 01:50:24.589955  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:24.519791 12195 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:24.500909 12145 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:24.592218  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 91789.9us
I20260808 01:50:24.520861 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:24.534301 12123 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:24.533540 12000 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:24.611986 12000 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:24.592746  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:24.597474 12190 log.cc:927] Time spent T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260808 01:50:24.597892 12060 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:24.598410 12193 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260808 01:50:24.614228  9638 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
I20260808 01:50:24.524969 12012 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:24.599220 12145 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:24.600148 12093 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:24.521606 11955 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:24.611004  9601 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:24.595310 12083 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260808 01:50:24.526754 12042 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:24.612551  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2015) took 3069 ms. Trace:
I20260808 01:50:24.633890  9559 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:24.660161  9550 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:24.663218  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.664482  9552 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:24.525260 11757 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.612614 12000 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:24.667411  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 56602.4us
I20260808 01:50:24.668670  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.671077  9550 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260808 01:50:24.595975 12094 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:24.670040 12042 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election won for term 5
W20260808 01:50:24.677103 12173 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.125s	user 0.022s	sys 0.003s
I20260808 01:50:24.677521  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1836) took 2054 ms. Trace:
I20260808 01:50:24.677713  9550 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 01:50:24.678607 12113 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:24.691154  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:24.678361 12093 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.680634 12199 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:24.699725  9638 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:24.692863  9481 rpcz_store.cc:276] 0808 01:50:22.622565 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.045495 (+1422930us) service_pool.cc:224] Handling call
0808 01:50:24.069056 (+ 23561us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:24.069610 (+   554us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:24.069644 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:24.070414 (+   770us) log.cc:844] Serialized 26 byte log entry
0808 01:50:24.094911 (+ 24497us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:24.094968 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.095087 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:24.677276 (+582189us) raft_consensus.cc:1685] finished
0808 01:50:24.677295 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:24.677458 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":23765,"threads_started":1,"wal-append.queue_time_us":105453}
I20260808 01:50:24.691041 11867 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:24.674651 12145 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:24.730916  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:24.660065 11955 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:24.725921  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 294) took 2062 ms. Trace:
I20260808 01:50:24.650331 12105 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.653985 12060 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:24.687315 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 306796us
I20260808 01:50:24.750418  9601 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:24.719381 11809 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:24.755692  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:24.655611  9480 rpcz_store.cc:276] 0808 01:50:21.543236 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:21.786360 (+243124us) service_pool.cc:224] Handling call
0808 01:50:21.786729 (+   369us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:21.797865 (+ 11136us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:22.065207 (+267342us) log.cc:844] Serialized 14 byte log entry
0808 01:50:22.306408 (+241201us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:22.761079 (+454671us) spinlock_profiling.cc:243] Waited 6.73 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:23.941173 (+1180094us) spinlock_profiling.cc:243] Waited 2.02 ms on lock 0x7b2c00134ff8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:50:23.942287 (+  1114us) log.cc:844] Serialized 26 byte log entry
0808 01:50:24.130807 (+188520us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:24.130873 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.136023 (+  5150us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:24.601888 (+465865us) raft_consensus.cc:1685] finished
0808 01:50:24.601910 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:24.612337 (+ 10427us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":380250,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":128,"raft.queue_time_us":26554,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":12286,"spinlock_wait_cycles":20122752,"thread_start_us":528356,"threads_started":3,"wal-append.queue_time_us":565493,"wal-append.run_cpu_time_us":5596,"wal-append.run_wall_time_us":1421599}
W20260808 01:50:24.764926  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:24.765587  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:24.766007  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:24.766470  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:24.723289 11565 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:24.726337 12042 raft_consensus.cc:697] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:24.776274 12042 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.777452 12105 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:24.779309  9635 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:24.786633 11809 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:24.788432  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1841) took 2164 ms. Trace:
I20260808 01:50:24.788316 11809 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.788947  9473 rpcz_store.cc:276] 0808 01:50:22.623477 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.108695 (+1485218us) service_pool.cc:224] Handling call
0808 01:50:24.175298 (+ 66603us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:24.175853 (+   555us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:24.175888 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:24.178137 (+  2249us) log.cc:844] Serialized 26 byte log entry
0808 01:50:24.438161 (+260024us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:24.438207 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.438289 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:24.788156 (+349867us) raft_consensus.cc:1685] finished
0808 01:50:24.788183 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:24.788353 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11075,"raft.queue_time_us":3817,"raft.run_cpu_time_us":434,"raft.run_wall_time_us":50592,"thread_start_us":259591,"threads_started":1,"wal-append.queue_time_us":513682}
I20260808 01:50:24.791355  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:24.792187  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 98602.7us
I20260808 01:50:24.792917  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1837) took 2170 ms. Trace:
I20260808 01:50:24.746351 11941 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:24.803503  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1845) took 1900 ms. Trace:
I20260808 01:50:24.803174  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:24.741355  9476 rpcz_store.cc:276] 0808 01:50:22.653750 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.215813 (+1562063us) service_pool.cc:224] Handling call
0808 01:50:24.301746 (+ 85933us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:24.311874 (+ 10128us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:50:24.588400 (+276526us) log.cc:844] Serialized 14 byte log entry
0808 01:50:24.702794 (+114394us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:24.702870 (+    76us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:24.715343 (+ 12473us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.715494 (+   151us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:24.715731 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":292376,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":838,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":13479,"thread_start_us":331305,"threads_started":2}
I20260808 01:50:24.812074  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.812306  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.805517  9472 rpcz_store.cc:276] 0808 01:50:22.903534 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.335086 (+1431552us) service_pool.cc:224] Handling call
0808 01:50:24.443453 (+108367us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:24.487901 (+ 44448us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:50:24.634707 (+146806us) log.cc:844] Serialized 14 byte log entry
0808 01:50:24.770518 (+135811us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:24.770585 (+    67us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:24.803239 (+ 32654us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.803320 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:24.803437 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":118005,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":52,"raft.queue_time_us":13104,"raft.run_cpu_time_us":387,"raft.run_wall_time_us":19633,"thread_start_us":160500,"threads_started":2}
I20260808 01:50:24.762259 12113 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:24.769433 11867 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.814528  9480 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 4: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:24.815383  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:24.814580  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:24.821044  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:24.794098  9479 rpcz_store.cc:276] 0808 01:50:22.622800 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.080811 (+1458011us) service_pool.cc:224] Handling call
0808 01:50:24.116345 (+ 35534us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:24.116821 (+   476us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:24.116852 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:24.117615 (+   763us) log.cc:844] Serialized 26 byte log entry
0808 01:50:24.190072 (+ 72457us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:24.489852 (+299780us) log.cc:844] Serialized 14 byte log entry
0808 01:50:24.493921 (+  4069us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.494179 (+   258us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:24.792646 (+298467us) raft_consensus.cc:1685] finished
0808 01:50:24.792670 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:24.792847 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":422134,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":6120,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":61488,"thread_start_us":344876,"threads_started":2,"wal-append.queue_time_us":246334}
I20260808 01:50:24.841922  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:24.842898  9473 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:24.864114  9473 raft_consensus.cc:2410] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260808 01:50:24.861634  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.741686 12198 consensus_queue.cc:1059] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.234s
I20260808 01:50:24.867244  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.732105 12199 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:24.868965  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:24.868256 12113 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.870131  9481 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 2.
I20260808 01:50:24.873226  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:24.874784  9552 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:24.877653 11867 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:24.887318  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:24.879863 12206 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
W20260808 01:50:24.900431  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:24.881017 12093 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:24.816828 12145 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:24.881999 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 64610.6us
I20260808 01:50:24.873999  9551 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:24.826056 12202 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:24.839973 12105 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:24.872725  9601 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:24.900493  9476 raft_consensus.cc:2410] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:50:24.906940  9472 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 5: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:24.899935  9550 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:24.903383  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.824568 11941 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:24.911669  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:24.918962 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 84823.1us
I20260808 01:50:24.924777  9552 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:24.930364  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:24.930083  9480 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:24.952688  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2032) took 2560 ms. Trace:
I20260808 01:50:24.951678  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:24.955207  9638 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:24.955829 12202 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:25.443252  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 541071us
W20260808 01:50:25.444567  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:24.953188  9478 rpcz_store.cc:276] 0808 01:50:22.392600 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.001834 (+1609234us) service_pool.cc:224] Handling call
0808 01:50:24.002440 (+   606us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:24.095265 (+ 92825us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:24.480092 (+384827us) log.cc:844] Serialized 14 byte log entry
0808 01:50:24.721616 (+241524us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:24.761971 (+ 40355us) log.cc:844] Serialized 26 byte log entry
0808 01:50:24.762146 (+   175us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:24.762177 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.762248 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:24.952247 (+189999us) raft_consensus.cc:1685] finished
0808 01:50:24.952271 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:24.952581 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14492,"prepare.queue_time_us":516523,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":93332,"raft.run_cpu_time_us":454,"raft.run_wall_time_us":8359,"thread_start_us":547423,"threads_started":3,"wal-append.queue_time_us":308881}
W20260808 01:50:24.964350  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:25.021204  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:25.443776  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:24.962585  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 144403us
I20260808 01:50:24.911538 12105 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:25.145033  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 255) took 2963 ms. Trace:
W20260808 01:50:25.445354  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:25.448449  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:25.024459  9551 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:24.961867 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45949.2us
I20260808 01:50:24.962249  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2031) took 2570 ms. Trace:
W20260808 01:50:24.936829 12104 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.656s	user 0.011s	sys 0.011s
I20260808 01:50:25.454568  9477 rpcz_store.cc:276] 0808 01:50:22.392293 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:23.993095 (+1600802us) service_pool.cc:224] Handling call
0808 01:50:23.993770 (+   675us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:24.140544 (+146774us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:24.140583 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:24.185186 (+ 44603us) log.cc:844] Serialized 26 byte log entry
0808 01:50:24.248649 (+ 63463us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:24.248700 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.248787 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:24.961915 (+713128us) raft_consensus.cc:1685] finished
0808 01:50:24.961946 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:24.962130 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39987,"raft.queue_time_us":147057,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":10381,"thread_start_us":119326,"threads_started":2,"wal-append.queue_time_us":746857}
W20260808 01:50:25.456554 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 536467us
W20260808 01:50:25.444792 12094 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.733s	user 0.008s	sys 0.007s
I20260808 01:50:24.961242 11941 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:25.281275 12225 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
W20260808 01:50:24.961719 12201 log.cc:927] Time spent T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.438s	user 0.001s	sys 0.000s
W20260808 01:50:25.078125 12206 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
W20260808 01:50:25.111599 12169 consensus_meta.cc:287] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.774s	user 0.016s	sys 0.020s
I20260808 01:50:25.474581  9477 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:50:25.475776 12169 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:25.477144 12124 raft_consensus.cc:3055] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:25.477635  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 296) took 1468 ms. Trace:
I20260808 01:50:25.457504  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1842) took 2833 ms. Trace:
I20260808 01:50:25.474982  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:25.479130  9477 rpcz_store.cc:276] 0808 01:50:24.009329 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:25.459651 (+1450322us) service_pool.cc:224] Handling call
0808 01:50:25.472255 (+ 12604us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:25.476678 (+  4423us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:25.477400 (+   722us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:25.481884  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:25.482772  9551 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
W20260808 01:50:25.488703 12215 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.810s	user 0.002s	sys 0.000s
I20260808 01:50:24.963122 11980 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:25.490473  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 248) took 4077 ms. Trace:
W20260808 01:50:25.454890  9475 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.017s	sys 0.024s
I20260808 01:50:25.413357 12221 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.087s
I20260808 01:50:25.454617 12105 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:25.170598 11955 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:25.490942  9629 rpcz_store.cc:276] 0808 01:50:21.412828 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.111539 (+2698711us) service_pool.cc:224] Handling call
0808 01:50:24.112234 (+   695us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:24.175077 (+ 62843us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:24.175111 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:24.175793 (+   682us) log.cc:844] Serialized 26 byte log entry
0808 01:50:24.517671 (+341878us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:24.517729 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.518885 (+  1156us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:25.487895 (+969010us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:25.490065 (+  2170us) raft_consensus.cc:1685] finished
0808 01:50:25.490095 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:25.490394 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16126,"raft.queue_time_us":35165,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":172,"spinlock_wait_cycles":281820544,"thread_start_us":341378,"threads_started":1,"wal-append.queue_time_us":501878}
W20260808 01:50:25.493881 12212 log.cc:927] Time spent T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.819s	user 0.002s	sys 0.000s
W20260808 01:50:25.493606  9475 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.019s	sys 0.025s
I20260808 01:50:25.450639  9636 rpcz_store.cc:276] 0808 01:50:22.181602 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.310081 (+2128479us) service_pool.cc:224] Handling call
0808 01:50:24.491064 (+180983us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:24.599993 (+108929us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:24.600028 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:24.600686 (+   658us) log.cc:844] Serialized 26 byte log entry
0808 01:50:24.735433 (+134747us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:24.735489 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.735596 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:25.144655 (+409059us) raft_consensus.cc:1685] finished
0808 01:50:25.144688 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:25.144930 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1406,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":54533,"thread_start_us":134182,"threads_started":1,"wal-append.queue_time_us":540980}
I20260808 01:50:25.450266 12202 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:25.479171 12169 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:25.498371 12169 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:25.499625 11980 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:25.501039 11980 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:25.490669  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:25.493382  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:25.503989 12139 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:25.472532  9524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.791s	user 0.025s	sys 0.022s
I20260808 01:50:25.504689 12139 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:25.504731  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:25.505816  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:25.506713  9551 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:25.507102 12159 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:25.160144 12145 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:25.479399  9474 rpcz_store.cc:276] 0808 01:50:22.623625 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.131840 (+1508215us) service_pool.cc:224] Handling call
0808 01:50:24.206447 (+ 74607us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:24.207201 (+   754us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:24.207238 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:24.208001 (+   763us) log.cc:844] Serialized 52 byte log entry
0808 01:50:24.376062 (+168061us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b4800211158. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:50:24.708453 (+332391us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:24.708528 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.708636 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:25.177352 (+468716us) raft_consensus.cc:1685] finished
0808 01:50:25.177387 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:25.457398 (+280011us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4846,"raft.run_cpu_time_us":3637,"raft.run_wall_time_us":162148,"spinlock_wait_cycles":39761280,"thread_start_us":499948,"threads_started":1,"wal-append.queue_time_us":869750}
I20260808 01:50:25.494207 11867 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:25.508363 11867 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:25.485323 12218 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.808s	user 0.003s	sys 0.000s
I20260808 01:50:25.499126 12169 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:25.506275  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:25.543205  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1738) took 2857 ms. Trace:
I20260808 01:50:25.543746  9632 rpcz_store.cc:276] 0808 01:50:22.686079 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.314895 (+1628816us) service_pool.cc:224] Handling call
0808 01:50:24.315529 (+   634us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:24.372033 (+ 56504us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:24.372079 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:24.480068 (+107989us) log.cc:844] Serialized 52 byte log entry
0808 01:50:24.650929 (+170861us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:24.651016 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.651169 (+   153us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:25.484368 (+833199us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:25.542820 (+ 58452us) raft_consensus.cc:1685] finished
0808 01:50:25.542847 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:25.543131 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":15598,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":132,"spinlock_wait_cycles":264405760,"thread_start_us":91511,"threads_started":1,"wal-append.queue_time_us":169114}
I20260808 01:50:25.546020 12202 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:25.546949 12202 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:25.541254 12233 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260808 01:50:25.549402 12202 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:25.549602  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:25.555917 10367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 425) took 1334 ms. Trace:
I20260808 01:50:25.513298 12231 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:25.556457 10367 rpcz_store.cc:276] 0808 01:50:24.221622 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.226955 (+  5333us) service_pool.cc:224] Handling call
0808 01:50:24.255922 (+ 28967us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:24.256614 (+   692us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:24.256652 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:24.257369 (+   717us) log.cc:844] Serialized 26 byte log entry
0808 01:50:24.354901 (+ 97532us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:24.354955 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.355030 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:24.960775 (+605745us) spinlock_profiling.cc:243] Waited 51.6 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:25.555550 (+594775us) raft_consensus.cc:1685] finished
0808 01:50:25.555580 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:25.555835 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":115610,"raft.run_cpu_time_us":540,"raft.run_wall_time_us":126164,"spinlock_wait_cycles":118772480,"thread_start_us":92548,"threads_started":1,"wal-append.queue_time_us":168155}
I20260808 01:50:25.558002 12231 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:25.558597 12113 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:25.562839  9524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.881s	user 0.036s	sys 0.025s
W20260808 01:50:25.564685  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 109309us
I20260808 01:50:25.566596  9554 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:25.573342 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:25.577778 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:25.578611 10369 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:25.570289 10332 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:25.578830 12042 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.006s
I20260808 01:50:25.582976 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:25.584623 11867 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:25.584935 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:25.586367  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:25.587106 10334 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:25.477674 12124 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:25.566488 11941 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:25.591850 10361 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:25.592913 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:25.598979  9474 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: [2.3-4.4]
I20260808 01:50:25.595887 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:25.602466 10366 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:25.603425 10367 raft_consensus.cc:2393] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:25.605630  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:25.600292  9524 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:25.610919  9554 raft_consensus.cc:2410] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:50:25.630478  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:25.631960 12239 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:25.632659 12239 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:25.633224 12239 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:25.633565  9559 raft_consensus.cc:2410] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:25.635371 11941 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:25.636463 12000 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.925s	user 0.015s	sys 0.000s
I20260808 01:50:25.636633  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 2113) took 1067 ms. Trace:
I20260808 01:50:25.637135  9559 rpcz_store.cc:276] 0808 01:50:24.569658 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.668319 (+ 98661us) service_pool.cc:224] Handling call
0808 01:50:25.636357 (+968038us) spinlock_profiling.cc:243] Waited 936 ms on lock 0x7b540015bb54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:25.636531 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2153897216}
I20260808 01:50:25.638813 12114 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:25.636986 12000 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:25.639686 11867 raft_consensus.cc:1027] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: attempt to promote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260808 01:50:25.641990 12000 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:25.644527  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2083) took 1541 ms. Trace:
I20260808 01:50:25.645015  9557 rpcz_store.cc:276] 0808 01:50:24.111851 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.112107 (+   256us) service_pool.cc:224] Handling call
0808 01:50:24.112615 (+   508us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:24.576479 (+463864us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:25.644203 (+1067724us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b54000b45d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:25.644418 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":619457280}
I20260808 01:50:25.649010  9559 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6. Ops: [5.2-5.2]
I20260808 01:50:25.636090 11941 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:25.666610  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 117162us
I20260808 01:50:25.642395 12191 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:25.695541  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 64914us
W20260808 01:50:25.644845 11649 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.526s	user 0.016s	sys 0.004s
I20260808 01:50:25.714480 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:25.716411 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:25.727749  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 186015us
W20260808 01:50:25.735533 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 267689us
I20260808 01:50:25.732160  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1846) took 2828 ms. Trace:
I20260808 01:50:25.692122 12180 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:25.736307  9476 rpcz_store.cc:276] 0808 01:50:22.903919 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.920485 (+2016566us) service_pool.cc:224] Handling call
0808 01:50:24.920999 (+   514us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:24.921342 (+   343us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:25.451513 (+530171us) log.cc:844] Serialized 14 byte log entry
0808 01:50:25.731528 (+280015us) log.cc:844] Serialized 14 byte log entry
0808 01:50:25.731789 (+   261us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:25.731846 (+    57us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:25.731879 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:25.731965 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:25.732094 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36224,"prepare.queue_time_us":542907,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":1325,"thread_start_us":246761,"threads_started":1}
W20260808 01:50:25.735584  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 146749us
I20260808 01:50:25.692893 12104 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 01:50:25.738550 12245 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260808 01:50:25.771808  9599 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:25.778951  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:25.776448 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:25.782289 10365 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:25.779953 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 312703us
W20260808 01:50:25.778719  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 60337.8us
I20260808 01:50:25.741948 11867 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:25.787511 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:25.715588 12234 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:25.773568 12159 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:25.717078 12000 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260808 01:50:25.774365 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:25.778297 12104 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:25.813680  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:25.809777  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:25.820523  9557 raft_consensus.cc:2410] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:50:25.814213  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:25.802920 11995 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.096s	user 0.017s	sys 0.038s
I20260808 01:50:25.835661  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:25.836819  9632 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
I20260808 01:50:25.822409  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:25.834371  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47698.4us
I20260808 01:50:25.837023  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:25.843020 12034 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:25.843910 12034 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:25.844074 12243 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260808 01:50:25.844483 12034 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:25.823956 12104 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:25.848133 12049 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:25.848831 12049 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:25.843138  9559 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:25.852877  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 297) took 1843 ms. Trace:
W20260808 01:50:25.833635 11995 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.153s	user 0.018s	sys 0.039s
I20260808 01:50:25.859001  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:25.820796 11980 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:25.865840  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:25.866633  9474 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:50:25.859697  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:25.858237  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:25.855878  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:25.850590 11941 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:25.821352 12238 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:25.821760 12114 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 01:50:25.860350  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 55649.3us
I20260808 01:50:25.832594 11867 raft_consensus.cc:697] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:25.840023 12191 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:25.857828 12159 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:25.869586  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:25.900202  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:25.869736  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 79590.7us
W20260808 01:50:25.873157  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 144262us
I20260808 01:50:25.875579  9472 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:25.854926  9481 rpcz_store.cc:276] 0808 01:50:24.009945 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:25.473151 (+1463206us) service_pool.cc:224] Handling call
0808 01:50:25.473449 (+   298us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:25.585494 (+112045us) raft_consensus.cc:1534] Early marking committed up to index 5
0808 01:50:25.592580 (+  7086us) log.cc:844] Serialized 14 byte log entry
0808 01:50:25.852527 (+259947us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:25.852581 (+    54us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:25.852628 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:25.852695 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:25.852812 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":8232,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":90,"raft.queue_time_us":7096,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":16902,"thread_start_us":259474,"threads_started":2,"wal-append.queue_time_us":246603}
I20260808 01:50:25.851037 12169 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:25.908179 10332 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:25.888779 10365 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:25.836541 12180 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:25.914184  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:25.912685 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:25.892402 12191 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:25.927062  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1849) took 1341 ms. Trace:
I20260808 01:50:25.924034  9600 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:25.862999 12105 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:25.905498 12246 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:25.947890  9599 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:25.947281 11867 consensus_queue.cc:237] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:25.835361 12117 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.093s
I20260808 01:50:25.950580 11809 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 01:50:25.951813 11809 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:25.952040 12051 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260808 01:50:25.953610 12244 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260808 01:50:25.957396  9524 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:25.854208 12094 consensus_queue.cc:1059] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.077s
I20260808 01:50:25.919610 12221 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:25.953526 12051 raft_consensus.cc:3055] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:25.960093  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1847) took 3055 ms. Trace:
I20260808 01:50:25.960179  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:25.928934  9477 rpcz_store.cc:276] 0808 01:50:24.585790 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:25.483153 (+897363us) service_pool.cc:224] Handling call
0808 01:50:25.484417 (+  1264us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:25.485865 (+  1448us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:25.575625 (+ 89760us) log.cc:844] Serialized 14 byte log entry
0808 01:50:25.911393 (+335768us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:25.915507 (+  4114us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:25.926652 (+ 11145us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:25.926805 (+   153us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:25.926985 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":84258,"prepare.queue_time_us":95361,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":69,"raft.queue_time_us":114988,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":14676,"thread_start_us":301521,"threads_started":2,"wal-append.queue_time_us":273165}
W20260808 01:50:25.962229  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:25.965633  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:25.972980  9479 rpcz_store.cc:276] 0808 01:50:22.904261 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:25.011960 (+2107699us) service_pool.cc:224] Handling call
0808 01:50:25.074663 (+ 62703us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:25.075458 (+   795us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:25.075501 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:25.076530 (+  1029us) log.cc:844] Serialized 52 byte log entry
0808 01:50:25.444044 (+367514us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:25.444086 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:25.444162 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:25.959725 (+515563us) raft_consensus.cc:1685] finished
0808 01:50:25.959757 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:25.959990 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2780,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":58423,"thread_start_us":269988,"threads_started":1,"wal-append.queue_time_us":653790}
I20260808 01:50:25.930157 12191 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:25.958918 11941 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:25.904877 11980 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:25.959556 12042 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:25.959391 12234 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 01:50:25.994349  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 61228.5us
W20260808 01:50:25.994922  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:25.964316 12221 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:25.972786 12051 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:25.932582 12114 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20260808 01:50:25.901618 12238 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:25.992388 12123 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 01:50:26.003414 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 83387.4us
I20260808 01:50:25.939303 12139 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:26.006376 10334 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:26.007468  9524 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:25.995395 12042 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:25.996822 12234 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:26.015769 12051 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:26.011219  9601 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:25.965207 12246 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.030792 12098 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260808 01:50:26.031656 12180 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:26.031929 12098 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:26.032337 12263 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:26.032888 12246 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:26.002426 12221 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:26.035559 12118 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:26.036157 12118 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:26.005816 12123 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:26.039060 12123 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:26.042310 12060 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:26.021605 12234 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:26.025158 12150 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:26.043467 12150 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:26.043931 12150 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:26.046345 12259 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260808 01:50:26.013895 12049 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:26.050561 12229 raft_consensus.cc:3055] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:26.051074 12229 raft_consensus.cc:740] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:26.051627 12105 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:26.052373 12247 log.cc:927] Time spent T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260808 01:50:26.052150 12229 consensus_queue.cc:260] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.055414 12229 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:26.027033 12235 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:26.058938 12235 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:26.025722 11980 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:26.025593  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:26.025153 12159 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:26.041918 12239 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.032363 12098 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:26.065830 10368 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:26.066213 12098 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:26.067330 12030 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:26.067759 12239 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:26.068063 12098 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:26.027961  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1784) took 1082 ms. Trace:
I20260808 01:50:26.058081  9600 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:26.068956 12034 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.014801 12042 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:26.073726 11757 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:26.074113  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:26.050292 12199 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:26.130177  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:26.076648 12147 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 3.317s	user 0.039s	sys 0.083s
I20260808 01:50:26.078963 12265 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:26.145262 12265 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:26.145809 12265 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:26.149168 10363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 497) took 4041 ms. Trace:
W20260808 01:50:26.080591 12213 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.481s	user 0.005s	sys 0.000s
I20260808 01:50:26.080746 12139 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:26.152567  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2100) took 1870 ms. Trace:
I20260808 01:50:26.152976  9556 rpcz_store.cc:276] 0808 01:50:24.282331 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.284815 (+  2484us) service_pool.cc:224] Handling call
0808 01:50:24.285203 (+   388us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:24.285828 (+   625us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:24.285861 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:24.286535 (+   674us) log.cc:844] Serialized 26 byte log entry
0808 01:50:24.452226 (+165691us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:24.452291 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.452485 (+   194us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:26.079527 (+1627042us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:26.152251 (+ 72724us) raft_consensus.cc:1685] finished
0808 01:50:26.152272 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:26.152499 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":17568,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":26485,"spinlock_wait_cycles":36170112,"thread_start_us":140515,"threads_started":1,"wal-append.queue_time_us":224614}
I20260808 01:50:26.081652 12238 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:26.082217 12157 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 3.114s	user 0.045s	sys 0.033s
I20260808 01:50:26.083614 12113 consensus_queue.cc:1059] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:26.155763 12113 raft_consensus.cc:3055] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:26.156159 12113 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:26.157166 12113 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
W20260808 01:50:26.086048  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:26.059520 12235 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:50:26.159108  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:26.159834  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:26.160567  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:26.127357  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 112544us
I20260808 01:50:26.128796  9472 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
I20260808 01:50:26.131166 12051 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:26.058020 12150 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:26.135355 12034 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:26.143216 11757 consensus_queue.cc:1059] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20260808 01:50:26.069761 10332 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.849s (SENT) [suppressed 34 similar messages]
I20260808 01:50:26.132929 12104 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:26.070070  9551 rpcz_store.cc:276] 0808 01:50:24.945510 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:25.507422 (+561912us) service_pool.cc:224] Handling call
0808 01:50:25.507823 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:25.559732 (+ 51909us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:25.559763 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:25.560323 (+   560us) log.cc:844] Serialized 26 byte log entry
0808 01:50:25.987371 (+427048us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:25.987705 (+   334us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:25.987786 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:26.027688 (+ 39902us) raft_consensus.cc:1685] finished
0808 01:50:26.027707 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:26.027880 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9231,"raft.queue_time_us":10546,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":110,"thread_start_us":426576,"threads_started":1,"wal-append.queue_time_us":462304}
W20260808 01:50:26.347693  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 217591us
I20260808 01:50:26.348729  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:26.353964 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:26.138196 12191 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:26.369050  9601 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:26.342928 11980 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:26.367654 12113 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:26.367458 12030 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:26.149781 10363 rpcz_store.cc:276] 0808 01:50:22.107937 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:22.572848 (+464911us) service_pool.cc:224] Handling call
0808 01:50:22.573394 (+   546us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:22.574050 (+   656us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:22.574093 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:22.587333 (+ 13240us) log.cc:844] Serialized 26 byte log entry
0808 01:50:22.692152 (+104819us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:22.692203 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:22.693832 (+  1629us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:26.075540 (+3381708us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:26.148492 (+ 72952us) raft_consensus.cc:1685] finished
0808 01:50:26.148522 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:26.148934 (+   412us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12943,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":128,"spinlock_wait_cycles":27720064,"thread_start_us":101284,"threads_started":1,"wal-append.queue_time_us":171246}
I20260808 01:50:26.387395 12030 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:26.364799 12094 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:26.391206 12094 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:26.393997 12094 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:26.399058 12253 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260808 01:50:26.154492 12221 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.405463 12221 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:26.407667 11980 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:26.407964 12104 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:26.409811  9599 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:26.377910 11809 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:26.378460 12123 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:26.437899 12123 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:26.390568 12030 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:26.451057 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 381304us
I20260808 01:50:26.420149 12191 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:26.456699 12150 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:26.456550 12118 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.457520 12265 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.460034 12265 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:26.460975  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46764.1us
I20260808 01:50:26.441591 12114 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:26.463423 12114 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:50:26.464447  9599 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:26.465793 11809 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:26.466229 11809 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:26.466617 11809 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:50:26.469310  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:26.479449  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:26.481411  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:26.482234 12265 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:26.483163 12265 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:26.483640 12265 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:26.484643 12150 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:26.485100 12150 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:26.485419 12150 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:26.486907 12104 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:26.487529 12104 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.488739 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:26.489675 10363 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 6: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:26.495904 12104 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:26.497890 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:26.500769  9599 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:26.501477 12234 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.501935 12104 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:26.507094  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:26.508945 12061 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 3.680s	user 0.013s	sys 0.024s
I20260808 01:50:26.509460 12042 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.512861 12042 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:26.514540 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:26.515377 10366 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:26.515924 12235 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:26.519101  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:26.519268 12006 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:26.509474 12061 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 3.750s	user 0.013s	sys 0.024s
I20260808 01:50:26.453138  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 1905) took 1544 ms. Trace:
I20260808 01:50:26.518458 12235 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:26.545174  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 176214us
I20260808 01:50:26.545729  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:26.551169  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:26.555214 12098 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:26.557528  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:26.526391  9638 rpcz_store.cc:276] 0808 01:50:24.909030 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.956157 (+ 47127us) service_pool.cc:224] Handling call
0808 01:50:25.452387 (+496230us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:25.991888 (+539501us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:25.991941 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:26.012592 (+ 20651us) log.cc:844] Serialized 26 byte log entry
0808 01:50:26.446857 (+434265us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:26.446895 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:26.446973 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:26.450778 (+  3805us) raft_consensus.cc:1685] finished
0808 01:50:26.450800 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:26.453061 (+  2261us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7695,"raft.queue_time_us":50326,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":2919,"thread_start_us":786765,"threads_started":2,"wal-append.queue_time_us":424237}
I20260808 01:50:26.555290  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:26.560356  9556 raft_consensus.cc:2376] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6. Re-sending same reply.
I20260808 01:50:26.561834  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:26.564986  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:26.566496  9552 raft_consensus.cc:2410] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260808 01:50:26.567977  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:26.569394  9559 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 5: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:26.579818  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:26.580829  9551 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:26.583527  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:26.583988  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:26.585788  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:26.580638  9554 raft_consensus.cc:2376] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5. Re-sending same reply.
I20260808 01:50:26.590490 10366 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:26.586792  9558 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:26.592548  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:26.601125 12234 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:26.601683 12124 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.592123  9600 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:26.542413 11809 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:26.609707  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:26.610613  9551 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:50:26.613879  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:26.614231  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:26.615631  9557 raft_consensus.cc:2410] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260808 01:50:26.615653  9599 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:26.615995  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 295) took 3939 ms. Trace:
I20260808 01:50:26.617663  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:26.620486 12124 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:26.621119  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:26.622798  9551 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 01:50:26.624545  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:26.626897  9554 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:26.631093  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:26.632130  9556 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:26.618391  9475 rpcz_store.cc:276] 0808 01:50:22.676191 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.244040 (+1567849us) service_pool.cc:224] Handling call
0808 01:50:24.348817 (+104777us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:24.613719 (+264902us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:24.613747 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:24.712407 (+ 98660us) log.cc:844] Serialized 26 byte log entry
0808 01:50:25.496732 (+784325us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:25.496776 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:26.603936 (+1107160us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b540038d1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:26.603990 (+    54us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:26.604051 (+    61us) raft_consensus.cc:1685] finished
0808 01:50:26.604072 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:26.615915 (+ 11843us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25437,"raft.queue_time_us":14123,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":43015,"spinlock_wait_cycles":576623360,"thread_start_us":704342,"threads_started":1,"wal-append.queue_time_us":881804}
I20260808 01:50:26.617849 12051 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:26.638587  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:26.642055  9599 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:26.643316  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:26.644027  9556 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:26.651953  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:26.652671  9554 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:26.656143 11756 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:26.649111 12051 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:26.658615 12051 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.663079 10366 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:26.665747  9523 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:26.683279 12030 consensus_queue.cc:1059] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:50:26.684492 12030 raft_consensus.cc:3055] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:26.684916 12030 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:26.691155 11809 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:26.691272 10368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 430) took 1097 ms. Trace:
I20260808 01:50:26.692003 10368 rpcz_store.cc:276] 0808 01:50:25.594345 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:25.594517 (+   172us) service_pool.cc:224] Handling call
0808 01:50:25.600694 (+  6177us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:26.066312 (+465618us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:26.690879 (+624567us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b54005122d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:26.691174 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1069186816}
I20260808 01:50:26.667845 12098 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:26.707044 12113 consensus_queue.cc:1059] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.006s
I20260808 01:50:26.709475 12113 raft_consensus.cc:3055] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:26.709883 12113 raft_consensus.cc:740] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:26.710768 12113 consensus_queue.cc:260] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.712178 12113 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:26.671362 12235 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:26.714236 12235 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:26.715293 12264 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.698s	user 0.007s	sys 0.005s
I20260808 01:50:26.716910 12235 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:26.716774 12150 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:26.718887 12230 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:26.719201 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:26.719863 12230 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:26.685995 12030 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.728574  9556 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:26.729604  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:26.730249  9556 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:26.744280  9473 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:26.744643 10334 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:26.746407  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:26.747165  9472 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:26.748394 12036 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:26.748839 12036 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:26.747042  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:26.744305  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:26.753863  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:26.754520  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:26.755059  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:26.752563  9475 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:26.743057 12265 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:26.770406  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:26.749222 12036 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:26.781492  9473 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:26.782408 12265 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:26.781428  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:26.766260 12235 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:26.788892 12235 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:26.790740 12030 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:26.793814 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:26.795418 11864 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:26.795881 11864 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:26.796262 11864 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:26.796635 12117 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260808 01:50:26.787027 11987 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.261s	user 0.020s	sys 0.026s
I20260808 01:50:26.805810  9472 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:26.812618  9554 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:26.814356  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:26.815472  9554 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:26.821965  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:26.822293  9472 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:26.822026  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:26.823805 12150 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:26.824222  9473 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:26.824308  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:26.826182  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:26.827507 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:26.828023 12229 consensus_queue.cc:1059] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 01:50:26.828714 10334 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:26.829787 11961 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:50:26.830523 12199 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:26.831092 12199 raft_consensus.cc:697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:26.830030 12221 raft_consensus.cc:3055] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:26.840286 12221 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:26.836035 12198 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:26.838192  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:26.839061  9477 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:26.832082 12199 consensus_queue.cc:237] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.839511 12235 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:26.820912 12118 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:26.812803 11987 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.286s	user 0.020s	sys 0.027s
I20260808 01:50:26.841702 12221 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.846419  9474 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
W20260808 01:50:26.851002  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:26.852746  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:26.853606  9558 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:26.845571 12198 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:26.856664  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:26.857631  9472 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:26.859203  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1727) took 5527 ms. Trace:
I20260808 01:50:26.859843  9554 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:50:26.860196  9634 rpcz_store.cc:276] 0808 01:50:21.332207 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:24.006382 (+2674175us) service_pool.cc:224] Handling call
0808 01:50:24.380999 (+374617us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:24.435373 (+ 54374us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:26.857482 (+2422109us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b540008d250. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:26.859122 (+  1640us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":820027776}
I20260808 01:50:26.861569  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:26.862622  9481 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:26.864457 11568 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:26.865999  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:26.867031  9475 raft_consensus.cc:2410] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:26.868909  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:26.874583 10332 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:26.877990  9477 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:26.872182  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:26.879374 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:26.884538  9634 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:26.885133  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:26.886912  9476 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:50:26.890252  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:26.891680  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:26.893909  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:26.847939 12234 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260808 01:50:26.894598  9555 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 5
I20260808 01:50:26.894454  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:26.895478 12198 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.896474  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:26.854955 12230 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.882020 12177 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:26.910480  9475 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:50:26.913643  9472 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260808 01:50:26.918520  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:26.919239  9472 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:26.919507  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:26.913838  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:26.920270  9560 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:50:26.916738  9638 raft_consensus.cc:2410] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260808 01:50:26.916519  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:26.922359  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:26.922716  9477 raft_consensus.cc:2410] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:26.910540  9554 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:26.923015  9476 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:26.919081  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:26.924700  9634 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:26.926024  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:26.926506  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:26.927392  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:26.927230 11864 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.927734  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:26.928256  9632 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:26.928962 12177 raft_consensus.cc:697] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:26.929665 11864 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:26.928920  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:26.931007  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:26.882239 12214 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:26.933856 10368 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: []
W20260808 01:50:26.911913 12185 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 2.588s	user 0.011s	sys 0.002s
I20260808 01:50:26.935895  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:26.936044  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:26.960990 12030 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:26.961793 12030 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.930336 12177 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:26.961726 12113 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:26.963677 12113 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:26.964262 12030 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:26.964912  9521 connection.cc:256] RPC call timeout handler was delayed by -2.90033757200581s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15360000000000001s for a response
W20260808 01:50:26.965394  9521 connection.cc:256] RPC call timeout handler was delayed by -2.7878375720174517s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16610000000000003s for a response
W20260808 01:50:26.965677  9521 connection.cc:256] RPC call timeout handler was delayed by -2.5619375719979871s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1912s for a response
I20260808 01:50:26.965762 12113 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:26.965991  9521 connection.cc:256] RPC call timeout handler was delayed by -2.5574375720170792s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1917s for a response
W20260808 01:50:26.966320  9521 connection.cc:256] RPC call timeout handler was delayed by -2.4907383329991717s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19820000000000002s for a response
W20260808 01:50:26.966593  9521 connection.cc:256] RPC call timeout handler was delayed by -1.599837571993703s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2981s for a response
I20260808 01:50:26.967711 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:26.968060  9521 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:26.968945 12118 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:26.962375  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:26.973839  9560 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:26.976527  9556 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:26.982143  9551 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:26.987685  9629 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:50:26.912779 12119 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:26.996790  9635 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:26.953469  9472 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:27.021162  9555 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:27.024655  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:27.025135  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:27.026189  9472 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 5: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:26.931668 12221 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:27.031616  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:27.031279  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:26.924552 12234 consensus_queue.cc:1243] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260808 01:50:27.035322  9555 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:27.038072  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:27.037106  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:27.026607  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:27.053737  9632 raft_consensus.cc:2410] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:50:27.055023  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:27.055953  9472 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:26.988292 12198 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:27.051144  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1848) took 4146 ms. Trace:
I20260808 01:50:26.992738 12214 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:27.079905  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:27.081629  9554 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:27.078922  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.086802  9638 raft_consensus.cc:2410] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260808 01:50:27.061687  9478 rpcz_store.cc:276] 0808 01:50:22.904591 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:25.457267 (+2552676us) service_pool.cc:224] Handling call
0808 01:50:25.551107 (+ 93840us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:25.551389 (+   282us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:25.551744 (+   355us) log.cc:844] Serialized 14 byte log entry
0808 01:50:26.661743 (+1109999us) spinlock_profiling.cc:243] Waited 4.85 ms on lock 0x7b18004409e0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4185ac56 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4453a6ac 00007f5d4186fe3b
0808 01:50:27.023137 (+361394us) spinlock_profiling.cc:243] Waited 4.18 ms on lock 0x7b18004409e0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4453a6ac 00007f5d4186fe3b 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667
0808 01:50:27.050226 (+ 27089us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:27.050343 (+   117us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:27.050412 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:27.050582 (+   170us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:27.051041 (+   459us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16831,"prepare.queue_time_us":14431,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":76,"raft.queue_time_us":125965,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":9506,"spinlock_wait_cycles":53493120,"thread_start_us":355890,"threads_started":2,"wal-append.queue_time_us":413183,"wal-append.run_cpu_time_us":2131,"wal-append.run_wall_time_us":1045854}
I20260808 01:50:27.089066  9554 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:50:27.091163  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:27.091907  9475 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:27.094275  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 67813.3us
I20260808 01:50:27.095351  9558 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:27.097895  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:27.101051 12292 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260808 01:50:27.109225  9560 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:27.110788  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:27.112402  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.113508 11995 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:27.114120  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.091272 12124 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260808 01:50:27.120329  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 1.1896e+06us
I20260808 01:50:27.127101  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:27.127663  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:27.130707  9521 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:27.116266  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:27.137089  9481 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:27.135944  9635 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:27.131182  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.121125  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1723) took 5838 ms. Trace:
I20260808 01:50:27.115190  9634 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:27.120078  9599 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:27.144834 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:27.145744 10368 raft_consensus.cc:2410] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260808 01:50:27.145877  9599 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:27.146994  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:27.114076 11995 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:27.105060 12231 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:27.145956  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:27.123119  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:50:27.160708  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.163527  9635 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:27.162716  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:27.154302  9472 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:27.165728  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:27.166489  9472 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 6
I20260808 01:50:27.118249 12072 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:27.170150  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.122102 12266 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:27.119007 12261 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:27.155951  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:27.169700  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:27.143165  9633 rpcz_store.cc:276] 0808 01:50:21.282618 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:23.925746 (+2643128us) service_pool.cc:224] Handling call
0808 01:50:23.966803 (+ 41057us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:24.060438 (+ 93635us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:24.060484 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:24.061365 (+   881us) log.cc:844] Serialized 26 byte log entry
0808 01:50:24.140752 (+ 79387us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:24.140813 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:24.140919 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:26.911192 (+2770273us) spinlock_profiling.cc:243] Waited 3.07 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:27.084029 (+172837us) raft_consensus.cc:1685] finished
0808 01:50:27.084060 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:27.120921 (+ 36861us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35704,"raft.queue_time_us":42660,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":339,"spinlock_wait_cycles":7072128,"thread_start_us":78804,"threads_started":1,"wal-append.queue_time_us":154823}
I20260808 01:50:27.158279 12051 consensus_queue.cc:1059] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20260808 01:50:27.173734  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:27.167075 12150 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:27.174795 12118 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:27.175782  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.174322  9599 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:27.168542  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:27.179466  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.121763 12260 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:27.209175  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:27.167270 12265 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:27.211527  9481 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:27.190240  9473 raft_consensus.cc:2410] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260808 01:50:27.174126  9478 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:27.193384 12125 raft_consensus.cc:3055] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Stepping down as leader of term 4
W20260808 01:50:27.196213 12295 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260808 01:50:27.197597  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:27.199106  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 75524.3us
I20260808 01:50:27.120455 12063 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:27.233438 12098 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:27.200135 12006 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:27.235054 11809 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:27.171166 11980 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:27.236321 11980 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:27.236805 11980 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:27.175736 11568 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:27.179301  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.211892 12214 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:27.239512  9635 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:27.187638 12072 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:27.241218  9635 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260808 01:50:27.241442  9637 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
W20260808 01:50:27.217535 12296 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260808 01:50:27.155296 12205 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.035s
I20260808 01:50:27.203210 12150 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:27.242997 12150 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:27.185825 12266 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:27.244520 12266 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:27.217056 12260 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:27.178511 11995 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:27.247517  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:27.248586 12260 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:27.248759  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 51270.2us
I20260808 01:50:27.250921 12191 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:50:27.252151 12191 raft_consensus.cc:3055] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:27.252570 12191 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:27.253615 12191 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:27.254226  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:27.248613  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:27.256788  9473 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:27.256038  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:27.220832  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 51229.7us
I20260808 01:50:27.261484  9601 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:27.232642  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 58392.2us
I20260808 01:50:27.221436  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:27.215812 12125 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:27.223520  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.225310 12036 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:27.203616 12261 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:27.271615 12125 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:27.273681 12261 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:27.263763  9635 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 01:50:27.276043 11962 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:27.276598 11962 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:27.277678 11962 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:27.277678  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:27.280300  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:27.207754 11941 raft_consensus.cc:697] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 6 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:27.282909  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:27.273208 12125 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:27.289420  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:27.293079  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:27.311471  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:27.360195  9476 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:27.360805  9481 raft_consensus.cc:2393] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 7: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:50:27.326761  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 58227.8us
W20260808 01:50:27.356854  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 95458.9us
I20260808 01:50:27.357211  9475 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
W20260808 01:50:27.358872  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 83294.7us
I20260808 01:50:27.359360 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:27.237782 11803 raft_consensus.cc:3055] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Stepping down as leader of term 4
W20260808 01:50:27.364421  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 91206.9us
W20260808 01:50:27.368444  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:27.373466  9477 raft_consensus.cc:2410] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:27.375469 12030 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:50:27.376403 12030 consensus_queue.cc:1243] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260808 01:50:27.377063 12036 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:27.378011  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:27.379544  9481 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 01:50:27.380949 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:27.367916  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 104976us
I20260808 01:50:27.383594  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:27.387146  9551 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:27.385250  9524 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:27.394026  9474 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:27.397230 11962 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:27.398222 11941 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:27.395061  9479 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
W20260808 01:50:27.395648  9637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1893) took 5246 ms (client timeout 2786 ms). Trace:
W20260808 01:50:27.422173  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 59747.5us
I20260808 01:50:27.414312  9472 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:27.423723  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:27.449144  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:27.400436  9473 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:27.400305  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:27.420261  9637 rpcz_store.cc:269] 0808 01:50:22.121762 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:22.300178 (+178416us) service_pool.cc:224] Handling call
0808 01:50:27.294540 (+4994362us) spinlock_profiling.cc:243] Waited 4.86 s on lock 0x7b54000db1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:27.367948 (+ 73408us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11176110464}
W20260808 01:50:27.460598 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 79752.7us
I20260808 01:50:27.461859 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:27.440061  9599 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:27.468142  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:27.469177  9551 raft_consensus.cc:2368] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6.
I20260808 01:50:27.371032 11803 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:27.399735 12098 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:27.399787 12030 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:27.477272  9550 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6. Ops: [4.6-5.7]
I20260808 01:50:27.393258 11867 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:27.483955  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:27.419907 12036 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:27.502271  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:27.485500  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 85258.8us
W20260808 01:50:27.491365  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:27.525305  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:27.488477  9478 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:27.527011  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:27.527139 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 65391.4us
I20260808 01:50:27.482625  9634 raft_consensus.cc:2410] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 6 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:27.481241  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:27.481551  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.528793 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:27.529356  9632 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:27.507553  9479 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 01:50:27.508525 12068 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:27.509335  9374 catalog_manager.cc:5697] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 1 to 3, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 3 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:50:27.511168  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.512575 12072 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:27.513931  9476 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260808 01:50:27.378167 11955 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.106s
I20260808 01:50:27.501611 12235 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:27.521479 11980 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:27.522774 11649 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:27.544706  9479 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:27.545305  9637 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:27.404234 12191 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:27.498317 11809 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:27.419271 12113 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:27.505013  9472 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 01:50:27.548149  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:27.532395  9630 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:27.549015  9555 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:27.481256 12006 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:27.477422 12034 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:27.477945 12099 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:27.459947 12260 raft_consensus.cc:2764] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260808 01:50:27.544174 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:27.559865  9638 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:27.470495 12265 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.075s
I20260808 01:50:27.477296 12071 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:27.523386 12119 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260808 01:50:27.551625  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:27.567193  9556 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:27.554881  9550 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260808 01:50:27.550710 12036 raft_consensus.cc:697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:27.478896 12030 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:27.533100 11867 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:27.561184  9472 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:50:27.544898  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.563939  9477 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:27.564643  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:27.575742  9552 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:27.565534  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:27.576843  9555 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:27.553668 12113 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:27.558113  9479 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:27.569011  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.579599  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:27.581910  9472 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:27.567987  9550 pending_rounds.cc:85] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 4
I20260808 01:50:27.521522 12150 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:27.583196 10332 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:27.595760 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:27.566445 12071 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:27.568300 12178 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:27.561723 11809 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:27.603101  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:27.566193 12067 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:27.595830  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:27.598980  9550 pending_rounds.cc:107] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260808 01:50:27.606859  9550 raft_consensus.cc:2889] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:27.575433  9374 catalog_manager.cc:5697] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:50:27.585085  9551 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:27.603303  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.512612 11803 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:27.573586 12235 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260808 01:50:27.570451  9444 connection.cc:256] RPC call timeout handler was delayed by -2.7885599069995806s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2786s for a response
W20260808 01:50:27.631532 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:27.600452  9472 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260808 01:50:27.636739  9472 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260808 01:50:27.642040  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:27.605214 12221 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:27.573778 12068 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:27.596488 12066 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:27.575650 12030 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:27.666360  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.667335  9632 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:27.583281 12113 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:27.564323 12099 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:27.682027  9444 connection.cc:256] RPC call timeout handler was delayed by -1.8849616110092029s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1545s for a response
W20260808 01:50:27.682696  9444 connection.cc:256] RPC call timeout handler was delayed by -1.8678616110119037s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1564s for a response
W20260808 01:50:27.683074  9444 connection.cc:256] RPC call timeout handler was delayed by -1.7688616110244766s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1674s for a response
W20260808 01:50:27.683472  9444 connection.cc:256] RPC call timeout handler was delayed by -1.4799616110103671s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1995s for a response
W20260808 01:50:27.683835  9444 connection.cc:256] RPC call timeout handler was delayed by -1.4628616110130679s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2014s for a response
W20260808 01:50:27.684206  9444 connection.cc:256] RPC call timeout handler was delayed by -1.4601616110303439s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2017s for a response
W20260808 01:50:27.684571  9444 connection.cc:256] RPC call timeout handler was delayed by -1.441261611005757s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20379999999999998s for a response
W20260808 01:50:27.684890  9444 connection.cc:256] RPC call timeout handler was delayed by -1.4187616110139061s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20630000000000004s for a response
W20260808 01:50:27.685258  9444 connection.cc:256] RPC call timeout handler was delayed by -1.2522616110218223s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22480000000000003s for a response
W20260808 01:50:27.685629  9444 connection.cc:256] RPC call timeout handler was delayed by -1.2369616110227071s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22650000000000003s for a response
W20260808 01:50:27.685988  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0254616110178176s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25s for a response
I20260808 01:50:27.689342  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:27.601493 12071 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:27.651175 12199 consensus_queue.cc:1059] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20260808 01:50:27.652560 12186 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:27.654071 12235 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:27.706403  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.701473  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:27.706463  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.711064  9637 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:27.711289  9550 raft_consensus.cc:2410] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:50:27.661654 12125 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:27.661023 11864 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:27.712886  9374 catalog_manager.cc:5697] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 4 to 5, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 5 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:27.700852  9477 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:27.662397 12068 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:27.661152 12067 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:27.691373 12261 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:27.722632  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.730888  9374 catalog_manager.cc:5697] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 3 to 4, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 4 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:50:27.732618  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:27.701426 12298 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:27.714139 11803 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:27.714368 12235 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:27.750257  9374 catalog_manager.cc:5697] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 3 to 5, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 5 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:50:27.738430  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:27.755842  9637 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:27.740195  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:27.741210 10368 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260808 01:50:27.744311  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:27.764008  9472 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:50:27.768042  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.769019  9632 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:27.729417 12006 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:27.691879 12221 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:27.813578 12221 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:27.816570  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:27.817281  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.818284  9629 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
I20260808 01:50:27.723441 12266 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:27.819649  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.821372  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.822206  9629 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
I20260808 01:50:27.775869  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:27.834120  9374 catalog_manager.cc:5697] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 2 to 4, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 4 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:50:27.850437  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:27.783464  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:27.784255 12068 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:27.787247 12099 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:27.788128 11864 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:50:27.790349 12150 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:27.795608 12299 log.cc:927] Time spent T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.193s	user 0.000s	sys 0.001s
I20260808 01:50:27.733947 12052 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.248s
I20260808 01:50:27.726226 12036 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:27.773336  9635 raft_consensus.cc:2368] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6.
I20260808 01:50:27.778187  9472 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 01:50:27.878013  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:27.839840  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.840435 12298 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:27.782696  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.844976  9479 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 4. Ops: []
W20260808 01:50:27.770990  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 81751.5us
I20260808 01:50:27.852566  9550 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:27.879106  9631 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:27.821731 12266 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:27.851492  9476 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
W20260808 01:50:27.858530  9630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1735) took 5173 ms (client timeout 1917 ms). Trace:
W20260808 01:50:27.868829  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 419896us
I20260808 01:50:27.871762  9552 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:27.874571  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:27.876997  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:27.893663  9601 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:27.835353  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:27.765333 12261 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:27.883715  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 367044us
I20260808 01:50:27.854537 10334 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:27.852133 11980 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:27.895416  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:27.881898  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 355023us
W20260808 01:50:27.892608  9630 rpcz_store.cc:269] 0808 01:50:22.684893 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:22.882116 (+197223us) service_pool.cc:224] Handling call
0808 01:50:27.848791 (+4966675us) spinlock_profiling.cc:243] Waited 4.62 s on lock 0x7b54003fa554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:27.858450 (+  9659us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10621089536}
I20260808 01:50:27.897668  9601 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:27.897833  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 434039us
I20260808 01:50:27.907171  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:27.898607  9374 catalog_manager.cc:5697] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:50:27.898567  9633 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.005s	sys 0.029s
W20260808 01:50:27.909245  9633 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.005s	sys 0.029s
I20260808 01:50:27.904688  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:27.910391  9551 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:27.902873  9634 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:27.907068  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:27.912206 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 64535.1us
I20260808 01:50:27.908834  9374 catalog_manager.cc:5697] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 1 to 5, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 5 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:27.856163 12072 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.043s
I20260808 01:50:27.934321  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:27.937546  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:27.939056  9637 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:27.941205  9524 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:27.898161 11757 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260808 01:50:27.948449 12301 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.199s	user 0.000s	sys 0.001s
I20260808 01:50:27.951617 12124 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:27.952312  9478 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:27.952860  9478 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:27.950095  9599 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:27.959656  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:27.953948  9478 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:27.961170  9478 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:27.842864 11809 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:27.942514  9524 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:27.963882  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:27.925308 12234 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:27.973852  9524 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:27.974807  9601 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:27.932387 12118 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:27.947868 12150 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:27.983207  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:27.990638  9551 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:27.991879  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:27.945547 12231 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:27.945941 12266 raft_consensus.cc:2764] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260808 01:50:27.952327 12124 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:27.982565  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:27.951650 12261 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:27.976300  9374 catalog_manager.cc:5697] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 3 to 4, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 4 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:27.966095 11980 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:28.018168  9550 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:27.969460 12265 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:28.008625 11987 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:28.021474 11867 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:28.014389 12124 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:27.975235 11568 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:27.970194 10361 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:27.990360 12119 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:28.027870  9374 catalog_manager.cc:5697] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 1 to 2, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 2 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:28.010872 12234 raft_consensus.cc:697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:28.017697 12205 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:28.018332 12051 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election won for term 6
W20260808 01:50:28.041710  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 155323us
I20260808 01:50:28.060586  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:28.003672 12118 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:28.030105 12265 raft_consensus.cc:697] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:28.032963 11980 raft_consensus.cc:697] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:28.036118 12119 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:28.036453 12125 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:28.039321 12318 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:28.052284 11568 raft_consensus.cc:697] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:28.043432 11867 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.048s
I20260808 01:50:28.091177 12051 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:28.090379 11980 consensus_queue.cc:237] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:28.095502 12305 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
W20260808 01:50:28.093710 12302 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.188s	user 0.000s	sys 0.001s
I20260808 01:50:28.095422 12265 consensus_queue.cc:237] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:28.147208  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 304) took 2103 ms. Trace:
I20260808 01:50:28.096084  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:28.096709  9478 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:50:28.096884 12298 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:28.054543  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2067) took 1141 ms. Trace:
I20260808 01:50:28.157146  9481 rpcz_store.cc:276] 0808 01:50:26.912917 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:27.459371 (+546454us) service_pool.cc:224] Handling call
0808 01:50:27.484253 (+ 24882us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:27.484481 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 4
0808 01:50:27.859322 (+374841us) log.cc:844] Serialized 14 byte log entry
0808 01:50:28.054078 (+194756us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:28.054156 (+    78us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:28.054219 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:28.054299 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:28.054465 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61038,"prepare.queue_time_us":582399,"prepare.run_wall_time_us":41,"raft.queue_time_us":5429,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":4058,"spinlock_wait_cycles":16634624,"thread_start_us":507738,"threads_started":2}
I20260808 01:50:28.097563 12119 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:28.071425  9374 catalog_manager.cc:5697] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 1 to 2, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 2 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:50:28.147886  9480 rpcz_store.cc:276] 0808 01:50:26.043908 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:26.747453 (+703545us) service_pool.cc:224] Handling call
0808 01:50:26.747823 (+   370us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:27.155345 (+407522us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:27.155376 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:27.204651 (+ 49275us) log.cc:844] Serialized 52 byte log entry
0808 01:50:27.602101 (+397450us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:27.602142 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:27.602207 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:28.146779 (+544572us) raft_consensus.cc:1685] finished
0808 01:50:28.146814 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:28.147105 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":394823,"raft.queue_time_us":407464,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":15038,"thread_start_us":377429,"threads_started":1,"wal-append.queue_time_us":555546}
I20260808 01:50:28.093160 12118 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:28.104116 12314 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:28.148773 12230 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:28.165830  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 189189us
W20260808 01:50:28.183707  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 87720.6us
I20260808 01:50:28.183254  9551 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:50:28.066684 12308 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.109s
I20260808 01:50:28.136436 12318 raft_consensus.cc:697] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:28.194536  9558 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:50:28.193320  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:28.196717  9551 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 01:50:28.167326  9374 catalog_manager.cc:5697] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 2 to 5, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 5 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:28.182276  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:28.198086  9550 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:28.198894  9552 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 01:50:28.094252 12030 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:28.191373  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:28.196202  9554 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:50:28.201721  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:28.201673  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:28.202631  9555 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:28.067780 12310 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.099s
I20260808 01:50:28.205299  9557 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 01:50:28.193171  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:28.179287 12191 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.625s	user 0.011s	sys 0.004s
I20260808 01:50:28.200227  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1740) took 1220 ms. Trace:
I20260808 01:50:28.210613  9633 rpcz_store.cc:276] 0808 01:50:26.980380 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:27.215977 (+235597us) service_pool.cc:224] Handling call
0808 01:50:27.250006 (+ 34029us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:27.250339 (+   333us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:50:27.909740 (+659401us) log.cc:844] Serialized 14 byte log entry
0808 01:50:28.199881 (+290141us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:28.199944 (+    63us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:28.200000 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:28.200064 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:28.200165 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25047,"prepare.queue_time_us":23749,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"thread_start_us":874562,"threads_started":2}
I20260808 01:50:28.166170  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2069) took 1182 ms. Trace:
W20260808 01:50:28.183208  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 206468us
I20260808 01:50:28.202927  9560 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:28.215220  9475 rpcz_store.cc:276] 0808 01:50:26.983584 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:27.475257 (+491673us) service_pool.cc:224] Handling call
0808 01:50:27.475684 (+   427us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:27.475980 (+   296us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:50:27.856178 (+380198us) log.cc:844] Serialized 14 byte log entry
0808 01:50:28.165566 (+309388us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:28.165647 (+    81us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:28.165710 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:28.165778 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:28.166085 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11944,"prepare.queue_time_us":587830,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":17322,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":9510,"thread_start_us":616182,"threads_started":2}
W20260808 01:50:28.218616  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:28.222600  9601 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:28.206836  9374 catalog_manager.cc:5697] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:50:28.185104 12266 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 01:50:28.231981  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:28.240571 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:28.243806 10366 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260808 01:50:28.268728  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 274069us
I20260808 01:50:28.274271  9550 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:28.211604  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1863) took 1078 ms. Trace:
I20260808 01:50:28.210451 10361 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:50:28.285794  9555 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:28.281682  9474 rpcz_store.cc:276] 0808 01:50:27.133670 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:27.520703 (+387033us) service_pool.cc:224] Handling call
0808 01:50:27.521051 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:27.521268 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:50:27.977591 (+456323us) log.cc:844] Serialized 14 byte log entry
0808 01:50:28.210116 (+232525us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:28.210820 (+   704us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:28.210970 (+   150us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:28.211186 (+   216us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:28.211467 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52671,"prepare.queue_time_us":569995,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":21178,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":4554,"thread_start_us":626233,"threads_started":2}
W20260808 01:50:28.291626  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:28.195746 12230 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:28.191286 12314 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:28.242014 12266 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:28.225275 12261 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:28.351155  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:28.201120  9478 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "244e080726394210b5467bf688813cf0"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:38182
W20260808 01:50:28.359934  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 166832us
I20260808 01:50:28.261804 11803 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.136s
I20260808 01:50:28.392338 12266 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:50:28.409641 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 384334us
I20260808 01:50:28.309758 12125 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.095s
I20260808 01:50:28.411437  9473 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:28.411903  9478 raft_consensus.cc:606] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:28.400403  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:28.261650 12006 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:28.270426 11568 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:28.309820 11757 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.099s
I20260808 01:50:28.352741 12331 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.102s
I20260808 01:50:28.243037 12234 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:28.418526  9481 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 7: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:50:28.413872 12312 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.176s	user 0.001s	sys 0.000s
I20260808 01:50:28.417186  9478 raft_consensus.cc:614] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:50:28.420425 10332 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:28.418218  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:28.432996  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:28.423625 12099 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
W20260808 01:50:28.439731 12329 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260808 01:50:28.437368 12221 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:28.442760 12035 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:28.443150 12221 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:28.443514 12035 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:28.443763 12221 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:28.444542 12035 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:28.442485 11987 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:28.450763 12198 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:50:28.309474 12051 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:28.452279 12323 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260808 01:50:28.452483 12071 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.738s	user 0.013s	sys 0.000s
W20260808 01:50:28.453320 12321 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.215s	user 0.000s	sys 0.001s
I20260808 01:50:28.459798  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:28.458238 12230 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:28.460978  9473 raft_consensus.cc:2393] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 5: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:28.462494  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:28.463327  9481 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:28.435083 11867 raft_consensus.cc:3055] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:28.416254 12231 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:28.453168 12071 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:28.477795 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46036.8us
I20260808 01:50:28.476758 11867 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:28.481316  9636 raft_consensus.cc:2393] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:28.482954 12177 raft_consensus.cc:3055] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:28.428876 12186 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.057s
W20260808 01:50:28.476639 12113 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.771s	user 0.013s	sys 0.002s
I20260808 01:50:28.484175 12113 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:28.485517  9550 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:28.483440 12177 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:28.485576 12071 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:28.487679 12325 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260808 01:50:28.487207 12177 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:28.488840 12177 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:28.481992 11867 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:28.493292 11867 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:28.401971 12318 consensus_queue.cc:237] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:28.519912  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:28.520180  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 302230us
I20260808 01:50:28.520710  9473 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:28.521304 12349 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:50:28.522967 12186 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:28.523574 12113 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:28.478226 12119 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:28.525960 12067 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:28.526335 12119 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:28.488863 12334 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260808 01:50:28.530611 12333 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.005s
I20260808 01:50:28.534230 12099 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:50:28.535267  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:28.536795 12333 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:28.538301  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:28.539623 12333 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260808 01:50:28.541499 12235 consensus_meta.cc:287] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.760s	user 0.010s	sys 0.001s
I20260808 01:50:28.544092  9475 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:28.544461 12030 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:28.548579  9600 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.005s	sys 0.024s
W20260808 01:50:28.549155  9600 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.005s	sys 0.025s
I20260808 01:50:28.551169  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:28.553256  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:28.542032 12235 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:28.661562 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 104002us
I20260808 01:50:28.554088  9475 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:28.663247  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:28.664134  9475 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:28.555898  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:28.637723  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 119086us
W20260808 01:50:28.676239  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 120473us
I20260808 01:50:28.678548  9521 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:28.680051  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 247178us
I20260808 01:50:28.684355  9473 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
W20260808 01:50:28.546525 12124 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.508s	user 0.012s	sys 0.000s
I20260808 01:50:28.690982  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:28.693584 10364 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:28.687582  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:28.703539  9559 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:28.730088  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:28.747390  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:28.675588 12030 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:28.686522 12260 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:28.517702 12231 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
W20260808 01:50:28.771409  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:28.693073 12119 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260808 01:50:28.844324 12350 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260808 01:50:28.825544 11864 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:28.846127 11864 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:28.848711 11864 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:28.840448  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:28.850574  9477 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:28.843047  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:28.854552  9479 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:28.859064 11980 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:28.859797 11980 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:28.761456 12334 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260808 01:50:28.698073 12331 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:28.886911 12314 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:28.693455 11941 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.172s
I20260808 01:50:28.894253 12235 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:28.552456 12186 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:28.706754 12298 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:28.952705  9475 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:28.961014  9473 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:28.965607  9477 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
W20260808 01:50:28.967190  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:28.695256 12124 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:28.975839  9481 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:28.987252  9478 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20260808 01:50:28.981369  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:28.837801 12328 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:29.002916 10334 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:28.976634  9479 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
W20260808 01:50:28.990059  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 290418us
I20260808 01:50:29.004630  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:28.995270  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:28.989302  9473 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:28.995914 11864 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:28.963714  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:29.004954  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:28.994570 12327 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:29.018011  9557 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:29.006245  9560 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 7
I20260808 01:50:28.995353 11649 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:29.007109  9476 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:29.025209  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:29.027029  9477 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:28.966400 12298 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:28.976240 12260 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 01:50:29.032451  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:29.010372 12266 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.587s	user 0.010s	sys 0.001s
I20260808 01:50:29.017467 12072 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:29.022202 12036 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.009s
I20260808 01:50:29.023376 12099 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:29.017997 12328 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.022022 11864 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.062390 12099 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:29.076661  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 398197us
I20260808 01:50:29.037569 12298 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:50:29.016598  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 244e080726394210b5467bf688813cf0 (from 6e7856bed1be43acaa821290d506ee97 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:50:29.054385 12169 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:29.062485 12150 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:29.063861 12331 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:29.123646  9556 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 2. Current term is 3. Ops: []
W20260808 01:50:29.095781 12351 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260808 01:50:29.097731 12099 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:29.097570 11864 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:29.084527 12235 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:29.050482 11649 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.030438 12327 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:29.103638 12309 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:29.099040 12328 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:29.096587 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:29.056550 12266 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:29.159106 11864 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:29.115074 12344 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20260808 01:50:29.145282 12030 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:29.169845 12309 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:29.196610 11864 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.191167 11980 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:29.145857  9521 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:29.069830 12072 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.131469 12150 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:29.218638 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 200811us
W20260808 01:50:29.222405  9480 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.008s	sys 0.058s
I20260808 01:50:29.202282 12328 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:29.139659 12331 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.158804 12235 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:29.212582  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 66838.2us
I20260808 01:50:29.234205  9521 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:29.222985  9480 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.009s	sys 0.058s
I20260808 01:50:29.151151  9560 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:29.204725 12030 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:29.224503 12318 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:29.247617  9521 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:29.224304 12231 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:29.227679 12186 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:29.207563 12309 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:29.229683 12124 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:29.238600 11980 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:29.246217 12119 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 01:50:29.264911  9636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1918) took 1785 ms (client timeout 2046 ms). Trace:
W20260808 01:50:29.266604  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 302981us
W20260808 01:50:29.310365  9636 rpcz_store.cc:269] 0808 01:50:27.473987 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:27.583679 (+109692us) service_pool.cc:224] Handling call
0808 01:50:29.258790 (+1675111us) spinlock_profiling.cc:243] Waited 870 ms on lock 0x7b54001b49d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:29.258952 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2000379136}
I20260808 01:50:29.251014 12169 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:29.281222  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 276323us
I20260808 01:50:29.242609 12310 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260808 01:50:29.236111 12328 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.262336 12231 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:29.264242 10334 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:29.282037 12177 consensus_meta.cc:287] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.793s	user 0.014s	sys 0.003s
I20260808 01:50:29.251603 12125 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.031s
I20260808 01:50:29.275794 12260 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:29.297657  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:29.288195 11867 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.794s	user 0.008s	sys 0.008s
I20260808 01:50:29.304121 11864 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:29.316435  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:29.317325  9599 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:29.320269  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:29.326571  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:29.287901 12359 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:29.331604  9560 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20260808 01:50:29.331578 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 67462.3us
I20260808 01:50:29.335583  9631 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:50:29.336328  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:29.414543  9634 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:29.335609  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:29.416280  9630 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:50:29.265583 11980 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.346608  9557 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:50:29.343076  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:29.349455  9555 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
W20260808 01:50:29.418602  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 98990.8us
I20260808 01:50:29.419898  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:29.354189 12260 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:29.358762 11649 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:29.387372 12352 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.211s	user 0.000s	sys 0.002s
I20260808 01:50:29.390880  9637 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 2.
W20260808 01:50:29.391122  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 73908.8us
I20260808 01:50:29.351085 12327 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:29.345127 12355 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:29.433945  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:29.449911  9560 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:29.447899  9634 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: [5.4-7.5]
I20260808 01:50:29.453763  9631 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:50:29.461300  9560 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260808 01:50:29.464128  9560 pending_rounds.cc:85] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 3
I20260808 01:50:29.466634  9560 pending_rounds.cc:107] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260808 01:50:29.467955  9560 raft_consensus.cc:2889] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:50:29.468979  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:29.452353 11864 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:29.359280 12266 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:29.446691 12358 log.cc:927] Time spent T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
I20260808 01:50:29.333966 12169 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.477752  9632 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: []
W20260808 01:50:29.485807  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:29.429797 11941 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:29.504649  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:29.492443  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:29.493054  9560 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260808 01:50:29.493500  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 245995us
I20260808 01:50:29.430663 12367 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:29.510922 12231 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:29.511344 12349 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:29.511616 12231 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:29.511973 12349 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:29.512889 11987 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:29.513281 12124 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:29.513515 11987 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:29.513935 12124 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.514355 12349 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:29.455219 12355 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:29.516744  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:29.476503 12327 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:29.484498 12072 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:29.501362 12119 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:29.525601  9473 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:29.549490  9632 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:29.553578  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:29.554337  9473 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:50:29.554113 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:29.517293 12235 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:29.517096 12150 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:29.498762 11649 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20260808 01:50:29.541657 12266 raft_consensus.cc:2764] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260808 01:50:29.516381 12124 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:29.502337 12331 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:29.573343 12234 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:29.525504 12375 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:29.573872 12234 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:29.574327 12234 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:29.574646 12260 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:29.575342 12362 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:29.575945 12362 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.580996 12099 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.431504 12359 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:29.500495 11864 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.584013 12359 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:29.584527 11864 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:29.586349 12221 consensus_meta.cc:287] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.142s	user 0.012s	sys 0.004s
I20260808 01:50:29.586500  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1886) took 1234 ms. Trace:
I20260808 01:50:29.586879 12221 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.587045  9480 rpcz_store.cc:276] 0808 01:50:28.351920 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:28.352091 (+   171us) service_pool.cc:224] Handling call
0808 01:50:28.396150 (+ 44059us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:28.516480 (+120330us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:50:28.545939 (+ 29459us) log.cc:844] Serialized 14 byte log entry
0808 01:50:29.236697 (+690758us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:29.239720 (+  3023us) log.cc:844] Serialized 26 byte log entry
0808 01:50:29.239896 (+   176us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:29.239938 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:29.240014 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:29.586252 (+346238us) raft_consensus.cc:1685] finished
0808 01:50:29.586278 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:29.586429 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":101087,"prepare.queue_time_us":16781,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":122138,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":3213,"thread_start_us":688959,"threads_started":1,"wal-append.queue_time_us":900903}
I20260808 01:50:29.589232 12221 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:29.588730 12234 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.561367 12119 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.591570 12234 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:29.591886 12119 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:29.593050 12169 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:29.514636 11987 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:29.526505 12318 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:29.596678 12318 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:29.598287 12328 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:29.572374  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:29.582728 12366 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
I20260808 01:50:29.526815 12355 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:29.558821 10334 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:29.599298 12369 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.154s	user 0.000s	sys 0.001s
I20260808 01:50:29.599189  9634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:50:29.631897  9634 raft_consensus.cc:606] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:29.632004  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:29.601606 12318 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:29.603606 12099 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:29.566637 12327 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:29.616298 10368 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260808 01:50:29.642103  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:29.643059  9637 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:29.603647 12231 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:29.608037 12362 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:29.636956 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 78360.9us
I20260808 01:50:29.632460  9634 raft_consensus.cc:614] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:50:29.574139 12375 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:29.610080 12371 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:29.654969  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 162743us
I20260808 01:50:29.666347  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:29.697450  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:29.698405  9560 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:29.660218  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:29.656255 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 159862us
I20260808 01:50:29.675249  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:29.734838  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:29.649891 12036 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:29.741681  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:29.750320  9636 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:29.743819  9550 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:50:29.752381  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:29.754968  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:29.755882  9554 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:29.760810  9600 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:29.679311 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:29.663703 12328 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:29.656857 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 438) took 1113 ms. Trace:
W20260808 01:50:29.766412  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 2f740cd3515641a8b2c4add8f809dd82 (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:50:29.771323  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:29.769819  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:29.774211  9480 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 7
I20260808 01:50:29.755569 12150 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:29.776361 12150 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:29.777343 10365 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:50:29.779640 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:29.701392 12359 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.071s
I20260808 01:50:29.739379 12099 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:29.772480 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:29.750128 12036 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:29.654218 12266 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.111s
I20260808 01:50:29.785089  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:29.801083 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:29.801828 12177 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:29.774224  9550 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:29.763787 12051 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:29.802498 12177 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.802711 12051 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.803077 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:29.802079 10367 raft_consensus.cc:2368] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4.
I20260808 01:50:29.805152 12051 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:29.806635  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:29.807531  9551 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:29.805152 12177 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:29.787523 10368 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 4: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:29.811409  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:29.808465  9635 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:50:29.813853  9635 raft_consensus.cc:606] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:29.816306 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:29.816627 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:29.818085  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 337148us
I20260808 01:50:29.822990  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:29.823285 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:29.824010  9472 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:29.824852  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:29.796152 12036 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:29.831429 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:29.835209 12375 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:29.835830 12370 log.cc:927] Time spent T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
W20260808 01:50:29.829339 12298 consensus_meta.cc:287] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.705s	user 0.015s	sys 0.000s
I20260808 01:50:29.837272 12298 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.837801  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1892) took 1160 ms. Trace:
I20260808 01:50:29.840539  9476 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:29.829628 12349 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.016s
I20260808 01:50:29.808259 12385 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.142s
I20260808 01:50:29.843400  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:29.844143  9634 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:29.826452 12327 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:29.846415 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:29.838487  9475 rpcz_store.cc:276] 0808 01:50:28.677614 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:28.956775 (+279161us) service_pool.cc:224] Handling call
0808 01:50:28.957215 (+   440us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:28.996897 (+ 39682us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:28.996950 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:29.019083 (+ 22133us) log.cc:844] Serialized 26 byte log entry
0808 01:50:29.259787 (+240704us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:29.259831 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:29.259895 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:29.837470 (+577575us) raft_consensus.cc:1685] finished
0808 01:50:29.837508 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:29.837712 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":155,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":28326,"thread_start_us":196323,"threads_started":1,"wal-append.queue_time_us":603878}
I20260808 01:50:29.852530  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:29.853456  9550 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:29.854835 12150 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:29.857096  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:29.858179  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:29.857744  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:29.864100  9637 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:29.870287  9600 leader_election.cc:400] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4.
I20260808 01:50:29.870735  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:29.870785  9600 leader_election.cc:403] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:50:29.872314  9634 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:29.771471 12328 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:29.874588  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:29.879169  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 54162.5us
I20260808 01:50:29.882926  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:29.885571  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 63387us
I20260808 01:50:29.794515 12099 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.848819 12298 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:29.874516 12309 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.608s	user 0.014s	sys 0.000s
I20260808 01:50:29.886837  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:29.914889  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 60198.4us
I20260808 01:50:29.697772 12371 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:29.977782 10366 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:29.954907 12309 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:29.946625  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59931.1us
I20260808 01:50:29.964044  9480 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:50:29.964635  9472 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:30.015873  9629 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 5: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:29.921641 12150 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 4
I20260808 01:50:29.935066  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2073) took 1394 ms. Trace:
I20260808 01:50:29.955507 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:29.934223  9637 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:30.058373  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:30.061071  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:29.973814  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:30.007076  9636 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:30.057860  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:29.920372 12373 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.072s	user 0.001s	sys 0.001s
I20260808 01:50:29.970923  9635 raft_consensus.cc:3055] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:30.055627  9478 rpcz_store.cc:276] 0808 01:50:28.540894 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:28.541123 (+   229us) service_pool.cc:224] Handling call
0808 01:50:28.986997 (+445874us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:28.987890 (+   893us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:29.934786 (+946896us) spinlock_profiling.cc:243] Waited 445 ms on lock 0x7b540009a1d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:29.934987 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1024262528}
I20260808 01:50:30.018450 12309 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:30.075316 12396 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 01:50:30.084836 12365 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.734s	user 0.003s	sys 0.003s
I20260808 01:50:30.086768 12118 consensus_queue.cc:1059] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260808 01:50:30.079663 12380 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.283s	user 0.000s	sys 0.001s
I20260808 01:50:30.087108 12376 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:50:30.087507 12375 raft_consensus.cc:3055] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:30.016934 12355 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:30.094575 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 139139us
I20260808 01:50:30.056473  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:30.017891 12371 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:29.919204 12367 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260808 01:50:30.099884  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:30.099496  9635 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:29.921795 12266 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:30.029978 12310 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:30.113761  9521 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:30.116104  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:30.118860  9632 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.566s	user 0.013s	sys 0.000s
I20260808 01:50:30.121708  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:30.126802  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 107959us
I20260808 01:50:30.089542 12231 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:30.079560 12354 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:29.977401 12381 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.153s
I20260808 01:50:30.136812  9635 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:30.161356  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:30.161294 12380 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
W20260808 01:50:30.154387  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:30.122434 12375 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:30.173964  9635 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:30.173316  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1894) took 1207 ms. Trace:
W20260808 01:50:30.174891  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 61175.6us
I20260808 01:50:30.175192  9477 rpcz_store.cc:276] 0808 01:50:28.965465 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:29.050949 (+ 85484us) service_pool.cc:224] Handling call
0808 01:50:29.071549 (+ 20600us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:29.105576 (+ 34027us) raft_consensus.cc:1534] Early marking committed up to index 5
0808 01:50:29.132821 (+ 27245us) log.cc:844] Serialized 14 byte log entry
0808 01:50:29.674833 (+542012us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:29.690512 (+ 15679us) log.cc:844] Serialized 26 byte log entry
0808 01:50:29.690692 (+   180us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:29.690730 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:29.690801 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:30.173028 (+482227us) raft_consensus.cc:1685] finished
0808 01:50:30.173058 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:30.173228 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27056,"prepare.queue_time_us":15459,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":97753,"raft.run_cpu_time_us":436,"raft.run_wall_time_us":38056,"thread_start_us":461109,"threads_started":2,"wal-append.queue_time_us":588127}
I20260808 01:50:30.132444 12367 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:30.179898  9637 raft_consensus.cc:2410] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:30.184590 12393 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:30.185226 12099 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:29.771188 10361 rpcz_store.cc:276] 0808 01:50:28.543085 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:28.543245 (+   160us) service_pool.cc:224] Handling call
0808 01:50:28.543667 (+   422us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:28.544210 (+   543us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:28.544252 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:28.545047 (+   795us) log.cc:844] Serialized 52 byte log entry
0808 01:50:28.874050 (+329003us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:28.874119 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:28.874253 (+   134us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:29.637010 (+762757us) raft_consensus.cc:1685] finished
0808 01:50:29.638584 (+  1574us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:29.653549 (+ 14965us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14575,"raft.queue_time_us":8723,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":24753,"thread_start_us":315751,"threads_started":1,"wal-append.queue_time_us":558310}
W20260808 01:50:30.186399 12406 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260808 01:50:30.187116 12230 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:30.187577 12230 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:30.187940 12230 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:30.191152 12400 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.089s	user 0.001s	sys 0.001s
I20260808 01:50:30.162698 12266 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:30.181385  9632 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260808 01:50:30.193356  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:30.186422 12328 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:30.194124  9478 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:30.194587 11955 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:30.196158 12030 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:30.165699 12388 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:30.198880 12036 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:30.202710 12068 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:30.198834  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2080) took 1497 ms. Trace:
I20260808 01:50:30.203970  9479 rpcz_store.cc:276] 0808 01:50:28.701956 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:29.007659 (+305703us) service_pool.cc:224] Handling call
0808 01:50:29.034124 (+ 26465us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:29.346956 (+312832us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:29.347003 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:29.347792 (+   789us) log.cc:844] Serialized 52 byte log entry
0808 01:50:29.815260 (+467468us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:30.103368 (+288108us) log.cc:844] Serialized 14 byte log entry
0808 01:50:30.105372 (+  2004us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:30.105481 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:30.198548 (+ 93067us) raft_consensus.cc:1685] finished
0808 01:50:30.198568 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:30.198764 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12036,"prepare.queue_time_us":385088,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":91722,"raft.run_cpu_time_us":434,"raft.run_wall_time_us":18362,"thread_start_us":990631,"threads_started":3,"wal-append.queue_time_us":848136}
I20260808 01:50:30.206146  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:30.209867  9479 raft_consensus.cc:2368] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4.
I20260808 01:50:30.207614  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2162) took 2019 ms. Trace:
I20260808 01:50:30.214871  9556 rpcz_store.cc:276] 0808 01:50:28.187779 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:28.189679 (+  1900us) service_pool.cc:224] Handling call
0808 01:50:28.203424 (+ 13745us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:29.125154 (+921730us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:30.207316 (+1082162us) spinlock_profiling.cc:243] Waited 920 ms on lock 0x7b54000de154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:30.207544 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2115807104}
I20260808 01:50:30.206195  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:30.216712  9473 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:30.217005  9635 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:50:30.172294 12298 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:30.207558 12375 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:30.218101  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:30.221586  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:30.160986 12398 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:30.235795  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2171) took 1255 ms. Trace:
I20260808 01:50:30.242810  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:30.218113 12310 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:30.199800 12388 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:30.183892 12150 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4.
I20260808 01:50:30.243753  9558 rpcz_store.cc:276] 0808 01:50:28.980460 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:28.980631 (+   171us) service_pool.cc:224] Handling call
0808 01:50:28.986048 (+  5417us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:28.998750 (+ 12702us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:28.998796 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:28.999328 (+   532us) log.cc:844] Serialized 26 byte log entry
0808 01:50:29.243019 (+243691us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:29.243067 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:29.243152 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:30.084100 (+840948us) spinlock_profiling.cc:243] Waited 8.02 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:30.128637 (+ 44537us) spinlock_profiling.cc:243] Waited 4.69 ms on lock 0x7b48000c03d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:50:30.235510 (+106873us) raft_consensus.cc:1685] finished
0808 01:50:30.235536 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:30.235705 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12161,"raft.queue_time_us":23477,"raft.run_cpu_time_us":5304,"raft.run_wall_time_us":1114077,"spinlock_wait_cycles":29226624,"thread_start_us":220178,"threads_started":1,"wal-append.queue_time_us":279037}
I20260808 01:50:30.261667  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:30.234385 12191 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:30.174299 12231 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:30.213204 12367 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:30.273432  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:30.268750 12191 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:30.278321 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:30.260177 12375 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:30.299548 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:30.311553 10365 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:30.296799  9473 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:30.257910 12398 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:30.274791 12261 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:30.275496 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:30.357491 12261 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:30.357721 10363 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:30.369848  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 278) took 1056 ms. Trace:
I20260808 01:50:30.335717 12191 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:30.343994 12314 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:30.350631  9638 raft_consensus.cc:2410] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:50:30.370512  9633 rpcz_store.cc:276] 0808 01:50:29.313033 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:29.313284 (+   251us) service_pool.cc:224] Handling call
0808 01:50:29.321622 (+  8338us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:29.476950 (+155328us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:29.476994 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:29.477754 (+   760us) log.cc:844] Serialized 52 byte log entry
0808 01:50:29.679130 (+201376us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:29.679178 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:30.367503 (+688325us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b540003c0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:30.367570 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:30.367638 (+    68us) raft_consensus.cc:1685] finished
0808 01:50:30.367660 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:30.369743 (+  2083us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":34226,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":115,"spinlock_wait_cycles":355879680,"thread_start_us":143698,"threads_started":1,"wal-append.queue_time_us":305036}
I20260808 01:50:30.367624 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:30.388356 11757 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:30.417812 12118 consensus_queue.cc:1243] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20260808 01:50:30.435204  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 213728us
I20260808 01:50:30.440577  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:30.392266 12314 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:30.457348  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:30.418603 12355 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:30.470557  9559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 2166) took 2273 ms (client timeout 1924 ms). Trace:
W20260808 01:50:30.471311  9559 rpcz_store.cc:269] 0808 01:50:28.189219 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:28.191089 (+  1870us) service_pool.cc:224] Handling call
0808 01:50:30.462092 (+2271003us) spinlock_profiling.cc:243] Waited 502 ms on lock 0x7b54000b45d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:30.462339 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1155412224}
I20260808 01:50:30.433245 11757 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:30.475044 12388 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:30.503451  9523 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:30.501224 12333 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:30.508507 12333 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:30.511234  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 329) took 1535 ms. Trace:
I20260808 01:50:30.511734  9476 rpcz_store.cc:276] 0808 01:50:28.975441 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:29.042303 (+ 66862us) service_pool.cc:224] Handling call
0808 01:50:29.047959 (+  5656us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:29.841080 (+793121us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:30.510814 (+669734us) spinlock_profiling.cc:243] Waited 792 ms on lock 0x7b5400093dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:30.511149 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1822207872}
I20260808 01:50:30.502301 12030 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:30.515285 12030 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:30.503640 12421 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:30.516856 12421 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:30.518478 12030 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:30.519018 12398 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:30.491315 12230 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:30.519775 12314 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:30.515590 12376 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:30.520321 12376 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:30.520640 11980 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:30.521711 12230 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:30.522927 12205 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:30.522588 12261 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:30.523340 12205 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:30.524329 12205 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:30.525360 12261 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:30.520714 12376 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:30.526186 12261 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:30.531368 10364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 521) took 2960 ms. Trace:
W20260808 01:50:30.534924  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 234678us
I20260808 01:50:30.505280  9523 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:30.535609 10364 rpcz_store.cc:276] 0808 01:50:27.571244 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:27.574887 (+  3643us) service_pool.cc:224] Handling call
0808 01:50:27.597160 (+ 22273us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:28.700229 (+1103069us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:29.572250 (+872021us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b54004e00a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490b1a0 00007f5d44996de2 00007f5d449931b6 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:50:30.531082 (+958832us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540041f0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:30.531292 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2538110080}
I20260808 01:50:30.542346 12266 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:30.543388 12266 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:30.544060  9551 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:30.543927 12266 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:30.544348  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:30.544777  9558 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:50:30.545089  9554 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:30.545931  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:30.547605  9550 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260808 01:50:30.548079  9550 pending_rounds.cc:85] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 2
I20260808 01:50:30.548339  9550 pending_rounds.cc:107] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260808 01:50:30.548611  9550 raft_consensus.cc:2889] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:30.550081  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:30.550833  9557 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:30.526486 12191 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:30.551610  9479 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:30.552039 12393 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:30.552563 12393 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:30.553004 12393 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:30.553807  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:30.554471  9479 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:30.555361  9479 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:38182
I20260808 01:50:30.555874  9479 raft_consensus.cc:606] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:30.556205  9479 raft_consensus.cc:614] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:50:30.556509 12421 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:30.557885  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:30.558761  9556 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:30.558665  9554 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:30.557528  9479 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 01:50:30.560096  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:30.560843  9558 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:30.561033  9479 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:30.562343  9479 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:50:30.564190 12429 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:30.566864  9479 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 01:50:30.568751  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:30.564785  9600 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:30.572053  9479 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:50:30.546346 12266 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:30.574481  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 69295.3us
I20260808 01:50:30.581267  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:30.582265  9551 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:50:30.586225  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:30.588943  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:30.589316  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:30.590041  9552 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:30.590497  9558 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 5: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:30.561920 12372 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.749s	user 0.001s	sys 0.001s
I20260808 01:50:30.591638  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:30.592007  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:30.526625 12261 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:30.597342  9560 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260808 01:50:30.594445  9552 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:30.597033  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 9949664c54584858ae8fc74c9dc07a4a (from 6e7856bed1be43acaa821290d506ee97 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:50:30.628461  9556 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:50:30.632910  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:30.633821  9551 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:30.525362 12205 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:30.561959 12231 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:30.656422  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2078) took 1965 ms. Trace:
I20260808 01:50:30.682123  9481 rpcz_store.cc:276] 0808 01:50:28.691011 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:28.987260 (+296249us) service_pool.cc:224] Handling call
0808 01:50:28.987762 (+   502us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:29.051609 (+ 63847us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:29.051660 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:29.107008 (+ 55348us) log.cc:844] Serialized 26 byte log entry
0808 01:50:29.330544 (+223536us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:29.330607 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:29.330727 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:30.561211 (+1230484us) spinlock_profiling.cc:243] Waited 2.83 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:30.656062 (+ 94851us) raft_consensus.cc:1685] finished
0808 01:50:30.656091 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:30.656340 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39668,"raft.queue_time_us":74461,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":15031,"spinlock_wait_cycles":6500096,"thread_start_us":176309,"threads_started":1,"wal-append.queue_time_us":574097}
I20260808 01:50:30.625370 12261 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:30.630437  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260808 01:50:30.573372 12314 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:30.593288 12421 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:30.706465  9557 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:30.596822 12419 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:30.533946 12367 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:30.629142 12333 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:30.720732 12376 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:30.698541  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:50:30.728175  9552 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
W20260808 01:50:30.731585  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 291084us
I20260808 01:50:30.733752  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:30.736865  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:30.707919 12191 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:30.739907  9444 leader_election.cc:400] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6.
I20260808 01:50:30.760866  9444 leader_election.cc:403] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260808 01:50:30.576434 12429 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:30.711380 12419 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:30.768929 12419 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:30.771658  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:50:30.775792  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:30.776181 11757 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:30.777138  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:30.777336 12266 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6.
I20260808 01:50:30.778169 12266 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:30.778565 12266 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:30.778959 12266 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:30.779320 12006 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:30.779850  9638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1926) took 3049 ms (client timeout 2048 ms). Trace:
W20260808 01:50:30.780293  9638 rpcz_store.cc:269] 0808 01:50:27.730809 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:27.739541 (+  8732us) service_pool.cc:224] Handling call
0808 01:50:30.779603 (+3040062us) spinlock_profiling.cc:243] Waited 2.5 s on lock 0x7b54001b3fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:30.779756 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5752926464}
I20260808 01:50:30.780891  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:30.781512  9556 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:30.783010  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:30.783645  9472 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:30.785398  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:30.786077  9560 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:30.790210  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:30.791290  9560 raft_consensus.cc:2410] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 6 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:50:30.795555  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:30.796653  9555 raft_consensus.cc:2410] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:30.781464 12375 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:30.804245  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:30.806260  9599 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:30.783545 10334 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:30.808323 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:30.809448  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:30.810348  9558 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:30.804430  9556 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:30.820837  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:30.821600  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:30.707535 12314 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:30.824518 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:30.827067  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:30.830276  9560 raft_consensus.cc:2410] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260808 01:50:30.827486  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:30.830739  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:30.831207  9638 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:30.831517  9635 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:30.828301  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:30.799414 12367 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:30.834350  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:30.834992  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:30.837284  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:30.839742 12376 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:30.841239  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:30.798234 12393 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:30.846253  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:30.847091  9480 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:30.841984  9550 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:30.835726  9556 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
I20260808 01:50:30.845153  9630 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 8: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:30.760627 12425 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:30.851085  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:30.855397  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:30.822230 12421 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:30.827353 12371 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:30.806403 12435 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:30.828663 12355 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:30.823935 12068 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election won for term 6
W20260808 01:50:30.907264  9444 connection.cc:256] RPC call timeout handler was delayed by -2.0400622999877669s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15000000000000002s for a response
I20260808 01:50:30.898064 11980 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:50:30.907996  9444 connection.cc:256] RPC call timeout handler was delayed by -1.8690622999856714s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.169s for a response
I20260808 01:50:30.908558 11980 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:30.908576  9444 connection.cc:256] RPC call timeout handler was delayed by -1.8510623000038322s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.171s for a response
W20260808 01:50:30.909528  9444 connection.cc:256] RPC call timeout handler was delayed by -1.7592622999800369s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18120000000000003s for a response
W20260808 01:50:30.909922  9444 connection.cc:256] RPC call timeout handler was delayed by -1.6620622999907937s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.192s for a response
W20260808 01:50:30.910287  9444 connection.cc:256] RPC call timeout handler was delayed by -1.5837622999970336s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20070000000000002s for a response
I20260808 01:50:30.825131 12364 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.073s
I20260808 01:50:30.870458 12327 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.042s
I20260808 01:50:30.914074 10360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 531) took 1114 ms. Trace:
I20260808 01:50:30.914345 11757 raft_consensus.cc:993] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:50:30.914799 12434 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:30.915120 11757 raft_consensus.cc:1081] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Signalling peer 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:50:30.914567 10360 rpcz_store.cc:276] 0808 01:50:29.799983 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:29.800156 (+   173us) service_pool.cc:224] Handling call
0808 01:50:29.800585 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:30.054664 (+254079us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:30.057593 (+  2929us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:30.075457 (+ 17864us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:30.086337 (+ 10880us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:30.913771 (+827434us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b540046c154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:30.913826 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:30.913990 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":504413440}
I20260808 01:50:30.919766  9480 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
W20260808 01:50:30.896608  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 47454.4us
I20260808 01:50:30.896821  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:30.924819  9630 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:30.925043 12434 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:30.853536  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 44180.9us
I20260808 01:50:30.898767  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:30.841193 12363 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:30.898797  9635 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:30.828848 12375 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:30.840893 12072 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.065s
I20260808 01:50:30.846055 12381 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:30.905329 12349 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:30.903918  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:30.927796  9601 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:30.832671 12344 consensus_queue.cc:1059] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260808 01:50:30.929500  9599 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:50:30.997716  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 69076.8us
W20260808 01:50:30.910547  9444 connection.cc:256] RPC call timeout handler was delayed by -1.5513623000006191s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20430000000000004s for a response
I20260808 01:50:30.990052  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:30.987691  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:31.000110  9481 raft_consensus.cc:2410] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:31.000257  9480 raft_consensus.cc:2410] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260808 01:50:30.990918  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:31.001281  9475 raft_consensus.cc:2410] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:30.779860 12006 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:30.939213  9638 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:31.025077  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:31.027889  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:31.028151  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:31.028961  9481 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:31.030836  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:30.940279  9472 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:50:30.891567 12425 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:31.033387  9481 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:30.951700  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:30.958083  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:31.035315  9477 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:30.959409  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:30.959815 12434 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:30.961694 12314 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:30.939823 12363 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:30.907635 12435 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:30.898460 11809 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20260808 01:50:30.968143 12393 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:30.942158 11980 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:30.975622 12433 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
W20260808 01:50:31.076900  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 49090.7us
I20260808 01:50:30.947016 12364 consensus_queue.cc:1243] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260808 01:50:31.077975 12364 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:31.078495 12364 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:31.038839 12422 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:31.079013 12364 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:31.079183 12422 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:31.079319  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:31.079715 12422 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:30.948042 12072 consensus_queue.cc:1243] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 2.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260808 01:50:30.903393 12421 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:30.946996  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:30.999369  9444 connection.cc:256] RPC call timeout handler was delayed by -1.5486622999887913s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2046s for a response
I20260808 01:50:30.982844 12429 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:31.084087  9444 connection.cc:256] RPC call timeout handler was delayed by -1.5468622999906074s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2048s for a response
W20260808 01:50:31.086835  9444 connection.cc:256] RPC call timeout handler was delayed by -1.5468622999906074s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2048s for a response
W20260808 01:50:31.087424  9444 connection.cc:256] RPC call timeout handler was delayed by -1.4604622999904677s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21440000000000003s for a response
W20260808 01:50:31.087815  9444 connection.cc:256] RPC call timeout handler was delayed by -1.4370622999849729s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.217s for a response
W20260808 01:50:31.088124  9444 connection.cc:256] RPC call timeout handler was delayed by -1.3884623000049032s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22240000000000004s for a response
I20260808 01:50:30.996256  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:30.994450 10332 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:31.037467  9556 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:31.038470  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:31.090034  9552 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:31.051620  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:31.052892  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:31.053535  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:31.094522  9475 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:31.065651  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:31.095890  9480 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:30.947947 12381 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:31.067305  9524 connection.cc:256] RPC call timeout handler was delayed by -1.0698846210143529s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1685s for a response
W20260808 01:50:31.099035  9524 connection.cc:256] RPC call timeout handler was delayed by -1.0456939930154476s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1516s for a response
I20260808 01:50:30.989183 12318 raft_consensus.cc:3055] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:30.948356 12068 raft_consensus.cc:697] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:31.038798 12425 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:31.069407  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:31.065286 12363 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:31.075191 12435 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:31.070551  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:30.850519 12354 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260808 01:50:31.116783  9599 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:31.116456 11809 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260808 01:50:30.997145  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 69435.4us
I20260808 01:50:31.084215  9558 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
W20260808 01:50:31.088433  9444 connection.cc:256] RPC call timeout handler was delayed by -1.3668622999975923s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22480000000000003s for a response
W20260808 01:50:31.119480  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1319622999872081s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2509s for a response
W20260808 01:50:31.119846  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1220622999826446s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.252s for a response
W20260808 01:50:31.120126  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0626622999843676s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2586s for a response
W20260808 01:50:31.120460  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0005623000033665s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2655s for a response
I20260808 01:50:31.121693  9444 leader_election.cc:400] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6.
I20260808 01:50:31.066382 12349 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260808 01:50:31.086107 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 244370us
I20260808 01:50:31.126895 10334 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:30.994793 12375 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:31.103459 12381 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:31.128192 10334 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:31.105211  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:31.118552 12327 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.010s
I20260808 01:50:31.139367  9524 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:31.101058 11568 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:31.090754 12398 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20260808 01:50:31.101971 12318 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:31.130707 12420 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260808 01:50:31.133757  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:31.134573  9472 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:50:31.126194 12349 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:31.136775  9552 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:50:31.144198 11568 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:31.137390  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:31.137811  9601 leader_election.cc:400] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 3. Current term is 4.
I20260808 01:50:31.178311  9480 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 7
W20260808 01:50:31.178601  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 43943.4us
I20260808 01:50:31.148082  9630 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
I20260808 01:50:31.131356  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:31.147482  9524 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:31.145632 12437 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:31.149814 12314 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:31.097442 12419 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:31.148900 12006 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:31.136353  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:31.181821  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:31.189740  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:31.188894  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:31.159415 12349 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:31.187132 12205 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:31.202936  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:31.137063 12068 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:31.203871  9635 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:31.193385 12448 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:31.177008 12393 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:31.205928 12440 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260808 01:50:31.202033  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:31.202361 11867 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:31.187948 12437 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:31.194036  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:31.160722 12371 raft_consensus.cc:697] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:31.217881  9521 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:31.219151  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:31.219578  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:31.220330  9633 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:31.221855  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:31.205886 12448 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:31.299664  9477 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260808 01:50:31.188956 11568 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:31.214913  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 83666.6us
W20260808 01:50:31.233837  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 97779.7us
I20260808 01:50:31.223405  9634 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 5: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:31.306844  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:31.330453  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:31.353289  9599 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:31.366009  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:31.368221  9630 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:31.194726 12333 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:31.375499  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:31.376574  9472 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:31.375442  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:31.207647 11980 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:31.368948  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:31.388995  9480 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
W20260808 01:50:31.399175  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46025.1us
I20260808 01:50:31.396595  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:31.224203 12052 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:31.215194 12437 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:31.469130  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 337) took 2123 ms. Trace:
I20260808 01:50:31.406689  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:31.469825  9474 rpcz_store.cc:276] 0808 01:50:29.346134 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:29.579171 (+233037us) service_pool.cc:224] Handling call
0808 01:50:29.579647 (+   476us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:29.580035 (+   388us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:29.580064 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:30.717312 (+1137248us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b2c00134ff8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:50:30.717768 (+   456us) log.cc:844] Serialized 26 byte log entry
0808 01:50:30.849199 (+131431us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:30.849235 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:30.849306 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:31.447416 (+598110us) raft_consensus.cc:1685] finished
0808 01:50:31.447440 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:31.469012 (+ 21572us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1712,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"spinlock_wait_cycles":6406400,"thread_start_us":131111,"threads_started":1,"wal-append.queue_time_us":324818}
I20260808 01:50:31.471581  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:31.390931 12266 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.612s	user 0.012s	sys 0.001s
I20260808 01:50:31.206782 12393 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:31.378518 12429 raft_consensus.cc:2764] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260808 01:50:31.390755 12367 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:31.310765 12448 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:31.390690 12434 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:31.449000 12344 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:31.430603  9630 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:50:31.452705 12450 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:31.248159 12355 raft_consensus.cc:697] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:50:31.444144 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 298594us
W20260808 01:50:31.445868  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 228045us
I20260808 01:50:31.403540 12327 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:50:31.420472 12261 consensus_meta.cc:287] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.731s	user 0.012s	sys 0.004s
I20260808 01:50:31.462313 12421 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:31.327005 12006 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:50:31.451112 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 456453us
I20260808 01:50:31.455343 10366 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260808 01:50:31.421742 12376 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:31.458122 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:31.464641  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:31.403492 12432 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20260808 01:50:31.472445  9477 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:31.479024 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:31.329293 12318 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:31.398442 12420 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:31.441177  9472 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
W20260808 01:50:31.469640 12417 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 1.033s	user 0.006s	sys 0.000s
I20260808 01:50:31.407685 12418 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260808 01:50:31.501919  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 286752us
I20260808 01:50:31.507308 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:31.510615 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:31.528946 10365 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:31.484198 12052 raft_consensus.cc:697] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:31.522432 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:31.522698 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:31.484694 12461 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:50:31.532797 12447 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.202s	user 0.000s	sys 0.000s
W20260808 01:50:31.533972 12443 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260808 01:50:31.534404 12421 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:31.538604 12067 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:31.540287 12036 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:31.597683 12067 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:31.509876 12261 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:31.540875 12318 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:31.506958 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:31.544754 12442 log.cc:927] Time spent T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.476s	user 0.002s	sys 0.000s
I20260808 01:50:31.545197 12418 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:31.456398 12371 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:31.488435 12434 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:31.554772 12420 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:31.554965 11568 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:31.558876 12006 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:31.497645 12344 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:31.561817 12419 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:31.562889 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:31.564240 12393 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:31.660321  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2082) took 1716 ms. Trace:
I20260808 01:50:31.661526  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2188) took 1119 ms. Trace:
I20260808 01:50:31.660818  9476 rpcz_store.cc:276] 0808 01:50:29.943506 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:30.512895 (+569389us) service_pool.cc:224] Handling call
0808 01:50:30.513268 (+   373us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:30.730064 (+216796us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:30.730474 (+   410us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:30.735239 (+  4765us) log.cc:844] Serialized 52 byte log entry
0808 01:50:31.092688 (+357449us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:31.092733 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:31.092803 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:31.660084 (+567281us) raft_consensus.cc:1685] finished
0808 01:50:31.660106 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:31.660254 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":50095,"raft.run_cpu_time_us":331,"raft.run_wall_time_us":332,"thread_start_us":462855,"threads_started":2,"wal-append.queue_time_us":404156}
I20260808 01:50:31.567157 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:31.591642  9524 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:31.505079 12327 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:31.506340 12450 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:31.483455 12266 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:31.601135  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:31.605747  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:31.549361 12355 consensus_queue.cc:237] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:31.526625 12376 raft_consensus.cc:697] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:31.658087 12344 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:31.615670 12418 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:31.707391  9631 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 7: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:31.614946 12457 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:31.669103 12421 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:31.656569 12464 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:31.742332 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 235077us
I20260808 01:50:31.598876 12461 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:31.747152  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 155632us
W20260808 01:50:31.759954  9629 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.570s	user 0.019s	sys 0.006s
I20260808 01:50:31.764117  9629 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:31.662312  9559 rpcz_store.cc:276] 0808 01:50:30.541972 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:30.555195 (+ 13223us) service_pool.cc:224] Handling call
0808 01:50:30.555537 (+   342us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:30.597497 (+ 41960us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:30.597537 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:30.636967 (+ 39430us) log.cc:844] Serialized 52 byte log entry
0808 01:50:30.845859 (+208892us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:30.898883 (+ 53024us) log.cc:844] Serialized 14 byte log entry
0808 01:50:30.899526 (+   643us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:30.899606 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:31.544120 (+644514us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:31.661124 (+117004us) raft_consensus.cc:1685] finished
0808 01:50:31.661271 (+   147us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:31.661446 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18007,"prepare.queue_time_us":17599,"prepare.run_wall_time_us":52,"raft.queue_time_us":32164,"raft.run_cpu_time_us":331,"raft.run_wall_time_us":43707,"spinlock_wait_cycles":70201856,"thread_start_us":118717,"threads_started":1,"wal-append.queue_time_us":299627}
I20260808 01:50:31.712913 12419 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:31.774188  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:31.774601 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:31.781322  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:31.782828  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:31.775669  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:31.785857 10332 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:31.805787  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 342592us
I20260808 01:50:31.813349  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:31.817540  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:31.822228  9472 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:50:31.786453 12261 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:31.799362 12457 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:31.793741 12333 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:31.872951  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:31.792019 12036 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20260808 01:50:31.838426  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:31.808566 12459 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:31.772683 12421 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:31.899441  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 61160.7us
I20260808 01:50:31.900357  9601 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:31.901170  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:31.902156  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:31.893322 12030 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:31.904089  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:31.837752 11867 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:31.798591 12460 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.119s
I20260808 01:50:31.836632 11568 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:31.843578 12457 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:31.767475 12464 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:31.925927  9633 consensus_meta.cc:287] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.705s	user 0.018s	sys 0.012s
I20260808 01:50:31.926368  9633 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:50:31.792943 12419 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:31.735590 12052 consensus_queue.cc:237] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:31.904695 11757 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:31.907927 12098 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:31.913103 12051 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:31.929390  9631 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: [5.4-7.5]
I20260808 01:50:31.909924 12030 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:31.925786 12309 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:31.911347  9554 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:53208
I20260808 01:50:31.910215 11809 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:50:31.927448 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 102436us
I20260808 01:50:31.920918 12459 raft_consensus.cc:697] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:50:31.931869 12449 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.241s	user 0.000s	sys 0.002s
I20260808 01:50:31.948844  9479 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:31.801035 12067 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:31.940593  9554 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
W20260808 01:50:31.975940  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 43504.8us
I20260808 01:50:31.965792  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:31.979496  9554 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:50:31.992621  9555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 1812) took 1179 ms (client timeout 1515 ms). Trace:
I20260808 01:50:31.922612 12248 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.028s
W20260808 01:50:31.946771  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 495552us
I20260808 01:50:31.952265 12467 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:32.008404 12467 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:32.009991 12419 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:31.938225 12469 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.015s
W20260808 01:50:31.957243 12446 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.557s	user 0.001s	sys 0.001s
I20260808 01:50:31.957767 12459 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:31.958909 12439 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.868s	user 0.002s	sys 0.007s
W20260808 01:50:31.959384 12465 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260808 01:50:31.959138 11867 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:32.031339 10332 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:31.928491 12457 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:31.948161 12434 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:31.902045 12376 consensus_queue.cc:237] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:31.993199  9555 rpcz_store.cc:269] 0808 01:50:30.799349 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:30.799571 (+   222us) service_pool.cc:224] Handling call
0808 01:50:31.965625 (+1166054us) spinlock_profiling.cc:243] Waited 2.03 ms on lock 0x7b5800003c98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d4185589e 000000000038d209 00007f5d49914940 00007f5d499dced4 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:50:31.978739 (+ 13114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4713728}
I20260808 01:50:31.939366 12051 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:31.946951 11809 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:31.988448 12470 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260808 01:50:31.836792 12450 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:31.998350  9601 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:31.944967 12420 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:32.018595  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.019s	sys 0.044s
W20260808 01:50:32.083081  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.019s	sys 0.044s
W20260808 01:50:31.998469 12363 consensus_meta.cc:287] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.872s	user 0.007s	sys 0.012s
I20260808 01:50:31.962807  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:32.032442  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 268290us
I20260808 01:50:31.989324 11568 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.078s
I20260808 01:50:32.085678  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:32.059471 10365 consensus_meta.cc:287] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.530s	user 0.004s	sys 0.019s
W20260808 01:50:32.081663 10332 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.971s (SENT) [suppressed 10 similar messages]
W20260808 01:50:31.939999  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 488572us
I20260808 01:50:32.014058 12266 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:32.090806  9524 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:32.087430 10365 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:32.027843 12327 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:32.092754  9524 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:32.093768  9554 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:32.106660  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 108388us
I20260808 01:50:32.081185 12051 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:32.075560 12473 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:32.080201 12451 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:31.989811 12333 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:32.068955 12452 consensus_queue.cc:1059] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260808 01:50:32.082240 12419 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:32.123096  9554 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:32.125347  9523 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:32.126600  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:32.116482 12437 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.630s	user 0.007s	sys 0.005s
I20260808 01:50:32.104884 12461 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:32.120318 12448 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.629s	user 0.013s	sys 0.000s
I20260808 01:50:32.104835 12467 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:32.133433  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:32.124363 12480 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:32.121613  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 337) took 1083 ms. Trace:
I20260808 01:50:32.120782 12472 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:32.179893  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:32.156334  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 350) took 1016 ms. Trace:
I20260808 01:50:32.134716 12310 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:32.091161 12363 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:32.087086 12030 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:32.180761  9551 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:32.145141 12036 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:32.182981  9557 rpcz_store.cc:276] 0808 01:50:31.054566 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:31.055293 (+   727us) service_pool.cc:224] Handling call
0808 01:50:31.055742 (+   449us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:31.056054 (+   312us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:50:31.220355 (+164301us) log.cc:844] Serialized 14 byte log entry
0808 01:50:32.121016 (+900661us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:32.121123 (+   107us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:32.121188 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:32.121291 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:32.121528 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46614,"prepare.queue_time_us":428853,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":277536,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":5538,"thread_start_us":1009675,"threads_started":3,"wal-append.queue_time_us":678559}
I20260808 01:50:32.180863  9475 rpcz_store.cc:276] 0808 01:50:31.140222 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:31.140995 (+   773us) service_pool.cc:224] Handling call
0808 01:50:31.141309 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:31.141530 (+   221us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:50:31.160079 (+ 18549us) log.cc:844] Serialized 14 byte log entry
0808 01:50:32.141892 (+981813us) log.cc:844] Serialized 14 byte log entry
0808 01:50:32.155862 (+ 13970us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:32.155919 (+    57us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:32.155975 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:32.156045 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:32.156159 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4435,"prepare.queue_time_us":13859,"prepare.run_cpu_time_us":118,"prepare.run_wall_time_us":6006,"raft.queue_time_us":57393,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":12354,"spinlock_wait_cycles":26543360,"thread_start_us":976936,"threads_started":3,"wal-append.queue_time_us":395951}
I20260808 01:50:32.129739 12419 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:32.185127 12063 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.036s	user 0.010s	sys 0.038s
I20260808 01:50:32.137223 12476 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.077s
I20260808 01:50:32.210781  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:32.212328  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:32.213495  9557 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:32.169658 12421 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:32.219575  9638 raft_consensus.cc:2410] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:50:32.182778 12479 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:32.125597 12473 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:32.236219  9551 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
W20260808 01:50:32.229220  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 145290us
W20260808 01:50:32.211844 12063 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.050s	user 0.010s	sys 0.038s
I20260808 01:50:32.209995 12310 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 01:50:32.248927 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 167366us
I20260808 01:50:32.169454 12437 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:32.240437  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:32.178905 12448 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:32.254705  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2081) took 2311 ms. Trace:
I20260808 01:50:32.256296  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:32.259466  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:32.227422 12464 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:32.211097 12036 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:32.255419  9473 rpcz_store.cc:276] 0808 01:50:29.943171 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:30.336312 (+393141us) service_pool.cc:224] Handling call
0808 01:50:30.366013 (+ 29701us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:30.569978 (+203965us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:30.570016 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:30.570669 (+   653us) log.cc:844] Serialized 52 byte log entry
0808 01:50:30.896336 (+325667us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:30.945742 (+ 49406us) log.cc:844] Serialized 14 byte log entry
0808 01:50:30.946206 (+   464us) log.cc:844] Serialized 14 byte log entry
0808 01:50:30.949864 (+  3658us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:30.950003 (+   139us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:31.958200 (+1008197us) spinlock_profiling.cc:243] Waited 9.93 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:32.254252 (+296052us) raft_consensus.cc:1685] finished
0808 01:50:32.254292 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:32.254588 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3199,"prepare.queue_time_us":98156,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":88,"raft.queue_time_us":188756,"raft.run_cpu_time_us":393,"raft.run_wall_time_us":49803,"spinlock_wait_cycles":22861440,"thread_start_us":443006,"threads_started":2,"wal-append.queue_time_us":365273}
W20260808 01:50:32.238330  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:32.308558 11757 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:32.232245 12418 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:32.316505  9521 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:32.242692 12498 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:32.191419 12480 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:32.334705  9551 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:32.241053 12479 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:32.343813 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:32.345825 12318 consensus_meta.cc:287] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.731s	user 0.010s	sys 0.000s
I20260808 01:50:32.337154 12472 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:32.349485 12435 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.223s	user 0.025s	sys 0.000s
W20260808 01:50:32.350226 12425 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.227s	user 0.008s	sys 0.016s
I20260808 01:50:32.320714 12421 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election won for term 5
W20260808 01:50:32.362368 12468 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.246s	user 0.001s	sys 0.000s
W20260808 01:50:32.397641 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 280785us
I20260808 01:50:32.308382  9557 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
W20260808 01:50:32.336709 12364 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.257s	user 0.016s	sys 0.003s
W20260808 01:50:32.362149  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 269442us
I20260808 01:50:32.326840 11757 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:32.373643  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 244806us
I20260808 01:50:32.402887  9524 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:32.403615 12420 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:50:32.403568 11757 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:32.402554 12364 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:32.410984  9632 raft_consensus.cc:2368] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6.
I20260808 01:50:32.411164  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:32.412003  9473 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 3: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:32.412669 12364 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:32.416368  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:32.417560  9473 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 6: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:32.423923  9524 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:32.423918  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:32.376901 12318 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:50:32.421778 12422 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.341s	user 0.021s	sys 0.000s
I20260808 01:50:32.426784  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:32.442977  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:32.444407  9557 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:32.445386  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:32.453709  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:32.421203 12421 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:32.432729  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:32.377533 12435 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:32.431228  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 305955us
I20260808 01:50:32.425426 12487 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:32.464057  9473 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:32.475255  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:32.476567  9473 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:50:32.456434 12497 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260808 01:50:32.432816 12364 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:32.429548 12363 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:32.436602  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:32.385216 12425 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:32.441445 12071 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:32.376636 12472 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:32.447113 12419 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:32.489056  9480 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 5: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:32.452207 12473 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:32.447449 12318 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:32.450981 12422 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:32.505487  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 290) took 1316 ms. Trace:
I20260808 01:50:32.506048  9634 rpcz_store.cc:276] 0808 01:50:31.189219 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:31.189409 (+   190us) service_pool.cc:224] Handling call
0808 01:50:32.505217 (+1315808us) spinlock_profiling.cc:243] Waited 28.6 ms on lock 0x7b54001b3ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:32.505403 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":65844992}
I20260808 01:50:32.530575 12505 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:32.532491 12505 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:50:32.534155 12494 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260808 01:50:32.500916 12437 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:32.512566 12492 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.231s	user 0.000s	sys 0.002s
I20260808 01:50:32.502588 12364 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:32.549163 12475 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:32.489435 12071 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:32.586759 12513 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260808 01:50:32.586503 12475 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:32.587913 12205 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:32.600608 12444 log.cc:927] Time spent T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 1.196s	user 0.001s	sys 0.001s
I20260808 01:50:32.600920 12205 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:32.602310 12475 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:32.596414 12509 log.cc:927] Time spent T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
W20260808 01:50:32.598145 12495 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.174s	user 0.000s	sys 0.001s
I20260808 01:50:32.599391  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:32.641371 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:32.642518 10365 raft_consensus.cc:2368] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5.
I20260808 01:50:32.550323 12422 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:32.555166 12500 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260808 01:50:32.645068 12248 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:32.539780 12473 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260808 01:50:32.550887 12314 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.359s	user 0.010s	sys 0.005s
I20260808 01:50:32.605130 12205 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:32.616501  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 292) took 1424 ms. Trace:
I20260808 01:50:32.544045 12418 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:32.552866 12472 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:32.551317 12036 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:32.548238 12425 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:32.619599 12364 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:32.624356 12318 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:32.625608 12435 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:32.625972 12354 raft_consensus.cc:3055] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 LEADER]: Stepping down as leader of term 3
W20260808 01:50:32.627496 12381 consensus_meta.cc:287] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.482s	user 0.010s	sys 0.021s
W20260808 01:50:32.628574 12507 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
W20260808 01:50:32.630540 12349 consensus_meta.cc:287] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.404s	user 0.022s	sys 0.007s
I20260808 01:50:32.631815 12477 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:32.687376 12477 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:32.632818 12071 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:32.636195 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:50:32.633901  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.170s	user 0.011s	sys 0.018s
I20260808 01:50:32.539197 12421 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:32.643369 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:32.636659  9601 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:32.652797  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:32.693353  9637 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
W20260808 01:50:32.654399 12501 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.270s	user 0.001s	sys 0.000s
I20260808 01:50:32.656020 12479 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:32.645697 12248 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:32.646121 12473 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:32.658572  9635 rpcz_store.cc:276] 0808 01:50:31.192419 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:31.202172 (+  9753us) service_pool.cc:224] Handling call
0808 01:50:31.202530 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:31.204268 (+  1738us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:32.616240 (+1411972us) spinlock_profiling.cc:243] Waited 1.13 ms on lock 0x7b54003faf54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:32.616430 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2590976}
I20260808 01:50:32.543705 12437 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:32.660126 12445 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.322s	user 0.003s	sys 0.003s
W20260808 01:50:32.662853 12457 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.587s	user 0.011s	sys 0.003s
I20260808 01:50:32.554452 12480 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:32.546729  9557 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:32.669901  9600 leader_election.cc:400] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5.
I20260808 01:50:32.683364 12354 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:32.687202 12349 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:32.689949 12381 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:32.529943 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:32.696192  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 59628.8us
I20260808 01:50:32.721344  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:32.664907 12314 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:32.712870  9600 leader_election.cc:403] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20260808 01:50:32.722823  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 53240us
I20260808 01:50:32.714119  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:32.701020 12418 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:32.702467 12473 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:32.741120 12472 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:32.743399  9636 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260808 01:50:32.727033  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:32.729252  9559 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:32.729624 12510 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:32.729947 12477 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:32.744746  9601 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:32.749550 10367 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 7: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:32.743505 12328 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:32.742055 12472 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:32.745142 12205 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:32.712095 12457 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:32.756886 12051 consensus_meta.cc:287] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.631s	user 0.015s	sys 0.000s
I20260808 01:50:32.718237 12437 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:32.770859  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:32.783437  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:32.770346  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:32.770011  9632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 1940) took 1779 ms (client timeout 1568 ms). Trace:
I20260808 01:50:32.809520 10366 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 5: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:50:32.809343  9632 rpcz_store.cc:269] 0808 01:50:30.990107 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:30.995940 (+  5833us) service_pool.cc:224] Handling call
0808 01:50:32.769797 (+1773857us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54004218d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:32.769934 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3040352000}
I20260808 01:50:32.772863 11649 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:32.811538 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:32.819808  9557 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:32.799454 12483 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.019s	sys 0.042s
W20260808 01:50:32.834878 12483 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.709s	user 0.021s	sys 0.042s
I20260808 01:50:32.836915  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 291) took 1645 ms. Trace:
I20260808 01:50:32.837520  9636 rpcz_store.cc:276] 0808 01:50:31.191900 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:31.192062 (+   162us) service_pool.cc:224] Handling call
0808 01:50:31.193755 (+  1693us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:32.743922 (+1550167us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:32.836586 (+ 92664us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b540008d754. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:32.836844 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3563280128}
I20260808 01:50:32.847623 12071 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:32.848460 12071 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:32.813235 11649 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:32.851202 11649 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:32.787637 12248 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:32.855445 12349 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:32.856468 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:32.812922 11568 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:50:32.866983 12458 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.246s	user 0.003s	sys 0.002s
I20260808 01:50:32.787942 12051 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:32.884253 12477 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:32.885355 12477 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:32.769168 12354 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:32.895620 12354 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:32.896085 12381 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:32.898869 12036 consensus_queue.cc:1175] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "259bcb3f0d254e8b9effac3277e654e0" responder_term: 4 status { last_received { term: 3 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 01:50:32.904994  9559 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:32.915153 12457 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:32.882195 12472 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:32.921522 12071 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:32.923102 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:32.923696 12466 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:32.924427 12466 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:32.935074 12437 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:32.935745  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:32.936235 12462 log.cc:927] Time spent T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 1.380s	user 0.003s	sys 0.000s
I20260808 01:50:32.936633  9478 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:50:32.940806  9638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 284) took 1760 ms (client timeout 1971 ms). Trace:
W20260808 01:50:32.941421  9638 rpcz_store.cc:269] 0808 01:50:31.180302 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:31.180449 (+   147us) service_pool.cc:224] Handling call
0808 01:50:32.940592 (+1760143us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540032c654. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:32.940737 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2319685376}
I20260808 01:50:32.942137 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:32.943656  9480 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2106) took 1518 ms (client timeout 1905 ms). Trace:
W20260808 01:50:32.944093  9480 rpcz_store.cc:269] 0808 01:50:31.425512 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:31.463371 (+ 37859us) service_pool.cc:224] Handling call
0808 01:50:32.943425 (+1480054us) spinlock_profiling.cc:243] Waited 973 ms on lock 0x7b540021dc54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:32.943590 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2238647168}
I20260808 01:50:32.945528  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:32.948387  9475 raft_consensus.cc:2393] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 5: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:32.950742 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:32.950915  9601 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:32.944666  9599 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:32.957116 12051 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:32.958223 12398 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:32.957566 12067 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:32.984999  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:32.987910  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:32.988891  9480 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:32.990403  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:32.993956  9480 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:32.902123 12186 consensus_queue.cc:1059] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.087s
W20260808 01:50:33.024483  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:32.985239 12309 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:33.040824  9557 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:33.049808  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:33.018924 11980 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:33.014875 12098 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:33.063855 12098 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:33.065424 10362 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:33.068385  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:33.017789 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:33.073287 12448 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:33.156893  9638 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260808 01:50:33.079130 10367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 449) took 1520 ms. Trace:
I20260808 01:50:33.159133 10367 rpcz_store.cc:276] 0808 01:50:31.559109 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:31.562421 (+  3312us) service_pool.cc:224] Handling call
0808 01:50:33.078730 (+1516309us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540064aad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:33.078928 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2505511936}
I20260808 01:50:33.051951 12367 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:33.084578  9551 pending_rounds.cc:85] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 2
I20260808 01:50:33.091117 12309 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:33.063210 11980 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:33.098582  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:33.102217  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:33.107873  9478 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:33.108887  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:33.215996  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:33.216686  9478 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
I20260808 01:50:33.217527  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:33.217640  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1791) took 1614 ms. Trace:
I20260808 01:50:33.218236  9630 rpcz_store.cc:276] 0808 01:50:31.603259 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:31.706172 (+102913us) service_pool.cc:224] Handling call
0808 01:50:31.706586 (+   414us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:31.740960 (+ 34374us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:31.741007 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:31.741069 (+    62us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:31.741122 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:33.217285 (+1476163us) spinlock_profiling.cc:243] Waited 34.1 ms on lock 0x7b540008c354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:33.217358 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:33.217557 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":78389632}
I20260808 01:50:33.218259  9478 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 6: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:50:33.161373 12491 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.549s	user 0.002s	sys 0.002s
I20260808 01:50:33.162137  9637 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:50:33.164847 11649 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:33.199469  9551 pending_rounds.cc:107] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 2.3
I20260808 01:50:33.226287  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:33.189913 12418 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:33.031035 12466 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:33.191571  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:33.200863  9480 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:33.203576  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:33.208434  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:33.262246  9475 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:33.262619  9559 raft_consensus.cc:2410] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 8 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:50:33.211232 12381 raft_consensus.cc:3055] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 LEADER]: Stepping down as leader of term 5
W20260808 01:50:33.213470 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:33.221796  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:33.239363  9551 raft_consensus.cc:2917] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Aborting config change with OpId 2.3: Aborted: Op aborted by new leader
I20260808 01:50:33.275378  9551 raft_consensus.cc:2889] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:33.274304  9478 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:33.313892  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:33.236222 12266 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:33.330157  9599 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:33.331626  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:33.332834 12072 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:33.336095  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:33.352566  9475 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:33.354912  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:60546 (request call id 15) took 29821 ms. Trace:
I20260808 01:50:33.355360 11649 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:33.355440  9551 rpcz_store.cc:276] 0808 01:50:03.533954 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:03.641978 (+108024us) service_pool.cc:224] Handling call
0808 01:50:04.016514 (+374536us) log.cc:844] Serialized 506 byte log entry
0808 01:50:04.668796 (+652282us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b180033c160. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d448eceeb 00007f5d44971f43 00007f5d44999b32 00007f5d4998bb39
0808 01:50:33.318706 (+28649910us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":145194,"raft.queue_time_us":418095,"raft.run_cpu_time_us":2396,"raft.run_wall_time_us":270854,"spinlock_wait_cycles":5032704,"thread_start_us":132094,"threads_started":1,"wal-append.queue_time_us":328867,"wal-append.run_cpu_time_us":2406,"wal-append.run_wall_time_us":1082429}
I20260808 01:50:33.269898 12381 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:33.359681 12248 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:50:33.360090 12381 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:33.361675 12248 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:33.358937 12349 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:33.365764 12071 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:33.366271  9600 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:33.366788 12071 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:33.367827 11649 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:33.368498 11649 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:33.368808 12072 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:33.371016 12072 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:33.371997 12309 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:33.369059 11649 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:33.376765 12205 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.597s	user 0.010s	sys 0.004s
W20260808 01:50:33.377605  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 3759209338e14437aff5f92a2fff36da: Aborted: Op aborted by new leader
I20260808 01:50:33.377699 12205 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:33.221971 12367 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:33.415189  9478 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:33.362728 12067 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:33.354450  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:33.428308  9475 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:33.451351  9601 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:33.389519 12381 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:33.453544  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:33.457342  9480 raft_consensus.cc:2410] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 8 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:50:33.457863  9601 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:33.459223  9601 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:33.486070  9601 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:33.420542 12328 consensus_meta.cc:287] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.652s	user 0.007s	sys 0.011s
I20260808 01:50:33.379161 12266 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:33.446585 12205 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:33.421146 12067 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:33.468984 12349 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:33.489727 12051 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:33.495286 12476 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:33.534399 12052 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:33.510668 12328 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5.
I20260808 01:50:33.536717 12266 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:33.554840 12071 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:33.453111 12036 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.076s
I20260808 01:50:33.556325 12098 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:33.554049 12266 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:33.538457 12477 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:33.552960 12052 raft_consensus.cc:697] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:33.456964 12354 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20260808 01:50:33.550491  9480 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 01:50:33.549142 12051 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:33.622140 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:33.624449  9636 raft_consensus.cc:2410] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260808 01:50:33.640969 12067 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:33.644656 12051 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:33.529861 12205 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:33.670238 12367 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:33.675212 11809 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:33.677584  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.214s	user 0.011s	sys 0.018s
W20260808 01:50:33.678521  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 2.28185e+06us
I20260808 01:50:33.680192  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:33.570148 12310 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.145s	user 0.007s	sys 0.079s
I20260808 01:50:33.688375  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:33.753582  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 65317.8us
I20260808 01:50:33.759001 12367 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:33.760134 12367 raft_consensus.cc:2764] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
W20260808 01:50:33.763581 11980 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.516s	user 0.008s	sys 0.005s
W20260808 01:50:33.767938  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:33.768359  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2126) took 1132 ms. Trace:
I20260808 01:50:33.807752  9480 rpcz_store.cc:276] 0808 01:50:32.635852 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:33.525132 (+889280us) service_pool.cc:224] Handling call
0808 01:50:33.525513 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:33.565805 (+ 40292us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:33.754562 (+188757us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b54002dbed4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:33.768283 (+ 13721us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":51441536}
I20260808 01:50:33.808532  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:33.773679 12477 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:33.777598 10367 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:50:33.783252 12052 consensus_queue.cc:237] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:33.787925 12473 consensus_meta.cc:287] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.039s	user 0.010s	sys 0.006s
W20260808 01:50:33.791450 12398 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.833s	user 0.000s	sys 0.014s
W20260808 01:50:33.746623 12310 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.326s	user 0.007s	sys 0.080s
I20260808 01:50:33.794080 12205 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:33.797598 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:33.798245 11568 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.939s	user 0.012s	sys 0.004s
I20260808 01:50:33.558405 12266 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:33.805274 12067 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:33.811831  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:50:33.821918 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:33.823529  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2129) took 1182 ms. Trace:
I20260808 01:50:33.868705  9480 rpcz_store.cc:276] 0808 01:50:32.640831 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:33.822389 (+1181558us) service_pool.cc:224] Handling call
0808 01:50:33.822856 (+   467us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:33.823116 (+   260us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:33.823138 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:33.823177 (+    39us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:33.823214 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:33.823288 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:33.823448 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:33.869982  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:33.871049  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2130) took 1059 ms. Trace:
I20260808 01:50:33.871656  9480 rpcz_store.cc:276] 0808 01:50:32.811339 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:33.869805 (+1058466us) service_pool.cc:224] Handling call
0808 01:50:33.870185 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:33.870374 (+   189us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:33.870400 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:33.870460 (+    60us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:33.870496 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:33.870557 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:33.870966 (+   409us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:33.874030  9480 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260808 01:50:33.874962  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2133) took 1055 ms. Trace:
I20260808 01:50:33.875447  9480 rpcz_store.cc:276] 0808 01:50:32.819139 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:33.873120 (+1053981us) service_pool.cc:224] Handling call
0808 01:50:33.873566 (+   446us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:33.874590 (+  1024us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:33.874891 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:33.793154 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:33.878046 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 84990.6us
W20260808 01:50:33.846359  9473 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.342s	user 0.017s	sys 0.038s
W20260808 01:50:33.878947  9473 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.374s	user 0.017s	sys 0.038s
W20260808 01:50:33.857410  9558 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.537s	user 0.026s	sys 0.071s
W20260808 01:50:33.879982  9558 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.560s	user 0.027s	sys 0.072s
W20260808 01:50:33.902690  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 82043e5428784a1eb53a98811ecbbbb4: Illegal state: Replica 259bcb3f0d254e8b9effac3277e654e0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:50:33.907306  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 95555.5us
I20260808 01:50:33.909930  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:33.914337  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:33.846550 12398 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:33.819552 12477 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:33.943114  9474 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.486s	user 0.017s	sys 0.041s
W20260808 01:50:33.943575  9474 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.487s	user 0.017s	sys 0.041s
I20260808 01:50:33.945022  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2117) took 1509 ms. Trace:
I20260808 01:50:33.824223 11980 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:33.918300  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:33.906442 11757 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.491s	user 0.040s	sys 0.055s
I20260808 01:50:33.945654  9474 rpcz_store.cc:276] 0808 01:50:32.435631 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:32.436603 (+   972us) service_pool.cc:224] Handling call
0808 01:50:32.437110 (+   507us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:32.437442 (+   332us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:32.456557 (+ 19115us) log.cc:844] Serialized 14 byte log entry
0808 01:50:33.944119 (+1487562us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:33.944207 (+    88us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:33.944281 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:33.944387 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:33.944565 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":25068,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"thread_start_us":1487095,"threads_started":1}
I20260808 01:50:33.884884 12451 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:33.883956 11568 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:33.846216 12473 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:33.868799 12418 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.099s
I20260808 01:50:33.996315  9372 catalog_manager.cc:5697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 4 to 5, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 5 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:50:33.999661  9372 catalog_manager.cc:5697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 5 to 6, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 6 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 01:50:33.973870 11757 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.558s	user 0.041s	sys 0.055s
W20260808 01:50:34.050933  9554 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.895s	user 0.024s	sys 0.037s
W20260808 01:50:34.051697  9554 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.896s	user 0.024s	sys 0.037s
W20260808 01:50:34.053385  9479 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.554s	user 0.011s	sys 0.049s
W20260808 01:50:34.054112  9479 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.555s	user 0.011s	sys 0.049s
W20260808 01:50:34.013167 12381 consensus_meta.cc:287] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.522s	user 0.016s	sys 0.000s
W20260808 01:50:34.056068  9550 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.615s	user 0.023s	sys 0.072s
W20260808 01:50:34.092581  9550 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.652s	user 0.023s	sys 0.072s
W20260808 01:50:34.056381  9523 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.585s	user 0.018s	sys 0.055s
I20260808 01:50:34.007309 12349 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:50:34.093513  9523 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.622s	user 0.019s	sys 0.055s
I20260808 01:50:34.006250  9372 catalog_manager.cc:5697] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 2 to 4, now has a pending config: VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193), VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194), VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195), NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 4 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:50:34.104151  9372 catalog_manager.cc:5697] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 5 to 6, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 6 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:50:34.106045  9372 catalog_manager.cc:5697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 1 to 2, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 2 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 01:50:34.055869 10361 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.844s	user 0.017s	sys 0.061s
W20260808 01:50:34.124091  9374 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50064 (request call id 236) took 12746 ms (client timeout 15000 ms). Trace:
I20260808 01:50:34.007019 12051 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:34.112404  9372 catalog_manager.cc:5697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:50:34.153024  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2134) took 1332 ms. Trace:
I20260808 01:50:34.153649  9480 rpcz_store.cc:276] 0808 01:50:32.820673 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:33.876500 (+1055827us) service_pool.cc:224] Handling call
0808 01:50:33.876916 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:33.924339 (+ 47423us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:33.924391 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:33.924467 (+    76us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:33.924518 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:34.152619 (+228101us) spinlock_profiling.cc:243] Waited 46.9 ms on lock 0x7b540019c8d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:34.152685 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:34.152913 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":107790720}
I20260808 01:50:34.157701  9480 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:50:34.164294  9480 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 3. Current term is 4. Ops: []
W20260808 01:50:34.176410  9557 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.877s	user 0.005s	sys 0.000s
W20260808 01:50:34.176971  9557 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.878s	user 0.005s	sys 0.000s
W20260808 01:50:34.178292  9445 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.864s	user 0.002s	sys 0.000s
W20260808 01:50:34.178805  9445 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.864s	user 0.002s	sys 0.000s
I20260808 01:50:34.177090 12563 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:34.181480 12563 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:34.182057 12563 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:34.166379  9372 catalog_manager.cc:5697] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 1 to 4, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 4 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 01:50:34.169356  9524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.720s	user 0.023s	sys 0.043s
W20260808 01:50:34.185444  9524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.736s	user 0.023s	sys 0.043s
W20260808 01:50:34.186154  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 1.75934e+06us
I20260808 01:50:34.186987  9524 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:34.186800  9372 catalog_manager.cc:5697] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 1 to 4, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 4 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:50:34.233345  9372 catalog_manager.cc:5697] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 4 to 5, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 5 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:50:34.189393  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:34.129009 10361 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.924s	user 0.017s	sys 0.061s
W20260808 01:50:34.197562  9560 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.551s	user 0.015s	sys 0.028s
I20260808 01:50:34.173214  9480 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
W20260808 01:50:34.188421  9477 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.583s	user 0.010s	sys 0.036s
W20260808 01:50:34.265858  9477 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.661s	user 0.012s	sys 0.037s
W20260808 01:50:34.265836  9560 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.620s	user 0.018s	sys 0.029s
W20260808 01:50:34.267369  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 77861us
W20260808 01:50:34.280725  9552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.785s	user 0.009s	sys 0.046s
W20260808 01:50:34.281340  9552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.787s	user 0.009s	sys 0.047s
I20260808 01:50:34.282485  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2211) took 2146 ms. Trace:
W20260808 01:50:34.284578  9634 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.651s	user 0.015s	sys 0.032s
W20260808 01:50:34.285174  9634 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.652s	user 0.016s	sys 0.032s
W20260808 01:50:34.230947 12534 log.cc:927] Time spent T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.208s	user 0.002s	sys 0.000s
I20260808 01:50:34.288143  9372 catalog_manager.cc:5697] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 3 to 5, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 5 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:50:34.296790  9481 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.805s	user 0.007s	sys 0.021s
W20260808 01:50:34.257755 11649 consensus_meta.cc:287] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.882s	user 0.008s	sys 0.008s
I20260808 01:50:34.282544  9480 pending_rounds.cc:85] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 6
I20260808 01:50:34.363245  9480 pending_rounds.cc:107] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260808 01:50:34.363672  9480 raft_consensus.cc:2889] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:50:34.328377  9481 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.836s	user 0.008s	sys 0.022s
W20260808 01:50:34.331416  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:34.283275  9552 rpcz_store.cc:276] 0808 01:50:32.135995 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:32.482371 (+346376us) service_pool.cc:224] Handling call
0808 01:50:32.482900 (+   529us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:32.491149 (+  8249us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:32.491182 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:32.494347 (+  3165us) log.cc:844] Serialized 26 byte log entry
0808 01:50:34.281946 (+1787599us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:34.282015 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:34.282130 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:34.282199 (+    69us) raft_consensus.cc:1685] finished
0808 01:50:34.282218 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:34.282406 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":1787117,"threads_started":1,"wal-append.queue_time_us":1684217}
W20260808 01:50:34.343827 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:34.369704  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.033s	user 0.014s	sys 0.014s
W20260808 01:50:35.452728  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.116s	user 0.027s	sys 0.031s
I20260808 01:50:34.382809  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 363) took 1121 ms. Trace:
W20260808 01:50:35.453747  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 3.11734e+06us
W20260808 01:50:35.443455  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 1.17463e+06us
I20260808 01:50:35.454355  9480 rpcz_store.cc:276] 0808 01:50:33.263743 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:34.172304 (+908561us) service_pool.cc:224] Handling call
0808 01:50:34.172739 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:34.382420 (+209681us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:34.382723 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:35.437458  9557 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.259s	user 0.011s	sys 0.013s
I20260808 01:50:35.455871  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2208) took 3320 ms. Trace:
I20260808 01:50:35.404742  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
W20260808 01:50:35.398998  9446 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.085s	user 0.013s	sys 0.028s
W20260808 01:50:35.458101  9446 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.144s	user 0.014s	sys 0.036s
I20260808 01:50:35.459100  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 365) took 2178 ms. Trace:
W20260808 01:50:35.398998  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.435s	user 0.001s	sys 0.018s
W20260808 01:50:35.470458  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.507s	user 0.005s	sys 0.021s
W20260808 01:50:35.471383  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 1.55306e+06us
I20260808 01:50:34.327211  9372 catalog_manager.cc:5697] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:50:35.479409  9630 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.610s	user 0.009s	sys 0.020s
W20260808 01:50:35.480101  9630 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.611s	user 0.009s	sys 0.020s
W20260808 01:50:35.481446 10333 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.257s	user 0.000s	sys 0.057s
W20260808 01:50:35.481868  9636 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.568s	user 0.013s	sys 0.016s
I20260808 01:50:35.456475  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:35.482560  9636 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.571s	user 0.013s	sys 0.016s
W20260808 01:50:35.405463  9479 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.307s	user 0.010s	sys 0.023s
W20260808 01:50:35.463786  9599 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.428s	user 0.000s	sys 0.010s
W20260808 01:50:35.408512  9478 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.943s	user 0.008s	sys 0.019s
W20260808 01:50:35.408200  9477 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.141s	user 0.009s	sys 0.007s
W20260808 01:50:35.407384  9475 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.788s	user 0.010s	sys 0.014s
I20260808 01:50:34.772287 12051 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:35.486748 11867 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:35.053438 12063 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.464s	user 0.000s	sys 0.046s
W20260808 01:50:35.488324 12063 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.898s	user 0.000s	sys 0.047s
I20260808 01:50:35.488229 11867 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:34.261910 11955 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.931s	user 0.007s	sys 0.018s
W20260808 01:50:35.491370 11955 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 4.193s	user 0.014s	sys 0.052s
I20260808 01:50:34.665622 11980 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:34.694989 12030 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.140s	user 0.022s	sys 0.030s
W20260808 01:50:35.494350 12030 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.939s	user 0.024s	sys 0.033s
I20260808 01:50:35.496198 12030 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260808 01:50:34.280462 12068 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.623s	user 0.017s	sys 0.042s
W20260808 01:50:35.498665 12068 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.842s	user 0.025s	sys 0.057s
I20260808 01:50:35.456552  9560 rpcz_store.cc:276] 0808 01:50:32.135248 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:32.157199 (+ 21951us) service_pool.cc:224] Handling call
0808 01:50:32.170996 (+ 13797us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:32.263154 (+ 92158us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:50:32.308905 (+ 45751us) log.cc:844] Serialized 14 byte log entry
0808 01:50:32.646194 (+337289us) spinlock_profiling.cc:243] Waited 3.43 ms on lock 0x7b18001e8ec0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d4453242a 00007f5d4499de0a
0808 01:50:34.268074 (+1621880us) log.cc:844] Serialized 14 byte log entry
0808 01:50:35.120664 (+852590us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x7b18001e8ec0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4185ac56 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121
0808 01:50:35.122585 (+  1921us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b18001e8ec0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:50:35.442192 (+319607us) spinlock_profiling.cc:243] Waited 4.81 ms on lock 0x7b18001e8ec0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667 00007f5d449ae5e8
0808 01:50:35.452762 (+ 10570us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:35.452884 (+   122us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:35.453058 (+   174us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:35.455270 (+  2212us) spinlock_profiling.cc:243] Waited 91.9 ms on lock 0x7b54000b40d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:35.455342 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:35.455595 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1156736,"prepare.queue_time_us":55325,"prepare.run_cpu_time_us":144,"prepare.run_wall_time_us":2538,"raft.queue_time_us":1217681,"raft.run_cpu_time_us":672,"raft.run_wall_time_us":67560,"spinlock_wait_cycles":313620096,"thread_start_us":1621279,"threads_started":1,"wal-append.queue_time_us":1874264}
W20260808 01:50:35.407797  9476 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.916s	user 0.011s	sys 0.032s
W20260808 01:50:34.135030  9374 rpcz_store.cc:269] 0808 01:50:21.517161 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:21.541246 (+ 24085us) service_pool.cc:224] Handling call
0808 01:50:34.098713 (+12557467us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:50:29.296267 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:50:29.355567 (+ 59300us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:50:29.355635 (+    68us) write_op.cc:435] Acquired schema lock
0808 01:50:29.355675 (+    40us) tablet.cc:662] Decoding operations
0808 01:50:29.357462 (+  1787us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:50:29.357716 (+   254us) write_op.cc:641] Partition lock acquired for write op
0808 01:50:29.357771 (+    55us) tablet.cc:685] Acquiring locks for 13 operations
0808 01:50:29.479064 (+121293us) tablet.cc:701] Row locks acquired
0808 01:50:29.499501 (+ 20437us) write_op.cc:260] PREPARE: finished
0808 01:50:29.548702 (+ 49201us) write_op.cc:270] Start()
0808 01:50:29.573613 (+ 24911us) write_op.cc:276] Timestamp: P: 1786153829534793 usec, L: 0
0808 01:50:29.573661 (+    48us) op_driver.cc:348] REPLICATION: starting
0808 01:50:29.574481 (+   820us) log.cc:844] Serialized 5183 byte log entry
0808 01:50:30.338994 (+764513us) op_driver.cc:464] REPLICATION: finished
0808 01:50:30.689220 (+350226us) write_op.cc:301] APPLY: starting
0808 01:50:30.689363 (+   143us) tablet.cc:1367] starting BulkCheckPresence
0808 01:50:30.689808 (+   445us) tablet.cc:1370] finished BulkCheckPresence
0808 01:50:30.689991 (+   183us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:50:30.960243 (+270252us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:50:30.960697 (+   454us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=13
0808 01:50:30.960792 (+    95us) write_op.cc:312] APPLY: finished
0808 01:50:30.961366 (+   574us) log.cc:844] Serialized 95 byte log entry
0808 01:50:30.961634 (+   268us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:30.961845 (+   211us) write_op.cc:454] Released schema lock
0808 01:50:30.961947 (+   102us) write_op.cc:341] FINISH: Updating metrics
0808 01:50:30.962551 (+   604us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:30.962593 (+    42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":877,"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":306169,"apply.run_cpu_time_us":7206,"apply.run_wall_time_us":305124,"num_ops":13,"prepare.queue_time_us":388581,"prepare.run_cpu_time_us":24907,"prepare.run_wall_time_us":518561,"raft.queue_time_us":175451,"raft.run_cpu_time_us":25086,"raft.run_wall_time_us":223876,"replication_time_us":703041,"thread_start_us":652901,"threads_started":4,"wal-append.queue_time_us":315400,"wal-append.run_cpu_time_us":24521,"wal-append.run_wall_time_us":2067005}]]}
W20260808 01:50:35.400103  9443 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.945s	user 0.007s	sys 0.014s
W20260808 01:50:35.400452  9445 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.220s	user 0.011s	sys 0.014s
W20260808 01:50:35.456409  9557 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.279s	user 0.014s	sys 0.015s
I20260808 01:50:35.452793  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 342) took 3328 ms. Trace:
W20260808 01:50:35.455881  9552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 2245) took 2076 ms (client timeout 1648 ms). Trace:
W20260808 01:50:35.436571  9551 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.058s	user 0.018s	sys 0.011s
W20260808 01:50:34.532285  9472 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.068s	user 0.023s	sys 0.018s
I20260808 01:50:35.415843  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2116) took 2980 ms. Trace:
W20260808 01:50:35.482016 10333 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.258s	user 0.000s	sys 0.057s
W20260808 01:50:35.437305  9558 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.476s	user 0.003s	sys 0.024s
I20260808 01:50:35.501724 12030 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:35.568612  9552 rpcz_store.cc:269] 0808 01:50:33.379726 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:35.455375 (+2075649us) service_pool.cc:211] Skipping call since client already timed out
0808 01:50:35.455598 (+   223us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260808 01:50:35.407878  9474 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.405s	user 0.006s	sys 0.003s
I20260808 01:50:35.545342  9480 rpcz_store.cc:276] 0808 01:50:33.280832 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:35.457081 (+2176249us) service_pool.cc:224] Handling call
0808 01:50:35.457923 (+   842us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:35.458457 (+   534us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:35.458505 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:35.458579 (+    74us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:35.458668 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:35.458799 (+   131us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:35.458992 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:35.563733  9478 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.098s	user 0.012s	sys 0.019s
W20260808 01:50:35.566797  9477 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.300s	user 0.013s	sys 0.009s
W20260808 01:50:35.566272 10332 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.315s	user 0.006s	sys 0.010s
W20260808 01:50:35.567222  9476 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 3.076s	user 0.012s	sys 0.036s
W20260808 01:50:35.570576  9599 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.535s	user 0.001s	sys 0.010s
W20260808 01:50:35.585529  9443 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.130s	user 0.007s	sys 0.019s
W20260808 01:50:34.545951 12524 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.537s	user 0.001s	sys 0.001s
W20260808 01:50:35.581301  9479 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.483s	user 0.018s	sys 0.027s
W20260808 01:50:34.884343 12526 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.347s	user 0.000s	sys 0.001s
W20260808 01:50:35.581950  9475 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.962s	user 0.012s	sys 0.016s
W20260808 01:50:35.580657  9523 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.144s	user 0.009s	sys 0.035s
W20260808 01:50:35.596895 10332 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.346s	user 0.007s	sys 0.011s
I20260808 01:50:35.633780  9480 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260808 01:50:35.085608 12072 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.677s	user 0.004s	sys 0.009s
W20260808 01:50:35.635481 12072 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.227s	user 0.006s	sys 0.009s
W20260808 01:50:35.637880  9552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 2248) took 1865 ms (client timeout 1647 ms). Trace:
W20260808 01:50:35.638485  9552 rpcz_store.cc:269] 0808 01:50:33.772746 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:35.637607 (+1864861us) service_pool.cc:211] Skipping call since client already timed out
0808 01:50:35.637789 (+   182us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260808 01:50:34.388068 12266 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:34.580785 12327 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.664s	user 0.015s	sys 0.023s
I20260808 01:50:35.640048  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:35.640550 12327 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.724s	user 0.017s	sys 0.034s
I20260808 01:50:35.641717  9552 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:35.595350  9372 catalog_manager.cc:5697] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 4 to 6, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 6 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:50:34.257813 12530 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:34.381966 11649 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:35.597199  9445 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.416s	user 0.016s	sys 0.014s
W20260808 01:50:35.164438 12563 consensus_meta.cc:287] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.982s	user 0.010s	sys 0.004s
W20260808 01:50:34.915653 12476 consensus_meta.cc:287] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.304s	user 0.009s	sys 0.008s
I20260808 01:50:35.651856 12476 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:35.651695 12563 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:35.639268  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 340) took 3545 ms. Trace:
I20260808 01:50:35.654991  9556 rpcz_store.cc:276] 0808 01:50:32.093475 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:32.097937 (+  4462us) service_pool.cc:224] Handling call
0808 01:50:32.098503 (+   566us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:32.368482 (+269979us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:32.368518 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:32.369048 (+   530us) log.cc:844] Serialized 26 byte log entry
0808 01:50:32.802517 (+433469us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:32.802562 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:32.802637 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:35.638732 (+2836095us) raft_consensus.cc:1685] finished
0808 01:50:35.638780 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:35.639062 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60573,"raft.queue_time_us":143008,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":17278,"thread_start_us":613281,"threads_started":2,"wal-append.queue_time_us":1495004}
W20260808 01:50:35.596375  9560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 2247) took 1824 ms (client timeout 1570 ms). Trace:
I20260808 01:50:35.656570  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:35.596685  9551 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.218s	user 0.022s	sys 0.011s
I20260808 01:50:35.645936  9372 catalog_manager.cc:5697] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:50:35.657894  9560 rpcz_store.cc:269] 0808 01:50:33.772405 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:35.596067 (+1823662us) service_pool.cc:211] Skipping call since client already timed out
0808 01:50:35.596289 (+   222us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260808 01:50:35.642445  9552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 349) took 1505 ms (client timeout 1697 ms). Trace:
W20260808 01:50:35.595180  9472 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 3.131s	user 0.023s	sys 0.026s
W20260808 01:50:35.661170  9552 rpcz_store.cc:269] 0808 01:50:34.136740 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:35.639398 (+1502658us) service_pool.cc:224] Handling call
0808 01:50:35.642320 (+  2922us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:34.286957 12375 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.729s	user 0.013s	sys 0.027s
W20260808 01:50:35.660163  9474 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.658s	user 0.010s	sys 0.006s
W20260808 01:50:34.854570 10368 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.187s	user 0.018s	sys 0.020s
W20260808 01:50:35.595350  9558 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.634s	user 0.006s	sys 0.027s
W20260808 01:50:35.630882  9634 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.292s	user 0.007s	sys 0.015s
W20260808 01:50:34.678820 12355 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.959s	user 0.017s	sys 0.034s
I20260808 01:50:35.655982 11649 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:35.115810 12309 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.117s	user 0.007s	sys 0.006s
W20260808 01:50:34.656817 12364 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.714s	user 0.016s	sys 0.033s
W20260808 01:50:34.900739 12529 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.369s	user 0.002s	sys 0.000s
W20260808 01:50:34.388651 12544 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.200s	user 0.000s	sys 0.001s
W20260808 01:50:34.553766 12421 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.626s	user 0.018s	sys 0.022s
W20260808 01:50:34.400322 12450 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.049s	user 0.029s	sys 0.047s
W20260808 01:50:35.135200 12472 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.145s	user 0.007s	sys 0.013s
W20260808 01:50:34.388499 12523 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.487s	user 0.002s	sys 0.000s
W20260808 01:50:35.115705 12493 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.780s	user 0.001s	sys 0.020s
I20260808 01:50:35.669199  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 366) took 2379 ms. Trace:
W20260808 01:50:34.581174 12499 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.615s	user 0.018s	sys 0.024s
W20260808 01:50:34.656780 12483 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.820s	user 0.016s	sys 0.025s
I20260808 01:50:34.292588 12505 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:35.670521  9523 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.234s	user 0.011s	sys 0.035s
W20260808 01:50:35.686318  9634 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.400s	user 0.007s	sys 0.016s
W20260808 01:50:34.959740 12469 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.839s	user 0.020s	sys 0.046s
W20260808 01:50:34.488270 12546 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
W20260808 01:50:34.905156 12520 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.356s	user 0.000s	sys 0.001s
W20260808 01:50:35.681696  9554 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.487s	user 0.020s	sys 0.007s
I20260808 01:50:35.705339  9480 rpcz_store.cc:276] 0808 01:50:33.281275 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:35.596471 (+2315196us) service_pool.cc:224] Handling call
0808 01:50:35.632809 (+ 36338us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:35.634505 (+  1696us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:35.660269 (+ 25764us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:35.713771 12493 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.393s	user 0.016s	sys 0.020s
W20260808 01:50:35.723001  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 1.68742e+06us
I20260808 01:50:35.569435  9481 rpcz_store.cc:276] 0808 01:50:32.435142 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:32.447160 (+ 12018us) service_pool.cc:224] Handling call
0808 01:50:32.447553 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:32.460512 (+ 12959us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:32.460562 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:32.461175 (+   613us) log.cc:844] Serialized 26 byte log entry
0808 01:50:34.365802 (+1904627us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:34.365842 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:34.365916 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:35.414814 (+1048898us) raft_consensus.cc:1685] finished
0808 01:50:35.414916 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:35.415461 (+   545us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12449,"raft.queue_time_us":15054,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":5087,"thread_start_us":1873581,"threads_started":1,"wal-append.queue_time_us":2545848}
I20260808 01:50:35.594698 12310 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:35.688071  9559 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.169s	user 0.007s	sys 0.023s
I20260808 01:50:35.595539 12419 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:35.724335  9637 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.748s	user 0.017s	sys 0.012s
I20260808 01:50:35.720434  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:35.733071  9554 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.680s	user 0.022s	sys 0.008s
W20260808 01:50:35.738966  9637 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.891s	user 0.018s	sys 0.012s
W20260808 01:50:35.745587  9559 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.334s	user 0.008s	sys 0.023s
W20260808 01:50:35.725796 12472 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.716s	user 0.009s	sys 0.016s
W20260808 01:50:35.747776 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 1.49698e+06us
I20260808 01:50:35.744436  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 307) took 1894 ms. Trace:
W20260808 01:50:35.752152 12355 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.975s	user 0.029s	sys 0.061s
W20260808 01:50:35.752210 12499 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.751s	user 0.021s	sys 0.037s
W20260808 01:50:35.770005  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 121055us
I20260808 01:50:35.770504 12532 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260808 01:50:35.778904 12375 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 4.137s	user 0.030s	sys 0.034s
I20260808 01:50:35.779096  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2213) took 3619 ms. Trace:
W20260808 01:50:35.773837  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:35.780270 12589 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:35.779898  9557 rpcz_store.cc:276] 0808 01:50:32.159584 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:33.224259 (+1064675us) service_pool.cc:224] Handling call
0808 01:50:33.224665 (+   406us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:33.224957 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:34.177490 (+952533us) log.cc:844] Serialized 14 byte log entry
0808 01:50:35.778203 (+1600713us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:35.778351 (+   148us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:35.778586 (+   235us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:35.778728 (+   142us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:35.778996 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42011,"prepare.queue_time_us":752439,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":71194,"thread_start_us":2268696,"threads_started":2}
I20260808 01:50:35.781801 11980 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:35.781590 12589 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:35.739236 12477 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:35.782876 12531 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:35.782986 11980 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:35.783535  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:35.766136 12483 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.880s	user 0.029s	sys 0.030s
I20260808 01:50:35.784864  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 2250) took 2011 ms. Trace:
I20260808 01:50:35.785401  9557 rpcz_store.cc:276] 0808 01:50:33.773375 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:35.783030 (+2009655us) service_pool.cc:224] Handling call
0808 01:50:35.784782 (+  1752us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:35.787266 12067 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.886s	user 0.014s	sys 0.016s
I20260808 01:50:35.776136  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:35.719563 12266 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 01:50:35.797338 12364 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.770s	user 0.024s	sys 0.038s
I20260808 01:50:35.798005 11568 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:35.805358 10368 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 3.020s	user 0.018s	sys 0.024s
I20260808 01:50:35.568228  9550 rpcz_store.cc:276] 0808 01:50:32.124101 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:32.124585 (+   484us) service_pool.cc:224] Handling call
0808 01:50:32.124991 (+   406us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:32.359150 (+234159us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:32.359203 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:32.371537 (+ 12334us) log.cc:844] Serialized 26 byte log entry
0808 01:50:34.093119 (+1721582us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:34.093162 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:34.161080 (+ 67918us) spinlock_profiling.cc:243] Waited 228 ms on lock 0x7b54000fee54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:34.161129 (+    49us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:35.451596 (+1290467us) raft_consensus.cc:1685] finished
0808 01:50:35.451649 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:35.452426 (+   777us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7793,"raft.queue_time_us":142383,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":8797,"spinlock_wait_cycles":523888512,"thread_start_us":1652431,"threads_started":1,"wal-append.queue_time_us":2126807}
I20260808 01:50:35.756636  9630 rpcz_store.cc:276] 0808 01:50:33.810316 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:33.819959 (+  9643us) service_pool.cc:224] Handling call
0808 01:50:33.820337 (+   378us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:33.820769 (+   432us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:33.820810 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:33.863795 (+ 42985us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b2c000b67f8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:50:33.869251 (+  5456us) log.cc:844] Serialized 26 byte log entry
0808 01:50:35.481080 (+1611829us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:35.481211 (+   131us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:35.481443 (+   232us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:35.659499 (+178056us) raft_consensus.cc:1685] finished
0808 01:50:35.659532 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:35.704397 (+ 44865us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":216,"raft.queue_time_us":2530,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":23475,"spinlock_wait_cycles":29473408,"thread_start_us":1611024,"threads_started":1,"wal-append.queue_time_us":1773626}
I20260808 01:50:35.791909 12586 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:35.747663 12030 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:35.822829  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:35.774606 12052 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:35.827219 12421 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.778s	user 0.034s	sys 0.055s
W20260808 01:50:35.822271 12450 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 3.354s	user 0.031s	sys 0.067s
I20260808 01:50:35.802945 11980 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:35.827968  9556 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
W20260808 01:50:35.839263  9556 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 344) took 1735 ms (client timeout 1745 ms). Trace:
W20260808 01:50:35.839823  9556 rpcz_store.cc:269] 0808 01:50:34.103869 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:35.657320 (+1553451us) service_pool.cc:224] Handling call
0808 01:50:35.839003 (+181683us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:35.820072  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 1.38433e+06us
I20260808 01:50:35.829830 12473 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:35.819707 12309 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.728s	user 0.009s	sys 0.007s
I20260808 01:50:35.745600 12310 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:35.834214 12498 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:35.849524 12576 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:35.849937 12498 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:35.850113 12583 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:35.850417 12576 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:35.850977 12583 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:35.853226 12052 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:35.853909 12583 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:35.853678 12576 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:35.839007 12530 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260808 01:50:35.861598  9560 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:35.846124 12469 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 3.020s	user 0.026s	sys 0.055s
I20260808 01:50:35.804087 12505 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:35.865229 12541 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:35.878865 11568 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:35.881953 12568 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
W20260808 01:50:35.882045  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 93408.3us
I20260808 01:50:35.856202 12563 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:35.834604 12493 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:35.886857  9480 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260808 01:50:35.856703 12375 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:35.891556  9557 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260808 01:50:35.885128  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:35.888361 12591 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
W20260808 01:50:35.893043 12581 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.080s	user 0.001s	sys 0.001s
I20260808 01:50:35.889374  9480 pending_rounds.cc:85] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 6
I20260808 01:50:35.854580 12583 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:35.897640  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2207) took 3762 ms. Trace:
I20260808 01:50:35.898239  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2215) took 3732 ms. Trace:
I20260808 01:50:35.902730  9557 rpcz_store.cc:276] 0808 01:50:32.160251 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:35.786429 (+3626178us) service_pool.cc:224] Handling call
0808 01:50:35.828960 (+ 42531us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:35.892074 (+ 63114us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:35.892345 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:35.897591 12592 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260808 01:50:35.903237  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 305) took 2093 ms. Trace:
I20260808 01:50:35.896502  9480 pending_rounds.cc:107] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260808 01:50:35.909018  9480 raft_consensus.cc:2889] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:35.910269 12364 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:35.912859  9550 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:35.840236 12355 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.060s
I20260808 01:50:35.923744 12375 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:35.924530  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2122) took 3484 ms. Trace:
I20260808 01:50:35.924598 12589 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:35.925060 12540 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260808 01:50:35.925733 12493 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:35.905556 11867 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:35.900364  9554 rpcz_store.cc:276] 0808 01:50:32.134919 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:32.142001 (+  7082us) service_pool.cc:224] Handling call
0808 01:50:32.142471 (+   470us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:34.052336 (+1909865us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:34.052389 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:34.053072 (+   683us) log.cc:844] Serialized 26 byte log entry
0808 01:50:35.847733 (+1794661us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:35.854534 (+  6801us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:35.854800 (+   266us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:35.895042 (+ 40242us) raft_consensus.cc:1685] finished
0808 01:50:35.895065 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:35.897557 (+  2492us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11931,"raft.queue_time_us":168359,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":5444,"thread_start_us":3586227,"threads_started":2,"wal-append.queue_time_us":1664867}
I20260808 01:50:35.926327  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:35.931717  9556 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:35.895582 12571 raft_consensus.cc:2764] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260808 01:50:35.891036 12309 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:35.929963 11649 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:35.961344 12309 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:35.891599 12586 raft_consensus.cc:697] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:35.980365  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2119) took 3543 ms. Trace:
I20260808 01:50:35.855159 12419 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:35.980944  9473 rpcz_store.cc:276] 0808 01:50:32.437981 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:32.477431 (+ 39450us) service_pool.cc:224] Handling call
0808 01:50:32.477930 (+   499us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:32.478290 (+   360us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:32.478334 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:32.489115 (+ 10781us) log.cc:844] Serialized 26 byte log entry
0808 01:50:33.879534 (+1390419us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:33.879594 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:33.879692 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:35.934812 (+2055120us) raft_consensus.cc:1685] finished
0808 01:50:35.941336 (+  6524us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:35.980088 (+ 38752us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":1374642,"threads_started":1,"wal-append.queue_time_us":1866796}
I20260808 01:50:35.983428 12030 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:35.943115  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2216) took 3782 ms. Trace:
I20260808 01:50:35.945138  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2212) took 3808 ms. Trace:
I20260808 01:50:35.962441 11649 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:35.978772 12067 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:35.908859  9637 rpcz_store.cc:276] 0808 01:50:33.809513 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:33.810788 (+  1275us) service_pool.cc:224] Handling call
0808 01:50:33.811172 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:33.814873 (+  3701us) raft_consensus.cc:1534] Early marking committed up to index 5
0808 01:50:33.846834 (+ 31961us) log.cc:844] Serialized 14 byte log entry
0808 01:50:35.886316 (+2039482us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:35.894587 (+  8271us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:35.894740 (+   153us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:35.894873 (+   133us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:35.903078 (+  8205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34874,"prepare.queue_time_us":53145,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":34158,"raft.run_cpu_time_us":396,"raft.run_wall_time_us":13792,"spinlock_wait_cycles":1642368,"thread_start_us":1901930,"threads_started":1}
I20260808 01:50:36.007330  9550 rpcz_store.cc:276] 0808 01:50:32.160541 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:35.835210 (+3674669us) service_pool.cc:224] Handling call
0808 01:50:35.877783 (+ 42573us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:35.942668 (+ 64885us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:35.943021 (+   353us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:35.940030  9475 rpcz_store.cc:276] 0808 01:50:32.439977 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:33.490833 (+1050856us) service_pool.cc:224] Handling call
0808 01:50:33.491879 (+  1046us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:33.492317 (+   438us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:33.492358 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:33.509820 (+ 17462us) log.cc:844] Serialized 26 byte log entry
0808 01:50:35.758880 (+2249060us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:35.790831 (+ 31951us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:35.804972 (+ 14141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:35.924213 (+119241us) raft_consensus.cc:1685] finished
0808 01:50:35.924236 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:35.924443 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":2009684,"threads_started":1,"wal-append.queue_time_us":2330413}
I20260808 01:50:36.008567  9551 rpcz_store.cc:276] 0808 01:50:32.136279 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:33.083913 (+947634us) service_pool.cc:224] Handling call
0808 01:50:33.084342 (+   429us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:33.376747 (+292405us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:33.376791 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:33.377735 (+   944us) log.cc:844] Serialized 52 byte log entry
0808 01:50:35.795064 (+2417329us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:35.826648 (+ 31584us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:35.831567 (+  4919us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:35.929238 (+ 97671us) raft_consensus.cc:1685] finished
0808 01:50:35.929262 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:35.945023 (+ 15761us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":157,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":149,"thread_start_us":2292397,"threads_started":1,"wal-append.queue_time_us":2431705}
I20260808 01:50:36.012025  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:36.015687  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2217) took 3854 ms. Trace:
I20260808 01:50:35.987299  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 367) took 2705 ms. Trace:
I20260808 01:50:36.050151  9480 rpcz_store.cc:276] 0808 01:50:33.281691 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:35.734748 (+2453057us) service_pool.cc:224] Handling call
0808 01:50:35.826439 (+ 91691us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:35.986953 (+160514us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:35.987200 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:36.017710  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:36.018762 12310 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:36.019359 12583 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:36.021222 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:36.022169 12493 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:36.023099 12052 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:36.023321 11980 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:36.023845  9554 raft_consensus.cc:1217] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Deduplicated request from leader. Original: 2.2->[4.3-6.4]   Dedup: 4.3->[6.4-6.4]
I20260808 01:50:36.058519  9554 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20260808 01:50:36.023975 12477 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:36.024686 12072 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:36.031090  9552 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 5: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:36.032750 12498 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:36.033569 12364 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:36.062248  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 350) took 1920 ms. Trace:
I20260808 01:50:36.036645 12355 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260808 01:50:36.039551  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 104649us
I20260808 01:50:36.040660 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:36.042222  9560 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:36.031002  9556 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:36.052155  9551 rpcz_store.cc:276] 0808 01:50:32.160840 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.013934 (+3853094us) service_pool.cc:224] Handling call
0808 01:50:36.014600 (+   666us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:36.015139 (+   539us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:36.015178 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:36.015242 (+    64us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:36.015331 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.015420 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.015572 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:36.059429  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2115) took 3627 ms. Trace:
I20260808 01:50:36.064604  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:36.063663  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2218) took 3898 ms. Trace:
I20260808 01:50:36.072415 12072 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:36.092998  9560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 345) took 1982 ms (client timeout 2201 ms). Trace:
I20260808 01:50:35.980973 12586 consensus_queue.cc:237] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:36.099689 12072 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:36.073764 11980 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:36.074250  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2114) took 3642 ms. Trace:
I20260808 01:50:36.075762  9552 rpcz_store.cc:276] 0808 01:50:34.141926 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:35.925902 (+1783976us) service_pool.cc:224] Handling call
0808 01:50:36.062185 (+136283us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:36.081890  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 140377us
W20260808 01:50:36.090962 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:36.091228  9554 rpcz_store.cc:276] 0808 01:50:32.161129 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.022873 (+3861744us) service_pool.cc:224] Handling call
0808 01:50:36.023409 (+   536us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:36.059008 (+ 35599us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.059264 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:36.084192 12493 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:36.075704 10332 connection.cc:256] RPC call timeout handler was delayed by -1.2118586600117851s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16970000000000002s for a response
W20260808 01:50:36.093490  9560 rpcz_store.cc:269] 0808 01:50:34.104314 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:35.695588 (+1591274us) service_pool.cc:224] Handling call
0808 01:50:36.087041 (+391453us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:35.986239 12450 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.090s
I20260808 01:50:36.076406 12375 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:36.112946 10366 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:36.125432 10367 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:36.120404  9557 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260808 01:50:36.126667  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2220) took 3965 ms. Trace:
I20260808 01:50:36.127033  9557 rpcz_store.cc:276] 0808 01:50:32.161715 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.072615 (+3910900us) service_pool.cc:224] Handling call
0808 01:50:36.076662 (+  4047us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:36.126375 (+ 49713us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.126603 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:36.111385  9476 rpcz_store.cc:276] 0808 01:50:32.431758 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:32.433334 (+  1576us) service_pool.cc:224] Handling call
0808 01:50:32.433786 (+   452us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:32.459056 (+ 25270us) raft_consensus.cc:1534] Early marking committed up to index 1
0808 01:50:32.459100 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:32.463375 (+  4275us) log.cc:844] Serialized 26 byte log entry
0808 01:50:32.767645 (+304270us) spinlock_profiling.cc:243] Waited 9.08 ms on lock 0x7b4800211158. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:50:35.749777 (+2982132us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:35.755410 (+  5633us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:35.774964 (+ 19554us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:36.008781 (+233817us) raft_consensus.cc:1685] finished
0808 01:50:36.008805 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.074151 (+ 65346us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8093,"raft.queue_time_us":27150,"raft.run_cpu_time_us":3606,"raft.run_wall_time_us":305176,"spinlock_wait_cycles":20881792,"thread_start_us":3106968,"threads_started":1,"wal-append.queue_time_us":3229251}
I20260808 01:50:36.137118 12493 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:36.136194 12309 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:36.138229 12364 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:36.075469 12477 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:36.139812 12493 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:36.140714  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 306) took 2330 ms. Trace:
I20260808 01:50:36.129325 12052 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:36.141389 12477 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:36.141770  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2209) took 4006 ms. Trace:
I20260808 01:50:36.142220 12052 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:36.144742  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2235) took 3502 ms. Trace:
I20260808 01:50:36.144971 12052 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:36.145226  9560 rpcz_store.cc:276] 0808 01:50:32.642365 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.119219 (+3476854us) service_pool.cc:224] Handling call
0808 01:50:36.143972 (+ 24753us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:36.144347 (+   375us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:36.144376 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:36.144439 (+    63us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:36.144489 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.144549 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.144653 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:36.125023  9552 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 01:50:36.087967  9472 rpcz_store.cc:276] 0808 01:50:32.432175 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:32.434283 (+  2108us) service_pool.cc:224] Handling call
0808 01:50:32.457708 (+ 23425us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:32.463211 (+  5503us) raft_consensus.cc:1534] Early marking committed up to index 1
0808 01:50:32.463253 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:32.463801 (+   548us) log.cc:844] Serialized 26 byte log entry
0808 01:50:35.768935 (+3305134us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:35.782796 (+ 13861us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:35.782948 (+   152us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:36.041849 (+258901us) raft_consensus.cc:1685] finished
0808 01:50:36.041873 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.059318 (+ 17445us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16436,"raft.queue_time_us":6131,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":18689,"thread_start_us":3198866,"threads_started":1,"wal-append.queue_time_us":3559707}
I20260808 01:50:36.151296  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2233) took 3547 ms. Trace:
I20260808 01:50:36.151736  9552 rpcz_store.cc:276] 0808 01:50:32.603742 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.111680 (+3507938us) service_pool.cc:224] Handling call
0808 01:50:36.124440 (+ 12760us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:36.148180 (+ 23740us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.151220 (+  3040us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:36.085258 11568 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:36.141108  9636 rpcz_store.cc:276] 0808 01:50:33.809928 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:33.810082 (+   154us) service_pool.cc:224] Handling call
0808 01:50:33.817723 (+  7641us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:33.818387 (+   664us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:33.818472 (+    85us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:33.819406 (+   934us) log.cc:844] Serialized 26 byte log entry
0808 01:50:34.225156 (+405750us) spinlock_profiling.cc:243] Waited 9.15 ms on lock 0x7b48002379d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:50:35.598230 (+1373074us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:35.638931 (+ 40701us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:35.639090 (+   159us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:36.140351 (+501261us) raft_consensus.cc:1685] finished
0808 01:50:36.140378 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.140625 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2600,"raft.run_cpu_time_us":5033,"raft.run_wall_time_us":394551,"spinlock_wait_cycles":21053824,"thread_start_us":1571823,"threads_started":1,"wal-append.queue_time_us":2181759}
I20260808 01:50:36.124933  9601 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:36.157429  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:36.119429 12067 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:36.165292 12067 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:36.165839 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:36.171062 11649 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:36.170416 12560 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
W20260808 01:50:36.177083  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 48339.2us
I20260808 01:50:36.174911  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2225) took 3924 ms. Trace:
I20260808 01:50:36.180971  9556 rpcz_store.cc:276] 0808 01:50:32.251061 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.087422 (+3836361us) service_pool.cc:224] Handling call
0808 01:50:36.112162 (+ 24740us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:36.147681 (+ 35519us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:36.147707 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:36.156965 (+  9258us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:36.174606 (+ 17641us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.174699 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.174814 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:36.178527  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:36.185377  9524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.014s	sys 0.014s
W20260808 01:50:36.191347  9524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.708s	user 0.014s	sys 0.015s
I20260808 01:50:36.142246  9558 rpcz_store.cc:276] 0808 01:50:32.135505 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:32.213185 (+ 77680us) service_pool.cc:224] Handling call
0808 01:50:32.236470 (+ 23285us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:33.880518 (+1644048us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:33.880565 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:33.881174 (+   609us) log.cc:844] Serialized 26 byte log entry
0808 01:50:35.810486 (+1929312us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:35.826148 (+ 15662us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.141360 (+315212us) spinlock_profiling.cc:243] Waited 81.8 ms on lock 0x7b540002d5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:36.141408 (+    48us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:36.141463 (+    55us) raft_consensus.cc:1685] finished
0808 01:50:36.141477 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.141710 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4543,"raft.queue_time_us":143803,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":6636,"spinlock_wait_cycles":188196096,"thread_start_us":3274132,"threads_started":2,"wal-append.queue_time_us":2083625}
I20260808 01:50:36.194872  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:36.205134  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 368) took 2866 ms. Trace:
I20260808 01:50:36.205703  9481 rpcz_store.cc:276] 0808 01:50:33.339041 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:35.729871 (+2390830us) service_pool.cc:224] Handling call
0808 01:50:35.735791 (+  5920us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:35.834813 (+ 99022us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:50:35.889564 (+ 54751us) log.cc:844] Serialized 14 byte log entry
0808 01:50:36.204666 (+315102us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:36.204729 (+    63us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:36.204793 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.204889 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.205056 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":38095,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":88,"thread_start_us":280729,"threads_started":1}
I20260808 01:50:36.212477 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:36.204000  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:36.211723 12576 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:36.198431 12583 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:36.240272  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2232) took 3637 ms. Trace:
I20260808 01:50:36.240903  9558 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260808 01:50:36.240803  9551 rpcz_store.cc:276] 0808 01:50:32.603406 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.120969 (+3517563us) service_pool.cc:224] Handling call
0808 01:50:36.194291 (+ 73322us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:36.233029 (+ 38738us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:36.233066 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:36.239960 (+  6894us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:36.240016 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.240079 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.240208 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:36.242620  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2244) took 2902 ms. Trace:
I20260808 01:50:36.246187  9558 rpcz_store.cc:276] 0808 01:50:33.340080 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.199260 (+2859180us) service_pool.cc:224] Handling call
0808 01:50:36.240401 (+ 41141us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:36.241787 (+  1386us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.242464 (+   677us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:36.250790  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 49232.8us
I20260808 01:50:36.261782  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2214) took 4101 ms. Trace:
I20260808 01:50:36.233520 12469 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 01:50:36.293934  9446 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.001s	sys 0.010s
I20260808 01:50:36.269336 10368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 548) took 3609 ms. Trace:
W20260808 01:50:36.298514  9446 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.001s	sys 0.010s
I20260808 01:50:36.235852 12030 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:36.256060 11568 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:36.297137  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2136) took 3469 ms. Trace:
I20260808 01:50:36.281028  9559 rpcz_store.cc:276] 0808 01:50:32.159949 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:33.263713 (+1103764us) service_pool.cc:224] Handling call
0808 01:50:33.306017 (+ 42304us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:33.306743 (+   726us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:33.306792 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:33.307700 (+   908us) log.cc:844] Serialized 52 byte log entry
0808 01:50:35.812376 (+2504676us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:36.243838 (+431462us) log.cc:844] Serialized 14 byte log entry
0808 01:50:36.248162 (+  4324us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.255422 (+  7260us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:36.255487 (+    65us) raft_consensus.cc:1685] finished
0808 01:50:36.255506 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.261721 (+  6215us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":15223,"raft.run_cpu_time_us":1551,"raft.run_wall_time_us":39908,"thread_start_us":2698763,"threads_started":2,"wal-append.queue_time_us":2788752}
W20260808 01:50:36.315605  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 859456us
I20260808 01:50:36.316185  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2118) took 3880 ms. Trace:
W20260808 01:50:36.324849 10363 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.009s	sys 0.043s
I20260808 01:50:36.310246 12310 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:36.336234  9557 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:50:36.337442  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2246) took 2938 ms. Trace:
I20260808 01:50:36.337921  9557 rpcz_store.cc:276] 0808 01:50:33.398906 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.234652 (+2835746us) service_pool.cc:224] Handling call
0808 01:50:36.316486 (+ 81834us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:36.337095 (+ 20609us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.337365 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:36.335183 12587 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
W20260808 01:50:36.364277  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:36.325443 10363 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.010s	sys 0.044s
I20260808 01:50:36.356132 10366 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:36.315770 11568 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:36.298007 10368 rpcz_store.cc:276] 0808 01:50:32.650730 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:32.650944 (+   214us) service_pool.cc:224] Handling call
0808 01:50:32.651397 (+   453us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:32.651656 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:50:32.667200 (+ 15544us) log.cc:844] Serialized 14 byte log entry
0808 01:50:36.196519 (+3529319us) log.cc:844] Serialized 14 byte log entry
0808 01:50:36.204654 (+  8135us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:36.221095 (+ 16441us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:36.237927 (+ 16832us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.245320 (+  7393us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.259720 (+ 14400us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":19987,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":63,"raft.queue_time_us":10803,"raft.run_cpu_time_us":454,"raft.run_wall_time_us":12806,"thread_start_us":3402612,"threads_started":2,"wal-append.queue_time_us":3333262}
W20260808 01:50:36.366873  9445 connection.cc:256] RPC call timeout handler was delayed by -1.6756134540191852s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19s for a response
I20260808 01:50:36.365183 12530 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:36.360994 12592 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260808 01:50:36.298151 12583 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:36.434049  9445 connection.cc:256] RPC call timeout handler was delayed by -1.6112984589999542s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260808 01:50:36.415232 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:36.312464  9474 rpcz_store.cc:276] 0808 01:50:32.828061 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:33.974788 (+1146727us) service_pool.cc:224] Handling call
0808 01:50:33.975245 (+   457us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:33.975545 (+   300us) raft_consensus.cc:1534] Early marking committed up to index 4
0808 01:50:35.727717 (+1752172us) log.cc:844] Serialized 14 byte log entry
0808 01:50:36.219466 (+491749us) log.cc:844] Serialized 14 byte log entry
0808 01:50:36.228672 (+  9206us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:36.236212 (+  7540us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:36.245676 (+  9464us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.254559 (+  8883us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.297028 (+ 42469us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7493,"prepare.queue_time_us":1774084,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":7531,"raft.queue_time_us":4687,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":22940,"spinlock_wait_cycles":43310208,"thread_start_us":2076555,"threads_started":2}
I20260808 01:50:36.320483  9479 rpcz_store.cc:276] 0808 01:50:32.436081 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:32.437740 (+  1659us) service_pool.cc:224] Handling call
0808 01:50:32.457080 (+ 19340us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:32.461620 (+  4540us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:32.461665 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:32.467667 (+  6002us) log.cc:844] Serialized 26 byte log entry
0808 01:50:34.055610 (+1587943us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:35.717608 (+1661998us) log.cc:844] Serialized 14 byte log entry
0808 01:50:36.275468 (+557860us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.300325 (+ 24857us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:36.306627 (+  6302us) raft_consensus.cc:1685] finished
0808 01:50:36.306655 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.316113 (+  9458us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25093,"prepare.queue_time_us":1769429,"prepare.run_cpu_time_us":124,"prepare.run_wall_time_us":125,"raft.queue_time_us":21608,"raft.run_cpu_time_us":429,"raft.run_wall_time_us":39503,"thread_start_us":3562174,"threads_started":3,"wal-append.queue_time_us":1534198,"wal-append.run_cpu_time_us":5860,"wal-append.run_wall_time_us":1668712}
I20260808 01:50:36.396332  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:36.336365 12571 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:36.379043  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:36.383926  9555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 2226) took 3945 ms (client timeout 1665 ms). Trace:
I20260808 01:50:36.388670  9599 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:50:36.435719  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 962974us
I20260808 01:50:36.387082  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:36.435997  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 310972us
I20260808 01:50:36.384093 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:36.383332 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:36.446168 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 111098us
W20260808 01:50:36.474232  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 109137us
W20260808 01:50:36.382840 12565 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
W20260808 01:50:36.445818  9634 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.003s	sys 0.008s
W20260808 01:50:36.480166  9634 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.004s	sys 0.008s
W20260808 01:50:36.466849  9555 rpcz_store.cc:269] 0808 01:50:32.370099 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:32.425828 (+ 55729us) service_pool.cc:224] Handling call
0808 01:50:36.259159 (+3833331us) spinlock_profiling.cc:243] Waited 3.4 s on lock 0x7b540008a7d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:36.315235 (+ 56076us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7812647808}
I20260808 01:50:36.434935 11568 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:36.482995 11980 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:36.483340 10369 raft_consensus.cc:2410] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260808 01:50:36.480964  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:36.483248 12583 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:36.485663  9559 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:50:36.507802  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 348) took 2383 ms. Trace:
I20260808 01:50:36.508375  9559 rpcz_store.cc:276] 0808 01:50:34.124603 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.362005 (+2237402us) service_pool.cc:224] Handling call
0808 01:50:36.445459 (+ 83454us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:36.507417 (+ 61958us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.507714 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:36.486049  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:36.486334 12310 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:36.381079 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:36.519173 10367 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
W20260808 01:50:36.510622  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 395986us
I20260808 01:50:36.488999  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 346) took 2372 ms. Trace:
I20260808 01:50:36.383637 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:36.487250 11867 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:36.389153 12052 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.110s
W20260808 01:50:36.532361  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 143775us
I20260808 01:50:36.524720  9473 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:36.384292 12367 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:36.529268  9558 rpcz_store.cc:276] 0808 01:50:34.104633 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.261003 (+2156370us) service_pool.cc:224] Handling call
0808 01:50:36.461181 (+200178us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:36.461522 (+   341us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:36.461565 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:36.472810 (+ 11245us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:36.472913 (+   103us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.473026 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.476858 (+  3832us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:36.529604  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:36.518298 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:36.534332  9481 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:36.383127 12448 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:36.529867  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:36.551770 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:36.390257 12309 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.134s
I20260808 01:50:36.546108  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:36.521718 12450 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:36.549441  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2121) took 4110 ms. Trace:
I20260808 01:50:36.555735 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:50:36.542048 12551 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.612s	user 0.011s	sys 0.005s
I20260808 01:50:36.568504  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 347) took 2450 ms. Trace:
I20260808 01:50:36.572392 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:36.571923  9557 rpcz_store.cc:276] 0808 01:50:34.117849 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.368006 (+2250157us) service_pool.cc:224] Handling call
0808 01:50:36.473613 (+105607us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:36.551578 (+ 77965us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:36.551607 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:36.551657 (+    50us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:50:36.568244 (+ 16587us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.568330 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.568445 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:36.573267 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:36.562397 11867 raft_consensus.cc:697] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:36.575821 12420 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:36.576330  9475 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 7: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:36.527515 12571 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:36.577571  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:36.531755 12530 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:36.578333  9475 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:36.579295 12571 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:36.572826 12450 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:36.580603 12530 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:36.582273 10360 raft_consensus.cc:2410] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260808 01:50:36.583057 12067 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:36.583434  9474 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:36.584893 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:36.543527 10363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 563) took 2380 ms. Trace:
I20260808 01:50:36.576689 12420 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:36.589855  9480 raft_consensus.cc:2410] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:36.576695 11867 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:36.599074 10367 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 7: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:36.557801  9478 rpcz_store.cc:276] 0808 01:50:32.439559 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:33.458395 (+1018836us) service_pool.cc:224] Handling call
0808 01:50:33.459115 (+   720us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:35.743752 (+2284637us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:35.750535 (+  6783us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:35.861028 (+110493us) log.cc:844] Serialized 26 byte log entry
0808 01:50:36.228872 (+367844us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:36.228926 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.229013 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:36.549088 (+320075us) raft_consensus.cc:1685] finished
0808 01:50:36.549117 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.549354 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":208958,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":167,"thread_start_us":2491037,"threads_started":2,"wal-append.queue_time_us":608810}
I20260808 01:50:36.599738 12448 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:36.600281 12375 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:36.587909  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:36.602016 12541 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:36.602931  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:36.604359  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:36.607301 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:36.609079 12355 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:50:36.611600  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:36.612232  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2105) took 5196 ms. Trace:
I20260808 01:50:36.613642 12576 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:36.654917 12576 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:36.660597 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:36.660955  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:36.667222  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:36.591471  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 295) took 3962 ms. Trace:
W20260808 01:50:36.621969 12556 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.715s	user 0.000s	sys 0.002s
I20260808 01:50:36.620925 10363 rpcz_store.cc:276] 0808 01:50:34.160914 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:34.227503 (+ 66589us) service_pool.cc:224] Handling call
0808 01:50:34.390401 (+162898us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:34.549368 (+158967us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:34.549406 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:35.710642 (+1161236us) log.cc:844] Serialized 52 byte log entry
0808 01:50:36.424588 (+713946us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:36.441020 (+ 16432us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.491674 (+ 50654us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:36.511861 (+ 20187us) raft_consensus.cc:1685] finished
0808 01:50:36.511890 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.541386 (+ 29496us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":23299,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":33861,"thread_start_us":654526,"threads_started":1,"wal-append.queue_time_us":641395}
W20260808 01:50:36.622956 12601 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.139s	user 0.002s	sys 0.000s
I20260808 01:50:36.624703  9551 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20260808 01:50:36.627163  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 48259us
I20260808 01:50:36.627738 12052 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:50:36.627208  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:36.629439 12310 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:36.712071  9559 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:36.591327  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:36.630216 12067 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:36.736446  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:36.650570  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:36.651726  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:36.653728  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:36.648141 12375 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:36.674172  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:36.674507 11568 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:36.676483  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:36.677628  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 110685us
I20260808 01:50:36.736672 12530 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:36.673907 12505 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:36.648903  9477 rpcz_store.cc:276] 0808 01:50:31.415703 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:31.518475 (+102772us) service_pool.cc:224] Handling call
0808 01:50:31.518813 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:32.604470 (+1085657us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:32.604828 (+   358us) log.cc:844] Serialized 14 byte log entry
0808 01:50:35.730161 (+3125333us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:35.755615 (+ 25454us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:35.808102 (+ 52487us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.611830 (+803728us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54002dc3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:36.611898 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.612108 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":780,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":1584970,"raft.run_cpu_time_us":338,"raft.run_wall_time_us":32669,"spinlock_wait_cycles":2495758208,"thread_start_us":2982481,"threads_started":2,"wal-append.queue_time_us":1583324}
I20260808 01:50:36.816632 12072 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Stepping down as leader of term 5
W20260808 01:50:36.814574  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 75099.6us
I20260808 01:50:36.648054 12541 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:50:36.819651  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:36.823540 12498 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:36.824155  9557 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:36.824510  9637 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:36.825304  9481 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:36.812656 10332 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:36.737673  9630 raft_consensus.cc:2393] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:36.838189 12448 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:36.843137 11980 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:36.830318  9475 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
W20260808 01:50:36.844707  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 299990us
I20260808 01:50:36.672231  9634 rpcz_store.cc:276] 0808 01:50:32.628809 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:32.632402 (+  3593us) service_pool.cc:224] Handling call
0808 01:50:32.632740 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:32.632977 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:50:34.285764 (+1652787us) log.cc:844] Serialized 14 byte log entry
0808 01:50:36.481106 (+2195342us) log.cc:844] Serialized 14 byte log entry
0808 01:50:36.534052 (+ 52946us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:36.543112 (+  9060us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:36.543253 (+   141us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.561803 (+ 18550us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.591396 (+ 29593us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53804,"prepare.queue_time_us":510849,"prepare.run_cpu_time_us":180,"prepare.run_wall_time_us":34363,"raft.queue_time_us":82265,"raft.run_cpu_time_us":365,"raft.run_wall_time_us":15108,"thread_start_us":3667160,"threads_started":3,"wal-append.queue_time_us":1891219}
I20260808 01:50:36.781404  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2234) took 4123 ms. Trace:
I20260808 01:50:36.792209  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:36.812496 12419 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:36.739374  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2249) took 2925 ms. Trace:
I20260808 01:50:36.896346  9551 rpcz_store.cc:276] 0808 01:50:33.773078 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.255884 (+2482806us) service_pool.cc:224] Handling call
0808 01:50:36.623790 (+367906us) spinlock_profiling.cc:243] Waited 3.89 ms on lock 0x7b5800003c98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:50:36.624127 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:36.697936 (+ 73809us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.698265 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8936704}
I20260808 01:50:36.737437  9480 raft_consensus.cc:3055] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:36.811719  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:36.777904  9473 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:36.847671 12072 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:36.912632 12072 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:36.914057  9478 raft_consensus.cc:2410] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 8 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:50:36.914227 12072 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:36.919636 12450 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:36.847574 12375 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:50:36.875901  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 344357us
I20260808 01:50:36.889436 12052 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:36.892161 12063 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:50:36.737838 12067 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:36.911151 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 131174us
I20260808 01:50:36.884604 11980 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:36.930140  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:36.930912  9552 raft_consensus.cc:2376] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5. Re-sending same reply.
I20260808 01:50:36.930135  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:36.925016  9480 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:36.886891  9554 rpcz_store.cc:276] 0808 01:50:32.604052 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.119219 (+3515167us) service_pool.cc:224] Handling call
0808 01:50:36.153489 (+ 34270us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:36.153906 (+   417us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:36.314608 (+160702us) log.cc:844] Serialized 14 byte log entry
0808 01:50:36.516784 (+202176us) log.cc:844] Serialized 14 byte log entry
0808 01:50:36.629762 (+112978us) spinlock_profiling.cc:243] Waited 1.02 ms on lock 0x7b1800291880. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667 00007f5d449ae5e8
0808 01:50:36.726403 (+ 96641us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:36.726502 (+    99us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:36.726572 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.726672 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.726890 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":648474,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":58,"raft.queue_time_us":48600,"raft.run_cpu_time_us":372,"raft.run_wall_time_us":32748,"spinlock_wait_cycles":2345216,"thread_start_us":410202,"threads_started":3,"wal-append.queue_time_us":249390}
I20260808 01:50:36.928309  9636 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:50:36.935571 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:36.932355  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:36.946081  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2237) took 4296 ms. Trace:
I20260808 01:50:36.947762  9556 rpcz_store.cc:276] 0808 01:50:32.650189 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.190552 (+3540363us) service_pool.cc:224] Handling call
0808 01:50:36.255767 (+ 65215us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:36.383908 (+128141us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:36.540324 (+156416us) log.cc:844] Serialized 14 byte log entry
0808 01:50:36.945560 (+405236us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:36.945643 (+    83us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:36.945720 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.945816 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.945992 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5572,"prepare.queue_time_us":275636,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"thread_start_us":451430,"threads_started":2}
I20260808 01:50:36.951644  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:36.948518  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2242) took 4020 ms. Trace:
I20260808 01:50:36.954180  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:36.955382  9557 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 5: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:36.963397  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:36.967729  9559 raft_consensus.cc:2410] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260808 01:50:36.953239  9550 rpcz_store.cc:276] 0808 01:50:32.928215 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.197174 (+3268959us) service_pool.cc:224] Handling call
0808 01:50:36.243446 (+ 46272us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:36.260802 (+ 17356us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:50:36.538183 (+277381us) log.cc:844] Serialized 14 byte log entry
0808 01:50:36.826491 (+288308us) spinlock_profiling.cc:243] Waited 9.48 ms on lock 0x7b18003b52a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188ce62 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:50:36.948120 (+121629us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:36.948199 (+    79us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:36.948264 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.948340 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.948460 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16474,"prepare.queue_time_us":382413,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":53583,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":18773,"spinlock_wait_cycles":21806336,"thread_start_us":516589,"threads_started":2}
I20260808 01:50:36.981324  9556 raft_consensus.cc:3055] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:36.981794  9556 raft_consensus.cc:740] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:36.982264  9637 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:36.983490  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:36.936556 12571 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:36.990455  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 49468.7us
I20260808 01:50:36.964483 12450 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260808 01:50:36.999879  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 106039us
I20260808 01:50:36.954394 12583 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:36.986526  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 352) took 2792 ms. Trace:
I20260808 01:50:37.003017  9551 rpcz_store.cc:276] 0808 01:50:34.194387 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.933519 (+2739132us) service_pool.cc:224] Handling call
0808 01:50:36.985803 (+ 52284us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:36.986071 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:36.986106 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:36.986169 (+    63us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:36.986214 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:36.986290 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:36.986435 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:36.996083  9557 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260808 01:50:36.996662  9555 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260808 01:50:37.006387  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 351) took 2847 ms. Trace:
I20260808 01:50:37.011293  9555 rpcz_store.cc:276] 0808 01:50:34.159502 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.944737 (+2785235us) service_pool.cc:224] Handling call
0808 01:50:36.990772 (+ 46035us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:37.006052 (+ 15280us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:37.006312 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:37.013618  9558 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260808 01:50:37.019544 12603 consensus_queue.cc:1059] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260808 01:50:37.007688  9480 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:37.020963 12603 raft_consensus.cc:3055] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:37.021116  9480 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:37.021452 12603 raft_consensus.cc:740] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:37.014362  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2253) took 1169 ms. Trace:
I20260808 01:50:37.022917  9558 rpcz_store.cc:276] 0808 01:50:35.857902 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.945361 (+1087459us) service_pool.cc:224] Handling call
0808 01:50:36.984679 (+ 39318us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:37.014103 (+ 29424us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:37.014302 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:37.023524 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 98918us
I20260808 01:50:37.004068 11867 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:37.033066 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:37.059167  9481 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:37.041854 10332 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:37.064061  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2254) took 1186 ms. Trace:
I20260808 01:50:37.018313 12571 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:37.064832  9554 rpcz_store.cc:276] 0808 01:50:35.877433 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.945531 (+1068098us) service_pool.cc:224] Handling call
0808 01:50:36.975901 (+ 30370us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:37.062339 (+ 86438us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:50:37.062391 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:37.062458 (+    67us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:37.062506 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:37.062600 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:37.063972 (+  1372us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5295,"raft.queue_time_us":15598}
W20260808 01:50:37.068796  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 146001us
I20260808 01:50:37.069011  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:37.071794  9475 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:37.083858  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:37.091879 12460 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:37.097652 11980 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:37.101502  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:37.076054  9556 consensus_queue.cc:260] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:37.103401 11568 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:37.109297  9481 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:37.111616  9636 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:37.114358 12355 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:37.114924 12355 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:37.115295 12355 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:37.070355 12450 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:37.125381  9634 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
W20260808 01:50:37.123132 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 81594us
I20260808 01:50:37.123675  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:37.124464  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:37.122181 11980 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:37.022488 12603 consensus_queue.cc:260] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:37.135169 11568 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:37.136138  9601 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:37.141168 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:37.177673  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 239696us
I20260808 01:50:37.178965  9556 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:37.199156  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:37.200738 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 128782us
I20260808 01:50:37.178254 11568 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:37.230877 11568 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:37.235611  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 115794us
I20260808 01:50:37.235498 12541 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:37.237291  9521 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:37.239045  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:37.251785 12571 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:37.252029 11867 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:37.252514  9556 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260808 01:50:37.254245  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 118259us
I20260808 01:50:37.254549  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2258) took 1141 ms. Trace:
W20260808 01:50:37.253584 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 127086us
I20260808 01:50:37.253511  9636 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:37.256235  9630 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:37.257844  9556 rpcz_store.cc:276] 0808 01:50:36.113555 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.961262 (+847707us) service_pool.cc:224] Handling call
0808 01:50:36.981092 (+ 19830us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:37.254258 (+273166us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:37.254486 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:37.261838  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:37.265424  9637 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:37.269696 11568 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:37.262926  9444 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.048s (SENT) [suppressed 14 similar messages]
W20260808 01:50:37.256428 12576 consensus_meta.cc:287] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.601s	user 0.014s	sys 0.000s
W20260808 01:50:37.276311  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 46160.4us
I20260808 01:50:37.278271 11568 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:37.283192  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:37.287384  9474 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:37.294513  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:37.281461 12450 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:37.279548 12603 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:37.300137  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:37.305035  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:37.311721  9481 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:37.286630 12541 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:37.280326 12477 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.027s
I20260808 01:50:37.318958  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:37.323791  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:37.323966  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:37.324707  9630 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:37.324672  9637 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 2.
W20260808 01:50:37.324079  9444 connection.cc:256] RPC call timeout handler was delayed by -4.4672207239782438s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1502s for a response
W20260808 01:50:37.326422  9444 connection.cc:256] RPC call timeout handler was delayed by -4.4555207239754964s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1515s for a response
W20260808 01:50:37.326730  9444 connection.cc:256] RPC call timeout handler was delayed by -4.4078207239799667s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15680000000000002s for a response
W20260808 01:50:37.327054  9444 connection.cc:256] RPC call timeout handler was delayed by -4.0613207239948679s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19530000000000003s for a response
W20260808 01:50:37.327288  9444 connection.cc:256] RPC call timeout handler was delayed by -4.040620723972097s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1976s for a response
W20260808 01:50:37.327601  9444 connection.cc:256] RPC call timeout handler was delayed by -4.0226207239902578s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1996s for a response
W20260808 01:50:37.327841  9444 connection.cc:256] RPC call timeout handler was delayed by -3.0488207239832263s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3078s for a response
W20260808 01:50:37.328083  9444 connection.cc:256] RPC call timeout handler was delayed by -2.9444207239721436s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3194s for a response
W20260808 01:50:37.328307  9444 connection.cc:256] RPC call timeout handler was delayed by -2.8895207239838783s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3255s for a response
W20260808 01:50:37.328534  9444 connection.cc:256] RPC call timeout handler was delayed by -2.8022207239700947s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3352s for a response
W20260808 01:50:37.329664  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:37.283375 12576 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:37.335739  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 78114.1us
W20260808 01:50:37.339588 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 137849us
I20260808 01:50:37.344805  9475 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7. Ops: []
W20260808 01:50:37.347961  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:37.346235  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:37.330957 12583 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:37.329106 12367 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:37.355562  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 72134.6us
I20260808 01:50:37.327222 12571 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:37.335522 12634 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:37.388093  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:37.385012 11568 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.004s
I20260808 01:50:37.389892  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:37.349207 12420 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:37.394906  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:37.349884 12621 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:37.398658  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:37.400775  9630 raft_consensus.cc:2410] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260808 01:50:37.399047  9475 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:37.405861  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:37.406926  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:37.388144 12627 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:37.387187 12541 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:37.393910 12598 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:37.393064 12634 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:37.417641  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 71503.2us
I20260808 01:50:37.410307 11568 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.009s
I20260808 01:50:37.419730  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:37.422554  9630 raft_consensus.cc:2410] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260808 01:50:37.390177 12355 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:37.439782  9477 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:37.440780 11568 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:50:37.395592 12583 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:37.396431 12643 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
W20260808 01:50:37.447283  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 45763.8us
I20260808 01:50:37.420533 12063 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:37.460711 12561 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:37.425375 12634 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:37.416983 12571 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:37.424449 12627 raft_consensus.cc:697] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:37.463747  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:37.465943  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 78493.9us
I20260808 01:50:37.427639 12564 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:37.480930  9473 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:37.481328  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:37.482717 12072 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.538s	user 0.009s	sys 0.003s
I20260808 01:50:37.484403 12355 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:37.485486  9476 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 01:50:37.487001  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:37.414400 12367 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:37.491868  9479 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260808 01:50:37.500612  9481 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
W20260808 01:50:37.488473 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 194546us
I20260808 01:50:37.506246  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:37.511777  9630 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
W20260808 01:50:37.511798 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 171081us
W20260808 01:50:37.467034 12375 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.542s	user 0.015s	sys 0.000s
I20260808 01:50:37.443604 12598 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:37.518023 11568 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:37.518522  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:37.522295  9473 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:37.525513  9601 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:37.528649  9477 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:37.531608  9475 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:37.535348  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:37.535482  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:37.536576  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 374) took 1360 ms. Trace:
I20260808 01:50:37.536944  9472 rpcz_store.cc:276] 0808 01:50:36.167892 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.675172 (+507280us) service_pool.cc:224] Handling call
0808 01:50:36.675605 (+   433us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:36.848757 (+173152us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:36.848786 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:36.902009 (+ 53223us) log.cc:844] Serialized 52 byte log entry
0808 01:50:37.183127 (+281118us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:37.183198 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:37.183305 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:37.527889 (+344584us) raft_consensus.cc:1685] finished
0808 01:50:37.527916 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:37.528101 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28911,"raft.queue_time_us":134291,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":30167,"thread_start_us":280593,"threads_started":1,"wal-append.queue_time_us":427397}
I20260808 01:50:37.545964  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:37.546474  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:37.546098  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:37.547824  9475 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 01:50:37.553042  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:37.554286 12460 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:37.554953 12460 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:37.559284  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:37.520452 12564 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:37.563686  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:37.564488 12571 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:37.564728  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:37.565651  9560 raft_consensus.cc:2410] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260808 01:50:37.565819  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:37.566525  9555 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
W20260808 01:50:37.566982  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 48457.5us
I20260808 01:50:37.567830  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:37.564834  9550 raft_consensus.cc:2410] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260808 01:50:37.568629 12063 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:37.569605 12564 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:37.575659 12639 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260808 01:50:37.575173 12627 consensus_queue.cc:237] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:37.577682 12642 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260808 01:50:37.577816 12460 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:37.579254  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:37.580614  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:37.582273  9556 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:37.553258  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:37.584611  9481 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:37.583886  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:37.586084  9474 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:37.587344 12583 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:37.588145 12367 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:37.591708  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2142) took 1220 ms. Trace:
I20260808 01:50:37.536260 12072 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20260808 01:50:37.523727 12067 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.060s
I20260808 01:50:37.608392 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 444) took 6128 ms. Trace:
I20260808 01:50:37.601253 12063 raft_consensus.cc:697] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:37.612142 12063 consensus_queue.cc:237] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:37.613049  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:37.614733  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:37.615761 12530 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:37.616595  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:37.616503 12530 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:37.598316  9473 raft_consensus.cc:2410] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260808 01:50:37.599813  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:37.597321  9480 rpcz_store.cc:276] 0808 01:50:36.371578 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:36.688271 (+316693us) service_pool.cc:224] Handling call
0808 01:50:36.737218 (+ 48947us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:37.591003 (+853785us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:37.591066 (+    63us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:37.591180 (+   114us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:37.591242 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:37.591379 (+   137us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:37.591592 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2221,"raft.queue_time_us":89283,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":8335,"thread_start_us":167549,"threads_started":1}
W20260808 01:50:37.554979 12640 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20260808 01:50:37.522925 12375 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:37.603662  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 84672.5us
I20260808 01:50:37.604732  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:37.605093 12629 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:37.608068 12641 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
I20260808 01:50:37.619349  9474 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:37.618602  9478 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:37.545174 12576 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:37.600730 12571 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:37.634604  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:37.633886 10334 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:37.642236  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:37.645267  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:37.623387 12460 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:37.621429 12498 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:37.651758  9476 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:37.630909 12598 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:37.660179  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:37.660990 11867 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:37.673424 12632 log.cc:927] Time spent T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Append to log took a long time: real 0.274s	user 0.003s	sys 0.000s
I20260808 01:50:37.675051 12530 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:37.672791 11867 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:37.684967 11867 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:37.692344  9480 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:37.664808 12072 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:37.696776  9474 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:37.665663 12498 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:37.699321  9480 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:37.695855  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:37.717929  9481 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:50:37.702164 12375 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:37.766544  9473 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:37.771306  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:37.711997 12564 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:37.796079 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:37.835675  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 64499us
I20260808 01:50:37.834785  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:37.815085  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 166064us
I20260808 01:50:37.805435  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:37.837960  9472 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:37.709594 12629 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:37.769351 12571 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:37.853855  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:37.857441  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:37.805663 12564 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:37.811460 12530 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:37.842629 12646 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:37.872449 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:37.871292  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 303540us
I20260808 01:50:37.865798  9600 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:37.896649  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:37.894872 10365 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:37.811244 11867 consensus_queue.cc:1059] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.081s
I20260808 01:50:37.905579  9601 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:37.917917  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:37.914863  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:37.918788  9551 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:37.919521 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:37.894609 12564 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:37.871491 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:37.609001 10361 rpcz_store.cc:276] 0808 01:50:31.479414 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:31.482847 (+  3433us) service_pool.cc:224] Handling call
0808 01:50:31.534876 (+ 52029us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:31.819505 (+284629us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:50:32.176819 (+357314us) log.cc:844] Serialized 14 byte log entry
0808 01:50:35.888511 (+3711692us) spinlock_profiling.cc:243] Waited 1.37 ms on lock 0x7b180037a080. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d41889e05 00007f5d483aada7 00007f5d4499dd87 00007f5d449ad49f 00007f5d449ad418
0808 01:50:36.432778 (+544267us) log.cc:844] Serialized 14 byte log entry
0808 01:50:37.066145 (+633367us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:37.080504 (+ 14359us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:37.091186 (+ 10682us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:37.607919 (+516733us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b5400512cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:37.607992 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:37.608171 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14828,"prepare.queue_time_us":2232419,"prepare.run_cpu_time_us":143,"prepare.run_wall_time_us":16563,"raft.queue_time_us":70014,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":69631,"spinlock_wait_cycles":595380352,"thread_start_us":3215993,"threads_started":5,"wal-append.queue_time_us":2901821,"wal-append.run_cpu_time_us":1869,"wal-append.run_wall_time_us":1163565}
I20260808 01:50:37.893203 12530 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:37.981930  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:37.989349  9475 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 5: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:50:37.997716  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 101232us
I20260808 01:50:37.917467 12498 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:38.001969  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:38.013470  9551 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:37.913902 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:38.009431 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:38.016201  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:38.016925 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 164852us
I20260808 01:50:38.017664  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:37.995594 12535 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260808 01:50:37.982745 12477 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:37.989096 12530 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:38.025804  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:38.024349  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:38.034595 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:38.036046  9474 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
W20260808 01:50:38.028540  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 181397us
I20260808 01:50:37.922536 12646 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:38.046473  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:38.047138  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:38.016078 12656 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:38.015246 12375 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:38.027794 12498 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260808 01:50:37.985476 12634 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:38.072544  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 49205.6us
W20260808 01:50:38.073112  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 56961.7us
W20260808 01:50:38.074779  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 219772us
I20260808 01:50:38.076263 10366 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260808 01:50:38.075340  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:38.084653  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:38.061352 12477 raft_consensus.cc:697] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:38.086453  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:38.093021 12629 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:38.093220  9664 heartbeater.cc:646] Failed to heartbeat to 127.7.165.254:43565 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.7.165.254:43565 timed out after 15.000s (SENT)
I20260808 01:50:38.095727  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:38.096969  9480 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:38.105386  9475 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:38.109648 12662 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:38.137341 12662 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:38.137768  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:38.138435  9559 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:38.143278  9472 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:38.155213  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:38.156353  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 81971.3us
I20260808 01:50:38.108413 11867 raft_consensus.cc:3055] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:38.120702 12646 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:38.163738  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:38.168851  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 79817.2us
I20260808 01:50:38.169770  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:38.120312 12477 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:38.171912 12662 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:38.172492  9552 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:38.173857  9552 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:38.174237  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:38.159078  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:38.162137  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:38.160398  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:38.120903 12661 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:38.177176  9559 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:38.186753  9550 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:38.186641  9479 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:38.156013 12364 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260808 01:50:38.196167  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 110859us
I20260808 01:50:38.202251  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:38.208339  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 46342.4us
I20260808 01:50:38.159242 11867 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:38.219909  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:38.224833 12674 raft_consensus.cc:3055] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:38.225445 12674 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:38.226192 12678 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:38.230615  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:38.231127  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:38.236969  9478 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:38.135257  9630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0ea307d9adc442539683547d0176a05b"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:50:38.238404  9630 raft_consensus.cc:606] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:38.238986  9630 raft_consensus.cc:614] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:50:38.239017 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:50:38.241398  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 0ea307d9adc442539683547d0176a05b (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:50:38.203629 12598 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:50:38.243559  9474 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 3.
W20260808 01:50:38.243906 12653 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.173s	user 0.001s	sys 0.000s
W20260808 01:50:38.245050 12660 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260808 01:50:38.245313  9630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:50:38.245987  9630 raft_consensus.cc:606] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:38.226559 12674 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:38.236090 12067 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:38.236137  9599 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:38.257683  9550 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:38.260068  9472 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
W20260808 01:50:38.258738 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 45772.1us
I20260808 01:50:38.205020 12364 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:38.270749 12674 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:38.271915 11867 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:38.272675 12603 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.959s	user 0.010s	sys 0.003s
I20260808 01:50:38.267711  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:38.274788  9475 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:38.276958  9480 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:38.279547  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:38.270927 12364 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:38.291358  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 88415.6us
I20260808 01:50:38.292750  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:38.294642  9444 leader_election.cc:400] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6.
I20260808 01:50:38.296804  9444 leader_election.cc:403] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260808 01:50:38.298926  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:38.299681 12571 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6.
I20260808 01:50:38.291818  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:38.294946 12603 raft_consensus.cc:2764] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260808 01:50:38.295291  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:38.304941  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:38.306205  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:38.291491 12658 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.212s	user 0.001s	sys 0.000s
I20260808 01:50:38.307974  9474 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:38.308919  9477 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:38.291774 12375 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:38.310807  9472 raft_consensus.cc:2410] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20260808 01:50:38.316659  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:38.317752 12634 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:38.318763 12667 consensus_queue.cc:1059] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:50:38.319195 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:38.322726  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:38.323467 12375 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election won for term 6
W20260808 01:50:38.286652  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 50652.9us
I20260808 01:50:38.325268 12634 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:50:38.326512 12684 raft_consensus.cc:3055] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:38.326967 12684 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:38.328085 12684 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:38.328936 12677 raft_consensus.cc:3055] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:38.329435 12677 raft_consensus.cc:740] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:38.341176  9478 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:38.285775 12067 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:38.330603 12677 consensus_queue.cc:260] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:38.345204 12655 consensus_queue.cc:1059] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.005s
I20260808 01:50:38.397221  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:38.419279  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:38.400425  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 304925us
I20260808 01:50:38.345654  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:38.389915  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:38.595240  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:38.388093  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:38.495862  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:38.598181  9474 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:50:38.419976  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:38.525007  9445 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:38.612320  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:38.636617 12672 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:38.638516 12672 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:38.640878 12683 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:38.641124  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:38.641934  9630 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:38.641556 12683 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:38.643530  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:38.644539 12683 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:38.338546 12375 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 6 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:38.645134  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:38.645998  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:38.647053  9629 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: []
W20260808 01:50:38.647332  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50563.3us
I20260808 01:50:38.340575 12603 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election won for term 9
W20260808 01:50:38.563491 12564 consensus_meta.cc:287] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.571s	user 0.013s	sys 0.000s
I20260808 01:50:38.652292  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:38.653134 12564 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:38.656668 12564 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:38.647094  9631 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:38.649417  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:38.581552 12627 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.909s	user 0.010s	sys 0.005s
I20260808 01:50:38.659737  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:38.664965  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:38.667276  9558 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 7
I20260808 01:50:38.670260  9636 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
W20260808 01:50:38.663998 12627 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.992s	user 0.011s	sys 0.006s
W20260808 01:50:38.663480 12656 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.011s	sys 0.025s
I20260808 01:50:38.506531 12530 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:38.678066  9637 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:38.680260  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2279) took 1108 ms. Trace:
I20260808 01:50:38.682967  9551 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 8: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:38.689848 10334 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:38.696825 12670 log.cc:927] Time spent T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.547s	user 0.003s	sys 0.000s
I20260808 01:50:38.697435  9524 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:38.699740 12678 raft_consensus.cc:3055] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:38.703562  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1939) took 1097 ms. Trace:
I20260808 01:50:38.704162 12692 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.016s
I20260808 01:50:38.680948  9552 rpcz_store.cc:276] 0808 01:50:37.571603 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:38.199321 (+627718us) service_pool.cc:224] Handling call
0808 01:50:38.216744 (+ 17423us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:38.223061 (+  6317us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:38.223092 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:38.223782 (+   690us) log.cc:844] Serialized 52 byte log entry
0808 01:50:38.434394 (+210612us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:38.441689 (+  7295us) log.cc:844] Serialized 14 byte log entry
0808 01:50:38.441925 (+   236us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:38.442007 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:38.518414 (+ 76407us) spinlock_profiling.cc:243] Waited 8.81 ms on lock 0x7b48000c03d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:50:38.679835 (+161421us) raft_consensus.cc:1685] finished
0808 01:50:38.679976 (+   141us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:38.680173 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50717,"prepare.queue_time_us":23402,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":10794,"raft.queue_time_us":233631,"raft.run_cpu_time_us":2743,"raft.run_wall_time_us":198060,"spinlock_wait_cycles":20267776,"thread_start_us":210160,"threads_started":1,"wal-append.queue_time_us":384099}
I20260808 01:50:38.663535 12364 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:38.661799 12663 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.017s
I20260808 01:50:38.721771  9476 rpcz_store.cc:276] 0808 01:50:37.606151 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:37.614271 (+  8120us) service_pool.cc:224] Handling call
0808 01:50:38.703238 (+1088967us) spinlock_profiling.cc:243] Waited 35.9 ms on lock 0x7b54004047d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:38.703468 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":82628480}
I20260808 01:50:38.706782  9475 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:38.688247 12530 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:38.727327  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:38.677033 12656 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.011s	sys 0.025s
W20260808 01:50:38.736644  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 57789.7us
I20260808 01:50:38.738430  9637 raft_consensus.cc:2410] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260808 01:50:38.738785  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:38.702080 12375 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:38.751385  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:38.752844  9633 raft_consensus.cc:2368] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8.
I20260808 01:50:38.724395 12678 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:38.760952 12678 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:38.726235 12627 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:38.762976 12627 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:38.748178 12656 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:38.763545 12627 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:38.764921 12656 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:38.742004 12672 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:38.767177  9476 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:38182
I20260808 01:50:38.767930  9476 raft_consensus.cc:606] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:38.768362  9476 raft_consensus.cc:614] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:50:38.771203 12698 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
W20260808 01:50:38.773455 12674 consensus_meta.cc:287] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.502s	user 0.009s	sys 0.005s
I20260808 01:50:38.792547  9558 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
W20260808 01:50:38.799808  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 61201.8us
I20260808 01:50:38.812017 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:38.823396 12692 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:50:38.824673 12692 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:50:38.825083 12692 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:38.749819 12691 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:38.827443 12691 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:38.827877 12691 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:38.799546 12695 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:38.831965 12364 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:38.832736 12676 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:38.815358 12663 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:38.835006 12663 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:38.836198 12663 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:38.840730  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:38.842430 12541 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:38.843022 12541 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:38.829939 12695 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:38.851915 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:38.852731 10360 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260808 01:50:38.856117  9635 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 01:50:38.856820  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:38.833364 12676 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:38.858753  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:38.860658 12676 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:38.801246 12678 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:38.862146  9631 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:38.863385  9629 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:38.865899 12684 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:38.868824  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:38.878191  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2281) took 1305 ms. Trace:
I20260808 01:50:38.870641  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:38.878868  9557 rpcz_store.cc:276] 0808 01:50:37.572318 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:38.227666 (+655348us) service_pool.cc:224] Handling call
0808 01:50:38.228104 (+   438us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:38.461865 (+233761us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:38.461908 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:38.549946 (+ 88038us) log.cc:844] Serialized 52 byte log entry
0808 01:50:38.734643 (+184697us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:38.734705 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:38.734788 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:38.877782 (+142994us) raft_consensus.cc:1685] finished
0808 01:50:38.877820 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:38.878100 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38781,"raft.queue_time_us":100119,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":119,"thread_start_us":323524,"threads_started":2,"wal-append.queue_time_us":243737}
I20260808 01:50:38.873245  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:38.873689  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:38.874095  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:38.906265  9524 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:38.912350  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:38.907361  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 59277.8us
I20260808 01:50:38.924254  9556 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:38.930336  9638 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:38.931875  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:38.932672  9638 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:38.933409  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2278) took 1362 ms. Trace:
I20260808 01:50:38.933917  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:38.933902  9559 rpcz_store.cc:276] 0808 01:50:37.571192 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:38.187593 (+616401us) service_pool.cc:224] Handling call
0808 01:50:38.215058 (+ 27465us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:38.474434 (+259376us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:38.474480 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:38.475359 (+   879us) log.cc:844] Serialized 52 byte log entry
0808 01:50:38.676214 (+200855us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:38.676271 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:38.676374 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:38.933149 (+256775us) raft_consensus.cc:1685] finished
0808 01:50:38.933176 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:38.933339 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45530,"raft.queue_time_us":106837,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":94,"thread_start_us":283027,"threads_started":2,"wal-append.queue_time_us":333202}
W20260808 01:50:38.941237  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 598325us
I20260808 01:50:38.943289  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:38.944024  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:38.945431  9631 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:38.947005  9629 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:38.948300  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:38.949436  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:38.952286  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:38.953857  9560 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:38.926570  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:38.957232  9524 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:38.911180 12645 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.271s	user 0.006s	sys 0.006s
I20260808 01:50:38.960270  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:38.974136  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:38.909126 12674 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:38.987150  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 1838) took 1153 ms. Trace:
I20260808 01:50:38.987900  9551 rpcz_store.cc:276] 0808 01:50:37.808135 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:38.017390 (+209255us) service_pool.cc:224] Handling call
0808 01:50:38.960898 (+943508us) spinlock_profiling.cc:243] Waited 664 ms on lock 0x7b5400217fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:38.961128 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1527531392}
I20260808 01:50:38.990995  9630 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:38.995786  9634 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260808 01:50:38.971182 12645 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.331s	user 0.007s	sys 0.007s
I20260808 01:50:38.892915 12667 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.093s
I20260808 01:50:38.976343 12598 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:38.844017 12541 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:38.970979 12693 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:38.983227 12530 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.086s
I20260808 01:50:39.046830 10366 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
W20260808 01:50:39.058084  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:38.982962 12662 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:38.990402 12683 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260808 01:50:39.115844  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 172094us
I20260808 01:50:38.826180 12692 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:39.098074 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 358701us
I20260808 01:50:39.148999  9635 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:39.067741 12571 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:39.155973  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:39.156545  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:38.997553 12675 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.151s
I20260808 01:50:39.004287 12674 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:39.041479 12576 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:39.102656 12683 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:39.349164 12598 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:39.350172 12598 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:39.195300 12655 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:39.351320 12598 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:39.221833 12603 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:39.351490 12655 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:39.353282 12603 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:39.349164 12576 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:39.356106 12576 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:39.315030 12704 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.165s	user 0.000s	sys 0.001s
I20260808 01:50:39.321237 11867 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:39.358662 12571 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:39.358762  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2283) took 1288 ms. Trace:
I20260808 01:50:39.359160 12571 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:39.359242  9554 rpcz_store.cc:276] 0808 01:50:38.070801 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:38.513834 (+443033us) service_pool.cc:224] Handling call
0808 01:50:38.514245 (+   411us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:38.564206 (+ 49961us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:38.564251 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:38.588694 (+ 24443us) log.cc:844] Serialized 26 byte log entry
0808 01:50:38.736657 (+147963us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:38.736730 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:38.736850 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:39.358491 (+621641us) raft_consensus.cc:1685] finished
0808 01:50:39.358515 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:39.358697 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":58256,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"thread_start_us":54949,"threads_started":1,"wal-append.queue_time_us":332850}
I20260808 01:50:39.349545  9634 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:39.191704  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2287) took 1113 ms. Trace:
I20260808 01:50:39.374989  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2174) took 1034 ms. Trace:
I20260808 01:50:39.362263  9559 rpcz_store.cc:276] 0808 01:50:38.078754 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:39.152654 (+1073900us) service_pool.cc:224] Handling call
0808 01:50:39.190936 (+ 38282us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:39.191208 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:39.191250 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:39.191316 (+    66us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:39.191367 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:39.191449 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:39.191610 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:39.321684  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:39.385810  9473 rpcz_store.cc:276] 0808 01:50:38.340648 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:38.618047 (+277399us) service_pool.cc:224] Handling call
0808 01:50:38.618448 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:38.618749 (+   301us) raft_consensus.cc:1534] Early marking committed up to index 1
0808 01:50:38.618792 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:38.666213 (+ 47421us) log.cc:844] Serialized 26 byte log entry
0808 01:50:38.849018 (+182805us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:38.849062 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:38.849138 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:39.374649 (+525511us) raft_consensus.cc:1685] finished
0808 01:50:39.374699 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:39.374892 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":140998,"threads_started":1,"wal-append.queue_time_us":390079}
W20260808 01:50:39.349265  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 209871us
I20260808 01:50:39.364444  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2286) took 1292 ms. Trace:
I20260808 01:50:39.390404  9560 rpcz_store.cc:276] 0808 01:50:38.072512 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:39.121081 (+1048569us) service_pool.cc:224] Handling call
0808 01:50:39.363742 (+242661us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:39.364008 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:39.364045 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:39.364103 (+    58us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:39.364147 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:39.364219 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:39.364363 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:39.047619 12375 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:38.994598 12498 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:39.101727 12662 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:39.362357  9554 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 7. Current term is 8. Ops: []
W20260808 01:50:39.409677  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 414573us
I20260808 01:50:39.404505  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:39.398378  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2169) took 1103 ms. Trace:
I20260808 01:50:39.415279  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2290) took 1074 ms. Trace:
I20260808 01:50:39.420076  9554 rpcz_store.cc:276] 0808 01:50:38.341029 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:39.361583 (+1020554us) service_pool.cc:224] Handling call
0808 01:50:39.362113 (+   530us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:39.414882 (+ 52769us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:39.415198 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:39.417075  9479 rpcz_store.cc:276] 0808 01:50:38.295431 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:38.399982 (+104551us) service_pool.cc:224] Handling call
0808 01:50:38.400453 (+   471us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:38.400822 (+   369us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:38.401248 (+   426us) log.cc:844] Serialized 14 byte log entry
0808 01:50:39.397925 (+996677us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:39.397993 (+    68us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:39.398053 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:39.398136 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:39.398296 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":300043,"prepare.queue_time_us":222,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":396253,"raft.run_cpu_time_us":57,"raft.run_wall_time_us":57,"thread_start_us":673641,"threads_started":2,"wal-append.queue_time_us":303559}
I20260808 01:50:39.422463  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2292) took 1079 ms. Trace:
I20260808 01:50:39.423252  9559 rpcz_store.cc:276] 0808 01:50:38.343007 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:39.386311 (+1043304us) service_pool.cc:224] Handling call
0808 01:50:39.416352 (+ 30041us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:39.421923 (+  5571us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:39.421969 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:39.422052 (+    83us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:39.422100 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:39.422205 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:39.422383 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:39.349360 12683 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:39.437790  9633 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:39.237514 12692 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:39.367384 12541 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:39.196595 12647 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:39.362846 12677 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:39.448745  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59262.2us
I20260808 01:50:39.408885 12655 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:39.452905  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:39.460542  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:39.456427  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:39.196053 12693 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.260s
I20260808 01:50:39.434513 12662 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:39.467244 12724 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election won for term 6
W20260808 01:50:39.467527 12711 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.235s	user 0.000s	sys 0.000s
I20260808 01:50:39.443953 12672 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:39.390735 12676 consensus_queue.cc:1059] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
W20260808 01:50:39.476739 10332 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.015s	sys 0.023s
W20260808 01:50:39.531343 10332 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.015s	sys 0.023s
W20260808 01:50:39.531754 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 563255us
W20260808 01:50:39.468338 12654 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.021s	sys 0.029s
W20260808 01:50:39.503650  9472 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.022s	sys 0.020s
W20260808 01:50:39.503055  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:39.536906  9472 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.022s	sys 0.020s
W20260808 01:50:39.531380  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 80209.5us
I20260808 01:50:39.537293  9633 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 5. Current term is 6. Ops: []
W20260808 01:50:39.540212  9523 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.020s	sys 0.056s
W20260808 01:50:39.540735  9523 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.020s	sys 0.056s
I20260808 01:50:39.414306 12375 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:39.541208  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 572314us
I20260808 01:50:39.533252 12672 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:50:39.549594  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:39.550485  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:39.551095  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:39.554262  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:39.559563 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:39.458175 12530 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.084s
I20260808 01:50:39.458840 12647 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:39.567176  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 372026us
W20260808 01:50:39.563236  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:39.564793 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:39.585441 12723 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:39.586294 12723 raft_consensus.cc:697] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:39.565584 12643 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:39.587843 12643 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:39.587488 12723 consensus_queue.cc:237] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:39.588274 12643 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:39.590781 12721 raft_consensus.cc:2764] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20260808 01:50:39.595973 12678 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.735s	user 0.013s	sys 0.000s
I20260808 01:50:39.600407  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2173) took 1303 ms. Trace:
W20260808 01:50:39.602180  9521 leader_election.cc:336] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.694s (SENT)
I20260808 01:50:39.600871  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2176) took 1256 ms. Trace:
I20260808 01:50:39.561486  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:39.536731 12654 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.022s	sys 0.029s
I20260808 01:50:39.533776 12724 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 6 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:39.602300  9474 rpcz_store.cc:276] 0808 01:50:38.297054 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:38.600418 (+303364us) service_pool.cc:224] Handling call
0808 01:50:38.600899 (+   481us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:39.068173 (+467274us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:39.068224 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:39.069162 (+   938us) log.cc:844] Serialized 52 byte log entry
0808 01:50:39.390235 (+321073us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:39.390278 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:39.390352 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:39.600100 (+209748us) raft_consensus.cc:1685] finished
0808 01:50:39.600138 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:39.600326 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":412157,"raft.queue_time_us":530794,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":25832,"thread_start_us":320579,"threads_started":1,"wal-append.queue_time_us":511617}
I20260808 01:50:39.541106 12695 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:39.604846  9476 rpcz_store.cc:276] 0808 01:50:38.344824 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:38.781575 (+436751us) service_pool.cc:224] Handling call
0808 01:50:38.809461 (+ 27886us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:38.809750 (+   289us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:38.809778 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:38.846252 (+ 36474us) log.cc:844] Serialized 26 byte log entry
0808 01:50:39.366998 (+520746us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:39.410866 (+ 43868us) log.cc:844] Serialized 14 byte log entry
0808 01:50:39.411101 (+   235us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:39.411181 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:39.600569 (+189388us) raft_consensus.cc:1685] finished
0808 01:50:39.600601 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:39.600788 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":42045,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":63,"raft.queue_time_us":55779,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":131,"thread_start_us":443550,"threads_started":1,"wal-append.queue_time_us":641393}
W20260808 01:50:39.633536  9445 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.001s	sys 0.018s
W20260808 01:50:39.640463  9445 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.001s	sys 0.019s
W20260808 01:50:39.641184  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 1.11623e+06us
I20260808 01:50:39.640692  9635 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: []
W20260808 01:50:39.642845  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:39.656689  9443 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.032s	sys 0.042s
W20260808 01:50:39.657547  9443 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.032s	sys 0.042s
W20260808 01:50:39.662573  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 101191us
I20260808 01:50:39.675484  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:39.639101 12721 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:50:39.644284 12715 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:39.659543  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 77923.3us
I20260808 01:50:39.687458  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:39.718058  9637 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:39.704242  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:39.642709 12695 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.041s
I20260808 01:50:39.571139 12672 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:39.742801  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 90465c2f160641a7901ab0815e723ce1 (from 6e7856bed1be43acaa821290d506ee97 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:50:39.737591  9474 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
W20260808 01:50:39.736362  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48974.8us
I20260808 01:50:39.748019  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:39.786535  9476 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:39.786783  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:39.751292 12375 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:39.680032 12721 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:39.724476 12603 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:39.773367  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:39.732712 12718 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:39.775235  9633 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:50:39.750435 12598 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:39.798611  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:39.726123 12715 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:39.789561  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:39.799293 12667 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.697s	user 0.005s	sys 0.009s
I20260808 01:50:39.874176  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:39.868480 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 183909us
I20260808 01:50:39.875116  9633 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:39.801909 12674 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:39.796329 12721 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:50:39.880730  9474 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cacb9622c47c4737beff104125e98b77"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:38182
I20260808 01:50:39.885217  9474 raft_consensus.cc:606] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:39.885828  9474 raft_consensus.cc:614] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:50:39.822649 12375 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 01:50:39.866456 12667 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.007s	sys 0.010s
W20260808 01:50:39.913265  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 496738us
I20260808 01:50:39.914296  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:39.825814 12603 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:39.935173  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet cacb9622c47c4737beff104125e98b77 (from 6e7856bed1be43acaa821290d506ee97 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:50:39.876927 12647 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:39.910386 11867 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.547s	user 0.012s	sys 0.003s
I20260808 01:50:39.943470  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:39.948637  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:39.953982 11568 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:39.954734 11568 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:39.955269 11568 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:39.956895  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:39.949026  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:39.963478  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 49295us
I20260808 01:50:39.964665  9524 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:39.895879 12674 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:39.913942 12667 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:39.770895 12724 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:39.983790  9556 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
W20260808 01:50:40.012714 10332 leader_election.cc:336] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.704s (SENT)
I20260808 01:50:40.047348 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:40.048837 12742 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260808 01:50:40.049749 12716 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.184s	user 0.001s	sys 0.000s
W20260808 01:50:40.050616 12730 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.161s	user 0.000s	sys 0.001s
I20260808 01:50:40.052121  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2285) took 1980 ms. Trace:
I20260808 01:50:40.052559  9557 rpcz_store.cc:276] 0808 01:50:38.071524 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:39.120263 (+1048739us) service_pool.cc:224] Handling call
0808 01:50:39.183606 (+ 63343us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:39.183919 (+   313us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:50:39.183954 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:39.347622 (+163668us) log.cc:844] Serialized 26 byte log entry
0808 01:50:39.653069 (+305447us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:39.653106 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:39.653170 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:40.051813 (+398643us) raft_consensus.cc:1685] finished
0808 01:50:40.051845 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:40.052044 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":304998,"threads_started":1,"wal-append.queue_time_us":421822}
I20260808 01:50:39.822160 12718 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:40.066764 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 54202.1us
I20260808 01:50:40.067555  9633 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:50:40.070851  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2289) took 1991 ms. Trace:
W20260808 01:50:40.071171  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 106640us
I20260808 01:50:40.071388  9556 rpcz_store.cc:276] 0808 01:50:38.079552 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:39.316221 (+1236669us) service_pool.cc:224] Handling call
0808 01:50:39.983173 (+666952us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b5800003c98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:50:39.983512 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:39.995939 (+ 12427us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:40.070770 (+ 74831us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5683072}
W20260808 01:50:40.063510 12731 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.193s	user 0.000s	sys 0.000s
I20260808 01:50:40.077514  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:40.072876  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:40.074913  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2288) took 1995 ms. Trace:
I20260808 01:50:40.099318  9558 rpcz_store.cc:276] 0808 01:50:38.079182 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:39.196716 (+1117534us) service_pool.cc:224] Handling call
0808 01:50:39.330763 (+134047us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:39.331216 (+   453us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:39.331248 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:39.331890 (+   642us) log.cc:844] Serialized 26 byte log entry
0808 01:50:39.659289 (+327399us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:39.659337 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:39.659433 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:40.074430 (+414997us) raft_consensus.cc:1685] finished
0808 01:50:40.074464 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:40.074799 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":261921,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"spinlock_wait_cycles":1920,"thread_start_us":326963,"threads_started":1,"wal-append.queue_time_us":548137}
I20260808 01:50:40.106099 11568 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:40.086344  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:40.107542  9634 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:40.087059  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:40.076655 12742 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:40.089380  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:40.108345  9631 raft_consensus.cc:2376] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7. Re-sending same reply.
I20260808 01:50:40.109064 12738 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260808 01:50:40.111027 12736 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
I20260808 01:50:40.115031 12676 raft_consensus.cc:3055] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:40.115657 12676 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:40.116819 12676 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:40.119987  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:39.990123 12674 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:39.981606 12654 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.115s
W20260808 01:50:40.070163 12675 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.720s	user 0.023s	sys 0.028s
I20260808 01:50:40.075977  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 367) took 1430 ms. Trace:
I20260808 01:50:40.125003  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 363) took 1481 ms. Trace:
I20260808 01:50:40.096429  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:40.135133  9476 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:50:40.136442 11568 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:40.080363 12745 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:40.140067  9556 rpcz_store.cc:276] 0808 01:50:38.645392 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:40.075260 (+1429868us) service_pool.cc:224] Handling call
0808 01:50:40.075607 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:40.075757 (+   150us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:40.075773 (+    16us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:40.075802 (+    29us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:50:40.075820 (+    18us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:40.075854 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:40.075924 (+    70us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:40.163264  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:40.144647  9559 rpcz_store.cc:276] 0808 01:50:38.643992 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:39.456160 (+812168us) service_pool.cc:224] Handling call
0808 01:50:39.525306 (+ 69146us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:39.525643 (+   337us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:50:39.822365 (+296722us) log.cc:844] Serialized 14 byte log entry
0808 01:50:40.124320 (+301955us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:40.124406 (+    86us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:40.124593 (+   187us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:40.124708 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:40.124901 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50697,"prepare.queue_time_us":394382,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"thread_start_us":448546,"threads_started":2}
I20260808 01:50:40.150653 12718 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:40.152276  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:40.068280 12667 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:40.159420 12715 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:40.100437 12643 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:40.121934 12684 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.252s	user 0.014s	sys 0.000s
I20260808 01:50:40.163281  9556 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260808 01:50:40.359587  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 371) took 1619 ms. Trace:
W20260808 01:50:40.278277  9637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1824) took 1417 ms (client timeout 1807 ms). Trace:
W20260808 01:50:40.361729  9637 rpcz_store.cc:269] 0808 01:50:38.860816 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:38.873781 (+ 12965us) service_pool.cc:224] Handling call
0808 01:50:40.278014 (+1404233us) spinlock_profiling.cc:243] Waited 715 ms on lock 0x7b54001b44d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:40.278167 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1645602560}
W20260808 01:50:40.185303 12627 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.421s	user 0.016s	sys 0.001s
I20260808 01:50:40.363739  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 366) took 1718 ms. Trace:
W20260808 01:50:40.358332  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 285598us
I20260808 01:50:40.218995 12717 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:40.139873 12675 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.790s	user 0.023s	sys 0.029s
I20260808 01:50:40.122638 12742 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:40.164553  9630 raft_consensus.cc:2376] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7. Re-sending same reply.
I20260808 01:50:40.363691 12627 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:40.364274  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:40.365247 12742 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:40.364419  9557 rpcz_store.cc:276] 0808 01:50:38.644993 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:40.054363 (+1409370us) service_pool.cc:224] Handling call
0808 01:50:40.063305 (+  8942us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:40.079339 (+ 16034us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:40.079382 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:40.082577 (+  3195us) log.cc:844] Serialized 26 byte log entry
0808 01:50:40.262644 (+180067us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:40.262757 (+   113us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:40.262876 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:40.363528 (+100652us) raft_consensus.cc:1685] finished
0808 01:50:40.363549 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:40.363675 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":92019,"raft.queue_time_us":39648,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":22300,"thread_start_us":96199,"threads_started":1,"wal-append.queue_time_us":277822}
W20260808 01:50:40.364778  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:40.368983  9524 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:40.370330 12693 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:40.181506  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 360) took 1586 ms. Trace:
I20260808 01:50:40.371786  9524 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:40.371677  9560 rpcz_store.cc:276] 0808 01:50:38.595090 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:39.392320 (+797230us) service_pool.cc:224] Handling call
0808 01:50:39.436120 (+ 43800us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:39.731794 (+295674us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:39.731844 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:39.749625 (+ 17781us) log.cc:844] Serialized 52 byte log entry
0808 01:50:39.882575 (+132950us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:39.882637 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:39.882789 (+   152us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:40.181207 (+298418us) raft_consensus.cc:1685] finished
0808 01:50:40.181233 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:40.181426 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":331524,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":164,"thread_start_us":426709,"threads_started":2,"wal-append.queue_time_us":379889}
W20260808 01:50:40.235425  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:40.313237  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:40.266908 10367 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260808 01:50:40.302983  9559 raft_consensus.cc:3055] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Stepping down as leader of term 3
W20260808 01:50:40.374459  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:40.221561 12571 consensus_meta.cc:287] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.862s	user 0.006s	sys 0.006s
W20260808 01:50:40.222481 12677 consensus_meta.cc:287] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.762s	user 0.009s	sys 0.004s
W20260808 01:50:40.352415  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 276755us
W20260808 01:50:40.374743  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:40.373273  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 368) took 1727 ms. Trace:
W20260808 01:50:40.375442  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:40.374787  9559 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:40.360049  9556 rpcz_store.cc:276] 0808 01:50:38.740421 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:40.162306 (+1421885us) service_pool.cc:224] Handling call
0808 01:50:40.162766 (+   460us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:40.359279 (+196513us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:40.359515 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:40.425221 10366 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
W20260808 01:50:40.618081  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:40.381670  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:40.381772  9558 rpcz_store.cc:276] 0808 01:50:38.645808 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:40.121196 (+1475388us) service_pool.cc:224] Handling call
0808 01:50:40.372729 (+251533us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:40.372945 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:40.372978 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:40.373022 (+    44us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:40.373050 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:40.373104 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:40.373215 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:40.456609  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:40.434473  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:40.385571  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:40.622758  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2295) took 1782 ms. Trace:
W20260808 01:50:40.609385  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 819864us
I20260808 01:50:40.623389  9560 rpcz_store.cc:276] 0808 01:50:38.840262 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:40.374152 (+1533890us) service_pool.cc:224] Handling call
0808 01:50:40.622224 (+248072us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:40.622400 (+   176us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:40.622425 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:40.622478 (+    53us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:40.622508 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:40.622561 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:40.622664 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:40.445462  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2170) took 2149 ms. Trace:
I20260808 01:50:40.625578  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:40.624593  9472 rpcz_store.cc:276] 0808 01:50:38.295959 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:38.592909 (+296950us) service_pool.cc:224] Handling call
0808 01:50:38.649503 (+ 56594us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:39.537468 (+887965us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:39.537511 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:39.547068 (+  9557us) log.cc:844] Serialized 52 byte log entry
0808 01:50:39.774776 (+227708us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:39.811618 (+ 36842us) log.cc:844] Serialized 14 byte log entry
0808 01:50:40.444530 (+632912us) spinlock_profiling.cc:243] Waited 3.9 ms on lock 0x7b1800252dc0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667 00007f5d449ae5e8
0808 01:50:40.444819 (+   289us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:40.444945 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:40.445009 (+    64us) raft_consensus.cc:1685] finished
0808 01:50:40.445141 (+   132us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:40.445348 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":312322,"prepare.queue_time_us":34495,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":68,"raft.queue_time_us":1026635,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":247,"spinlock_wait_cycles":8972544,"thread_start_us":811949,"threads_started":2,"wal-append.queue_time_us":360497}
W20260808 01:50:40.630892  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:40.632171 10332 leader_election.cc:336] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.156s (SENT)
W20260808 01:50:40.371198 12750 log.cc:927] Time spent T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.267s	user 0.000s	sys 0.002s
W20260808 01:50:40.273874 12702 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.907s	user 0.002s	sys 0.000s
I20260808 01:50:40.382608  9559 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:40.636874  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:40.637975  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:40.638234  9556 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:40.275275 12726 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Stepping down as leader of term 4
W20260808 01:50:40.378578 12541 consensus_meta.cc:287] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.927s	user 0.013s	sys 0.002s
I20260808 01:50:40.275132 12603 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:40.270514 12419 raft_consensus.cc:3055] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Stepping down as leader of term 5
W20260808 01:50:40.607733 12662 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.076s	user 0.013s	sys 0.000s
W20260808 01:50:40.643385  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:40.626909 12677 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:40.434898 12735 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:40.646752  9559 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:40.446492 12733 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:40.628352 12667 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:40.707018 12735 raft_consensus.cc:697] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:40.707806  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:40.708431 12735 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:40.709244  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:40.710140  9631 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:40.714210 12745 consensus_queue.cc:1059] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:50:40.715345  9560 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:50:40.716286  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2299) took 1768 ms. Trace:
I20260808 01:50:40.716702  9560 rpcz_store.cc:276] 0808 01:50:38.948273 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:40.675105 (+1726832us) service_pool.cc:224] Handling call
0808 01:50:40.686386 (+ 11281us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:40.715912 (+ 29526us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:40.716208 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:40.700664 12662 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:40.680791 12419 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:40.719825 12662 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:40.719493 12419 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:40.720628 12419 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:40.720769  9558 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:40.723706 12717 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:40.648854  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2280) took 3076 ms. Trace:
I20260808 01:50:40.649627  9550 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: []
W20260808 01:50:40.702557 12713 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 1.389s	user 0.000s	sys 0.013s
W20260808 01:50:40.651389 12761 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.204s	user 0.001s	sys 0.000s
I20260808 01:50:40.436448 12684 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:40.659205 12576 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:40.659955  9556 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
W20260808 01:50:40.662910  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:40.739346  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:40.666359  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:40.664642  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:40.666580  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet d0db7b84c3b040c788e0a3f66657d912: Illegal state: Replica 6e7856bed1be43acaa821290d506ee97 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:50:40.667603  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:40.740968  9637 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:40.668532  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:40.666386 12742 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:40.671406  9476 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:40.671756 12732 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:40.673324  9474 pending_rounds.cc:85] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 4
I20260808 01:50:40.750849  9474 pending_rounds.cc:107] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260808 01:50:40.751174  9474 raft_consensus.cc:2889] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:40.751680  9475 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:40.380000 12655 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:40.676647  9478 raft_consensus.cc:3055] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:40.756040  9478 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:40.183202 12674 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:40.371500 12693 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:40.762068 12693 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:40.762383  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:40.762928 12655 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:40.683727  9634 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260808 01:50:40.505103 12692 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.057s	user 0.014s	sys 0.000s
I20260808 01:50:40.441937 12627 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:40.436193 12643 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:40.687198  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:40.380339 12675 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.303s
W20260808 01:50:40.700268 12691 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.872s	user 0.007s	sys 0.010s
I20260808 01:50:40.700647 12634 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:40.685694  9554 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:40.641661 12420 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:40.706835 12733 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:40.647727 12748 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:40.728101  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2297) took 1887 ms. Trace:
I20260808 01:50:40.780462  9558 rpcz_store.cc:276] 0808 01:50:38.840915 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:40.620280 (+1779365us) service_pool.cc:224] Handling call
0808 01:50:40.640956 (+ 20676us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:40.721242 (+ 80286us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:40.727845 (+  6603us) spinlock_profiling.cc:243] Waited 79.6 ms on lock 0x7b5400409554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:40.728036 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":183070464}
I20260808 01:50:40.718386  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:40.730829  9559 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260808 01:50:40.736332  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:40.782321  9560 raft_consensus.cc:2376] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7. Re-sending same reply.
I20260808 01:50:40.783208  9559 pending_rounds.cc:85] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 2
I20260808 01:50:40.740212  9556 raft_consensus.cc:2368] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8.
W20260808 01:50:40.666963 12721 consensus_meta.cc:287] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.748s	user 0.010s	sys 0.000s
I20260808 01:50:40.745033  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:40.679019 12726 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:40.385521 12571 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:40.750236  9638 raft_consensus.cc:2368] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8.
I20260808 01:50:40.766996 12603 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:40.724848  9555 rpcz_store.cc:276] 0808 01:50:37.571972 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:38.223641 (+651669us) service_pool.cc:224] Handling call
0808 01:50:38.224048 (+   407us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:38.620399 (+396351us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:38.620449 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:38.658918 (+ 38469us) log.cc:844] Serialized 52 byte log entry
0808 01:50:38.726116 (+ 67198us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:38.726153 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:38.726233 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:40.272797 (+1546564us) spinlock_profiling.cc:243] Waited 69 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:40.648330 (+375533us) raft_consensus.cc:1685] finished
0808 01:50:40.648366 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:40.648761 (+   395us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42728,"raft.queue_time_us":104005,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":63,"spinlock_wait_cycles":158766976,"thread_start_us":415052,"threads_started":2,"wal-append.queue_time_us":438243}
W20260808 01:50:40.759382  9444 connection.cc:256] RPC call timeout handler was delayed by -2.6299334160285071s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1552s for a response
W20260808 01:50:40.787243  9444 connection.cc:256] RPC call timeout handler was delayed by -2.61103341600392s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1573s for a response
W20260808 01:50:40.787686  9444 connection.cc:256] RPC call timeout handler was delayed by -2.6065334160230123s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15780000000000002s for a response
W20260808 01:50:40.787971  9444 connection.cc:256] RPC call timeout handler was delayed by -2.550733416021103s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.164s for a response
W20260808 01:50:40.788267  9444 connection.cc:256] RPC call timeout handler was delayed by -2.5021334160119295s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1694s for a response
I20260808 01:50:40.763368  9473 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:40.685544  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:40.742815 12683 consensus_meta.cc:287] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.289s	user 0.014s	sys 0.001s
I20260808 01:50:40.750862 12662 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:40.724850 12677 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:40.770665  9478 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:40.753954 12541 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:40.752812 12667 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:40.778234  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:40.783641  9559 pending_rounds.cc:107] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 3.3
W20260808 01:50:40.788530  9444 connection.cc:256] RPC call timeout handler was delayed by -2.4517334160045721s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17500000000000002s for a response
I20260808 01:50:40.748189 12684 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:40.804787  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:40.808389  9559 raft_consensus.cc:2889] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:50:40.808808  9444 connection.cc:256] RPC call timeout handler was delayed by -2.40673341602087s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18000000000000002s for a response
W20260808 01:50:40.815402  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:40.817222  9478 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:40.738608 12656 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20260808 01:50:40.818979  9479 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260808 01:50:40.819450  9481 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:40.759622 12576 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:40.820963  9444 connection.cc:256] RPC call timeout handler was delayed by -2.3968334160163067s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1811s for a response
I20260808 01:50:40.832891  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:40.836015  9558 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:40.833766  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:40.837395  9444 connection.cc:256] RPC call timeout handler was delayed by -2.3770334160071798s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18330000000000002s for a response
I20260808 01:50:40.837481  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2293) took 1999 ms. Trace:
I20260808 01:50:40.838007  9559 rpcz_store.cc:276] 0808 01:50:38.838012 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:40.167870 (+1329858us) service_pool.cc:224] Handling call
0808 01:50:40.262004 (+ 94134us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:40.837067 (+575063us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:40.837403 (+   336us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:40.780088 12733 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:40.839623  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:40.843962  9444 connection.cc:256] RPC call timeout handler was delayed by -2.32843341602711s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1887s for a response
W20260808 01:50:40.844501  9444 connection.cc:256] RPC call timeout handler was delayed by -2.3167334160243627s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19s for a response
W20260808 01:50:40.845088  9444 connection.cc:256] RPC call timeout handler was delayed by -2.2537334160297178s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.197s for a response
W20260808 01:50:40.845530  9444 connection.cc:256] RPC call timeout handler was delayed by -2.2231334160023835s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20040000000000002s for a response
W20260808 01:50:40.845853  9444 connection.cc:256] RPC call timeout handler was delayed by -2.2033334160223603s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2026s for a response
W20260808 01:50:40.846114  9444 connection.cc:256] RPC call timeout handler was delayed by -2.1880334160232451s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20430000000000004s for a response
W20260808 01:50:40.846562  9444 connection.cc:256] RPC call timeout handler was delayed by -2.0935334160167258s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21480000000000002s for a response
W20260808 01:50:40.846940  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1647334160225s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31800000000000006s for a response
I20260808 01:50:40.759929 12732 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:40.804163 12756 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:40.845919  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:40.843601  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:40.848265  9473 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:40.847615  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:40.847638  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:40.785041 12634 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:40.854884  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:40.862623  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:40.871276  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:40.788471 12721 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:40.785698 12420 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:40.807228 12683 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:40.836560 12477 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260808 01:50:40.816951 12667 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:40.824599 12541 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:40.897790  9473 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:50:40.825735 12662 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:40.776535 12691 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:40.903887  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:40.900628  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:40.867834 12692 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:40.909157  9480 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:40.849207 12764 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20260808 01:50:40.920174  9560 raft_consensus.cc:2393] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 8: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:40.831756 12603 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:40.918056  9552 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
W20260808 01:50:40.915149  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:40.789014 12748 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:40.931300  9557 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:40.939396  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:40.940613  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:40.941411  9558 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:50:40.937287 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 247784us
I20260808 01:50:40.967636 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:40.967998  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:40.854993 12654 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.071s
W20260808 01:50:40.972820  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:40.975457  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 120695us
I20260808 01:50:40.977389  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:40.979804  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:40.925236 12656 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 01:50:40.987022  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:40.988298  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:40.982261  9601 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:40.931529 12643 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:40.983510 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:40.997493  9601 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:40.998602  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:41.009255  9473 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:38182
I20260808 01:50:41.010293  9473 raft_consensus.cc:606] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:40.886780 12726 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:41.016619 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 46421.5us
I20260808 01:50:41.026940  9632 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:41.028337 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:41.012436  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 164895us
I20260808 01:50:41.019371  9473 raft_consensus.cc:614] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:50:41.008427  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:40.991570 12477 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:40.934121 12692 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:41.031980  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:40.996276 12742 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:41.041659  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:41.045895  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:41.046976 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:41.047469  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:41.048242  9477 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 3.
I20260808 01:50:41.048708  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:41.049360  9474 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:41.049670 10362 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:41.051226  9479 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:41.051826 12735 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:50:41.052501  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:41.053205 12735 raft_consensus.cc:3055] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:41.053769 12735 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:41.053889  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:41.055352 12634 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:41.057245 12735 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:41.059135 12654 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260808 01:50:41.059491  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:41.059729 12654 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:41.059926 12776 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:41.060470 12735 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:41.010181 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:41.062048 12684 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:41.062726 12684 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:41.063112 12684 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:41.060308 12775 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:41.064018 12775 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:41.064872  9476 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:41.066154  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:41.066478 12775 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:41.067042 12771 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:41.071485  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:41.001767 12643 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:41.063453 12684 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:41.027079 12656 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:41.017333 12598 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:41.015811 12769 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:41.059415 12765 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.213s	user 0.000s	sys 0.001s
I20260808 01:50:41.014972 12756 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:41.035197 12571 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:41.076817  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46738.8us
I20260808 01:50:41.035516  9558 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
W20260808 01:50:41.080204  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:41.072983  9476 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:41.049386  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:41.091545  9481 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
W20260808 01:50:41.235371  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:41.335129  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:41.186203  9632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1829) took 2256 ms (client timeout 2087 ms). Trace:
I20260808 01:50:41.084386  9479 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:41.067097 12676 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:50:41.236788  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:41.336035  9477 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:41.345718  9474 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:41.187338  9473 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: [4.3-7.5]
I20260808 01:50:41.040315 12477 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:41.188877  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet d0db7b84c3b040c788e0a3f66657d912 (from 6e7856bed1be43acaa821290d506ee97 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:50:41.060264 12654 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:41.189414  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:41.151060  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:41.252611 12498 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260808 01:50:41.306070 12683 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:41.314492 12420 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:41.361982 12753 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.220s	user 0.002s	sys 0.000s
I20260808 01:50:41.354552 12364 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:41.358757  9476 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: []
W20260808 01:50:41.423677 12419 consensus_meta.cc:287] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.703s	user 0.016s	sys 0.001s
I20260808 01:50:41.306459 12715 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:41.257663 12708 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 1.889s	user 0.003s	sys 0.009s
I20260808 01:50:41.362752 12598 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:41.256589 12758 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.420s	user 0.000s	sys 0.001s
W20260808 01:50:41.336890  9632 rpcz_store.cc:269] 0808 01:50:38.929281 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:38.948005 (+ 18724us) service_pool.cc:224] Handling call
0808 01:50:41.185944 (+2237939us) spinlock_profiling.cc:243] Waited 1.82 s on lock 0x7b540027e054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:41.186067 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4176872832}
I20260808 01:50:41.363762  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:41.186064 12643 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:41.255000 12768 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.312s	user 0.000s	sys 0.001s
W20260808 01:50:41.253777 12773 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.162s	user 0.000s	sys 0.000s
I20260808 01:50:41.083250 12656 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:41.282477 12712 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.543s	user 0.004s	sys 0.008s
W20260808 01:50:41.367900  9554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 2302) took 1961 ms (client timeout 1965 ms). Trace:
I20260808 01:50:41.081614 12634 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:50:41.258879 12722 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:41.092130 12576 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:41.082661 12726 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:41.314363 12732 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:41.282562 12748 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:41.252785 12766 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.345s	user 0.000s	sys 0.001s
W20260808 01:50:41.335073  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:41.360163  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2303) took 1782 ms. Trace:
W20260808 01:50:41.577908  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 239220us
I20260808 01:50:41.387495  9555 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
W20260808 01:50:41.589541  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:41.384047  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 1
W20260808 01:50:41.590194  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:41.483522  9478 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.663s	user 0.013s	sys 0.001s
W20260808 01:50:41.591775  9554 rpcz_store.cc:269] 0808 01:50:39.406829 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:39.441258 (+ 34429us) service_pool.cc:224] Handling call
0808 01:50:41.367604 (+1926346us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54000b4ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:41.367768 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2825959296}
I20260808 01:50:41.060420 12776 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:41.091835 12771 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:41.283033 12541 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:41.488524 12364 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:41.491233 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:41.491683 12744 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.403s	user 0.000s	sys 0.004s
I20260808 01:50:41.355264 12692 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:41.491917 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:41.446249 12662 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:41.495807  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:41.499295 12655 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:41.499909 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 428705us
I20260808 01:50:41.502725 12603 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:41.510947  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1834) took 3704 ms. Trace:
I20260808 01:50:41.655411  9550 rpcz_store.cc:276] 0808 01:50:37.806909 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:38.269533 (+462624us) service_pool.cc:224] Handling call
0808 01:50:40.649339 (+2379806us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:40.725079 (+ 75740us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:41.510580 (+785501us) spinlock_profiling.cc:243] Waited 2.38 s on lock 0x7b540002f150. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:41.510845 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5471975680}
I20260808 01:50:41.656702 12771 raft_consensus.cc:697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:41.657205  9550 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:41.657680  9550 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:41.658465  9550 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:41.658203 12771 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:41.659607  9550 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:41.539724  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:41.661785  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:41.544023  9521 connection.cc:256] RPC call timeout handler was delayed by -1.497407539980486s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22290000000000001s for a response
W20260808 01:50:41.662271  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:41.430984 12477 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:41.384510 12769 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:41.663269  9524 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:41.665063 12769 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:41.283128 12782 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.276s
I20260808 01:50:41.665534  9524 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:41.282788 12721 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:41.666472 12541 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:41.578373 12767 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:41.585165 12598 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:41.041991 12677 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:41.602576  9478 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:50:41.604115  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:41.554987 12765 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
W20260808 01:50:41.558449 12693 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.791s	user 0.014s	sys 0.000s
W20260808 01:50:41.555783 12743 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 1.469s	user 0.008s	sys 0.012s
W20260808 01:50:41.556594 12733 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.702s	user 0.011s	sys 0.000s
I20260808 01:50:41.620368  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:41.679548  9477 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:41.667771  9601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.668s	user 0.152s	sys 0.246s
I20260808 01:50:41.603153 12776 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:41.680546  9601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.152s	sys 0.246s
I20260808 01:50:41.629709 12691 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:41.637691  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:41.684983  9473 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:50:41.638643  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:41.652086 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:41.650889 12419 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:41.667191  9475 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:41.656698  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2284) took 3585 ms. Trace:
I20260808 01:50:41.667778  9524 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:41.695327  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:41.696930  9524 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:41.668421  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:41.698194  9555 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:41.591413 12634 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:41.587788  9556 rpcz_store.cc:276] 0808 01:50:39.577988 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:41.358999 (+1781011us) service_pool.cc:224] Handling call
0808 01:50:41.359443 (+   444us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:41.359706 (+   263us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:41.359743 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:41.359801 (+    58us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:41.359858 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:41.359933 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:41.360085 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:41.595858 12722 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:41.666517  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2291) took 3324 ms. Trace:
I20260808 01:50:41.702153  9552 rpcz_store.cc:276] 0808 01:50:38.341897 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:39.373749 (+1031852us) service_pool.cc:224] Handling call
0808 01:50:39.412367 (+ 38618us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:40.928751 (+1516384us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:41.666221 (+737470us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54000b45d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:41.666448 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3461927680}
I20260808 01:50:41.624110  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:41.681190  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 682595us
I20260808 01:50:41.705657  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:41.558318 12756 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:41.706140 12691 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:50:41.678840 12772 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:41.707047  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:41.707430 12655 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:50:41.707103 12772 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:41.709131  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:41.710094  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2177) took 1074 ms. Trace:
I20260808 01:50:41.710597  9472 rpcz_store.cc:276] 0808 01:50:40.635350 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:40.635516 (+   166us) service_pool.cc:224] Handling call
0808 01:50:40.639652 (+  4136us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:40.642464 (+  2812us) raft_consensus.cc:1534] Early marking committed up to index 1
0808 01:50:40.642503 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:40.658320 (+ 15817us) log.cc:844] Serialized 26 byte log entry
0808 01:50:40.769356 (+111036us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:40.769418 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:40.769618 (+   200us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:41.709848 (+940230us) raft_consensus.cc:1685] finished
0808 01:50:41.709868 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:41.710028 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":110530,"threads_started":1,"wal-append.queue_time_us":217502}
I20260808 01:50:41.683284  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:41.684057  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:41.713056  9478 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
W20260808 01:50:41.560180  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:41.693465  9551 rpcz_store.cc:276] 0808 01:50:38.071221 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:39.037793 (+966572us) service_pool.cc:224] Handling call
0808 01:50:39.039179 (+  1386us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:39.601270 (+562091us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:39.601300 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:39.627130 (+ 25830us) log.cc:844] Serialized 52 byte log entry
0808 01:50:39.767468 (+140338us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:39.767556 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:39.769367 (+  1811us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:41.485501 (+1716134us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:41.656337 (+170836us) raft_consensus.cc:1685] finished
0808 01:50:41.656361 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:41.656604 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6067,"raft.queue_time_us":504011,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":18929,"spinlock_wait_cycles":27060480,"thread_start_us":576164,"threads_started":2,"wal-append.queue_time_us":443759}
I20260808 01:50:41.667117 12541 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:41.679160 12767 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:41.607208 12571 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:41.557190 12715 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:41.718390  9601 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:41.733206  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:41.734161  9472 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
W20260808 01:50:41.718474  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 94479.7us
W20260808 01:50:41.740037  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:41.740398  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:41.718945  9524 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:41.743166  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:41.744182  9555 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:41.746057  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:41.740716  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:41.747323  9474 raft_consensus.cc:2410] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260808 01:50:41.747627  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:41.768755  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 106098us
W20260808 01:50:41.769982  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:41.770408  9521 connection.cc:256] RPC call timeout handler was delayed by -1.0777281309710816s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1952s for a response
I20260808 01:50:41.773815 12691 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:41.774371 12691 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:41.554617 12654 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:41.780997  9550 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260808 01:50:41.775609 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:41.672475 12692 consensus_queue.cc:1059] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260808 01:50:41.710201 12662 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:41.792223  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2309) took 1407 ms. Trace:
W20260808 01:50:41.759245 12712 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260808 01:50:41.705238 12785 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:41.709862 12748 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:41.704871 12634 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:41.760001 12655 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:41.841388 12785 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:41.838553 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:50:41.843935  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 253052us
W20260808 01:50:41.844708  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:41.844220 12785 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:41.845475  9599 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:41.847093  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1816) took 3115 ms. Trace:
I20260808 01:50:41.838524 12364 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:41.841195 12779 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:41.849195 12364 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:41.849579 12779 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:41.850200 12779 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:41.840466  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 122176us
I20260808 01:50:41.852311  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:41.854775 12676 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:41.855592 12676 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:41.856877 12676 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:41.841718 12748 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:41.859534 12748 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:41.847515  9636 rpcz_store.cc:276] 0808 01:50:38.731372 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:38.731510 (+   138us) service_pool.cc:224] Handling call
0808 01:50:38.731895 (+   385us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:38.732846 (+   951us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:38.732874 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:38.739127 (+  6253us) log.cc:844] Serialized 26 byte log entry
0808 01:50:39.078356 (+339229us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:39.347151 (+268795us) log.cc:844] Serialized 14 byte log entry
0808 01:50:39.368130 (+ 20979us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:39.368237 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:40.926664 (+1558427us) spinlock_profiling.cc:243] Waited 6.04 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:41.846773 (+920109us) raft_consensus.cc:1685] finished
0808 01:50:41.846883 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:41.847031 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21156,"prepare.queue_time_us":267841,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":60,"raft.queue_time_us":25385,"raft.run_cpu_time_us":351,"raft.run_wall_time_us":28817,"spinlock_wait_cycles":13902592,"thread_start_us":278729,"threads_started":1,"wal-append.queue_time_us":864255}
I20260808 01:50:41.865936 12779 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:41.731367 12541 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:50:41.869940  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 129659us
I20260808 01:50:41.837692  9550 pending_rounds.cc:85] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 2
I20260808 01:50:41.880357  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:50:41.815868 12724 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.730s	user 0.015s	sys 0.020s
W20260808 01:50:41.895599  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 176738us
I20260808 01:50:41.895967  9521 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:41.840726  9555 rpcz_store.cc:276] 0808 01:50:40.384630 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:41.791082 (+1406452us) service_pool.cc:224] Handling call
0808 01:50:41.791554 (+   472us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:41.791819 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:41.791869 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:41.791935 (+    66us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:41.791978 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:41.792038 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:41.792156 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:41.898195  9550 pending_rounds.cc:107] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 4.3
W20260808 01:50:41.898358  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 51954.8us
I20260808 01:50:41.782773 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:41.902904  9444 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:41.903282  9473 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:41.905268 11568 consensus_queue.cc:1059] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:50:41.867878 12598 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:41.704370 12419 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:41.916124 10369 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:41.687934 12733 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:41.836078 12795 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:41.902825  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2300) took 2954 ms. Trace:
I20260808 01:50:41.975898  9560 rpcz_store.cc:276] 0808 01:50:38.948619 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:41.035180 (+2086561us) service_pool.cc:224] Handling call
0808 01:50:41.035597 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:41.035869 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:41.352305 (+316436us) log.cc:844] Serialized 14 byte log entry
0808 01:50:41.902426 (+550121us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:41.902503 (+    77us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:41.902548 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:41.902617 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:41.902760 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19345,"prepare.queue_time_us":304088,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":69,"thread_start_us":864340,"threads_started":3}
I20260808 01:50:41.906355 11568 raft_consensus.cc:3055] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:41.978901 11568 raft_consensus.cc:740] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:41.978560  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2311) took 1348 ms. Trace:
I20260808 01:50:41.979928  9560 rpcz_store.cc:276] 0808 01:50:40.630204 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:41.977548 (+1347344us) service_pool.cc:224] Handling call
0808 01:50:41.977870 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:41.978101 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:41.978140 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:41.978201 (+    61us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:41.978247 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:41.978324 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:41.978480 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:41.910482 12655 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:41.868624 12782 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:41.688577 12693 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:41.901116 12724 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.816s	user 0.016s	sys 0.021s
W20260808 01:50:41.942481  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:42.001835  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 99014.7us
I20260808 01:50:41.694351 12656 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:41.892604 12804 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:41.956307 10332 leader_election.cc:336] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.596s (SENT)
I20260808 01:50:41.899950  9550 raft_consensus.cc:2889] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:41.910868 12677 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:41.983662 12781 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:42.018811  9555 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:41.998867 11568 consensus_queue.cc:260] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:42.008973  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 113077us
W20260808 01:50:42.010234 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 54011.7us
W20260808 01:50:42.010356 12789 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.237s	user 0.000s	sys 0.002s
I20260808 01:50:42.011439  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2304) took 2409 ms. Trace:
I20260808 01:50:41.947526 12571 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:42.030548  9550 rpcz_store.cc:276] 0808 01:50:39.601962 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:41.656696 (+2054734us) service_pool.cc:224] Handling call
0808 01:50:41.657065 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:42.011077 (+354012us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:42.011358 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:42.031569  9479 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 6 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:42.033037 11568 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:42.034861  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:41.986908  9560 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:41.988833 12795 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:42.048177 12788 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:42.032058 12656 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:42.117530 10332 leader_election.cc:336] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.936s (SENT)
I20260808 01:50:41.988237 12777 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:50:42.119447  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 84731.4us
I20260808 01:50:42.125067  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2312) took 1485 ms. Trace:
W20260808 01:50:42.174032 12792 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260808 01:50:42.159873 12724 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260808 01:50:42.107508 12745 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.392s	user 0.035s	sys 0.058s
I20260808 01:50:41.988600 12655 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:42.002631 12782 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:42.179679  9560 rpcz_store.cc:276] 0808 01:50:40.640149 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:41.981512 (+1341363us) service_pool.cc:224] Handling call
0808 01:50:41.981970 (+   458us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:42.069057 (+ 87087us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:42.124973 (+ 55916us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:42.182862 10334 connection.cc:256] RPC call timeout handler was delayed by -1.5881488129671197s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15960000000000002s for a response
W20260808 01:50:42.177230 12801 log.cc:927] Time spent T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.270s	user 0.001s	sys 0.000s
W20260808 01:50:42.259944  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 77671.1us
I20260808 01:50:42.261026  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:42.116392  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:42.183622 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:42.201972  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:42.028723 12781 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:42.226526 12803 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.146s	user 0.004s	sys 0.004s
W20260808 01:50:42.230334 12791 log.cc:927] Time spent T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.337s	user 0.000s	sys 0.001s
W20260808 01:50:42.231429 12797 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
I20260808 01:50:42.233908 12677 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:42.016018 12804 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:42.240598  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2294) took 3402 ms. Trace:
I20260808 01:50:42.240736 12800 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:50:42.241840 10334 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:42.242547 12722 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:42.243983 12684 consensus_meta.cc:287] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.016s	user 0.010s	sys 0.004s
I20260808 01:50:42.181124 12598 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:42.245137  9477 consensus_meta.cc:287] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.565s	user 0.010s	sys 0.007s
I20260808 01:50:42.320881  9477 raft_consensus.cc:2410] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260808 01:50:42.321658  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:42.257586  9473 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
W20260808 01:50:42.267863  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:42.325814  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:42.271864  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 373) took 1584 ms. Trace:
W20260808 01:50:42.190152 12745 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.475s	user 0.036s	sys 0.059s
W20260808 01:50:42.284945  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 168638us
I20260808 01:50:42.285240  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2305) took 1901 ms. Trace:
I20260808 01:50:42.285830  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:42.298527  9557 rpcz_store.cc:276] 0808 01:50:38.838328 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:40.366984 (+1528656us) service_pool.cc:224] Handling call
0808 01:50:40.367381 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:40.933355 (+565974us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:42.237022 (+1303667us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b5400257e54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:42.240479 (+  3457us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1296282368}
I20260808 01:50:42.328055  9556 rpcz_store.cc:276] 0808 01:50:40.383572 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:41.699717 (+1316145us) service_pool.cc:224] Handling call
0808 01:50:41.710758 (+ 11041us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:41.711072 (+   314us) raft_consensus.cc:1534] Early marking committed up to index 4
0808 01:50:41.776488 (+ 65416us) log.cc:844] Serialized 14 byte log entry
0808 01:50:42.284463 (+507975us) log.cc:844] Serialized 14 byte log entry
0808 01:50:42.284751 (+   288us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:42.284834 (+    83us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:42.284900 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:42.284995 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:42.285154 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":171633,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":62,"raft.queue_time_us":392476,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":9351,"spinlock_wait_cycles":13015296,"thread_start_us":507156,"threads_started":2,"wal-append.queue_time_us":235805}
I20260808 01:50:42.329618  9472 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:42.330788  9556 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:42.331650  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2314) took 1603 ms. Trace:
I20260808 01:50:42.189033 12724 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:42.332559 12724 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:42.256445 12655 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:42.304987 11568 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:42.343045 11568 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:42.294034 12806 raft_consensus.cc:3055] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:42.276856 12571 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260808 01:50:42.295034 12789 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.110s	user 0.000s	sys 0.001s
W20260808 01:50:42.316890  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:42.327697  9560 rpcz_store.cc:276] 0808 01:50:40.687702 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:42.201810 (+1514108us) service_pool.cc:224] Handling call
0808 01:50:42.271133 (+ 69323us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:42.271397 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:42.271434 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:42.271507 (+    73us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:42.271554 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:42.271627 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:42.271779 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:42.359483  9560 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:42.360306  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2315) took 1631 ms. Trace:
I20260808 01:50:42.360687  9560 rpcz_store.cc:276] 0808 01:50:40.728484 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:42.358863 (+1630379us) service_pool.cc:224] Handling call
0808 01:50:42.359249 (+   386us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:42.360004 (+   755us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:42.360243 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:42.311038 12715 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:42.195668 12662 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:42.332041  9556 rpcz_store.cc:276] 0808 01:50:40.728118 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:42.330096 (+1601978us) service_pool.cc:224] Handling call
0808 01:50:42.330531 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:42.331308 (+   777us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:42.331573 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:42.365041  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:42.293680 12654 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:42.320693 12419 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:42.353539 11568 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:42.303807 12692 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:42.356383  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2313) took 1698 ms. Trace:
I20260808 01:50:42.265703 12634 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:42.295747  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2308) took 1911 ms. Trace:
I20260808 01:50:42.356765 12722 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:42.300627 12677 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:42.350402 12788 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:42.351647 12806 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:42.376034  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1954) took 1332 ms. Trace:
I20260808 01:50:42.375878  9550 rpcz_store.cc:276] 0808 01:50:40.658065 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:42.032384 (+1374319us) service_pool.cc:224] Handling call
0808 01:50:42.033202 (+   818us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:42.034725 (+  1523us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:42.165834 (+131109us) log.cc:844] Serialized 14 byte log entry
0808 01:50:42.355758 (+189924us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:42.355851 (+    93us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:42.355919 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:42.356054 (+   135us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:42.356279 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56474,"prepare.queue_time_us":135617,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":4889,"thread_start_us":132858,"threads_started":1}
I20260808 01:50:42.369243  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:42.356616 12571 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:42.374359  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 410448us
W20260808 01:50:42.385435 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 267916us
I20260808 01:50:42.438050  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.438997  9632 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:42.388859  9472 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
W20260808 01:50:42.397567  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:42.343138 12684 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:42.443368 12684 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:42.378873 12771 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:42.434367  9481 rpcz_store.cc:276] 0808 01:50:41.043832 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:41.335135 (+291303us) service_pool.cc:224] Handling call
0808 01:50:41.359519 (+ 24384us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:41.359901 (+   382us) raft_consensus.cc:1534] Early marking committed up to index 1
0808 01:50:41.359935 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:41.360521 (+   586us) log.cc:844] Serialized 26 byte log entry
0808 01:50:41.573785 (+213264us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:50:41.573866 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:41.574531 (+   665us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:42.375668 (+801137us) raft_consensus.cc:1685] finished
0808 01:50:42.375703 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:42.375945 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":104098,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":5950,"thread_start_us":212611,"threads_started":1,"wal-append.queue_time_us":333736}
I20260808 01:50:42.349838 12598 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:42.380684  9551 rpcz_store.cc:276] 0808 01:50:40.384389 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:41.714859 (+1330470us) service_pool.cc:224] Handling call
0808 01:50:41.715314 (+   455us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:41.718147 (+  2833us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:41.868951 (+150804us) log.cc:844] Serialized 14 byte log entry
0808 01:50:42.295356 (+426405us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:42.295420 (+    64us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:42.295469 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:42.295551 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:42.295683 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58119,"prepare.queue_time_us":179238,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":4640,"raft.queue_time_us":215431,"thread_start_us":385234,"threads_started":2,"wal-append.queue_time_us":375031}
I20260808 01:50:42.375114 12715 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election won for term 6
W20260808 01:50:42.450941  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:42.452903  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:42.381031 12795 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:42.386875 12733 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:42.476756  9551 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:42.477375  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:42.477622  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.479758  9632 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:42.474360  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.434430 12667 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:42.431676 12788 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:42.444792 12571 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:42.433866 12693 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:42.473102 12638 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 01:50:42.519165 12814 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:50:42.520227 12815 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:42.520994 12815 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:42.521461 12684 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:42.522572 12808 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
I20260808 01:50:42.522315 12684 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:42.525671  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 48414.5us
I20260808 01:50:42.526261 12777 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:50:42.527709 12777 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:42.529050  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:42.534449  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2301) took 3141 ms. Trace:
I20260808 01:50:42.541970  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.544394  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:42.545261  9631 raft_consensus.cc:2376] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6. Re-sending same reply.
I20260808 01:50:42.538156  9558 rpcz_store.cc:276] 0808 01:50:39.393454 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:41.335893 (+1942439us) service_pool.cc:224] Handling call
0808 01:50:41.336217 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:41.336439 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:41.673020 (+336581us) log.cc:844] Serialized 14 byte log entry
0808 01:50:42.278339 (+605319us) log.cc:844] Serialized 14 byte log entry
0808 01:50:42.534094 (+255755us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:42.534149 (+    55us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:42.534191 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:42.534254 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:42.534386 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":85494,"prepare.queue_time_us":163367,"prepare.run_cpu_time_us":120,"prepare.run_wall_time_us":4343,"raft.queue_time_us":576140,"raft.run_cpu_time_us":428,"raft.run_wall_time_us":64001,"thread_start_us":1006245,"threads_started":4,"wal-append.queue_time_us":338948}
I20260808 01:50:42.474959 12771 raft_consensus.cc:697] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:42.553992  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.546566  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:42.566612  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 198731us
I20260808 01:50:42.573587 10366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 591) took 2203 ms. Trace:
I20260808 01:50:42.574270  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.575955  9630 raft_consensus.cc:2368] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7.
W20260808 01:50:42.577673  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 208517us
I20260808 01:50:42.579270  9444 leader_election.cc:400] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8.
I20260808 01:50:42.579846  9444 leader_election.cc:403] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:50:42.581208  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:42.582813  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.539696 12733 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:42.535696 12748 consensus_meta.cc:287] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.675s	user 0.012s	sys 0.001s
I20260808 01:50:42.583585  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:42.552002 12678 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:42.585562  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.584367  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.529275 12788 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:42.591414  9632 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:42.590085  9524 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:42.587181  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.588869  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.605464  9630 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:42.596390  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.574170 10366 rpcz_store.cc:276] 0808 01:50:40.369718 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:40.370000 (+   282us) service_pool.cc:224] Handling call
0808 01:50:40.370403 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:40.631147 (+260744us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:42.573214 (+1942067us) spinlock_profiling.cc:243] Waited 54.5 ms on lock 0x7b5400690fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:42.573500 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":125443968}
W20260808 01:50:42.586843 12726 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.981s	user 0.007s	sys 0.007s
I20260808 01:50:42.597257  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 401) took 2865 ms. Trace:
I20260808 01:50:42.607120  9480 rpcz_store.cc:276] 0808 01:50:39.732405 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:39.732628 (+   223us) service_pool.cc:224] Handling call
0808 01:50:39.761153 (+ 28525us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:40.915227 (+1154074us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:42.597002 (+1681775us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54002bacd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:42.597193 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2639553408}
I20260808 01:50:42.608124  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:42.534615 12667 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.061s
I20260808 01:50:42.587597  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.582933  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.588171  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:42.594543 12762 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.647s	user 0.000s	sys 0.014s
W20260808 01:50:42.593385 12477 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.930s	user 0.008s	sys 0.005s
I20260808 01:50:42.621815  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:42.571612 12419 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
W20260808 01:50:42.633780 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 392014us
W20260808 01:50:42.627373 10332 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.891s (SENT) [suppressed 28 similar messages]
I20260808 01:50:42.625790  9633 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:42.586905 12809 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:42.639338  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:42.600557 12818 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:42.603235 12634 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:42.643567  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.647590  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:42.648010  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2307) took 2263 ms. Trace:
I20260808 01:50:42.648308  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:42.640296  9521 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:42.647927  9631 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:42.643285  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:42.655478  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 65490.1us
I20260808 01:50:42.657793  9637 raft_consensus.cc:2376] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6. Re-sending same reply.
I20260808 01:50:42.540303 12662 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:42.623111 12678 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:42.662957  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:42.657500  9552 rpcz_store.cc:276] 0808 01:50:40.384149 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:41.703639 (+1319490us) service_pool.cc:224] Handling call
0808 01:50:41.764745 (+ 61106us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:41.765040 (+   295us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:50:42.160738 (+395698us) log.cc:844] Serialized 14 byte log entry
0808 01:50:42.623774 (+463036us) log.cc:844] Serialized 14 byte log entry
0808 01:50:42.647628 (+ 23854us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:42.647687 (+    59us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:42.647732 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:42.647802 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:42.647926 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":171236,"prepare.queue_time_us":180172,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":103,"raft.queue_time_us":154571,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":36150,"thread_start_us":683077,"threads_started":3,"wal-append.queue_time_us":462356}
I20260808 01:50:42.663017  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2320) took 1309 ms. Trace:
I20260808 01:50:42.670456  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.601397 12748 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:42.669705  9559 rpcz_store.cc:276] 0808 01:50:41.353343 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:42.451989 (+1098646us) service_pool.cc:224] Handling call
0808 01:50:42.452280 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:42.452488 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 5
0808 01:50:42.491391 (+ 38903us) log.cc:844] Serialized 14 byte log entry
0808 01:50:42.662282 (+170891us) log.cc:844] Serialized 14 byte log entry
0808 01:50:42.662558 (+   276us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:42.662619 (+    61us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:42.662696 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:42.662799 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:42.662933 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64547,"prepare.queue_time_us":45809,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":5356,"thread_start_us":100630,"threads_started":1,"wal-append.queue_time_us":153079}
I20260808 01:50:42.647408 12693 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:42.647931 12695 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:42.641531 12477 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:42.678449  9524 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:42.680377  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 99245.9us
I20260808 01:50:42.640340 12834 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8.
I20260808 01:50:42.693861  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.737112  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:42.668376 12812 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:42.664551 12667 consensus_queue.cc:1243] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260808 01:50:42.667681 12684 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:42.760661 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:42.756153 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 128826us
W20260808 01:50:42.764569 10332 leader_election.cc:336] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.125s (SENT)
W20260808 01:50:42.759382  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 81049.8us
I20260808 01:50:42.775759  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:42.765810 10332 leader_election.cc:336] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.148s (SENT)
I20260808 01:50:42.777484 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:42.776782  9630 raft_consensus.cc:2410] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260808 01:50:42.778936  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.780462  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.689524 12693 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:42.605806 12806 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:42.786114  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:42.789561 10332 leader_election.cc:336] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.197s (SENT)
I20260808 01:50:42.804450  9633 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:50:42.805015 10332 leader_election.cc:336] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.224s (SENT)
W20260808 01:50:42.785344 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 135698us
I20260808 01:50:42.794220  9638 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:42.684198 12826 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260808 01:50:42.756186 12676 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.022s	sys 0.046s
I20260808 01:50:42.650067 12833 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.121s
I20260808 01:50:42.801209  9632 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:42.753512 12816 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:42.673867 12724 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:42.757094 12598 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:42.761441 12812 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:42.753659 12726 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.055s
I20260808 01:50:42.775759 12804 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:42.840883  9475 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 1946) took 1811 ms (client timeout 1916 ms). Trace:
W20260808 01:50:42.841594  9475 rpcz_store.cc:269] 0808 01:50:41.029781 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:41.031582 (+  1801us) service_pool.cc:224] Handling call
0808 01:50:42.840603 (+1809021us) spinlock_profiling.cc:243] Waited 627 ms on lock 0x7b54004051d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:42.840791 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1440994176}
I20260808 01:50:42.785126 12835 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:42.805819 12693 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:42.810003 12541 consensus_meta.cc:287] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.910s	user 0.015s	sys 0.000s
I20260808 01:50:42.835915  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.661068 12818 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:42.855178  9632 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:42.837909  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:42.810228 12676 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.023s	sys 0.046s
W20260808 01:50:42.801927 12824 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
I20260808 01:50:42.672220 12771 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:42.848569  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:42.805015 12834 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:42.797333 12831 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:42.656311 12809 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:42.816149 12795 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 01:50:42.856365 12800 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.007s	sys 0.024s
I20260808 01:50:42.936111  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:42.936818  9632 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:42.938030  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.940749  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.934906 12715 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 6 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:42.895182  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 375) took 1816 ms. Trace:
I20260808 01:50:42.825978 12820 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:42.848640 12806 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:42.856840 12835 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:42.825927 12842 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:42.903712  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:42.844779 12812 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:42.756409 12819 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.341s
I20260808 01:50:42.833217 12816 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:42.850992 12782 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:42.858664 12815 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:42.945808 12715 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:42.897313 12724 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:42.949512  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.983206  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 378) took 1291 ms. Trace:
I20260808 01:50:42.897614 12781 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:42.985448  9635 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:42.971757  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:42.980877  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2319) took 1644 ms. Trace:
W20260808 01:50:42.990116  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:42.977226  9559 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:50:42.984086 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:42.990525  9550 rpcz_store.cc:276] 0808 01:50:41.336593 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:42.435662 (+1099069us) service_pool.cc:224] Handling call
0808 01:50:42.451516 (+ 15854us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:42.618063 (+166547us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:42.618125 (+    62us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:42.618871 (+   746us) log.cc:844] Serialized 26 byte log entry
0808 01:50:42.824584 (+205713us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:42.824644 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:42.824746 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:42.980627 (+155881us) raft_consensus.cc:1685] finished
0808 01:50:42.980652 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:42.980809 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":71277,"raft.queue_time_us":99044,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":13136,"thread_start_us":287598,"threads_started":2,"wal-append.queue_time_us":260373}
I20260808 01:50:42.982185  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:50:42.993646 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:42.994158  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:42.994807 12830 log.cc:927] Time spent T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.206s	user 0.001s	sys 0.000s
I20260808 01:50:42.995930 12477 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:42.942498 12822 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.319s	user 0.002s	sys 0.000s
I20260808 01:50:42.961258 12807 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:42.997669  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:42.998716  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2316) took 2069 ms. Trace:
I20260808 01:50:42.862301 12541 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:42.844285 12678 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:42.998958  9557 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6. Ops: []
W20260808 01:50:43.257457 12779 consensus_meta.cc:287] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.390s	user 0.011s	sys 0.005s
I20260808 01:50:43.087534  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:43.258029 12779 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:43.194340 12654 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:42.950663 12838 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.265s	user 0.001s	sys 0.000s
I20260808 01:50:43.261833  9629 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:43.262035 12654 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:42.984258 12824 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
W20260808 01:50:42.950071 12763 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 2.235s	user 0.004s	sys 0.024s
I20260808 01:50:42.943925 12834 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:43.269506  9560 rpcz_store.cc:276] 0808 01:50:40.929734 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:42.361711 (+1431977us) service_pool.cc:224] Handling call
0808 01:50:42.362077 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:42.378543 (+ 16466us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:42.378589 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:42.386422 (+  7833us) log.cc:844] Serialized 26 byte log entry
0808 01:50:42.617527 (+231105us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:42.617595 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:42.617708 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:42.998453 (+380745us) raft_consensus.cc:1685] finished
0808 01:50:42.998476 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:42.998632 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9070,"raft.queue_time_us":166794,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":8466,"thread_start_us":141281,"threads_started":1,"wal-append.queue_time_us":408454}
I20260808 01:50:43.271932  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:42.951273 12827 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
I20260808 01:50:43.205040 12748 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:43.086946 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 102954us
I20260808 01:50:43.087069 12745 raft_consensus.cc:3055] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:43.282505 12745 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:43.267616 12807 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:43.289160 12779 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:43.268554 12834 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:43.292930 10360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 465) took 3737 ms. Trace:
I20260808 01:50:43.293699 10360 rpcz_store.cc:276] 0808 01:50:39.555993 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:39.566508 (+ 10515us) service_pool.cc:224] Handling call
0808 01:50:39.638555 (+ 72047us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:39.905886 (+267331us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:39.910790 (+  4904us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:40.121691 (+210901us) log.cc:844] Serialized 52 byte log entry
0808 01:50:40.379346 (+257655us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:40.380615 (+  1269us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:40.429013 (+ 48398us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:42.850174 (+2421161us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:43.292451 (+442277us) raft_consensus.cc:1685] finished
0808 01:50:43.292498 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:43.292833 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":93,"raft.queue_time_us":93162,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":19522,"spinlock_wait_cycles":28802432,"thread_start_us":195849,"threads_started":2,"wal-append.queue_time_us":414770}
I20260808 01:50:43.297871 12796 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260808 01:50:43.308887 12834 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:43.312120 12807 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:43.274199  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:43.316602  9560 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:43.322978  9550 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:43.324532  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:43.325369  9550 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:43.326772  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:43.327530  9550 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:43.328557  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:43.101727  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:43.330065  9560 raft_consensus.cc:2410] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260808 01:50:43.331032  9524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.000s	sys 0.018s
W20260808 01:50:43.331568  9524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.000s	sys 0.019s
W20260808 01:50:43.332096  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:42.944231 12676 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:42.993698 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:43.070111  9633 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 3.
W20260808 01:50:43.340852  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.000s	sys 0.017s
W20260808 01:50:43.341490  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.000s	sys 0.018s
I20260808 01:50:43.343768  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 379) took 1651 ms. Trace:
I20260808 01:50:43.343945  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:43.328719  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:42.983739  9551 rpcz_store.cc:276] 0808 01:50:41.692051 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:42.486577 (+794526us) service_pool.cc:224] Handling call
0808 01:50:42.487069 (+   492us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:42.487372 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:50:42.528126 (+ 40754us) log.cc:844] Serialized 14 byte log entry
0808 01:50:42.982852 (+454726us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:42.982905 (+    53us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:42.982960 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:42.983032 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:42.983145 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6936,"prepare.queue_time_us":40982,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":9737,"thread_start_us":454374,"threads_started":1}
I20260808 01:50:43.261371  9635 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
W20260808 01:50:43.338456  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 53993.4us
I20260808 01:50:42.998968 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:43.382798  9560 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:43.166107 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:43.384403  9560 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 01:50:43.130290  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:43.384577  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:43.344532  9558 rpcz_store.cc:276] 0808 01:50:41.692385 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:42.572038 (+879653us) service_pool.cc:224] Handling call
0808 01:50:42.593309 (+ 21271us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:42.642886 (+ 49577us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:42.642932 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:42.648543 (+  5611us) log.cc:844] Serialized 52 byte log entry
0808 01:50:42.790845 (+142302us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:42.810570 (+ 19725us) log.cc:844] Serialized 14 byte log entry
0808 01:50:43.343231 (+532661us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:43.343371 (+   140us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:43.343435 (+    64us) raft_consensus.cc:1685] finished
0808 01:50:43.343457 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:43.343677 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64553,"prepare.queue_time_us":17248,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":169908,"raft.run_cpu_time_us":482,"raft.run_wall_time_us":83011,"thread_start_us":598027,"threads_started":2,"wal-append.queue_time_us":522292}
W20260808 01:50:42.957484 12776 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.263s	user 0.000s	sys 0.011s
I20260808 01:50:43.388222  9632 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
W20260808 01:50:43.394467  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:43.255169 12847 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.274s	user 0.001s	sys 0.000s
I20260808 01:50:43.391711  9560 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260808 01:50:43.389144  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:42.968041 12835 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:43.399080  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:43.122295 12782 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:42.946646 12809 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:42.984202 12820 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:43.368187 12667 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:43.441396  9638 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: []
I20260808 01:50:43.364763 12748 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:43.445113  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 56058.9us
I20260808 01:50:43.457372  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:42.977015 12816 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:43.447161  9560 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:50:43.122509 12781 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:43.471556 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:43.368233 12807 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:43.459813  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:43.395625 12715 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:43.392952 12654 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:43.390573 12850 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:43.475111  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:43.476696 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:43.479465  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 403) took 1292 ms. Trace:
I20260808 01:50:43.451671 12724 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:43.458915 12598 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:43.485890  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:43.383982 12676 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:43.485497  9474 rpcz_store.cc:276] 0808 01:50:42.186749 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:42.187074 (+   325us) service_pool.cc:224] Handling call
0808 01:50:42.187526 (+   452us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:42.187829 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:42.187868 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:42.224899 (+ 37031us) log.cc:844] Serialized 26 byte log entry
0808 01:50:42.531207 (+306308us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:42.531259 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:42.533073 (+  1814us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:43.479218 (+946145us) raft_consensus.cc:1685] finished
0808 01:50:43.479243 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:43.479393 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":221790,"threads_started":1,"wal-append.queue_time_us":374566}
I20260808 01:50:43.449219 12782 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:43.478960 12788 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.856s	user 0.011s	sys 0.001s
I20260808 01:50:43.466841 12842 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 01:50:43.476902 12785 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.629s	user 0.014s	sys 0.020s
W20260808 01:50:43.519347 12785 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.672s	user 0.014s	sys 0.020s
I20260808 01:50:43.503274 12541 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:43.504846 12772 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:43.492267 12850 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:43.524856 12850 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:43.470110 12748 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:43.526624  9558 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:50:43.406869 12770 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.080s
I20260808 01:50:43.528609 12770 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:43.528990 12770 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:43.408320 12776 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:43.532522 12776 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:43.485819 12715 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:43.534958 12715 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:43.535662 12748 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:43.518601 10365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 592) took 1681 ms. Trace:
I20260808 01:50:43.536806 10365 rpcz_store.cc:276] 0808 01:50:41.837528 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:41.837941 (+   413us) service_pool.cc:224] Handling call
0808 01:50:41.896124 (+ 58183us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:42.186221 (+290097us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:42.186268 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:42.186329 (+    61us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:42.186374 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:43.518237 (+1331863us) spinlock_profiling.cc:243] Waited 290 ms on lock 0x7b5400785ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:43.518299 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:43.518487 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":666450176}
I20260808 01:50:43.529958 12770 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:43.539142 12770 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:43.484114 12834 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:43.544265  9560 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
W20260808 01:50:43.550141  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 173230us
I20260808 01:50:43.410880 12779 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
W20260808 01:50:43.542651 12677 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.105s	user 0.010s	sys 0.000s
W20260808 01:50:43.550499 12571 consensus_meta.cc:287] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.039s	user 0.013s	sys 0.001s
I20260808 01:50:43.512559 12782 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:43.480953 12807 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:43.562140  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:43.554107 12856 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:43.553027 12420 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:43.492784 12724 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:43.481722 12781 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:43.554008 12778 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 01:50:43.565508 12722 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.128s	user 0.015s	sys 0.000s
I20260808 01:50:43.557551 12818 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:43.587427  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 309985us
W20260808 01:50:43.589550  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:43.548265 12788 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:43.592782  9521 leader_election.cc:336] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.575s (SENT)
I20260808 01:50:43.571909 12856 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:43.593954 12855 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:43.580315 12420 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:43.603328  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:43.604368  9560 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:43.597892 12855 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:43.607486 12853 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:43.608399 12853 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:43.610162 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:43.611064 10365 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:50:43.595331  9521 leader_election.cc:336] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.588s (SENT)
W20260808 01:50:43.592882 12693 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.692s	user 0.012s	sys 0.000s
I20260808 01:50:43.613344  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:43.613999  9521 leader_election.cc:336] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.601s (SENT)
I20260808 01:50:43.621521  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:43.617192 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:43.569587 12677 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:43.614426 12775 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:43.572460 12571 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:43.619196 12477 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:50:43.621587  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:43.615968  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:43.632445  9474 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:43.632756  9475 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:43.633562  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:43.623301  9521 leader_election.cc:336] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.615s (SENT)
I20260808 01:50:43.623633  9472 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:50:43.635350  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:43.635501  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:43.636770  9472 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: [4.3-7.5]
I20260808 01:50:43.634927  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:43.638065  9472 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:43.632114  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:43.634277  9475 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:43.618168 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:43.633574  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:43.596077 12856 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:43.639626  9474 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:43.640278  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:43.638552  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 248684us
I20260808 01:50:43.640337 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:43.641635  9477 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260808 01:50:43.647176  9521 leader_election.cc:336] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.693s (SENT)
I20260808 01:50:43.647691  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:43.646710 12809 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260808 01:50:43.651245 12806 consensus_meta.cc:287] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.686s	user 0.011s	sys 0.000s
I20260808 01:50:43.652204  9550 raft_consensus.cc:2410] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260808 01:50:43.653350 12724 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:43.660290  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:43.661015 12807 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:43.630800 12693 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:43.662889  9552 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:43.665275  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:43.665957  9552 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:43.667433  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:43.668761  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:43.669770  9552 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:43.668862 12677 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:43.674028  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:43.628005 12722 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:43.619539 12818 consensus_queue.cc:1059] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20260808 01:50:43.657760 12691 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:43.669034  9633 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:50:43.688657  9474 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:43.684583  9478 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 7. Current term is 8. Ops: []
W20260808 01:50:43.668948 12733 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.069s	user 0.009s	sys 0.006s
I20260808 01:50:43.661039  9558 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:50:43.699668  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:43.693197  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:43.700408  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:43.700637  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 214830us
I20260808 01:50:43.690591  9524 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:43.701026  9477 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:43.694562  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:43.701109  9558 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:43.700467  9476 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:43.701876  9560 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
W20260808 01:50:43.684986 12655 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.338s	user 0.013s	sys 0.002s
I20260808 01:50:43.704300  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:43.701222 12812 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.726s	user 0.013s	sys 0.000s
I20260808 01:50:43.714052  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:43.718463  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 71334.9us
W20260808 01:50:43.721038  9521 leader_election.cc:336] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.731s (SENT)
I20260808 01:50:43.721604  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:43.636492 12775 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:43.652220 12477 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260808 01:50:43.705102 12733 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:43.718361  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:43.718003 12788 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:43.752880 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:43.753518 12855 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:43.753712 10369 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:43.755909  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:43.756795  9560 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:43.757217  9444 leader_election.cc:400] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8.
I20260808 01:50:43.716696 12778 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:43.718624 12726 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:43.765022  9524 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:43.766604  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 45623.8us
W20260808 01:50:43.768615  9521 leader_election.cc:336] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.807s (SENT)
I20260808 01:50:43.769068  9521 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:43.733136 12831 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.006s	sys 0.035s
I20260808 01:50:43.758932  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:43.716076 12854 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:43.774245  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:43.775593  9558 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:50:43.718272 12691 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:43.704571 12724 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.056s
I20260808 01:50:43.724006 12812 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:43.745301 12722 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.032s
W20260808 01:50:43.726506 12677 consensus_queue.cc:1175] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6e7856bed1be43acaa821290d506ee97" responder_term: 8 status { last_received { term: 2 index: 2 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: [4.3-7.5]" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 01:50:43.706475 12655 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:43.759519 12807 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:43.766721 12726 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:43.799494  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:43.856179  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 82026.1us
I20260808 01:50:43.803213 12855 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:43.858748  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:43.854506  9560 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:43.797058  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:43.858404  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:43.755098 12842 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 01:50:43.808746  9521 leader_election.cc:336] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.829s (SENT)
I20260808 01:50:43.864373  9521 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:43.803988  9524 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:43.776393 12781 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:43.809723 10334 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:43.804302 12864 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:43.793812 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:43.883057 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 89369.3us
I20260808 01:50:43.772930 12806 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20260808 01:50:43.819602 12767 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:43.906906  9637 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:50:43.777292 12831 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.690s	user 0.007s	sys 0.036s
I20260808 01:50:43.766057 12683 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260808 01:50:43.827430 12693 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:43.829334 12775 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:43.815766 12571 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:43.822044  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2310) took 3200 ms. Trace:
I20260808 01:50:43.916358  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:43.916879  9555 rpcz_store.cc:276] 0808 01:50:40.621283 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:41.913276 (+1291993us) service_pool.cc:224] Handling call
0808 01:50:41.913752 (+   476us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:42.019524 (+105772us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:43.821756 (+1802232us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b540002c6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:43.821962 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":240777984}
I20260808 01:50:43.822206 12420 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:43.822854 12560 raft_consensus.cc:3055] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:43.836869 12819 raft_consensus.cc:3055] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:43.837433  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:43.837785  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:43.882660 12864 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:43.825682  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 408) took 1537 ms. Trace:
I20260808 01:50:43.840725  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:43.841012  9475 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:43.946929  9478 rpcz_store.cc:276] 0808 01:50:42.288672 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:42.288814 (+   142us) service_pool.cc:224] Handling call
0808 01:50:42.289255 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:43.698931 (+1409676us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:43.825390 (+126459us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54000217d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:43.825610 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3208449920}
I20260808 01:50:43.842110  9477 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:43.948086  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:43.728515 12853 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:43.823578 12788 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:43.842043  9552 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:50:43.910231 12831 raft_consensus.cc:697] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:43.866246 12695 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:43.866050  9523 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:43.870090  9550 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:43.870437 12778 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:43.872996 12812 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:43.874459 12816 raft_consensus.cc:3055] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:42.995100  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
W20260808 01:50:43.879065  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 75180.6us
W20260808 01:50:43.881850 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 72210.8us
I20260808 01:50:43.884892  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:43.802268 12733 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:43.865242 12866 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:43.859760 12726 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:43.857618 12807 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:43.912640  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:43.880177 12814 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:43.918509  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:43.919984  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:43.921079  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:43.905982 12781 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:43.809141 12785 consensus_queue.cc:1059] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.100s
I20260808 01:50:43.838459  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:43.994100  9560 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:50:43.932753 12560 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:43.929629  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:43.943852  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 324082us
I20260808 01:50:43.906198 12796 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:43.906069 12746 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:43.952801  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:43.996304  9475 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:43.997334  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:43.997996  9475 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:50:43.954068  9635 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:43.979945  9632 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:50:43.759440 12857 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:43.958323  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:43.964560  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 155840us
I20260808 01:50:43.965582  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:43.969653  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 103719us
W20260808 01:50:43.971138 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47237.2us
I20260808 01:50:43.971060  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:44.000480  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:43.940805 12819 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:43.907219 12842 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
W20260808 01:50:44.033103  9521 leader_election.cc:336] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.834s (SENT)
I20260808 01:50:44.033684  9476 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:43.953681 12864 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:44.003832  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 119010us
I20260808 01:50:44.031816  9478 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 8: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:44.042657  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:44.034019  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:44.044406  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:43.943429 12800 consensus_queue.cc:1059] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.079s
I20260808 01:50:44.032644  9475 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 5: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:50:43.954593  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:44.045907  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 50192.3us
I20260808 01:50:44.034034  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:44.035427  9523 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:44.033697  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:44.038537  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:44.051093  9478 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
W20260808 01:50:44.038338  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 263716us
W20260808 01:50:44.071691  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:44.039443  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:44.073028  9476 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:44.040331  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:44.034709  9477 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:44.020864  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:44.119557 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:44.099329  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 63991.3us
W20260808 01:50:44.135831  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 102786us
I20260808 01:50:44.146239  9633 raft_consensus.cc:2376] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7. Re-sending same reply.
W20260808 01:50:44.146395  9521 leader_election.cc:336] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.849s (SENT)
I20260808 01:50:44.001613 12807 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:44.120903  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:44.098594  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:44.075086  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:44.147117  9521 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:44.074049  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:43.992933 12683 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:44.145679  9523 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:44.150318  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:44.156024  9632 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:50:43.974970 12816 raft_consensus.cc:740] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:44.001814 12866 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:44.159031  9631 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 10: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:50:44.160912  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62461.7us
I20260808 01:50:44.164371  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:44.168292  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:44.168591  9632 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:50:44.164918  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 65067.5us
I20260808 01:50:44.169977  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:43.977916 12722 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:50:44.002892 12796 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:44.134454 12677 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:44.172485  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:44.173630  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:44.096047 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 64835.8us
I20260808 01:50:44.157358  9523 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:44.161855  9521 leader_election.cc:336] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.872s (SENT)
I20260808 01:50:44.174376  9633 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:44.165139  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:44.030103 12724 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:44.178366  9599 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:44.166643  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:44.175992  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:44.179831  9601 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:44.112512 12818 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:44.177033 12796 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:44.112848 12809 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:44.125475 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:44.179060  9632 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:44.180413 12724 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:44.243278 12724 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:44.243850  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:44.244869  9474 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 4: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:44.186048 12722 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:44.099385 12782 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:44.187335 12818 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:44.197981 12745 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:44.075583 12863 consensus_queue.cc:1059] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.113s
I20260808 01:50:44.177798 12655 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:44.188952 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:44.257901  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:44.258458 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 70308.3us
I20260808 01:50:44.191584  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1957) took 3146 ms. Trace:
I20260808 01:50:44.191764 12738 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:44.279110 12738 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:44.279639 12738 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:44.195403 12820 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:44.196445 12869 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:44.201390 12692 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:44.284364 12869 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:44.284749 12692 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:44.204167 12775 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:44.145856 12560 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:44.278069 12864 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.162214 12812 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:44.133525 12785 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.080s
I20260808 01:50:44.286769  9555 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:44.159181 12819 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:44.288043 12819 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:44.287762 12722 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.171675 12678 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:44.291045 12678 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.291270 12795 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:50:44.293295 12795 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:44.293450 12678 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:44.231666 12835 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:44.181583  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:44.290500 12864 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:44.295107  9477 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:44.295511  9636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1852) took 1749 ms (client timeout 1970 ms). Trace:
W20260808 01:50:44.296018  9636 rpcz_store.cc:269] 0808 01:50:42.545603 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:42.621522 (+ 75919us) service_pool.cc:224] Handling call
0808 01:50:44.295270 (+1673748us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b5400338954. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:44.295442 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2927360256}
I20260808 01:50:44.296536 12818 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:44.171622 12842 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:50:44.303758  9634 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:42.944257  9556 rpcz_store.cc:276] 0808 01:50:41.078942 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:42.365310 (+1286368us) service_pool.cc:224] Handling call
0808 01:50:42.365762 (+   452us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:42.366101 (+   339us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:50:42.366529 (+   428us) log.cc:844] Serialized 14 byte log entry
0808 01:50:42.894458 (+527929us) log.cc:844] Serialized 14 byte log entry
0808 01:50:42.894655 (+   197us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:42.894754 (+    99us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:50:42.894840 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:42.894937 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:42.895098 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44662,"prepare.queue_time_us":785,"prepare.run_cpu_time_us":121,"prepare.run_wall_time_us":8931,"raft.queue_time_us":212777,"spinlock_wait_cycles":31649920,"thread_start_us":482293,"threads_started":1}
I20260808 01:50:44.304618  9635 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:50:44.181540 12767 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:44.211791 12816 consensus_queue.cc:260] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.212252  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:44.213248  9599 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:44.334544  9481 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
W20260808 01:50:44.335009  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 121848us
I20260808 01:50:44.335825  9599 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:44.184319  9601 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:44.153275 12857 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:44.020283 12733 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:44.022775 12781 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:44.219414 12879 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260808 01:50:44.221347  9637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1861) took 1559 ms (client timeout 1808 ms). Trace:
I20260808 01:50:44.145215 12831 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.176549  9631 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: [5.4-7.5]
W20260808 01:50:44.225013  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 67774.4us
W20260808 01:50:44.352793  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:44.227517  9633 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260808 01:50:44.184893  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:44.241775  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1840) took 3552 ms. Trace:
I20260808 01:50:44.354120  9633 pending_rounds.cc:85] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 5
I20260808 01:50:44.354494  9557 rpcz_store.cc:276] 0808 01:50:40.689453 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:42.348018 (+1658565us) service_pool.cc:224] Handling call
0808 01:50:42.998776 (+650758us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:43.256337 (+257561us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:44.241492 (+985155us) spinlock_profiling.cc:243] Waited 636 ms on lock 0x7b54003142d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:44.241684 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1463563136}
I20260808 01:50:44.171577 12866 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:44.228595 12856 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.527s	user 0.004s	sys 0.007s
W20260808 01:50:44.256466  9521 leader_election.cc:336] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.970s (SENT)
I20260808 01:50:44.304240 12746 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:44.262410  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:44.268980 12420 raft_consensus.cc:3055] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:44.275436 12807 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:44.278268 12676 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:44.284940 12869 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:44.286397 12560 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:44.177115 12770 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
W20260808 01:50:44.318025 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 321284us
I20260808 01:50:44.354040 12816 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:44.335668  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 163254us
W20260808 01:50:44.342803  9637 rpcz_store.cc:269] 0808 01:50:42.661713 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:42.663634 (+  1921us) service_pool.cc:224] Handling call
0808 01:50:44.221108 (+1557474us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b540025b7d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:44.221273 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2658009472}
I20260808 01:50:44.294564 12835 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:44.345412  9599 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:50:44.346654  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 162438us
I20260808 01:50:44.354557  9633 pending_rounds.cc:107] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260808 01:50:44.429380  9633 raft_consensus.cc:2889] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:44.358784  9480 raft_consensus.cc:2410] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260808 01:50:44.430651  9551 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 6. Ops: []
I20260808 01:50:44.433058  9480 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:44.359741 12864 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:44.243996 12726 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.362605 12781 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:44.364724 12733 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:44.368854  9629 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:50:44.371369 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 111176us
W20260808 01:50:44.377666  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 121265us
I20260808 01:50:44.259606  9479 rpcz_store.cc:276] 0808 01:50:41.044812 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:41.337469 (+292657us) service_pool.cc:224] Handling call
0808 01:50:41.364561 (+ 27092us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:42.112580 (+748019us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:44.191332 (+2078752us) spinlock_profiling.cc:243] Waited 664 ms on lock 0x7b540022b5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:44.191511 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1527963392}
I20260808 01:50:44.331104 12767 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:44.454969 12767 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:44.456383 12498 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:44.457443  9479 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:44.458974  9479 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:44.348718 12879 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:44.418560  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:44.460206  9479 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:44.420085  9475 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:44.422135  9477 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:44.464684  9638 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:44.465287  9638 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:44.338007  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:44.338423 12842 raft_consensus.cc:697] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:44.467785 12842 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:44.353629 12678 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:50:44.428830  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:44.413551 12788 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:44.406044 12420 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:44.469657 12678 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:44.471305 12678 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.472884 12420 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.474344 12420 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:50:44.433130 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:44.474962  9480 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: []
W20260808 01:50:44.476727  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47996.2us
I20260808 01:50:44.477098  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:44.477877  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:44.426618  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:44.479305  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:44.479377  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 52846.1us
I20260808 01:50:44.457106 12498 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.460268 12879 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:44.430747 12835 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:50:44.438480  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 93214.9us
I20260808 01:50:44.494160  9599 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:44.357488 12477 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:44.479640 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:44.505672  9480 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260808 01:50:44.480038 12779 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:44.480443 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:44.338881 12745 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:44.480782 12881 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260808 01:50:44.455090  9521 leader_election.cc:336] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.087s (SENT)
I20260808 01:50:44.522900 12881 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:44.523494  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 68451.6us
W20260808 01:50:44.524859  9521 leader_election.cc:336] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.143s (SENT)
I20260808 01:50:44.525074  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1858) took 1928 ms. Trace:
W20260808 01:50:44.525813  9521 leader_election.cc:336] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.150s (SENT)
I20260808 01:50:44.526280  9521 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:44.485610  9524 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:44.468556  9478 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260808 01:50:44.532475  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47001us
I20260808 01:50:44.533707  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:44.446163 12676 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:44.466563  9638 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:44.534981 12809 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:44.535091  9638 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:44.535539 12809 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:44.536041 12809 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:44.439935 12818 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:44.538894 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:44.539259 12857 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:44.539924 10364 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 8: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:44.540218 12818 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:44.540685 12818 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:44.540812  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:44.541399  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:44.379179 12856 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.543484  9373 catalog_manager.cc:5697] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 1 to 4, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 4 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:44.546058 12726 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:44.438577 12800 raft_consensus.cc:3055] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:50:44.546939 12800 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:44.547822 12807 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:44.547971 12800 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:44.549247 12800 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:50:44.540856 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:44.552420 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:44.480067 12882 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:44.523445 12881 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:44.532395  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:44.534233 10367 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:44.512080 12863 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:44.514353 12746 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:44.445358 12864 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.516359 12498 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:44.505753 12697 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:44.519243 12733 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:44.407660  9474 raft_consensus.cc:2410] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:50:44.527132  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:44.525506  9634 rpcz_store.cc:276] 0808 01:50:42.597098 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:42.638331 (+ 41233us) service_pool.cc:224] Handling call
0808 01:50:44.297585 (+1659254us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:44.304457 (+  6872us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:44.524859 (+220402us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b5400338950. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:44.525009 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3770480512}
I20260808 01:50:44.527645 10332 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:44.529317 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:44.627537 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 99964.9us
W20260808 01:50:44.529739  9372 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55058 (request call id 126) took 14832 ms (client timeout 15000 ms). Trace:
W20260808 01:50:44.629628  9372 rpcz_store.cc:269] 0808 01:50:29.697046 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:29.789916 (+ 92870us) service_pool.cc:224] Handling call
0808 01:50:44.529616 (+14739700us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:50:36.901646 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:50:36.902471 (+   825us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:50:36.908197 (+  5726us) write_op.cc:435] Acquired schema lock
0808 01:50:36.908243 (+    46us) tablet.cc:662] Decoding operations
0808 01:50:36.909847 (+  1604us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:50:36.909961 (+   114us) write_op.cc:641] Partition lock acquired for write op
0808 01:50:36.909991 (+    30us) tablet.cc:685] Acquiring locks for 13 operations
0808 01:50:37.018069 (+108078us) tablet.cc:701] Row locks acquired
0808 01:50:37.018117 (+    48us) write_op.cc:260] PREPARE: finished
0808 01:50:37.018356 (+   239us) write_op.cc:270] Start()
0808 01:50:37.018558 (+   202us) write_op.cc:276] Timestamp: P: 1786153837018279 usec, L: 0
0808 01:50:37.018605 (+    47us) op_driver.cc:348] REPLICATION: starting
0808 01:50:37.019553 (+   948us) log.cc:844] Serialized 5389 byte log entry
0808 01:50:37.658985 (+639432us) spinlock_profiling.cc:243] Waited 3.45 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:38.067943 (+408958us) op_driver.cc:464] REPLICATION: finished
0808 01:50:38.441402 (+373459us) write_op.cc:301] APPLY: starting
0808 01:50:38.441614 (+   212us) tablet.cc:1367] starting BulkCheckPresence
0808 01:50:38.442090 (+   476us) tablet.cc:1370] finished BulkCheckPresence
0808 01:50:38.442141 (+    51us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:50:38.508092 (+ 65951us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:50:38.508576 (+   484us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=13
0808 01:50:38.508658 (+    82us) write_op.cc:312] APPLY: finished
0808 01:50:38.509253 (+   595us) log.cc:844] Serialized 95 byte log entry
0808 01:50:38.509497 (+   244us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:38.509667 (+   170us) write_op.cc:454] Released schema lock
0808 01:50:38.509741 (+    74us) write_op.cc:341] FINISH: Updating metrics
0808 01:50:38.666134 (+156393us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:38.667745 (+  1611us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":109977,"reported_tablets":21,"child_traces":[["op",{"apply.queue_time_us":346932,"apply.run_cpu_time_us":11426,"apply.run_wall_time_us":184673,"num_ops":13,"prepare.queue_time_us":495627,"prepare.run_cpu_time_us":35498,"prepare.run_wall_time_us":357332,"raft.queue_time_us":341669,"raft.run_cpu_time_us":15952,"raft.run_wall_time_us":260529,"replication_time_us":1018439,"spinlock_wait_cycles":7926144,"thread_start_us":631002,"threads_started":4,"wal-append.queue_time_us":311095,"wal-append.run_cpu_time_us":50232,"wal-append.run_wall_time_us":2190529}]]}
I20260808 01:50:44.530256 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:44.531401 10363 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:44.531689 12787 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:44.534036  9524 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:44.534379  9550 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:44.637422 12787 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:44.640412 12787 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:44.573755 12692 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.556032 12882 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.642822 12884 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:44.643532 12884 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:44.643973 12884 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:44.644845 12692 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:44.544394 12856 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:44.645553  9555 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:44.489661  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:44.513233 12571 consensus_queue.cc:1059] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:50:44.555240  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:44.481056 12788 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election won for term 5
W20260808 01:50:44.588070  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:44.652179  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 118221us
I20260808 01:50:44.572660 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:44.572180 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:44.557384 12738 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.558450 12880 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
I20260808 01:50:44.568195  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:44.576049  9478 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: []
W20260808 01:50:44.569155  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 234916us
I20260808 01:50:44.658982  9551 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:44.579187  9521 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:44.576682  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 60907.1us
I20260808 01:50:44.659231  9552 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:44.646070  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:44.568771 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:44.647145  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 157591us
I20260808 01:50:44.661676  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:44.627238  9373 catalog_manager.cc:5697] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 2 to 4, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 4 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:44.556108 12779 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:44.554602 12724 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.570888 12477 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:44.660826  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 81786us
I20260808 01:50:44.668195  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:44.564122 12782 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:44.561120 12844 raft_consensus.cc:3055] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:44.666641  9550 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
W20260808 01:50:44.675467 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:44.676777  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:44.694600  9521 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:44.695073  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:44.685371  9601 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:44.696991  9555 raft_consensus.cc:2393] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:44.622588 12676 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:44.562839 12775 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.118s
I20260808 01:50:44.646524 12863 raft_consensus.cc:697] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:44.644873 12882 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:44.704861  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:44.655390 12684 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:50:44.657485 12498 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:44.562762 12655 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.118s
I20260808 01:50:44.555656 12796 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:44.730783  9472 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:44.557790 12771 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.121s
I20260808 01:50:44.740618  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:44.703994 12420 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:44.667409 12779 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:44.657994 12854 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:44.695870 12477 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:44.669313 12804 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:44.746001  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:44.767946  9552 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:44.705015 12844 raft_consensus.cc:740] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:44.709678 12692 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:50:44.758080  9373 catalog_manager.cc:5697] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 2 to 4, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 4 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:44.759966  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:44.739396 12693 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:44.772601  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:44.733235 12726 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:44.776665  9552 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:44.776851  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:44.757530 12420 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:44.780496  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:44.835801  9472 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:44.719830 12807 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:44.704216 12756 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
I20260808 01:50:44.789884 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:44.778426  9555 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:50:44.772315 12820 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:44.854071  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:44.865903  9478 raft_consensus.cc:2410] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 9 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260808 01:50:44.879276 12844 consensus_queue.cc:260] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:44.880775 12844 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:44.762529 12684 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.083s
I20260808 01:50:44.794056 12726 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:44.782418 12867 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:50:44.781297 12498 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:44.745791 12882 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20260808 01:50:44.789091 12693 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:50:44.886777 10332 leader_election.cc:336] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.891s (SENT)
W20260808 01:50:44.889424 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:44.766983 12854 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:44.904961  9373 catalog_manager.cc:5697] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 1 to 3, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 3 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:44.840306 12420 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:44.931548  9472 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:44.933429 10368 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
W20260808 01:50:44.946100 12866 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.532s	user 0.011s	sys 0.000s
I20260808 01:50:44.946610 12866 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.950910  9473 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 7: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:44.899127 12726 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:50:44.957542  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 272288us
I20260808 01:50:44.852896 12807 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.973680  9558 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:44.976886  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2197) took 1137 ms. Trace:
I20260808 01:50:44.978055  9481 rpcz_store.cc:276] 0808 01:50:43.839866 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:44.335473 (+495607us) service_pool.cc:224] Handling call
0808 01:50:44.335820 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:44.336069 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:44.371214 (+ 35145us) log.cc:844] Serialized 14 byte log entry
0808 01:50:44.975875 (+604661us) log.cc:844] Serialized 14 byte log entry
0808 01:50:44.976113 (+   238us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:44.976179 (+    66us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:44.976244 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:44.976604 (+   360us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:44.976793 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":22169,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":26185,"raft.queue_time_us":140728,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":11645,"thread_start_us":603394,"threads_started":2,"wal-append.queue_time_us":368540}
W20260808 01:50:44.986423  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:44.934279 12782 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:44.952709 12756 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:44.966620 12867 raft_consensus.cc:3055] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:44.992519  9481 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:45.003612  9373 catalog_manager.cc:5697] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 1 to 4, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 4 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:44.994240 12756 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:44.858353 12863 consensus_queue.cc:237] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.882589 12676 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.985275 12738 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:44.928028 12767 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:45.014427  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:45.311964  9555 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:45.020491  9481 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260808 01:50:45.313903  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1995) took 1145 ms. Trace:
I20260808 01:50:45.314467  9481 rpcz_store.cc:276] 0808 01:50:44.168708 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:45.019854 (+851146us) service_pool.cc:224] Handling call
0808 01:50:45.020219 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:45.313431 (+293212us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:45.313811 (+   380us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:44.884151 12820 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:45.027024  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2208) took 1159 ms. Trace:
I20260808 01:50:44.974344  9601 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:45.028855  9472 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260808 01:50:45.318112  9558 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
W20260808 01:50:45.318718  9445 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.013s	sys 0.009s
W20260808 01:50:45.319648  9445 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.013s	sys 0.009s
W20260808 01:50:45.176090  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:45.325291  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:45.318517  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 344256us
I20260808 01:50:45.331598  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2221) took 1202 ms. Trace:
I20260808 01:50:45.332155  9472 rpcz_store.cc:276] 0808 01:50:44.129149 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:45.027906 (+898757us) service_pool.cc:224] Handling call
0808 01:50:45.028373 (+   467us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:45.331211 (+302838us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:45.331503 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:45.333835  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:45.265232 12628 raft_consensus.cc:3055] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:45.335179 12628 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:45.336498 12628 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:45.337806 12628 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:45.340529 12884 consensus_meta.cc:287] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.696s	user 0.014s	sys 0.000s
I20260808 01:50:45.341147 12884 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:45.316001 12571 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.003s	sys 0.019s
W20260808 01:50:45.344614 12571 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.003s	sys 0.019s
I20260808 01:50:45.001972 12693 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:45.221144  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:45.346654 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:45.056445 12796 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:45.182291  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 513151us
I20260808 01:50:45.001715 12782 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:45.140344 10332 leader_election.cc:336] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.140s (SENT)
I20260808 01:50:45.071080 12906 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:45.359192 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 218893us
I20260808 01:50:45.359117 12906 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:45.360327 12879 consensus_meta.cc:287] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.801s	user 0.009s	sys 0.004s
I20260808 01:50:45.360931 12879 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:45.361605 12906 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:45.006361 12867 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:45.362874 12867 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:45.363906 12867 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:45.364662 12796 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:45.365234 12796 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:45.367580 12796 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:45.199572  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2192) took 1362 ms. Trace:
I20260808 01:50:45.200207 12420 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:44.994732 12881 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:45.371007 12881 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:45.371775 12420 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:45.209668  9474 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260808 01:50:45.067045 12778 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:45.373870 12778 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:45.374257  9473 rpcz_store.cc:276] 0808 01:50:43.837373 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:43.837518 (+   145us) service_pool.cc:224] Handling call
0808 01:50:45.199334 (+1361816us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54002bb6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:45.199488 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2319717760}
I20260808 01:50:45.374908 12778 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:45.376183  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:45.193080 12809 consensus_meta.cc:287] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.656s	user 0.013s	sys 0.000s
I20260808 01:50:45.376793  9473 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:45.376883 12809 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:45.378129  9560 raft_consensus.cc:2393] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:45.129185  9373 catalog_manager.cc:5697] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 2 to 4, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 4 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:50:45.380489 12809 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:45.381829  9373 catalog_manager.cc:5697] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 1 to 3, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 3 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:45.383667  9373 catalog_manager.cc:5697] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 2 to 5, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 5 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:45.192185 12807 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:45.004429 12864 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:45.074894 12498 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:45.388401  9555 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
W20260808 01:50:45.239477 12819 consensus_meta.cc:287] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.951s	user 0.012s	sys 0.003s
I20260808 01:50:45.005875 12724 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:45.122658 12831 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:45.391067 12831 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:45.392357 12820 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:45.393421 12831 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:45.262952 12892 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.409s	user 0.000s	sys 0.010s
W20260808 01:50:45.237084  9551 consensus_meta.cc:287] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.577s	user 0.011s	sys 0.000s
I20260808 01:50:45.394840 12767 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:45.315646  9638 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.008s	sys 0.013s
W20260808 01:50:45.347756 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:45.396042  9638 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.008s	sys 0.013s
W20260808 01:50:45.349611  9559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 1856) took 2365 ms (client timeout 1502 ms). Trace:
I20260808 01:50:45.354333 12692 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.003s
I20260808 01:50:45.397115  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 344) took 1476 ms. Trace:
W20260808 01:50:45.397469 12654 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260808 01:50:45.397867 12692 raft_consensus.cc:3055] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:45.398346 12692 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:45.397729  9638 rpcz_store.cc:276] 0808 01:50:43.921155 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:44.069724 (+148569us) service_pool.cc:224] Handling call
0808 01:50:44.119795 (+ 50071us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:44.464532 (+344737us) spinlock_profiling.cc:243] Waited 3.56 ms on lock 0x7b54003cd5a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490b1a0 00007f5d449955a0 00007f5d44996611 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c
0808 01:50:45.396616 (+932084us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:45.396664 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:45.396723 (+    59us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:45.396770 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:45.396883 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:45.397038 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":59475,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":46463,"spinlock_wait_cycles":8183040,"thread_start_us":667549,"threads_started":1}
W20260808 01:50:45.399226 12835 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.838s	user 0.013s	sys 0.000s
W20260808 01:50:45.398094  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:45.399525 12692 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:45.400601  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:45.400954 12692 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:45.356268  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:45.402643 12820 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:45.403232 12820 raft_consensus.cc:697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:45.390259 12796 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 01:50:45.394932  9551 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
W20260808 01:50:45.261698 12816 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.824s	user 0.009s	sys 0.005s
I20260808 01:50:45.404263 12820 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:45.264395 12889 log.cc:927] Time spent T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.383s	user 0.003s	sys 0.006s
I20260808 01:50:45.393265 12883 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:50:45.406996  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1873) took 1280 ms. Trace:
I20260808 01:50:45.407418  9632 rpcz_store.cc:276] 0808 01:50:44.127084 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:44.234298 (+107214us) service_pool.cc:224] Handling call
0808 01:50:44.234948 (+   650us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:44.235835 (+   887us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:44.235876 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:44.236683 (+   807us) log.cc:844] Serialized 52 byte log entry
0808 01:50:44.440982 (+204299us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:44.441048 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:44.441163 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:45.198105 (+756942us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:45.406618 (+208513us) raft_consensus.cc:1685] finished
0808 01:50:45.406643 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:45.406922 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":149,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":137,"spinlock_wait_cycles":408807168,"thread_start_us":203741,"threads_started":1,"wal-append.queue_time_us":482059}
W20260808 01:50:45.409247  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 53091us
W20260808 01:50:45.396885  9559 rpcz_store.cc:269] 0808 01:50:42.984230 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:43.028828 (+ 44598us) service_pool.cc:224] Handling call
0808 01:50:45.349378 (+2320550us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b5400258354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:45.349532 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3519626112}
I20260808 01:50:45.386263  9373 catalog_manager.cc:5697] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 3 to 5, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 5 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
W20260808 01:50:45.414237  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.831s	user 0.043s	sys 0.111s
W20260808 01:50:45.414656  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.832s	user 0.043s	sys 0.111s
W20260808 01:50:45.414999  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 909634us
I20260808 01:50:45.415715  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:45.418354  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:50:45.353555 12869 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.920s	user 0.011s	sys 0.002s
W20260808 01:50:45.429406 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:45.428131  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:45.414021  9373 catalog_manager.cc:5697] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 1 to 3, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 3 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
W20260808 01:50:45.433547  9476 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.005s	sys 0.015s
W20260808 01:50:45.434382  9476 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.005s	sys 0.016s
I20260808 01:50:45.429183  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:45.439781  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:45.439973 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:45.266040 12781 consensus_queue.cc:1059] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.091s
I20260808 01:50:45.353127 12916 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:45.306239 12678 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.120s
I20260808 01:50:45.355713 12738 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:45.353646 12866 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:45.469717  9478 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.015s	sys 0.023s
I20260808 01:50:45.469817  9632 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:45.470600  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:45.432369 12767 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:45.472425  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2210) took 1604 ms. Trace:
I20260808 01:50:45.458292  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:45.430423 12779 consensus_meta.cc:287] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.658s	user 0.013s	sys 0.001s
I20260808 01:50:45.460680  9633 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:45.300391 12910 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.131s
W20260808 01:50:45.464100  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:45.468344  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 390808us
I20260808 01:50:45.427429 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:45.470276  9478 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.015s	sys 0.023s
I20260808 01:50:45.479776  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:45.489331  9601 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:45.498536  9473 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:45.476526  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:45.515594  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45088.3us
I20260808 01:50:45.524847 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:45.600317  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:45.621913 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 194617us
W20260808 01:50:45.581187  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:45.516288  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:45.630131 10365 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:45.528945  9630 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 2. Current term is 3. Ops: []
I20260808 01:50:45.540391  9632 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260808 01:50:45.268028 12756 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:45.423947 12819 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.093s
I20260808 01:50:45.427346 12835 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:45.635988  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:45.638830  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:45.635054  9631 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:45.659628  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:45.660768  9631 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:45.462805 12854 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:45.439352 12869 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:45.704730  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 55654.9us
I20260808 01:50:45.526921 12571 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:45.705986  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:45.649806  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:45.665555  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:45.632926  9600 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.024s	sys 0.006s
I20260808 01:50:45.646845  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:45.708894  9600 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.725s	user 0.024s	sys 0.006s
W20260808 01:50:45.643471 12788 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.928s	user 0.028s	sys 0.036s
W20260808 01:50:45.650583 12842 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.687s	user 0.021s	sys 0.047s
I20260808 01:50:45.708776  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:45.707960  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:45.470248 10362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 477) took 1058 ms. Trace:
I20260808 01:50:45.711352  9473 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:50:45.783488 12781 raft_consensus.cc:3055] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:45.791896 12787 raft_consensus.cc:3055] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:45.792130 12781 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:45.792235  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:45.793471  9473 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 8: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:45.793332 12781 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:45.794719  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:45.470804 12916 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:45.796187 12782 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:45.797557 12894 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.498s	user 0.000s	sys 0.001s
I20260808 01:50:45.798219 12916 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:45.690490  9521 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:45.715101  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:45.654536  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:45.782455  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:45.784147  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:45.642791 12883 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:45.784685  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:45.829185  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 138842us
W20260808 01:50:45.829493  9445 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.012s	sys 0.036s
W20260808 01:50:45.829836  9445 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.012s	sys 0.036s
W20260808 01:50:45.742465 12842 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.784s	user 0.022s	sys 0.047s
I20260808 01:50:45.515931 12914 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:45.849311  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:45.789605  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:45.789773  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:45.798882 10368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 475) took 2348 ms (client timeout 2630 ms). Trace:
I20260808 01:50:45.799631 12913 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:45.632601 12915 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:45.800766 12854 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:45.796691 10362 rpcz_store.cc:276] 0808 01:50:44.409658 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:44.409905 (+   247us) service_pool.cc:224] Handling call
0808 01:50:44.410453 (+   548us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:44.410857 (+   404us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:50:44.749748 (+338891us) log.cc:844] Serialized 14 byte log entry
0808 01:50:45.313223 (+563475us) log.cc:844] Serialized 14 byte log entry
0808 01:50:45.355876 (+ 42653us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:45.417811 (+ 61935us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:45.425343 (+  7532us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:45.442205 (+ 16862us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:45.467484 (+ 25279us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":492994,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":98,"raft.queue_time_us":3412,"raft.run_cpu_time_us":427,"raft.run_wall_time_us":10060,"thread_start_us":770740,"threads_started":2,"wal-append.queue_time_us":701106}
W20260808 01:50:45.714651 12788 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.010s	user 0.028s	sys 0.037s
I20260808 01:50:45.803309 12884 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:45.805668 12900 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.747s	user 0.015s	sys 0.011s
I20260808 01:50:45.806288  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:45.807166 12898 log.cc:927] Time spent T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.736s	user 0.002s	sys 0.006s
I20260808 01:50:45.812075  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:45.645381 12779 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:45.812145  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:45.828513 12879 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:45.642314 12826 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:45.845052  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:45.787563 12767 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.103s
I20260808 01:50:45.854326 12804 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:45.846454  9552 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260808 01:50:45.891086  9552 pending_rounds.cc:85] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 3
I20260808 01:50:45.891451  9552 pending_rounds.cc:107] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260808 01:50:45.854516 12914 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:45.847517  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 388) took 1494 ms. Trace:
W20260808 01:50:45.850612  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:45.850587  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:45.664644 12906 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.176s
I20260808 01:50:45.792448 12787 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:45.858202 12915 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:45.902541  9632 raft_consensus.cc:2368] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8.
I20260808 01:50:45.676184 12864 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:45.671098 12831 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:45.843266 12818 consensus_meta.cc:287] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.302s	user 0.022s	sys 0.001s
I20260808 01:50:45.859181 10334 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:45.516245 12724 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:45.715875 12866 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.138s
I20260808 01:50:45.694476 12807 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:45.851235 12916 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260808 01:50:45.854952 12676 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.015s	sys 0.043s
I20260808 01:50:45.666616  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:46.025316  9630 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 4: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:45.888195 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:45.889830 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:45.891739  9552 raft_consensus.cc:2889] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:46.026361  9552 pending_rounds.cc:107] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260808 01:50:45.846572  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:45.877314 12901 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.990s	user 0.000s	sys 0.004s
I20260808 01:50:45.893345  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:45.904354 12879 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:45.894567  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:45.896396  9556 rpcz_store.cc:276] 0808 01:50:44.353650 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:44.377219 (+ 23569us) service_pool.cc:224] Handling call
0808 01:50:44.377679 (+   460us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:44.377965 (+   286us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:50:44.377992 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:44.415963 (+ 37971us) log.cc:844] Serialized 26 byte log entry
0808 01:50:44.782035 (+366072us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:44.782085 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:44.782170 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:45.796868 (+1014698us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:45.798928 (+  2060us) raft_consensus.cc:1685] finished
0808 01:50:45.798960 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:45.847403 (+ 48443us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25612672,"thread_start_us":336981,"threads_started":1,"wal-append.queue_time_us":585667}
I20260808 01:50:45.898888  9637 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:46.035699  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:45.897022  9557 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
W20260808 01:50:45.899854 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:45.857278 12788 raft_consensus.cc:697] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:50:45.884083 12844 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.003s	user 0.016s	sys 0.000s
I20260808 01:50:46.044179 12869 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:46.045444 12869 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:46.045993 12869 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:45.850577 10368 rpcz_store.cc:269] 0808 01:50:43.450503 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:43.450887 (+   384us) service_pool.cc:224] Handling call
0808 01:50:45.798604 (+2347717us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b5400681d54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:45.798758 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3330133888}
I20260808 01:50:45.912604  9633 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:45.856447 12678 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.048s
I20260808 01:50:45.845599 12908 consensus_queue.cc:1059] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.105s
W20260808 01:50:45.956739 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:46.050004  9633 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260808 01:50:46.012429 12628 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:45.994421  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:45.933594  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:45.891582 12726 consensus_meta.cc:287] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.908s	user 0.008s	sys 0.005s
I20260808 01:50:45.890851 12804 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:46.030321 12676 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.015s	sys 0.044s
I20260808 01:50:45.872223 12854 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:45.898319 12746 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.023s	sys 0.037s
I20260808 01:50:45.901664 12914 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:45.998420 12900 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260808 01:50:45.851377 12883 raft_consensus.cc:697] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:45.934175 12842 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:46.039289  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:45.934635 12884 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:50:46.032660 12920 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.375s	user 0.000s	sys 0.001s
W20260808 01:50:46.041958 12800 consensus_meta.cc:287] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.492s	user 0.018s	sys 0.000s
I20260808 01:50:46.026731 12807 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:45.885803 12656 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.048s	sys 0.110s
I20260808 01:50:45.915953 12915 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:45.960807 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 101734us
I20260808 01:50:46.017577 12831 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:46.051515 12724 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:46.032416 12787 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.016010 12818 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.135641 12818 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:45.898775 12826 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.138857 12826 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:46.139675 12800 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:46.139840 12826 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:50:46.075212 12804 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:46.140576 12800 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.144692 12800 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:46.056757 12628 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:46.146203 12628 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:46.094551 12746 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.749s	user 0.029s	sys 0.046s
I20260808 01:50:46.017127 12864 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:46.152451 12864 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:46.159653 12779 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:46.162451 12854 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:46.180173 12869 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:46.182291 12920 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
W20260808 01:50:46.185448 12867 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.821s	user 0.007s	sys 0.006s
I20260808 01:50:46.075474 12726 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:46.189231 12726 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:46.193302 12692 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.792s	user 0.009s	sys 0.004s
W20260808 01:50:46.003196 12498 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.035s	sys 0.041s
W20260808 01:50:46.209610 12498 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.820s	user 0.035s	sys 0.041s
W20260808 01:50:46.117308 12656 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.851s	user 0.049s	sys 0.111s
I20260808 01:50:46.109481 12842 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:46.119673 12883 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.117540 12926 raft_consensus.cc:3055] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:46.215828 12926 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:46.216269 12842 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:46.216967 12926 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:46.122118 12831 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:46.122272 12922 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:46.122711 12724 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:46.124135 12807 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:46.111943 12884 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.218461 12926 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:46.223196 12884 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:46.136029 12915 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.223963 12879 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:46.225929 12915 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:46.107577 12477 consensus_meta.cc:287] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.322s	user 0.012s	sys 0.005s
I20260808 01:50:46.112833 12835 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:46.131716 12788 consensus_queue.cc:237] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.230700 12724 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:46.232025 12477 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:46.097981 12820 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.684s	user 0.006s	sys 0.017s
W20260808 01:50:46.219542 12571 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.502s	user 0.013s	sys 0.000s
I20260808 01:50:46.249549 12571 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:46.248013 12628 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:45.788673  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 82772.3us
W20260808 01:50:45.914608  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:46.331208 12914 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:46.332566  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:46.333225  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:45.992105  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:46.027347  9552 raft_consensus.cc:2889] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:46.040619  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:46.051769  9551 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:46.298403  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2209) took 2430 ms. Trace:
I20260808 01:50:46.334448 12914 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:46.056946  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 567697us
I20260808 01:50:46.056972  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:46.305382  9600 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.005s	sys 0.048s
I20260808 01:50:46.046420  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:46.371217 12895 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.426s	user 0.004s	sys 0.000s
I20260808 01:50:46.371409  9634 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:46.308873  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:46.375038 12909 log.cc:927] Time spent T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 1.298s	user 0.008s	sys 0.015s
W20260808 01:50:46.052991  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 119563us
I20260808 01:50:46.380538  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 420) took 1944 ms. Trace:
I20260808 01:50:46.057487  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:46.314302 12804 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:46.384476  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 396) took 1023 ms. Trace:
I20260808 01:50:46.385100  9550 rpcz_store.cc:276] 0808 01:50:45.360932 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:45.361138 (+   206us) service_pool.cc:224] Handling call
0808 01:50:45.361543 (+   405us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:45.414560 (+ 53017us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:45.414593 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:45.414640 (+    47us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:50:45.414700 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:46.383737 (+969037us) spinlock_profiling.cc:243] Waited 52.8 ms on lock 0x7b5400217ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:46.383833 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:46.384225 (+   392us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":121351040}
I20260808 01:50:46.316370  9635 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:46.386366 12778 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:46.317880  9637 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
W20260808 01:50:46.320182 12921 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.379s	user 0.003s	sys 0.000s
I20260808 01:50:46.320554 12477 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:46.320816 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:46.252104 12820 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.840s	user 0.007s	sys 0.017s
W20260808 01:50:46.388616 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:46.330274 12628 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:46.303411  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 472859us
I20260808 01:50:46.391135  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1991) took 2272 ms. Trace:
I20260808 01:50:46.342216  9556 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 8: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:46.393477  9600 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.006s	sys 0.049s
I20260808 01:50:46.391760  9478 rpcz_store.cc:276] 0808 01:50:44.120204 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:44.866843 (+746639us) service_pool.cc:224] Handling call
0808 01:50:44.867311 (+   468us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:44.899984 (+ 32673us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:44.941068 (+ 41084us) log.cc:844] Serialized 14 byte log entry
0808 01:50:45.624429 (+683361us) log.cc:844] Serialized 14 byte log entry
0808 01:50:45.636083 (+ 11654us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:45.656010 (+ 19927us) log.cc:844] Serialized 26 byte log entry
0808 01:50:45.656279 (+   269us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:45.656332 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:45.656453 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:46.313609 (+657156us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:46.390555 (+ 76946us) raft_consensus.cc:1685] finished
0808 01:50:46.390594 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:46.390898 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":154798,"prepare.queue_time_us":45556,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":464851,"raft.run_cpu_time_us":375,"raft.run_wall_time_us":31252,"spinlock_wait_cycles":73690752,"thread_start_us":682490,"threads_started":1,"wal-append.queue_time_us":931460}
I20260808 01:50:46.395134  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:46.397459  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:46.344234 12884 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:46.399556 12884 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.400811  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:46.402427  9550 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:46.407621  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:46.352969 12926 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:46.409317 12926 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:46.354215 12571 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:46.340127  9479 rpcz_store.cc:276] 0808 01:50:43.868138 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:44.461344 (+593206us) service_pool.cc:224] Handling call
0808 01:50:44.461631 (+   287us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:44.462324 (+   693us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:44.462350 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:44.462900 (+   550us) log.cc:844] Serialized 26 byte log entry
0808 01:50:44.718903 (+256003us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:44.718970 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:44.719044 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:45.811265 (+1092221us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:46.297377 (+486112us) raft_consensus.cc:1685] finished
0808 01:50:46.297464 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:46.298022 (+   558us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":92,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":161,"spinlock_wait_cycles":382076544,"thread_start_us":255582,"threads_started":1,"wal-append.queue_time_us":402585}
I20260808 01:50:46.411729 12571 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.412300  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:46.398204  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:46.414559  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:46.415208  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:46.415920  9555 raft_consensus.cc:2368] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9.
I20260808 01:50:46.416146  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:46.417367  9478 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:46.417183 12926 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:46.418654 12571 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:46.419826  9636 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:46.414475  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:46.421416  9479 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:46.424897  9630 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:46.417117 12804 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:46.381328  9481 rpcz_store.cc:276] 0808 01:50:44.436018 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:45.343063 (+907045us) service_pool.cc:224] Handling call
0808 01:50:45.343473 (+   410us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:45.367233 (+ 23760us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:45.367262 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:45.367306 (+    44us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:50:45.367346 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:46.379773 (+1012427us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b540009a6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:46.379862 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:46.380252 (+   390us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":54069504}
W20260808 01:50:46.407966  9558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 393) took 1736 ms (client timeout 2074 ms). Trace:
I20260808 01:50:46.344808  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:46.398535  9556 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:46.441504  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 59338.1us
W20260808 01:50:46.432829  9558 rpcz_store.cc:269] 0808 01:50:44.671215 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:44.742141 (+ 70926us) service_pool.cc:224] Handling call
0808 01:50:46.407483 (+1665342us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54000fe954. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:46.407754 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":423692032}
W20260808 01:50:46.453155 12897 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.437s	user 0.013s	sys 0.000s
I20260808 01:50:46.453624  9599 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:46.453620  9472 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:46.460779 12881 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:46.456012  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:46.462105  9550 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:46.442260 12884 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:46.467815  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:46.473711  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:46.474337 12941 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:46.475611 12941 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.479825  9558 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260808 01:50:46.482883  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:46.483548  9552 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
W20260808 01:50:46.484489  9521 leader_election.cc:336] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.536s (SENT)
I20260808 01:50:46.496445  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:46.496759  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:46.484593  9444 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:46.486512 12914 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:46.491237  9555 raft_consensus.cc:2410] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260808 01:50:46.492169  9630 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:45.518224  9476 rpcz_store.cc:276] 0808 01:50:43.868470 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:44.462192 (+593722us) service_pool.cc:224] Handling call
0808 01:50:44.462553 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:44.462818 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:44.775160 (+312342us) log.cc:844] Serialized 14 byte log entry
0808 01:50:45.446242 (+671082us) log.cc:844] Serialized 14 byte log entry
0808 01:50:45.472001 (+ 25759us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:45.472071 (+    70us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:45.472139 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:45.472238 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:45.472364 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25361,"prepare.queue_time_us":4320,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":112,"raft.queue_time_us":25509,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":81116,"thread_start_us":971067,"threads_started":2}
I20260808 01:50:46.496505  9550 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:46.475759  9599 leader_election.cc:400] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9.
I20260808 01:50:46.530731  9599 leader_election.cc:403] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:50:46.529458  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:46.543119 12775 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.724s	user 0.014s	sys 0.044s
W20260808 01:50:46.543604 12775 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.808s	user 0.014s	sys 0.044s
W20260808 01:50:46.545841  9598 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.011s	sys 0.020s
W20260808 01:50:46.546362  9598 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.012s	sys 0.020s
W20260808 01:50:46.546765 12655 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.695s	user 0.019s	sys 0.041s
W20260808 01:50:46.547151 12655 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.790s	user 0.019s	sys 0.041s
I20260808 01:50:46.475652 12676 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:46.542564  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:46.490654 12923 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.623s	user 0.001s	sys 0.001s
W20260808 01:50:46.554770  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:45.316216  9475 rpcz_store.cc:276] 0808 01:50:43.867794 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:44.461295 (+593501us) service_pool.cc:224] Handling call
0808 01:50:44.461594 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:44.461802 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 5
0808 01:50:44.462161 (+   359us) log.cc:844] Serialized 14 byte log entry
0808 01:50:44.771353 (+309192us) log.cc:844] Serialized 14 byte log entry
0808 01:50:45.026591 (+255238us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:45.026659 (+    68us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:45.026732 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:45.026807 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:45.026951 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4838,"prepare.queue_time_us":4138,"prepare.run_cpu_time_us":146,"prepare.run_wall_time_us":6537,"raft.queue_time_us":546948,"thread_start_us":556339,"threads_started":2,"wal-append.queue_time_us":524112}
W20260808 01:50:46.556099  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:46.556658  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:46.559336  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1879) took 1081 ms. Trace:
I20260808 01:50:46.559834  9635 rpcz_store.cc:276] 0808 01:50:45.477735 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:45.524290 (+ 46555us) service_pool.cc:224] Handling call
0808 01:50:46.559027 (+1034737us) spinlock_profiling.cc:243] Waited 609 ms on lock 0x7b540003b1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:46.559220 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1400220416}
I20260808 01:50:46.484613 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 601) took 1958 ms. Trace:
I20260808 01:50:46.562995 10361 rpcz_store.cc:276] 0808 01:50:44.525479 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:44.548791 (+ 23312us) service_pool.cc:224] Handling call
0808 01:50:44.549319 (+   528us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:44.659702 (+110383us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:44.672381 (+ 12679us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:44.692631 (+ 20250us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:44.703791 (+ 11160us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:46.483923 (+1780132us) spinlock_profiling.cc:243] Waited 96 ms on lock 0x7b54004c3454. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:46.484025 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:46.484304 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":220869888}
I20260808 01:50:46.564158 12820 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:46.565644 12571 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:46.568774  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1883) took 1087 ms. Trace:
I20260808 01:50:46.569388  9557 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:50:46.569299  9637 rpcz_store.cc:276] 0808 01:50:45.481020 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:46.035284 (+554264us) service_pool.cc:224] Handling call
0808 01:50:46.568394 (+533110us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b540008cd54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:46.568669 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":645575552}
I20260808 01:50:46.571379  9633 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:46.568573  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:46.541175 12914 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:50:46.559633  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 75284.4us
W20260808 01:50:46.581774 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:46.582527  9551 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:46.582580  9559 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:46.583179  9630 raft_consensus.cc:2393] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:46.591418  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:46.551613 12676 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:46.593989  9479 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:46.557292 12917 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260808 01:50:46.601148  9634 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:46.603809 12477 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:46.607302 12812 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:46.608246 12812 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:46.610577  9636 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: [4.3-7.4]
I20260808 01:50:46.611824 12953 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:46.612800 12953 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.614522 12881 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:46.616518 12881 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:46.616906 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:46.617909 12941 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:46.618741  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 134421us
I20260808 01:50:46.624271 12745 consensus_queue.cc:1059] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:50:46.619155 12420 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:46.626530 12800 raft_consensus.cc:3055] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:46.627139 12800 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:46.626844 12420 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:46.628692 10361 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:46.628780 12800 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.631369 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:46.630244 12420 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:46.620909 12864 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:46.634960 12864 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:46.636431  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:46.636754 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:46.639528 12770 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:50:46.640628 12881 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:46.641017 12864 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:46.639587 12914 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:46.643067  9475 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:46.644600  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:46.646047  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:46.639640 12954 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 9
W20260808 01:50:46.621039  9600 connection.cc:256] RPC call timeout handler was delayed by -1.2051669290231075s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19230000000000003s for a response
I20260808 01:50:46.621912 12953 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:46.656277 12953 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:46.657224 12953 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.657949  9475 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 8: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:46.659132  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:46.666414 12745 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:46.667567 12745 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.622237  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 421) took 2185 ms. Trace:
I20260808 01:50:46.622555 12812 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:46.623276 12628 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 01:50:46.624464 12904 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 1.549s	user 0.007s	sys 0.019s
I20260808 01:50:46.671938  9521 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:46.623904 12676 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:46.673136  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2212) took 2803 ms. Trace:
W20260808 01:50:46.671793  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 460e1059f5504707b891c4b0e0ac6a09: Illegal state: Replica 52ccf4dcca1843f7bb29e9172e2d4ab4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:50:46.673733 12676 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:46.620276 12654 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:46.675860 12654 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.673725  9477 rpcz_store.cc:276] 0808 01:50:43.869149 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:44.463339 (+594190us) service_pool.cc:224] Handling call
0808 01:50:44.463735 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:44.464222 (+   487us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:44.464250 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:44.464834 (+   584us) log.cc:844] Serialized 26 byte log entry
0808 01:50:44.764578 (+299744us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:44.764631 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:44.764705 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:46.623251 (+1858546us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:46.672445 (+ 49194us) raft_consensus.cc:1685] finished
0808 01:50:46.672475 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:46.672839 (+   364us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":112,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"spinlock_wait_cycles":51544832,"thread_start_us":299196,"threads_started":1,"wal-append.queue_time_us":609877}
I20260808 01:50:46.625769  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:46.681318  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 55851.1us
I20260808 01:50:46.681715 12775 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:46.683475 12955 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:50:46.684299 12775 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:46.684839 12775 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:46.686813  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:46.688508  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:46.689632 12738 consensus_queue.cc:1059] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:50:46.691043 12654 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:46.691283 12738 consensus_queue.cc:1243] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 7201605bb2f54c949a4ebe6e9b3b4b2a log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260808 01:50:46.691872 12770 raft_consensus.cc:3055] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:46.692127  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:46.692313 12809 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:46.692334 12770 raft_consensus.cc:740] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:46.693255  9477 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:46.694422 12770 consensus_queue.cc:260] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.694398  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:46.696326 12770 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:46.696489  9479 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:46.663292  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:46.699584  9475 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:46.701057  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:46.694996  9524 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:46.687103  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:46.704257  9556 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
W20260808 01:50:46.678826  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 204432us
I20260808 01:50:46.726727  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:46.701422  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:46.669778  9472 rpcz_store.cc:276] 0808 01:50:44.436392 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:45.344989 (+908597us) service_pool.cc:224] Handling call
0808 01:50:45.345460 (+   471us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:46.460800 (+1115340us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:46.621885 (+161085us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54002bc0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:46.622156 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2547046784}
I20260808 01:50:46.730870  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1889) took 1247 ms. Trace:
I20260808 01:50:46.731474  9634 rpcz_store.cc:276] 0808 01:50:45.483072 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:46.046123 (+563051us) service_pool.cc:224] Handling call
0808 01:50:46.726624 (+680501us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:46.732888  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:46.702049 12923 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260808 01:50:46.730103  9481 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:46.729822 12942 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:46.723340 12804 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:46.740275  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:46.740867 12804 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:46.743099  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:46.745644 12907 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 1.525s	user 0.005s	sys 0.008s
I20260808 01:50:46.746507 12781 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:46.750329  9550 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:46.751420  9477 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:50:46.751904  9476 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:46.753576  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:46.756901  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 134428us
I20260808 01:50:46.760825 12955 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:46.761248  9477 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:46.762857  9475 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:50:46.757125 10332 leader_election.cc:336] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.096s (SENT)
I20260808 01:50:46.758862 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:46.767423  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 80554.9us
I20260808 01:50:46.768097  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 424) took 2190 ms. Trace:
I20260808 01:50:46.775355  9477 rpcz_store.cc:276] 0808 01:50:44.573734 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:46.752207 (+2178473us) service_pool.cc:224] Handling call
0808 01:50:46.761000 (+  8793us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:46.763072 (+  2072us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:46.763587 (+   515us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:46.769260  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 97523.5us
I20260808 01:50:46.756606 12745 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:46.772341 12937 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:46.772473 12940 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:46.773123  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 78338.3us
I20260808 01:50:46.783298 12809 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:50:46.783458 12420 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:50:46.787397  9521 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:46.793040  9481 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:50:46.795241  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 426) took 1788 ms. Trace:
I20260808 01:50:46.797978  9472 rpcz_store.cc:276] 0808 01:50:45.005213 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:46.770271 (+1765058us) service_pool.cc:224] Handling call
0808 01:50:46.770741 (+   470us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:46.792714 (+ 21973us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:46.792765 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:46.792832 (+    67us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:46.792871 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:46.792943 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:46.793116 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:46.789633 12942 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:46.787861 12955 raft_consensus.cc:697] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:46.800683  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:46.802879 12953 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:46.801011 12809 raft_consensus.cc:697] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:46.802140 12940 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:46.816545 12809 consensus_queue.cc:237] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:46.818061 12940 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:46.807253  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:46.819023 12942 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.822167  9521 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:46.822849 12857 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:46.823913 12857 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:46.825855  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:46.811880 12812 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:46.827293 12812 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:46.829938 12954 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9.
I20260808 01:50:46.830077  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:46.830896  9634 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:46.831014 12778 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:46.839597  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:46.840067  9630 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260808 01:50:46.841852  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:46.842499 12778 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:50:46.842942 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 92764.1us
I20260808 01:50:46.819500 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:46.802327 12937 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:46.843636  9636 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:46.843223  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:46.846036 10334 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:46.849354  9635 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:50:46.851939  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 54456.3us
W20260808 01:50:46.855166 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 102312us
I20260808 01:50:46.860370  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2224) took 1532 ms. Trace:
I20260808 01:50:46.860844  9472 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260808 01:50:46.860962  9481 rpcz_store.cc:276] 0808 01:50:45.328378 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:46.803202 (+1474824us) service_pool.cc:224] Handling call
0808 01:50:46.813409 (+ 10207us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:46.848542 (+ 35133us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:46.848582 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:46.853753 (+  5171us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:46.860026 (+  6273us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:46.860109 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:46.860269 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:46.856221  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:46.822350 12942 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:46.864600  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2225) took 1534 ms. Trace:
I20260808 01:50:46.865228  9472 rpcz_store.cc:276] 0808 01:50:45.330381 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:46.807548 (+1477167us) service_pool.cc:224] Handling call
0808 01:50:46.860412 (+ 52864us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:46.861272 (+   860us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:46.864488 (+  3216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:46.845257 12812 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:50:46.868698  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 59020.5us
I20260808 01:50:46.871584  9637 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260808 01:50:46.871886 12884 consensus_queue.cc:1059] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:50:46.872483  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 428) took 1519 ms. Trace:
I20260808 01:50:46.872901  9481 rpcz_store.cc:276] 0808 01:50:45.352909 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:46.862037 (+1509128us) service_pool.cc:224] Handling call
0808 01:50:46.871781 (+  9744us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:46.872143 (+   362us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:46.872169 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:46.872212 (+    43us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:46.872240 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:46.872298 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:46.872420 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:46.873962  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:46.875036 10365 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:46.877799  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 431) took 1126 ms. Trace:
I20260808 01:50:46.883872  9472 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260808 01:50:46.887522  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:46.848541 12745 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:46.889045 12934 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:46.891134 12940 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:46.892099  9521 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:46.892710  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:46.893081 12804 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:46.894930 12905 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.716s	user 0.003s	sys 0.008s
I20260808 01:50:46.897363 12800 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:46.898977  9524 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:46.899893  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1994) took 2777 ms. Trace:
I20260808 01:50:46.900919 12420 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:46.903461 12955 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.911734  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 1987) took 2794 ms. Trace:
I20260808 01:50:46.862123 12926 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:46.917725  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:46.919126  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:46.924392  9478 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260808 01:50:46.966189  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 74170.5us
I20260808 01:50:46.925802  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 422) took 2371 ms. Trace:
I20260808 01:50:46.926712  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:46.929311  9481 rpcz_store.cc:276] 0808 01:50:45.751312 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:46.874007 (+1122695us) service_pool.cc:224] Handling call
0808 01:50:46.874403 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:46.877371 (+  2968us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:46.877404 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:46.877475 (+    71us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:46.877510 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:46.877579 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:46.877713 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2383,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":146}
I20260808 01:50:46.942814  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:46.944990  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:46.950812  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2229) took 1545 ms. Trace:
I20260808 01:50:46.905855 12937 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:46.955433  9599 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:46.969072  9473 rpcz_store.cc:276] 0808 01:50:44.554423 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:45.795846 (+1241423us) service_pool.cc:224] Handling call
0808 01:50:45.796165 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:45.853819 (+ 57654us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:45.853867 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:45.853915 (+    48us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:50:45.853950 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:46.923844 (+1069894us) spinlock_profiling.cc:243] Waited 57.4 ms on lock 0x7b5400399754. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:46.925360 (+  1516us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:46.925606 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":131957120}
I20260808 01:50:46.974154  9635 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:50:46.977392  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2003) took 1146 ms. Trace:
I20260808 01:50:46.977950  9478 rpcz_store.cc:276] 0808 01:50:45.831392 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:46.878024 (+1046632us) service_pool.cc:224] Handling call
0808 01:50:46.878635 (+   611us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:46.968568 (+ 89933us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:46.977303 (+  8735us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:46.981521  9557 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:46.946717 12940 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:46.957719  9474 rpcz_store.cc:276] 0808 01:50:44.122770 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:44.907089 (+784319us) service_pool.cc:224] Handling call
0808 01:50:45.209266 (+302177us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:45.373849 (+164583us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:46.899371 (+1525522us) spinlock_profiling.cc:243] Waited 302 ms on lock 0x7b54002bb6d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:46.899701 (+   330us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":693726080}
I20260808 01:50:46.959414  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:46.961879  9480 rpcz_store.cc:276] 0808 01:50:44.117367 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:44.509509 (+392142us) service_pool.cc:224] Handling call
0808 01:50:44.509893 (+   384us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:44.510181 (+   288us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:44.510221 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:44.510791 (+   570us) log.cc:844] Serialized 26 byte log entry
0808 01:50:44.902309 (+391518us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:44.902385 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:44.902517 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:46.744699 (+1842182us) spinlock_profiling.cc:243] Waited 22.8 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:46.911168 (+166469us) raft_consensus.cc:1685] finished
0808 01:50:46.911195 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:46.911543 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":52417408,"thread_start_us":383187,"threads_started":1,"wal-append.queue_time_us":565619}
I20260808 01:50:46.963096 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:46.991217 10368 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:46.963775 12826 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:46.963801 12954 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:46.964620  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:46.965407 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:46.971446  9472 rpcz_store.cc:276] 0808 01:50:45.396783 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:46.866385 (+1469602us) service_pool.cc:224] Handling call
0808 01:50:46.883510 (+ 17125us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:46.941313 (+ 57803us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:46.941692 (+   379us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:46.866346 12775 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:46.984618  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 130085us
I20260808 01:50:46.923252 12778 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:46.997390  9551 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:46.958464  9634 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:46.960956 12804 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:47.000396  9524 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:46.975468 12942 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:47.006419  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:47.006817  9556 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 9: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:47.006946  9636 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 8: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:46.944023  9630 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:46.992969  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:46.988747 12571 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:47.010231  9557 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260808 01:50:47.011785 10365 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:47.011406 12954 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:46.992458 12940 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:47.013307  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:47.014191  9550 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260808 01:50:47.014839 12826 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:47.015863 10361 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:47.052263 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:47.054353 10361 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:50:47.018575  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:46.994230 12745 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:47.019505  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:47.020143  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 75914.1us
I20260808 01:50:47.020334 12778 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:47.020790 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:47.077713  9481 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 01:50:47.018601  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:47.022104  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:47.022753 12918 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:47.025447 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:46.985502 12934 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:47.026705 12914 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:47.026888 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:47.028074 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:47.028770 12804 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:47.029495  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:47.030364 12899 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.963s	user 0.003s	sys 0.007s
I20260808 01:50:47.000000  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2345) took 1178 ms. Trace:
I20260808 01:50:47.035897  9521 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:47.080866  9523 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:47.036567 12942 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:46.925895  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:50:47.040061  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:47.049942 12819 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:47.046720 10368 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:50:47.046072  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 423) took 2485 ms. Trace:
I20260808 01:50:47.048566  9551 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260808 01:50:47.046897  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:47.049889 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:47.056262  9631 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:47.056728 12826 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:47.088706  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:47.086089  9636 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 8: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:47.086098  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:47.057524  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:47.043366 12420 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:47.060213 12954 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:47.010337 12926 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:47.017730 12775 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:47.061498  9477 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:47.084998 12571 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:47.062021  9632 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
W20260808 01:50:47.068166 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 197247us
I20260808 01:50:47.098897  9556 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:47.080678 10365 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:50:47.084417  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:47.099418 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:47.082011 12804 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:47.088141  9634 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:47.103478  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:47.085667  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:47.088629  9559 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:47.087777 10367 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:47.088186  9479 rpcz_store.cc:276] 0808 01:50:44.554840 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:46.741031 (+2186191us) service_pool.cc:224] Handling call
0808 01:50:46.741592 (+   561us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:46.743992 (+  2400us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:50:46.762094 (+ 18102us) log.cc:844] Serialized 14 byte log entry
0808 01:50:47.003149 (+241055us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:47.003212 (+    63us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:47.039817 (+ 36605us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:47.039882 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:47.039978 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":17552,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":71,"raft.queue_time_us":4603,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":4701,"spinlock_wait_cycles":7040,"thread_start_us":196494,"threads_started":1,"wal-append.queue_time_us":224716}
I20260808 01:50:47.080727 12778 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:47.055537  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:47.116434  9523 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:47.096884 10366 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:47.107887  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:47.109136  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:47.110311  9637 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260808 01:50:47.118206  9551 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:47.110532 12826 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:47.111016  9635 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:47.116124  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:47.109242  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:47.112614  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 391) took 2655 ms. Trace:
I20260808 01:50:47.119273  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:47.094528 12864 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:47.114110 12788 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:47.115141  9599 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:50:47.115141  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:47.082517  9558 rpcz_store.cc:276] 0808 01:50:45.816888 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:46.481606 (+664718us) service_pool.cc:224] Handling call
0808 01:50:46.482289 (+   683us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:46.581510 (+ 99221us) raft_consensus.cc:1534] Early marking committed up to index 4
0808 01:50:46.750267 (+168757us) log.cc:844] Serialized 14 byte log entry
0808 01:50:46.995000 (+244733us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:46.995051 (+    51us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:46.995101 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:46.995176 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:46.995293 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29146,"prepare.queue_time_us":218195,"prepare.run_cpu_time_us":118,"prepare.run_wall_time_us":1987,"raft.queue_time_us":35826,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":4799,"thread_start_us":348667,"threads_started":2,"wal-append.queue_time_us":324858}
I20260808 01:50:47.113317  9552 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:47.126940  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:47.085844 12819 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:47.125011  9634 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260808 01:50:47.126081  9637 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260808 01:50:47.126281  9472 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:47.128615  9556 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:50:47.106431 12804 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:47.119676  9554 rpcz_store.cc:276] 0808 01:50:44.456641 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:44.457899 (+  1258us) service_pool.cc:224] Handling call
0808 01:50:44.458280 (+   381us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:44.458514 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:44.458690 (+   176us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:44.460008 (+  1318us) log.cc:844] Serialized 26 byte log entry
0808 01:50:44.797501 (+337493us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:44.797544 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:44.797640 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:47.029469 (+2231829us) spinlock_profiling.cc:243] Waited 22.9 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:47.112051 (+ 82582us) raft_consensus.cc:1685] finished
0808 01:50:47.112078 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:47.112396 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":52745088,"thread_start_us":337050,"threads_started":1,"wal-append.queue_time_us":533997}
I20260808 01:50:47.125771  9636 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:50:47.134490  9555 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:47.136515  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:47.138082  9630 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:47.139017  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:47.087864 12914 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:47.140136  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:47.137753  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:47.124341 12826 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:47.148382  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:47.131583 12819 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:47.144222 10332 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:47.150234  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:47.151291  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:47.153339  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 117610us
I20260808 01:50:47.154959  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:47.156790  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:47.142982 12934 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:47.159137 12812 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:47.162420  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:47.151080 12778 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:47.165087 12812 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:47.165334 12778 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:47.166591 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:47.167676 10361 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:47.159654 12781 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:47.173758 12914 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:47.173815 12781 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:47.175530 12781 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:47.181468 12812 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:47.166975 12745 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:47.182555 12812 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:47.139279 12809 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:50:47.186349 12809 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:47.187700 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:47.188190 12856 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:50:47.188687 12856 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:47.189252 12812 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:47.189628 12856 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:47.182652  9521 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:47.191134 12856 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:47.191744 12942 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:47.192444 12942 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260808 01:50:47.198321  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 205493us
I20260808 01:50:47.192438  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1897) took 1706 ms. Trace:
I20260808 01:50:47.199863  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:47.201800  9551 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 8: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:47.202607  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:47.194082 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:47.200095  9633 rpcz_store.cc:276] 0808 01:50:45.485723 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:46.051333 (+565610us) service_pool.cc:224] Handling call
0808 01:50:46.570950 (+519617us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:46.571986 (+  1036us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:47.192188 (+620202us) spinlock_profiling.cc:243] Waited 474 ms on lock 0x7b540008cd50. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:47.192334 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1089812224}
I20260808 01:50:47.144498 12826 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:47.207472 12914 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260808 01:50:47.208844 12809 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:47.210253  9632 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 7: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:47.206732  9479 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:47.211313  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:47.209321  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:47.207639 12812 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
W20260808 01:50:47.215692 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 71565.2us
I20260808 01:50:47.224485  9521 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:47.226902  9472 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:47.228312  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:47.228933  9472 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:47.230283  9633 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:50:47.234941  9481 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:47.237042  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:47.238518  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:47.240038  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:47.242388 10369 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:50:47.242825  9634 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:50:47.245244  9556 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:47.246745  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:47.238844 12942 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:47.214897 12745 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:47.249902  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:47.239136 12864 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260808 01:50:47.254071  9599 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:47.255312 12926 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:47.256686  9551 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260808 01:50:47.257007  9637 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:47.264137  9634 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260808 01:50:47.257218  9600 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:47.268731  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 57677.2us
I20260808 01:50:47.264431  9481 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:47.271859  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:47.275630  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:47.271893  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:47.139478 12788 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:47.276561  9632 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 7: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:50:47.278304  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 51750.9us
I20260808 01:50:47.276728  9637 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:47.282272 12733 raft_consensus.cc:3055] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:47.282831 12733 raft_consensus.cc:740] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:47.283891 12733 consensus_queue.cc:260] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:47.285260 12733 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:47.288142  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:47.288786 10332 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:47.291615 12420 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:47.268921 12914 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:47.298875  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:47.273447 12676 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260808 01:50:47.317224  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 43640.6us
I20260808 01:50:47.323814  9521 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:47.327132 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:47.327011 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:47.328663 12854 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:47.321486 12864 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:47.331441 12864 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:47.274849 12940 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:47.346284  9551 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:50:47.317965 12676 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:47.345186 12918 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election won for term 7
W20260808 01:50:47.370359 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 95771.8us
I20260808 01:50:47.384123 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:47.363267 12676 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:50:47.387672  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 51276.1us
I20260808 01:50:47.343572 12926 consensus_queue.cc:1059] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20260808 01:50:47.397927  9551 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:47.392774  9479 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:50:47.403760  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:47.354094 12937 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260808 01:50:47.348117 12973 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20260808 01:50:47.386348 12864 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:47.408885  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:47.387527 12854 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 01:50:47.414989  9479 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:47.396117 12918 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:47.395512 12942 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:47.420276  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 119814us
W20260808 01:50:47.421396 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 49589.7us
I20260808 01:50:47.390151 12819 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:47.424888 12914 raft_consensus.cc:3055] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:47.450654 10361 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:50:47.453401 12804 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:47.451601  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:47.460210  9559 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:47.433126 12778 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:47.429504 12971 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:47.461136  9556 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:47.464493  9550 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 9: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:47.466138  9523 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:47.466395 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:47.468885  9551 raft_consensus.cc:2376] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6. Re-sending same reply.
I20260808 01:50:47.470471  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:47.470599  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 104281us
I20260808 01:50:47.443321 12781 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:47.435402 12926 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:47.468062  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 79576.7us
I20260808 01:50:47.471434  9559 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:50:47.472252  9479 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:47.459897 12710 raft_consensus.cc:2764] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election decision vote started in defunct term 8: won
I20260808 01:50:47.475106 12857 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:50:47.477267  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:47.505389  9551 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:47.478436 12918 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:47.464943 12778 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:47.447553 12914 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:47.512487  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:47.467120 12971 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:47.503235  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:47.512717  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2349) took 1688 ms. Trace:
I20260808 01:50:47.523253  9552 rpcz_store.cc:276] 0808 01:50:45.824240 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:46.487500 (+663260us) service_pool.cc:224] Handling call
0808 01:50:47.113050 (+625550us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:47.127471 (+ 14421us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:47.512398 (+384927us) spinlock_profiling.cc:243] Waited 625 ms on lock 0x7b540002c1d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:47.512621 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1437385984}
I20260808 01:50:47.460376 12942 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:47.496261 12788 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:47.524793  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:47.526588 12973 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:47.521974 12857 raft_consensus.cc:697] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:47.544929  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:47.556798  9552 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:47.552188  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:47.542590  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:47.548569  9631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1893) took 2064 ms (client timeout 1953 ms). Trace:
I20260808 01:50:47.563591  9472 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
W20260808 01:50:47.562346  9631 rpcz_store.cc:269] 0808 01:50:45.484393 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:45.714643 (+230250us) service_pool.cc:224] Handling call
0808 01:50:47.548334 (+1833691us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54000db1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:47.548463 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2844812672}
I20260808 01:50:47.571707  9551 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:50:47.521965 12971 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:47.492069 12781 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:47.525410 12940 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:47.588826  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 76462.3us
I20260808 01:50:47.548904 12925 raft_consensus.cc:3055] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:47.594178  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:47.596194  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 90752.7us
W20260808 01:50:47.595692  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 156627us
I20260808 01:50:47.506013 12926 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:47.596727  9551 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:47.598248  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:47.584059 12785 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:47.582582 12864 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:47.592075 12925 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:47.601938 12816 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:47.602391  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:47.617966  9551 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:47.601765 12778 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:47.604072 12976 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260808 01:50:47.603895 12917 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:47.605768  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 427) took 2578 ms. Trace:
I20260808 01:50:47.622967  9477 rpcz_store.cc:276] 0808 01:50:45.027611 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:46.788670 (+1761059us) service_pool.cc:224] Handling call
0808 01:50:46.805280 (+ 16610us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:47.098212 (+292932us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:47.605461 (+507249us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b540019c3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:47.605670 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":588809088}
I20260808 01:50:47.606904 12955 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:47.608376 12693 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:47.609090 12942 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:47.613708  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:47.628403  9472 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:47.600569 12937 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:47.543339 12914 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:47.635417 12914 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:47.546396 12857 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:47.638625 12661 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Stepping down as leader of term 5
W20260808 01:50:47.641464 10332 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.991s (SENT) [suppressed 22 similar messages]
I20260808 01:50:47.642460 12661 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:47.642585  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:47.615556 12785 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:47.617246 12826 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:47.604017  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:47.644325 12819 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:47.650944 12917 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:47.644481 12955 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:47.651710 12917 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:47.652096 12781 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:47.652434 12917 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:47.632257  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:47.630476 12937 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:47.655550  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:47.656881 12937 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:47.658893 10369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 607) took 1068 ms. Trace:
W20260808 01:50:47.659358  9560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 1870) took 3127 ms (client timeout 3234 ms). Trace:
I20260808 01:50:47.660039 12778 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:47.659363 10369 rpcz_store.cc:276] 0808 01:50:46.625206 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:46.636207 (+ 11001us) service_pool.cc:224] Handling call
0808 01:50:47.658665 (+1022458us) spinlock_profiling.cc:243] Waited 530 ms on lock 0x7b5400646c54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:47.658802 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1218646784}
I20260808 01:50:47.664551  9523 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:47.659801  9560 rpcz_store.cc:269] 0808 01:50:44.531835 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:44.532106 (+   271us) service_pool.cc:224] Handling call
0808 01:50:47.658969 (+3126863us) spinlock_profiling.cc:243] Waited 812 ms on lock 0x7b54000de154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:47.659183 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1868261248}
I20260808 01:50:47.667106 12942 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:47.667708 12942 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:47.667889 12983 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:50:47.668118 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:47.668143 12942 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:47.672796 12854 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:50:47.651676 12955 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:47.645730  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:47.674585  9521 leader_election.cc:336] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.953s (SENT)
I20260808 01:50:47.633809  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:47.686312  9556 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 9: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:47.666385  9560 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:47.649201  9472 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
W20260808 01:50:47.650833  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 46936.7us
I20260808 01:50:47.688683  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:47.692453  9477 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:47.704439  9478 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: [4.3-8.5]
I20260808 01:50:47.648042 12785 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:50:47.709033  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 80684.7us
I20260808 01:50:47.650506 12826 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:47.737401  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 91751.2us
I20260808 01:50:47.732137 12973 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:47.745935  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:47.746877  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:47.747879  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:47.755440  9599 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:47.756791  9599 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:47.757689 12918 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:50:47.758255 12918 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:47.759050 12477 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:47.759569 12477 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:47.759253 12918 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:47.759938 12477 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:47.763805 12926 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:47.764122  9521 leader_election.cc:336] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.961s (SENT)
I20260808 01:50:47.764663  9521 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:47.767283 12955 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:47.767956  9521 leader_election.cc:336] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 3.046s (SENT)
I20260808 01:50:47.768924 12955 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:47.633127 12864 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:47.772974 12864 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:47.712635 12781 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:50:47.778422 12816 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:50:47.778800 12954 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:47.780297 12954 raft_consensus.cc:3055] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:47.780553  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:47.781154 12954 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:47.781843 12781 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:47.782127 12954 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:47.806980  9477 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:50:47.794366  9552 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:50:47.812363  9472 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 7. Current term is 8. Ops: []
W20260808 01:50:47.812372 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:47.810375  9601 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:47.912449  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 46987.2us
W20260808 01:50:47.918792  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 246822us
W20260808 01:50:47.926617  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:47.930577  9477 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:50:47.939915  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:47.942744  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 132459us
I20260808 01:50:47.896266 12864 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260808 01:50:47.981665  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 288605us
I20260808 01:50:47.983048  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:47.993665  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66295.2us
I20260808 01:50:47.962188 12420 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:50:47.917644 12937 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.083s
I20260808 01:50:48.008312  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:48.006426  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:47.934288 12917 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.023327  9524 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:48.000556 12994 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:48.027060  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:37234 (request call id 9) took 1042 ms. Trace:
I20260808 01:50:48.011889 12926 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:50:48.030263  9556 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:48.025597  9524 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:48.030776  9551 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:48.034148  9474 rpcz_store.cc:276] 0808 01:50:46.976052 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:47.031572 (+ 55520us) service_pool.cc:224] Handling call
0808 01:50:47.973899 (+942327us) spinlock_profiling.cc:243] Waited 7.34 ms on lock 0x7b5800005498. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999efd0 00007f5d4998b7e9 00007f5d4998bab2 00007f5d42ed8334
0808 01:50:48.018109 (+ 44210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16884480}
I20260808 01:50:48.050433  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:48.034497 12733 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:48.057253  9556 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:48.032354 12937 consensus_queue.cc:1243] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260808 01:50:48.059824  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:48.086736  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:48.087491  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:48.042397 12854 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:48.090504  9551 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: [6.4-8.5]
I20260808 01:50:48.044059 12819 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:48.061105  9524 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:48.062863 12999 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:48.067638  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:48.067904 12812 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:48.068672  9556 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260808 01:50:48.034126 12809 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.058s
I20260808 01:50:48.071822  9478 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:48.111222  9559 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260808 01:50:48.111642  9559 pending_rounds.cc:85] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 4
I20260808 01:50:48.112095  9559 pending_rounds.cc:107] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260808 01:50:48.112407  9559 raft_consensus.cc:2889] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:48.112562  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:48.113402  9478 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 8: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:48.015228 12477 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.114588 12940 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:48.115217 12940 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.115547  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:48.116180  9556 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:50:48.116583 12477 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:48.116824  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:48.119330  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 1890) took 1102 ms. Trace:
I20260808 01:50:48.119714 12940 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:48.120080  9558 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:48.048318 12971 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:48.080138  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 437) took 1034 ms. Trace:
W20260808 01:50:48.080863  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:48.080824 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:48.082038 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:48.083209 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:48.074173  9560 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:48.088078 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:48.123593  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 62578.7us
I20260808 01:50:48.063009  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:48.074216 12988 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260808 01:50:48.093515 12733 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:50:48.089540 12676 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.680s	user 0.007s	sys 0.007s
I20260808 01:50:48.101265 12819 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:48.069175  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:48.107427 12999 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:48.109911 12787 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:48.111507  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 1876) took 1194 ms. Trace:
I20260808 01:50:48.074082 12942 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:48.132814  9555 rpcz_store.cc:276] 0808 01:50:46.917185 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:46.917377 (+   192us) service_pool.cc:224] Handling call
0808 01:50:48.111253 (+1193876us) spinlock_profiling.cc:243] Waited 89.3 ms on lock 0x7b540008acd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:48.111429 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":205449472}
I20260808 01:50:48.126235  9524 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:48.136440  9599 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:48.114336  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:48.088317 12926 raft_consensus.cc:697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:48.077859 12826 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:48.043128 12994 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:48.076170 12917 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:48.147015  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 103018us
I20260808 01:50:48.147905  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:48.119771  9550 rpcz_store.cc:276] 0808 01:50:47.016665 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:47.055094 (+ 38429us) service_pool.cc:224] Handling call
0808 01:50:48.119108 (+1064014us) spinlock_profiling.cc:243] Waited 347 ms on lock 0x7b540002f154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:48.119267 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":798685952}
W20260808 01:50:48.125281  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:48.120806  9473 rpcz_store.cc:276] 0808 01:50:47.045442 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:47.549596 (+504154us) service_pool.cc:224] Handling call
0808 01:50:47.550065 (+   469us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:47.550393 (+   328us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:47.550417 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:47.589001 (+ 38584us) log.cc:844] Serialized 26 byte log entry
0808 01:50:47.923638 (+334637us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:47.923693 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:47.923797 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:48.079857 (+156060us) raft_consensus.cc:1685] finished
0808 01:50:48.079885 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:48.080068 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":301914,"threads_started":1,"wal-append.queue_time_us":433739}
I20260808 01:50:48.105827  9479 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 7. (term mismatch)
I20260808 01:50:48.150098  9479 pending_rounds.cc:85] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 6
I20260808 01:50:48.150506  9479 pending_rounds.cc:107] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 9.7
I20260808 01:50:48.131965  9560 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:48.150863  9479 raft_consensus.cc:2889] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:48.140628  9475 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:48.150944  9473 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260808 01:50:48.156164  9473 pending_rounds.cc:85] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 4
I20260808 01:50:48.156414  9473 pending_rounds.cc:107] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260808 01:50:48.156661  9473 raft_consensus.cc:2889] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:48.136565 12693 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:48.135799 12819 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:48.133767 12999 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:48.169831  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:48.178036  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 118206us
W20260808 01:50:48.178752  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:48.182545  9479 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260808 01:50:48.183058  9479 pending_rounds.cc:85] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 3
I20260808 01:50:48.183310  9479 pending_rounds.cc:107] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260808 01:50:48.183581  9479 raft_consensus.cc:2889] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:48.134053 12676 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:48.187098  9473 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:48.151384 13005 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:48.181094 12772 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:48.167650 12809 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:48.146209 12826 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:48.193032  9550 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:50:48.195295 12809 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:48.195789 12809 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:48.196151 12826 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.153661 12785 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:48.173286 12917 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260808 01:50:48.198045 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 72755us
I20260808 01:50:48.169723 12995 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:48.210256 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:48.213546  9521 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:48.214649  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:48.219384 12693 raft_consensus.cc:697] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:48.219636 13012 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:50:48.220778 12693 consensus_queue.cc:237] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.223481  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:48.225761  9555 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:48.224115 12998 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:48.231571 12998 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:48.232015 12998 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:48.225353  9524 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:48.224192 13005 raft_consensus.cc:697] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:48.185467 12934 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260808 01:50:48.199611 12772 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:48.235491 12772 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:48.235882 13005 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.185067 12420 consensus_queue.cc:1059] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260808 01:50:48.247512 13009 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:48.248368 13009 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:48.249606 13009 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:48.187215 12781 consensus_queue.cc:1059] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
W20260808 01:50:48.252475 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 50601.3us
I20260808 01:50:48.256606 10332 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:48.264168  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45904.7us
I20260808 01:50:48.264384 12917 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:50:48.228065 12995 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:48.212854 12940 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260808 01:50:48.253456 12826 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:48.274000  9555 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260808 01:50:48.261493 12778 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:48.263504 12934 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:48.263847 12942 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:48.225055 12733 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.294920  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:48.295101 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:50:48.300886 13003 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260808 01:50:48.304438  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:48.304438 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:50:48.311892  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:48.343022  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 117795us
I20260808 01:50:48.279487 12934 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:48.345469  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:48.278026 12778 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:48.277733 13016 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:50:48.284420 12971 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:48.351744  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 86646.9us
I20260808 01:50:48.357780  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:48.357868 12756 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:48.261531 12926 consensus_queue.cc:237] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.368013  9636 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:50:48.355311  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:48.370604  9631 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:50:48.370829  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:48.372685  9632 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 01:50:48.349532 12934 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:48.378266  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:48.380194  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:48.350775 12778 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:48.351809 13013 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:50:48.427762  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:48.367311 12955 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:48.419277  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:48.421767  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:48.421994  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:48.347299 12917 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:48.421123  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 2f740cd3515641a8b2c4add8f809dd82: Illegal state: Replica 6e7856bed1be43acaa821290d506ee97 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:50:48.419426  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:48.429052  9555 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 8: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:48.430979  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:48.387387 13005 raft_consensus.cc:3055] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:48.429672  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:48.435606  9632 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:48.373190 13016 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:48.341209 13011 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.115s
I20260808 01:50:48.449492  9629 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
W20260808 01:50:48.453521 13008 log.cc:927] Time spent T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.185s	user 0.001s	sys 0.000s
I20260808 01:50:48.453313  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:48.454020  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:48.418736 12994 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:48.455982  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:48.485132  9637 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:48.454025 12917 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:48.458132 13015 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election won for term 9
W20260808 01:50:48.458405  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 249435us
W20260808 01:50:48.459242 13006 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.185s	user 0.001s	sys 0.000s
I20260808 01:50:48.459867  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:48.463007  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:48.468103 12778 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:48.468653 12772 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:48.505831  9629 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 8: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:48.444348 13005 raft_consensus.cc:740] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:48.476151 13023 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:50:48.428454 12971 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260808 01:50:48.455293  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 75214.6us
I20260808 01:50:48.513399  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:48.370009 12998 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.482961  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:48.451469  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:48.517925  9555 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:48.434839 12676 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:48.504814 13029 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260808 01:50:48.427592  9425 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.BulkChangeConfig -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 54546.4us
I20260808 01:50:48.505389  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:48.508083  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:48.524628  9479 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:48.473246  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:48.444682 12420 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.075s
I20260808 01:50:48.372684 13021 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.083s
W20260808 01:50:48.515141  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:48.518235  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:48.454273 12785 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:48.519150  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:48.522289  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:48.516978 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 249192us
I20260808 01:50:48.518052  9631 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:48.536895  9551 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260808 01:50:48.516580 13015 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:48.517048 12971 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:48.546573 12967 log.cc:927] Time spent T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 1.423s	user 0.006s	sys 0.008s
I20260808 01:50:48.550872  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 425) took 3577 ms. Trace:
I20260808 01:50:48.551445  9476 rpcz_store.cc:276] 0808 01:50:44.973292 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:46.768012 (+1794720us) service_pool.cc:224] Handling call
0808 01:50:46.768510 (+   498us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:46.778301 (+  9791us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:46.778338 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:46.789774 (+ 11436us) log.cc:844] Serialized 26 byte log entry
0808 01:50:46.998453 (+208679us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:46.998497 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:46.998570 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:48.545680 (+1547110us) spinlock_profiling.cc:243] Waited 1.96 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:48.550476 (+  4796us) raft_consensus.cc:1685] finished
0808 01:50:48.550503 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:48.550795 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6181,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":7895,"spinlock_wait_cycles":4509440,"thread_start_us":186819,"threads_started":1,"wal-append.queue_time_us":272529}
I20260808 01:50:48.553823  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:48.558058  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 439) took 1318 ms. Trace:
I20260808 01:50:48.569692  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:48.571259 10369 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260808 01:50:48.554605  9476 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:48.572669  9475 rpcz_store.cc:276] 0808 01:50:47.240079 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:47.660549 (+420470us) service_pool.cc:224] Handling call
0808 01:50:47.660923 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:48.167173 (+506250us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:48.557652 (+390479us) spinlock_profiling.cc:243] Waited 475 ms on lock 0x7b54002a6cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:48.557967 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1092859136}
I20260808 01:50:48.545944 13016 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.521796 12955 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:48.564508  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:48.588934  9555 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260808 01:50:48.590164  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 46455.7us
I20260808 01:50:48.454900 12995 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.518921 13012 consensus_queue.cc:1059] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260808 01:50:48.530953 13020 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.079s
I20260808 01:50:48.601503 13020 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:48.602008 13020 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:48.578773 12745 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:48.572101 12971 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:48.607379  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:48.616055  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:48.616143  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:48.617405  9475 raft_consensus.cc:2410] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260808 01:50:48.591176 12955 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:48.570961 13005 consensus_queue.cc:260] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.627207  9521 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:48.630810 12804 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:48.630975 12997 raft_consensus.cc:2764] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
W20260808 01:50:48.633348 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 96840.6us
I20260808 01:50:48.639147 12804 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:48.639770 12999 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.631892 12925 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.641537 12772 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:48.642143 12925 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:48.643540 12809 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.644933  9560 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 8: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:48.646261 12809 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:48.615617 12917 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:48.625936 12955 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:50:48.668634  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 58424.5us
I20260808 01:50:48.669947  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:48.615888 12745 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:48.650826 12999 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:48.679201  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 245933us
W20260808 01:50:48.679355  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 52337.4us
I20260808 01:50:48.688481  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:48.689518  9481 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:48.694113  9521 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:48.680855 12788 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:50:48.685798 13005 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:48.682786 12778 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:48.702520  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:48.633720 13020 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:48.707917  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:48.712291  9481 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:50:48.714002  9481 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260808 01:50:48.718034  9555 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260808 01:50:48.713624 12788 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260808 01:50:48.712826 12864 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:50:48.721815 12788 raft_consensus.cc:3055] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:48.731766 12788 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:50:48.724732 12981 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.188s	user 0.004s	sys 0.009s
I20260808 01:50:48.733357 13020 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:48.693595 13012 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20260808 01:50:48.735428  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 367) took 1727 ms. Trace:
I20260808 01:50:48.736328 12998 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:48.706382 13029 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260808 01:50:48.735982  9633 rpcz_store.cc:276] 0808 01:50:47.008131 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:47.232239 (+224108us) service_pool.cc:224] Handling call
0808 01:50:47.232685 (+   446us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:47.237112 (+  4427us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:47.237149 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:47.237940 (+   791us) log.cc:844] Serialized 26 byte log entry
0808 01:50:47.402177 (+164237us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:47.402217 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:47.402291 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:48.723743 (+1321452us) spinlock_profiling.cc:243] Waited 26.2 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:48.734951 (+ 11208us) raft_consensus.cc:1685] finished
0808 01:50:48.734985 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:48.735340 (+   355us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":60145920,"thread_start_us":163500,"threads_started":1,"wal-append.queue_time_us":227012}
I20260808 01:50:48.724874 12778 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:48.738094 12778 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:48.743791  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:48.713928 12819 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.526s	user 0.013s	sys 0.002s
W20260808 01:50:48.751631  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 43772.3us
I20260808 01:50:48.703121 12999 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260808 01:50:48.769764  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:48.772145  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 69713.3us
I20260808 01:50:48.777941  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:48.777993 12809 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:50:48.778981 12942 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 01:50:48.780016  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 77258.3us
I20260808 01:50:48.780395  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:48.781070  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:48.782933  9601 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:48.671285 13015 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.720331 13016 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:50:48.757709 12819 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.794934  9559 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 7. Ops: []
I20260808 01:50:48.790282 12998 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:48.806912 12778 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:48.812222  9475 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:48.732851 12788 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.812599  9521 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:48.821195  9521 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:48.814728  9638 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:50:48.824223  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:48.811450 12661 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.828495 12809 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:48.829010 12661 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:48.829730 12864 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:50:48.822657 12918 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:48.831388 12826 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:50:48.831271 12918 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:48.822222 13016 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:48.833523  9521 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:48.834218 12942 raft_consensus.cc:3055] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:48.833945 13016 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.834438 13021 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:48.834669 12942 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:48.835107 13021 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.835779 12942 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.837149 13016 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:48.805851 12819 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:48.838137 13016 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:50:48.813596 12998 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:48.833118 13049 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.071s	user 0.000s	sys 0.002s
W20260808 01:50:48.847149  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 50696.4us
I20260808 01:50:48.816674 12973 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:48.838536 13021 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:48.838186 12942 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:48.873256  9479 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260808 01:50:48.869259  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:48.824926 12804 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:48.879457 12971 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:48.885337  9551 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: [6.4-8.5]
I20260808 01:50:48.886314 12971 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:48.890239 12995 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:48.869204 12998 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:48.880309 12788 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:48.822808 12934 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:48.903600  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:48.917754 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:48.919936  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 435) took 2346 ms. Trace:
I20260808 01:50:48.920429  9480 rpcz_store.cc:276] 0808 01:50:46.573523 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:47.000828 (+427305us) service_pool.cc:224] Handling call
0808 01:50:47.038073 (+ 37245us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:47.131045 (+ 92972us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:47.131076 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:47.131140 (+    64us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:47.131195 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:48.919588 (+1788393us) spinlock_profiling.cc:243] Waited 82.5 ms on lock 0x7b540021e154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:48.919651 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:48.919853 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":189667072}
I20260808 01:50:48.926044  9478 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
W20260808 01:50:48.927424  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:48.928633 10362 raft_consensus.cc:2393] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:48.928519 12926 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260808 01:50:48.930096  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:48.931458  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:48.933588  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:48.937623 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:48.923619 12804 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:48.911862 13016 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:48.948498  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:48.914434 12995 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:48.950376  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:48.952140  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:48.959651  9634 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
I20260808 01:50:48.947839 13034 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:48.955019  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:48.956923  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:48.963383  9480 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:48.962980  9630 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:48.925350 12973 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:48.963914  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:48.965834  9634 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:48.960220  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:48.970404  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:48.970115 13034 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.971318  9629 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:48.973335 13034 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:48.975422  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:48.976576  9476 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:48.981731  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:48.924983 13025 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:48.948166 13023 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:50:48.986764  9634 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:48.924805 12971 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20260808 01:50:48.959326 12995 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:49.003345  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:48.990607 13025 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.959019 13016 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.001269  9637 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260808 01:50:49.004276  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:49.009179 13030 raft_consensus.cc:3055] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:49.007812  9632 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260808 01:50:49.018340  9631 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:49.034310 13035 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.415s	user 0.000s	sys 0.001s
I20260808 01:50:49.072350  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2244) took 1014 ms. Trace:
I20260808 01:50:48.981436 12973 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:49.016678  9638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 351) took 3615 ms (client timeout 2096 ms). Trace:
W20260808 01:50:49.073267  9638 rpcz_store.cc:269] 0808 01:50:45.401672 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:45.427710 (+ 26038us) service_pool.cc:224] Handling call
0808 01:50:49.016314 (+3588604us) spinlock_profiling.cc:243] Waited 3.38 s on lock 0x7b54001b3ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:49.016530 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7772355840}
I20260808 01:50:49.074915 12973 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:49.075632 12973 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:49.076215 12973 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.076627 12819 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:49.077322 12819 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:49.034898  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:49.080009 12917 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:49.035425 12778 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:49.080500 12917 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:49.080945 12917 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:49.080703 12778 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:48.950997 12804 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:49.084271 12819 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:49.086221 12819 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:49.037072 12856 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:49.090660 12926 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:49.039247  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:49.080224  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 45446.1us
I20260808 01:50:49.095570  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:49.072993  9472 rpcz_store.cc:276] 0808 01:50:48.058030 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:48.062626 (+  4596us) service_pool.cc:224] Handling call
0808 01:50:49.072052 (+1009426us) spinlock_profiling.cc:243] Waited 454 ms on lock 0x7b5400399254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:49.072252 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1045149568}
W20260808 01:50:49.036115  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:49.017020 12955 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.045217  9631 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:49.047231 12928 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:49.101061 12928 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:49.102058 12928 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.103255 12928 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:49.024755 12738 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:50:49.107115 12738 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:49.025965  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2356) took 1586 ms. Trace:
I20260808 01:50:49.109158  9558 rpcz_store.cc:276] 0808 01:50:47.439516 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:47.475739 (+ 36223us) service_pool.cc:224] Handling call
0808 01:50:48.119855 (+644116us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:48.120550 (+   695us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:49.025654 (+905104us) spinlock_profiling.cc:243] Waited 644 ms on lock 0x7b540002f150. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:49.025850 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1480182016}
I20260808 01:50:49.030531  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 411) took 1584 ms. Trace:
I20260808 01:50:49.031819 13030 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:49.111538 13030 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.110864  9557 rpcz_store.cc:276] 0808 01:50:47.445905 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:47.483601 (+ 37696us) service_pool.cc:224] Handling call
0808 01:50:47.484030 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:47.595161 (+111131us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:50:47.616815 (+ 21654us) log.cc:844] Serialized 14 byte log entry
0808 01:50:47.922566 (+305751us) log.cc:844] Serialized 14 byte log entry
0808 01:50:47.932103 (+  9537us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:47.932182 (+    79us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:50:47.932251 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:49.030261 (+1098010us) spinlock_profiling.cc:243] Waited 90 ms on lock 0x7b54000fee54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:49.030315 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:49.030460 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14150,"prepare.queue_time_us":29894,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":97,"raft.queue_time_us":173175,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":16405,"spinlock_wait_cycles":206913792,"thread_start_us":286756,"threads_started":1,"wal-append.queue_time_us":402189,"wal-append.run_cpu_time_us":2252,"wal-append.run_wall_time_us":1039136}
I20260808 01:50:49.112720 13030 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:49.017915 12995 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:49.027324  9633 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:49.029398 12745 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:49.030361  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:49.060590  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2375) took 1412 ms. Trace:
I20260808 01:50:49.022233 13016 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:49.061017 10361 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260808 01:50:49.030258  9634 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:49.061502  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:49.127318  9477 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 8: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:49.063712 10366 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260808 01:50:49.062513 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:49.128533 10369 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:49.065955 13025 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:49.069108 13001 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.965s	user 0.000s	sys 0.004s
W20260808 01:50:49.070003 13041 log.cc:927] Time spent T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.288s	user 0.002s	sys 0.000s
I20260808 01:50:49.071642 12934 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:49.087679 12804 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:49.137224  9523 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:49.039866 12998 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:49.108141 12738 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.116092  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:49.121825  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:49.121652  9559 rpcz_store.cc:276] 0808 01:50:47.648528 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:48.114046 (+465518us) service_pool.cc:224] Handling call
0808 01:50:48.114390 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:48.795434 (+681044us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:49.060307 (+264873us) spinlock_profiling.cc:243] Waited 680 ms on lock 0x7b540002c6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:49.060521 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1564637952}
I20260808 01:50:49.091354 12926 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.130614  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2243) took 1075 ms. Trace:
I20260808 01:50:49.141492  9560 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:49.150342  9477 rpcz_store.cc:276] 0808 01:50:48.055249 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:49.061198 (+1005949us) service_pool.cc:224] Handling call
0808 01:50:49.130542 (+ 69344us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:49.134064 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:49.135900  9552 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:50:49.097254  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:49.137763  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 354921us
I20260808 01:50:49.122869 12971 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:49.125605 12973 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:49.132872 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:49.131934 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:49.184652  9556 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260808 01:50:49.180488 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:49.188059  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:49.189158  9481 raft_consensus.cc:2393] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:49.140515 12856 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:49.130280 12819 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:49.191905  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 94763.9us
W20260808 01:50:49.192701  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 7c508218361c446898fede7c5682fa9e: Illegal state: Leader has not yet committed an operation in its own term
I20260808 01:50:49.125718 12925 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20260808 01:50:49.140239 12997 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:49.150733 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:49.153581 12804 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:49.222180 12804 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:49.222786 12804 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:49.222012 10363 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:49.223374 13017 raft_consensus.cc:3055] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:49.158795 12971 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:49.223909 13017 raft_consensus.cc:740] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:49.224174 12971 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:49.208319  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:49.225119 13017 consensus_queue.cc:260] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:49.153597 13005 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.055s
W20260808 01:50:49.227397 13053 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260808 01:50:49.229585  9551 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:49.230530  9550 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:49.230942  9551 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260808 01:50:49.231474  9523 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:49.231832  9550 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:49.232024 12926 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:49.153201 13016 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
I20260808 01:50:49.152982 12942 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.058s
W20260808 01:50:49.235499  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 57929.1us
I20260808 01:50:49.212495 10367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 457) took 1545 ms. Trace:
I20260808 01:50:49.204391  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:49.152293 13025 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260808 01:50:49.178622 12778 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:49.214862 12997 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.215651  9472 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260808 01:50:49.218125 12788 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:49.221994 12785 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:50:49.272984 12785 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:49.273458 12785 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:49.232656 12926 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:49.214609 13058 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.213s	user 0.001s	sys 0.000s
I20260808 01:50:49.238626  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:49.210254  9559 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:49.183701 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:49.184005 12973 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:49.245421 13017 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:49.255765 12998 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:49.258080 13005 consensus_queue.cc:1243] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 9.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260808 01:50:49.263317 13034 raft_consensus.cc:3055] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:50:49.263510 12917 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.264017 12819 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:49.269528  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:49.296175  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:49.271469 12955 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:49.274457 12804 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:49.206548 12856 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.236610 10367 rpcz_store.cc:276] 0808 01:50:47.667218 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:47.670311 (+  3093us) service_pool.cc:224] Handling call
0808 01:50:47.671491 (+  1180us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:47.672106 (+   615us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:47.672142 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:47.673118 (+   976us) log.cc:844] Serialized 26 byte log entry
0808 01:50:48.039188 (+366070us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:48.039249 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:48.039352 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:49.068015 (+1028663us) spinlock_profiling.cc:243] Waited 1.9 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:49.197104 (+129089us) raft_consensus.cc:1685] finished
0808 01:50:49.197127 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:49.212406 (+ 15279us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1815,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":7019,"spinlock_wait_cycles":4370176,"thread_start_us":365378,"threads_started":1,"wal-append.queue_time_us":430050}
I20260808 01:50:49.277828 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:49.246052 13015 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:49.276280 12778 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:49.253854 12788 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:50:49.305145 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 271732us
I20260808 01:50:49.325968  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:49.327363  9554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 407) took 2295 ms (client timeout 2908 ms). Trace:
W20260808 01:50:49.346495  9554 rpcz_store.cc:269] 0808 01:50:47.031517 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:47.162074 (+130557us) service_pool.cc:224] Handling call
0808 01:50:49.327071 (+2164997us) spinlock_profiling.cc:243] Waited 864 ms on lock 0x7b540002cbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:49.327275 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1986786048}
I20260808 01:50:49.347710  9524 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:49.348685 12693 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:49.293171 13034 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:49.291663 10360 raft_consensus.cc:2376] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8. Re-sending same reply.
I20260808 01:50:49.320713 13012 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:49.363253  9601 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:49.371747  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:49.375203  9601 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:49.296473 13025 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20260808 01:50:49.274569 12785 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:49.343377 13015 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:49.378648 10332 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:49.293830 12973 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.350056 12788 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:49.385870 12676 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:49.411185  9600 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:49.411590 12917 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:49.412343 12917 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:49.412555 12676 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election won for term 8
W20260808 01:50:49.349684 13026 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
W20260808 01:50:49.413462 13047 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.637s	user 0.005s	sys 0.000s
I20260808 01:50:49.345320 13060 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:50:49.415210 12785 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:49.415467  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2032) took 1041 ms. Trace:
I20260808 01:50:49.416006  9473 rpcz_store.cc:276] 0808 01:50:48.374161 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:48.570051 (+195890us) service_pool.cc:224] Handling call
0808 01:50:48.570433 (+   382us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:48.570909 (+   476us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:48.570944 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:48.571461 (+   517us) log.cc:844] Serialized 26 byte log entry
0808 01:50:48.775938 (+204477us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:48.775979 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:48.776103 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:49.287236 (+511133us) spinlock_profiling.cc:243] Waited 6.45 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:49.415039 (+127803us) raft_consensus.cc:1685] finished
0808 01:50:49.415073 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:49.415374 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9782,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":11435,"spinlock_wait_cycles":14828416,"thread_start_us":204059,"threads_started":1,"wal-append.queue_time_us":204554}
I20260808 01:50:49.420783 13020 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:49.421738 13020 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:49.421907 12973 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:49.353888 13021 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:49.403595  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:49.404989  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 45569.4us
I20260808 01:50:49.294613 12926 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.408046  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:49.413242 12676 raft_consensus.cc:697] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:49.377254 13029 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:49.432132 12856 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:49.433143 12918 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:49.369452 12819 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.057s
I20260808 01:50:49.353597 12420 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.054s
I20260808 01:50:49.351003 12928 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.056s
I20260808 01:50:49.434983 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:49.435225 12477 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260808 01:50:49.437844  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:49.517371  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:49.437120  9554 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:49.440882  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:49.422473 13020 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:49.363600 13030 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20260808 01:50:49.408179 12816 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:49.408758 13015 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:49.412511 12778 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:49.409096 13069 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:49.452126 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 73584us
W20260808 01:50:49.493978  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 44672.3us
I20260808 01:50:49.548559  9445 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:49.426728 13021 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:49.432224 12804 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:49.530087 12997 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:49.530614 13064 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260808 01:50:49.553300 13021 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:49.526543 10360 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:49.502632 12856 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
W20260808 01:50:49.537528  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 96794.9us
I20260808 01:50:49.564046 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:49.521502 13029 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:49.537652 12819 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:49.568025 12983 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:49.446395 13023 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.081s
I20260808 01:50:49.563882 12772 consensus_queue.cc:1059] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:50:49.567071 13029 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:49.541805 12926 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:49.539722 13067 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20260808 01:50:49.555593 12816 raft_consensus.cc:697] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:49.451229 13060 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.083s
I20260808 01:50:49.575970  9473 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:49.531286 12676 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:49.589294 12819 raft_consensus.cc:697] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:49.590008 12942 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:49.599375  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 360812us
I20260808 01:50:49.606813  9472 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 7. Current term is 8. Ops: []
W20260808 01:50:49.610232  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 61744.5us
I20260808 01:50:49.611327  9445 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:49.616357  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:49.551163 12778 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.580853 12804 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:49.621819 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 57858.4us
I20260808 01:50:49.542443 13034 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.626816  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:49.632429  9477 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 8: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:49.627081  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:49.568884 12995 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.641381  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:49.642292  9479 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:49.622515 12954 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:49.643328  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 196003us
I20260808 01:50:49.644376  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:49.619927 12926 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:49.624877 12997 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:50:49.648000 12883 raft_consensus.cc:2764] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260808 01:50:49.608085 12942 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.581385 12983 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.648284 12816 consensus_queue.cc:237] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.706898  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:49.707782  9477 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:49.654395 12788 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.711120  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:49.664455 12995 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:49.665525 13071 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260808 01:50:49.657444 12778 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:49.713056  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:49.713761  9472 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:49.631291 12856 consensus_queue.cc:1059] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.033s
I20260808 01:50:49.554710 13069 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.715276 12856 raft_consensus.cc:3055] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:49.715667 12856 raft_consensus.cc:740] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:49.716821 13069 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:49.716584 12856 consensus_queue.cc:260] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:49.640281 13078 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260808 01:50:49.717931 12856 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:49.658934 13029 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:49.711848  9474 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
W20260808 01:50:49.657193 13068 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260808 01:50:49.651813  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:49.662550 13076 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260808 01:50:49.721324  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:49.736842  9479 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
W20260808 01:50:49.723284  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 71605.8us
W20260808 01:50:49.670336  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:49.712731 12788 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:49.742776  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:49.728538  9474 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
W20260808 01:50:49.750389  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 57575.9us
I20260808 01:50:49.751588  9601 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:49.752161  9477 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
W20260808 01:50:49.664394 13065 log.cc:927] Time spent T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260808 01:50:49.753790  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:49.756139  9555 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 8: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:49.645453 12772 raft_consensus.cc:3055] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:49.657541 12917 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:49.650192 12955 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:49.668196 13079 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:49.757254  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:49.667586 13012 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:49.774070  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:49.761649  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:49.774887  9558 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260808 01:50:49.775043  9552 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:49.765105  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:49.772511  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:49.772715  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:49.774142  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:49.759784  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:49.727629 13016 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:49.776150  9555 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:50:49.779374  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:49.751149 10360 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:50:49.645349 13033 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.138s
I20260808 01:50:49.778036 12955 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:49.762874 13029 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:49.751397 12778 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:49.787122  9479 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260808 01:50:49.830837  9479 pending_rounds.cc:85] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 3
I20260808 01:50:49.831151  9479 pending_rounds.cc:107] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260808 01:50:49.831415  9479 raft_consensus.cc:2889] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:49.775547 12917 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:49.761843 13017 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:49.816970  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:49.843812  9476 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:49.650421 12804 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:49.678315 12971 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.761130 12772 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:49.822268  9556 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:49.772173 13060 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:49.772655 12918 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:49.832242 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 172844us
I20260808 01:50:49.835613  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:49.859860  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:49.826489  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:49.758189 13020 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:49.663101 12819 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:49.757153 13021 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.784720 12995 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:49.860803  9551 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:49.778364 13011 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:49.861850 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 287640us
I20260808 01:50:49.860217  9555 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:49.883482  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:49.883786  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:49.845216  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:49.884723  9631 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 7: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:49.832544 12925 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:49.818181 13030 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:49.885129  9474 raft_consensus.cc:2376] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8. Re-sending same reply.
I20260808 01:50:49.885720  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:49.888892  9637 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:49.886431  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:49.796196 13079 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:49.861685 12778 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:49.795480 13016 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:49.896948  9476 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:49.897115  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:49.898614  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:49.899742  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:49.903312  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:49.904230  9630 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:49.906599  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:49.906410 12772 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:49.907840  9373 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50084 (request call id 231) took 28155 ms (client timeout 15000 ms). Trace:
I20260808 01:50:49.908506 13079 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.867110 13060 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:49.914016  9633 raft_consensus.cc:2376] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9. Re-sending same reply.
I20260808 01:50:49.915805  9637 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:49.916498  9521 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:49.872918 12781 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:49.842833 13029 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.925458  9475 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260808 01:50:49.924177  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:49.928145  9554 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:49.883097 13012 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:49.931660  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:49.925294  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:49.932924  9636 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:50:49.933517  9555 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260808 01:50:49.930513  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:49.934530 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 74348.5us
I20260808 01:50:49.930774  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:49.928584  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:49.943397  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:49.946002  9475 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260808 01:50:49.949519  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:49.868575 12918 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.904268 12926 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:49.949826  9481 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260808 01:50:49.944460  9556 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:49.904943 13095 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:49.970142  9631 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:50:49.892665 12995 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.894903 13011 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:49.936376 12756 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:49.986833 13091 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260808 01:50:49.955515 13015 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:49.989655 13060 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:49.990579 13021 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:49.991032 12917 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:49.991647 12917 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:49.992040 12917 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:49.992408 12917 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:49.993497 12918 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:49.994413 13078 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:50:49.974649 12926 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:49.926302 13086 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
W20260808 01:50:49.996577 13093 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
I20260808 01:50:49.996868 12926 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:49.918970 12934 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20260808 01:50:49.998581 13011 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:49.931306 12955 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:49.967590 12781 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:50.001793 12781 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:50.002604  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:50.003371  9630 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:49.935617 12785 consensus_queue.cc:1059] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.049s
I20260808 01:50:50.014546  9473 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:50.015723  9481 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9. Ops: []
W20260808 01:50:50.062842 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 162554us
I20260808 01:50:49.943181 13012 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:50.090734  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 174084us
I20260808 01:50:50.114018 13096 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:50.114758 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50900.8us
I20260808 01:50:49.983379 12954 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:50.114073 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:50.117210  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:50.008038 12756 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:50.125447 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:50.067303 12942 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:50.148432 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:49.977746 13095 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:50.135866 12785 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:50.121968 12983 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:50.170296  9630 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
W20260808 01:50:50.198261  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 81151.2us
W20260808 01:50:49.908268  9373 rpcz_store.cc:269] 0808 01:50:21.715247 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:21.756353 (+ 41106us) service_pool.cc:224] Handling call
0808 01:50:49.865803 (+28109450us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:50:46.207736 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:50:46.208505 (+   769us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:50:46.208585 (+    80us) write_op.cc:435] Acquired schema lock
0808 01:50:46.208629 (+    44us) tablet.cc:662] Decoding operations
0808 01:50:46.216392 (+  7763us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:50:46.216470 (+    78us) write_op.cc:641] Partition lock acquired for write op
0808 01:50:46.216513 (+    43us) tablet.cc:685] Acquiring locks for 10 operations
0808 01:50:46.217571 (+  1058us) tablet.cc:701] Row locks acquired
0808 01:50:46.217596 (+    25us) write_op.cc:260] PREPARE: finished
0808 01:50:46.217761 (+   165us) write_op.cc:270] Start()
0808 01:50:46.217881 (+   120us) write_op.cc:276] Timestamp: P: 1786153846217706 usec, L: 0
0808 01:50:46.217902 (+    21us) op_driver.cc:348] REPLICATION: starting
0808 01:50:46.218705 (+   803us) log.cc:844] Serialized 4019 byte log entry
0808 01:50:46.494733 (+276028us) op_driver.cc:464] REPLICATION: finished
0808 01:50:46.536079 (+ 41346us) write_op.cc:301] APPLY: starting
0808 01:50:46.536255 (+   176us) tablet.cc:1367] starting BulkCheckPresence
0808 01:50:46.536745 (+   490us) tablet.cc:1370] finished BulkCheckPresence
0808 01:50:46.536995 (+   250us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:50:46.594024 (+ 57029us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:50:46.646796 (+ 52772us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0808 01:50:46.646876 (+    80us) write_op.cc:312] APPLY: finished
0808 01:50:46.647489 (+   613us) log.cc:844] Serialized 77 byte log entry
0808 01:50:46.647789 (+   300us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:46.648437 (+   648us) write_op.cc:454] Released schema lock
0808 01:50:46.648549 (+   112us) write_op.cc:341] FINISH: Updating metrics
0808 01:50:46.649268 (+   719us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:46.649312 (+    44us) write_op.cc:454] Released schema lock
0808 01:50:47.028863 (+379551us) spinlock_profiling.cc:243] Waited 98.3 ms on lock 0x7b540005c3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4499070a 00007f5d449162f4 00007f5d449162a3 00007f5d44916218 00007f5d449161d3 00007f5d449154bc 00007f5d4493f1f6 00007f5d4492709b 00007f5d4491177d 00007f5d44915155
Metrics: {"mutex_wait_us":3423212,"reported_tablets":33,"child_traces":[["op",{"apply.queue_time_us":40747,"apply.run_cpu_time_us":6569,"apply.run_wall_time_us":113191,"num_ops":10,"prepare.queue_time_us":350092,"prepare.run_cpu_time_us":17092,"prepare.run_wall_time_us":144057,"raft.queue_time_us":1217,"raft.run_cpu_time_us":39154,"raft.run_wall_time_us":768082,"replication_time_us":230643,"spinlock_wait_cycles":226100992,"thread_start_us":747661,"threads_started":4,"wal-append.queue_time_us":39065,"wal-append.run_cpu_time_us":5017,"wal-append.run_wall_time_us":1392676}]]}
I20260808 01:50:50.171566 12942 raft_consensus.cc:2764] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260808 01:50:50.207743  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:50.141464 12954 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:50.211179  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 272653us
I20260808 01:50:50.224934  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:50.209903 12745 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:50.226387 12745 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:50.181263 12804 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:50.286605  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:50.121657 12778 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:50.209580 12971 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:50.387768 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:50.284574 12788 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.143s
I20260808 01:50:50.191074 13099 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.153s
I20260808 01:50:50.138016 13096 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:50.262264 12955 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:50:50.285528 12804 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:50.289801 12676 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:50.438324 12926 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:50.463236 13102 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:50.463594 12926 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:50.464151 12926 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:50.466944 13107 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:50.466805 12676 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:50.468565 13100 log.cc:927] Time spent T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.327s	user 0.000s	sys 0.000s
I20260808 01:50:50.469774 12676 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:50.451529 12955 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:50.470486 13120 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.243s	user 0.000s	sys 0.001s
I20260808 01:50:50.451927 12934 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:50.471032 12934 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:50.472365 13096 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:50.472126 12934 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:50.473582 12934 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:50.466823 13106 consensus_queue.cc:1059] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.213s
I20260808 01:50:50.474630  9474 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:50.475090 12955 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:50:50.478813 13121 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260808 01:50:50.457932 12804 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:50.479875 13106 raft_consensus.cc:3055] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:50.480418 13106 raft_consensus.cc:740] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:50.481245 12804 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:50.480549 13124 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.217s	user 0.000s	sys 0.000s
I20260808 01:50:50.437422 13029 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:50.490883 13115 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:50.493491 12995 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:50.545115  9637 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.627s	user 0.014s	sys 0.000s
I20260808 01:50:50.481588 13106 consensus_queue.cc:260] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:50.467756 13107 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:50.558972  9637 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:50:50.491082 12745 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:50.569550 13106 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:50.585839 13095 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:50.587067 12954 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:50.574146 13012 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:50.595675 13107 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:50.542152 12804 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:50.604219 12804 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:50.635819 12804 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:50.552882 13115 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:50.638532 13115 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:50.612663 12955 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:50.644155 12955 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:50.615532 13061 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:50.620303 13034 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:50.644562 13020 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:50.620792 12917 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.628s	user 0.012s	sys 0.000s
I20260808 01:50:50.653412  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:50.654313  9557 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:50.654469  9560 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260808 01:50:50.653343  9551 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:50.625360  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:50.656203  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:50.658564  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1928) took 2282 ms. Trace:
I20260808 01:50:50.658869  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:50.659024 13012 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:50.659744  9550 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:50.658257  9601 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:50.659871  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:50.660316  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:50.660650  9556 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:50.661556 12954 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:50:50.661942 12676 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:50.659103  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:50.662774 12676 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:50.663303  9560 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:50.659351  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:50.659013  9636 rpcz_store.cc:276] 0808 01:50:48.375776 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:48.509147 (+133371us) service_pool.cc:224] Handling call
0808 01:50:48.519561 (+ 10414us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:49.933378 (+1413817us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:50.658125 (+724747us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b540008dc54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:50.658439 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3249538304}
I20260808 01:50:50.665242 12676 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:50.665859 12937 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:50:50.666760  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:50.623978 13114 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:50.655186 12738 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:50.659929 13012 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:50.668707 13005 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:50:50.669247 13005 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:50.668843 13012 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:50.661872 13029 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:50.672334 13029 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:50:50.673038 13029 consensus_queue.cc:1243] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260808 01:50:50.673446 12954 consensus_queue.cc:1059] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:50:50.673815 13114 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:50.674849  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:50.676257 13114 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:50.676620  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:50.677307  9556 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:50.677809  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:50.679128  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:50.679464 13016 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:50.680392  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:50.680903  9556 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:50.680959  9638 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:50.678427  9633 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:50.682025  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:50.682173  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:50.670230 13005 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:50.635984  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:50.566632 12856 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.848s	user 0.010s	sys 0.003s
I20260808 01:50:50.683153  9636 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:50.684389 13005 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:50.684984  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:50.682632  9638 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:50.685670  9636 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 3.
I20260808 01:50:50.690192  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:50.683504  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:50.683995 12917 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:50.706454  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:50.707319  9629 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:50.709095 12917 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:50.691283  9638 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:50.691577  9633 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:50.604874 13102 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 01:50:50.693799  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:50.713160  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:45650 (request call id 11) took 1489 ms. Trace:
I20260808 01:50:50.692011 13061 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:50.710740  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:50.715288  9638 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:50.687083 12661 raft_consensus.cc:3055] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:50.695393 12926 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:50.727409  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 92121.5us
W20260808 01:50:50.699301  9521 leader_election.cc:336] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.544s (SENT)
I20260808 01:50:50.701594  9632 raft_consensus.cc:2410] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260808 01:50:50.702006 12738 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:50.711932 13029 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:50.713769  9634 rpcz_store.cc:276] 0808 01:50:49.222773 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:49.222969 (+   196us) service_pool.cc:224] Handling call
0808 01:50:50.711711 (+1488742us) spinlock_profiling.cc:243] Waited 48.4 ms on lock 0x7b540016b320. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44910900 00007f5d44999f22 00007f5d44998be0 00007f5d4998bb39 00007f5d42ed8334 00007f5d42ed82c2 00007f5d42ed81f1 00007f5d42ed816c 00007f5d42ed7422 00007f5d42ccd07c
0808 01:50:50.711953 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":111315712}
W20260808 01:50:50.748142  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48717.2us
I20260808 01:50:50.713956  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:50.770130 10331 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.869s	user 0.054s	sys 0.145s
W20260808 01:50:50.770728 10331 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.869s	user 0.055s	sys 0.145s
I20260808 01:50:50.725431  9633 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:50.690383  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:50.693543 13012 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:50.728487  9481 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:50:50.729812  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:50.731853  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:50.732674 13113 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:50.729095 12661 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:50.784724  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:50.785344  9638 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:50.783605  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:50.716948 13033 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:50.795538  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:50.792136  9629 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:50.786043 12738 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:50.798419  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:50.799364  9634 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:50.790043 13016 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:50.800387  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:50.801187  9634 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:50.801658 13016 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:50.802192  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:50.803922 13016 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:50.805454 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:50.807039  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:50.807336  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet c85ff498df484fc6ac0913aa57aa868b: Illegal state: Leader has not yet committed an operation in its own term
I20260808 01:50:50.813625  9479 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:50:50.815516  9636 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 7: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:50.807978 12661 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:50.808420  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:50.801560 12738 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:50.751492 13029 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:50.816329  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:50.821825  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:50.728247 13095 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.084s
I20260808 01:50:50.790277 13113 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:50.827963 13113 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:50.830294  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:50.831179  9634 raft_consensus.cc:2393] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:50.832065  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:50.832623  9634 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:50.837983  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:50.838009  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:50.839183  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:50.839785  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:50.785871 13086 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:50.824311 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:50.840240  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:50.856045 13113 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:50:50.857592  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:50.858217 13086 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:50.859304  9629 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:50.860374  9479 raft_consensus.cc:2410] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 9 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260808 01:50:50.860776 13086 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:50.841579  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:50.862071 13113 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:50.863397  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:50.864251 13113 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:50.864640 13113 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:50.865006 13113 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:50.789414 12804 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:50.867157 12804 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:50.868675 13096 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:50.869084  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:50.869313 13096 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:50.869594 13061 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:50.870916 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:50.871158 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:50.871444 10360 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260808 01:50:50.872766 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:50.873432 10365 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:50.873504 13012 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:50.869330 12804 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:50.783733 13115 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:50.789108 13102 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:50.876868  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:50.870245  9478 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:50.840240  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:50.888267  9631 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:50.889325  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:50.896306  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:50.858736  9634 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:50:50.921326  9524 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:50.871613 13096 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:50.933373  9524 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:50.934713 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 110508us
I20260808 01:50:50.878475 13061 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:50.936314 12942 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 01:50:50.937132 13099 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:50.939697 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:50.935650 12883 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:50.947588  9524 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:50.949656 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:50.899454 12693 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:50.953590 12826 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:50.898919 13078 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:50.961699 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:50.898408 12804 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:50.921099 12788 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:50.963649  9524 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:50.965723  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:50.966584 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:50.841743 12917 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:50.880812 13020 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:50.880509 12995 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:50:50.815626 13033 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:51.033005  9638 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:50.970124 12809 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:50.954430 12883 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:51.004086  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:50.964996 13112 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:50.966562 13107 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:51.017127  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:50.962831 12826 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:50.921700 13115 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:51.031184  9633 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:50:51.035097  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:51.015724  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52171.6us
I20260808 01:50:50.920733 13102 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:51.020548 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:51.047851  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:51.048611 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:51.048815 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:50.976253 13016 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:51.050427  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:51.022816 12942 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:51.048627  9524 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:51.053499  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:51.022192 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:51.040333 12809 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:51.042392 12883 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:51.058220  9638 raft_consensus.cc:2393] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:50.963222 13099 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:51.069818  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:51.072587  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:51.087970  9474 raft_consensus.cc:2393] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:51.088527  9638 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:50:51.058548  9524 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:51.079677 12934 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.605s	user 0.015s	sys 0.002s
I20260808 01:50:51.080050 13087 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:51.093693 13087 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:51.094200 13087 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:51.080534  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1950) took 1186 ms. Trace:
I20260808 01:50:51.099083  9636 rpcz_store.cc:276] 0808 01:50:49.893575 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:50.690096 (+796521us) service_pool.cc:224] Handling call
0808 01:50:51.079229 (+389133us) spinlock_profiling.cc:243] Waited 39.3 ms on lock 0x7b54000db1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:51.079434 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":90396416}
I20260808 01:50:51.100780  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:51.101823  9636 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:51.102221 13033 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:51.015918 12804 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.105193 12804 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:51.105499 13087 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.105986 12804 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:51.078088 12772 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:51.101182  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:51.108428  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:51.045802 13112 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.113296  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:50.969630 12693 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:51.090140  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:51.032032 13020 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:51.124105  9474 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260808 01:50:51.070356 12809 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:51.127269  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:51.120814 12745 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.002s	sys 0.044s
I20260808 01:50:51.098465 13102 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:51.109382  9638 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:51.117251 12804 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:51.049569 12826 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:51.038648 12917 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.117815  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:51.168655  9638 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260808 01:50:51.087718 12942 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:51.123351 12997 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:51.145448 13099 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:51.031816 12788 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.092744  9633 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:51.150996 10362 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260808 01:50:51.151571 13112 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:51.142408 13113 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:51.142763 12745 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.003s	sys 0.044s
I20260808 01:50:51.161573 13020 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:51.163523 12693 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:51.086831 13016 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.166328  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:51.051852 13107 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.169726  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1947) took 1283 ms. Trace:
I20260808 01:50:51.213404  9638 rpcz_store.cc:276] 0808 01:50:49.886467 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.167904 (+1281437us) service_pool.cc:224] Handling call
0808 01:50:51.168353 (+   449us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:51.169382 (+  1029us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:51.169652 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:51.183048  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:51.201280 13134 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:50:51.201591  9445 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:51.209223  9601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.023s	sys 0.016s
W20260808 01:50:51.220504  9601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.023s	sys 0.016s
W20260808 01:50:51.220932  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 551012us
W20260808 01:50:51.210448  9599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.015s	sys 0.024s
W20260808 01:50:51.221958  9599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.015s	sys 0.025s
W20260808 01:50:51.222342  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 558558us
W20260808 01:50:51.225039  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:51.219132  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:51.228691  9638 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260808 01:50:51.229643  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1959) took 1321 ms. Trace:
I20260808 01:50:51.230114  9638 rpcz_store.cc:276] 0808 01:50:49.908458 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.214498 (+1306040us) service_pool.cc:224] Handling call
0808 01:50:51.214878 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:51.229282 (+ 14404us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:51.229564 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:51.241017  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:51.252085 12788 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:51.253037 13134 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:51.253793 13134 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.256580 13134 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:51.257331 13134 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:51.259040 10363 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260808 01:50:51.264672 13113 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:51.265687 13113 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:51.266113 13113 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:51.238572 12995 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:51.271193 12995 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:51.271621 12995 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:51.275262 13107 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:51.277319 12756 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:51.277941 13115 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:51.245007 13078 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:50:51.279424 13078 raft_consensus.cc:697] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:51.280536 13016 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:51.280471 13078 consensus_queue.cc:237] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.281828 12926 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:51.197988 12997 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.286060 12997 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:51.247236 12937 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260808 01:50:51.288774 12661 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:51.290961 12942 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.242326  9638 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:51.293501  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1960) took 1384 ms. Trace:
I20260808 01:50:51.293907  9638 rpcz_store.cc:276] 0808 01:50:49.908765 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.241722 (+1332957us) service_pool.cc:224] Handling call
0808 01:50:51.242071 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:51.293199 (+ 51128us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:51.293430 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:51.300282  9638 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260808 01:50:51.306974  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:51.308756  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 374) took 1287 ms. Trace:
I20260808 01:50:51.309245  9638 rpcz_store.cc:276] 0808 01:50:50.021760 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.294978 (+1273218us) service_pool.cc:224] Handling call
0808 01:50:51.295320 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:51.301085 (+  5765us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:51.308663 (+  7578us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:51.325093 10332 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:51.326649 12745 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:50:51.327288 12745 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:51.327811 12745 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:51.257997 13134 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.266538 13113 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:51.342105 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:51.345184  9638 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:50:51.346069 10365 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:51.346273  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 375) took 1324 ms. Trace:
I20260808 01:50:51.346876  9638 rpcz_store.cc:276] 0808 01:50:50.022321 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.310087 (+1287766us) service_pool.cc:224] Handling call
0808 01:50:51.310472 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:51.345906 (+ 35434us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:51.346195 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:51.348804  9638 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260808 01:50:51.294489 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:51.357116 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:51.210963 13029 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.300238 12934 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:51.363680  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 377) took 1301 ms. Trace:
I20260808 01:50:51.364363  9638 rpcz_store.cc:276] 0808 01:50:50.062218 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.348000 (+1285782us) service_pool.cc:224] Handling call
0808 01:50:51.348481 (+   481us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:51.360332 (+ 11851us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:51.363479 (+  3147us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:51.365157 10366 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:51.370507 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:51.373112  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:51.397758 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:51.398128 13115 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:51.399233 13115 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:51.404927 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:51.407083 13115 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:51.425854 13134 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:51.427312 12917 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:51.440845  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46676us
W20260808 01:50:51.445462  9635 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.007s	sys 0.019s
W20260808 01:50:51.447137  9635 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.008s	sys 0.019s
W20260808 01:50:51.374307 13082 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.005s	sys 0.014s
W20260808 01:50:51.451057  9598 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.751s	user 0.008s	sys 0.017s
W20260808 01:50:51.538821  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 50830.6us
I20260808 01:50:51.469893  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1920) took 1571 ms. Trace:
W20260808 01:50:51.481396  9521 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.629s	user 0.019s	sys 0.017s
W20260808 01:50:51.542052  9521 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.690s	user 0.019s	sys 0.032s
I20260808 01:50:51.541344  9555 rpcz_store.cc:276] 0808 01:50:49.898801 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:49.899071 (+   270us) service_pool.cc:224] Handling call
0808 01:50:49.899381 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:49.933964 (+ 34583us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:51.469628 (+1535664us) spinlock_profiling.cc:243] Waited 33.8 ms on lock 0x7b540029efd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:51.469831 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":77711232}
W20260808 01:50:51.542657  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 691145us
W20260808 01:50:51.456566  9472 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2051) took 1734 ms (client timeout 1676 ms). Trace:
W20260808 01:50:51.544402  9472 rpcz_store.cc:269] 0808 01:50:49.721817 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:49.751729 (+ 29912us) service_pool.cc:224] Handling call
0808 01:50:51.456124 (+1704395us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540021d254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:51.456398 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2317290880}
I20260808 01:50:51.464443  9630 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260808 01:50:51.547648  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 379) took 1432 ms. Trace:
W20260808 01:50:51.548034  9521 leader_election.cc:336] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.162s (SENT)
W20260808 01:50:51.550496  9598 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.851s	user 0.009s	sys 0.017s
I20260808 01:50:51.548516  9630 rpcz_store.cc:276] 0808 01:50:50.115539 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.463662 (+1348123us) service_pool.cc:224] Handling call
0808 01:50:51.464129 (+   467us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:51.547203 (+ 83074us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:51.547545 (+   342us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:51.288861 12804 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.557958  9521 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:50:51.451792 13082 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.005s	sys 0.015s
I20260808 01:50:51.576133  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:51.379156 12973 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:51.588035 12973 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:51.591238 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:51.595108 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:51.582479  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:51.446846 13033 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:51.584102 13112 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:51.585114 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:51.612596 13112 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:51.613236 13112 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:51.586189 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:51.586994 13138 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:51.615361 13138 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.439759 13015 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:51.372735 12934 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:51.550830 12864 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:51.468076 13086 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:50:51.552403 12856 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:51.465991 12971 raft_consensus.cc:3055] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:50:51.619230 13015 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:51.599135 12693 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:51.571982 12883 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:51.550918 13089 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:51.618615 13138 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:51.625757 12934 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:51.626752 12864 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.624117 13033 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:51.658212 13015 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:51.642729 13012 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260808 01:50:51.679255  9481 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 452) took 1553 ms (client timeout 1589 ms). Trace:
I20260808 01:50:51.604843  9521 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:50:51.680581  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 75840.6us
I20260808 01:50:51.650993 12954 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:51.652275  9632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1940) took 1768 ms (client timeout 2162 ms). Trace:
I20260808 01:50:51.652566  9637 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 4. (term mismatch)
I20260808 01:50:51.681790  9637 pending_rounds.cc:85] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 3
I20260808 01:50:51.682245  9637 pending_rounds.cc:107] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260808 01:50:51.561767 13134 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:51.690120  9634 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.009s	sys 0.041s
W20260808 01:50:51.690701  9634 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.009s	sys 0.041s
W20260808 01:50:51.681689  9632 rpcz_store.cc:269] 0808 01:50:49.884321 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:49.885343 (+  1022us) service_pool.cc:224] Handling call
0808 01:50:51.651939 (+1766596us) spinlock_profiling.cc:243] Waited 769 ms on lock 0x7b54003f9b54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:51.652188 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1767834368}
W20260808 01:50:51.652798 12998 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.024s	sys 0.029s
I20260808 01:50:51.559084 12917 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.095s
I20260808 01:50:51.667197 13114 raft_consensus.cc:3055] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:50:51.557121 12997 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.099s
I20260808 01:50:51.561688 13016 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.095s
W20260808 01:50:51.655742 10360 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.020s	sys 0.034s
W20260808 01:50:51.669242  9523 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.012s	sys 0.034s
I20260808 01:50:51.646613 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:51.672559 10367 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.012s	sys 0.034s
I20260808 01:50:51.682719  9637 raft_consensus.cc:2889] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:51.678597 13033 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:51.678853 13015 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:51.679869  9481 rpcz_store.cc:269] 0808 01:50:50.125792 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:50.180079 (+ 54287us) service_pool.cc:224] Handling call
0808 01:50:51.679016 (+1498937us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b54002bc0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:51.679173 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1099850624}
I20260808 01:50:51.663569 12693 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:51.648504 13086 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:50:51.709231 10360 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.021s	sys 0.034s
I20260808 01:50:51.611600 12973 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:51.657800 12971 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:51.696728  9632 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: []
W20260808 01:50:51.715238 13061 consensus_meta.cc:287] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.748s	user 0.012s	sys 0.002s
I20260808 01:50:51.708356  9637 pending_rounds.cc:107] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260808 01:50:51.732645  9637 raft_consensus.cc:2889] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:51.683817  9445 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:51.734107  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 50410.7us
I20260808 01:50:51.720160 12693 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:51.708756 10367 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.013s	sys 0.035s
W20260808 01:50:51.695612 12998 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.655s	user 0.024s	sys 0.030s
I20260808 01:50:51.692589 13107 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:51.698127 13114 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:50:51.704571  9523 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.012s	sys 0.035s
I20260808 01:50:51.712849 12864 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:51.672845 13089 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.718361 13016 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260808 01:50:51.698228 13134 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.698982 12954 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:51.759734  9632 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260808 01:50:51.776371  9446 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.009s	sys 0.026s
I20260808 01:50:51.658485 12856 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:51.768258  9524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.013s	sys 0.030s
I20260808 01:50:51.778689  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:51.761049  9443 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.019s	sys 0.015s
I20260808 01:50:51.747865 12973 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:51.778651  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:51.782112  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:51.782879  9560 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:51.782191  9558 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:51.784036  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:51.784797  9443 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.019s	sys 0.015s
W20260808 01:50:51.778493  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.009s	sys 0.041s
W20260808 01:50:51.787276  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.010s	sys 0.041s
W20260808 01:50:51.787674  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 546739us
I20260808 01:50:51.782406  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:51.778726  9524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.013s	sys 0.030s
I20260808 01:50:51.788518  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:51.788684  9636 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.018s	sys 0.040s
W20260808 01:50:51.790921  9636 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.019s	sys 0.040s
I20260808 01:50:51.788638  9556 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
W20260808 01:50:51.791095  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 700032us
W20260808 01:50:51.777221  9446 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.010s	sys 0.026s
I20260808 01:50:51.792414  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:51.803041  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:51.794863  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:51.797389  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:51.820230  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:51.820958  9481 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260808 01:50:51.822427  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:51.771464 13107 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:51.832170  9551 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260808 01:50:51.834636  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:51.835274  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:51.835856  9481 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:51.837332  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:51.838618  9552 raft_consensus.cc:2410] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260808 01:50:51.840898  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:51.843781  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:51.813477 12738 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.020s	sys 0.015s
W20260808 01:50:51.865644 12738 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.021s	sys 0.015s
I20260808 01:50:51.835094 13114 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:51.857546  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:51.867509 13114 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:51.867579  9477 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:51.794855 12917 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:51.871676 12917 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:51.872218 12917 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:51.872722 12917 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:50:51.832963 12826 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.634s	user 0.011s	sys 0.002s
I20260808 01:50:51.782752 12973 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:51.860821  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:51.880182 13144 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
W20260808 01:50:51.881551 13142 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.164s	user 0.000s	sys 0.001s
W20260808 01:50:51.882778 12809 consensus_meta.cc:287] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.735s	user 0.013s	sys 0.000s
W20260808 01:50:51.962913  9599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.011s	sys 0.043s
W20260808 01:50:51.963467  9599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.011s	sys 0.043s
W20260808 01:50:51.832085 12995 consensus_meta.cc:287] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.560s	user 0.010s	sys 0.004s
W20260808 01:50:51.963806  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 522126us
I20260808 01:50:51.964076 12995 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:51.965507  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:50:51.888761  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:51.891355  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:51.892305  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:51.973263  9476 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:50:51.893471  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:51.859232  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:51.776625 12864 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20260808 01:50:51.912662  9479 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:51.833230 13082 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:51.755813 13086 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:51.805706 13140 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:51.805109 12997 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:51.832690 13115 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:50:51.860324 10362 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:50:51.758160 13061 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.070s
I20260808 01:50:51.884282  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:51.796013 13102 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.021s	sys 0.008s
I20260808 01:50:51.826990 13073 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:51.941200  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:51.962527  9560 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:50:51.884925 13138 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:51.939898 13103 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.026s	sys 0.026s
I20260808 01:50:51.966408  9599 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:51.979287  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:52.012465  9475 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 7: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:51.981000  9474 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:50:51.941339 12942 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.011s	sys 0.014s
I20260808 01:50:51.982478  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:52.018961 13159 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:52.020047  9552 raft_consensus.cc:2393] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 9: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:52.021137 13159 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:51.925755 13079 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:50:52.022935 13079 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:52.023464 13141 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260808 01:50:52.023654 13079 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.775803 12971 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:52.027460 12971 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:51.983601  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:52.035033  9631 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:52.036139  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:52.037246  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:52.037846  9631 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:50:51.933733 12973 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:52.026485 13079 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:52.041080 13079 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:51.991923 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:51.997515  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:52.042846 12942 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.012s	sys 0.015s
I20260808 01:50:52.043017  9478 raft_consensus.cc:2410] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260808 01:50:52.044299  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:51.969343 13078 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:52.045017  9631 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:52.045455 13078 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:52.045996 13078 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:52.041816 13079 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:52.048120  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:52.049667 13079 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:52.046272  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:52.017205  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:52.051438  9631 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:52.052899  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:52.053556  9601 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:52.017386  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:52.054811  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:52.055284  9476 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:50:52.056355  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:52.011154 13103 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.026s	sys 0.026s
I20260808 01:50:52.016165  9560 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:50:51.976653 12883 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:51.943032 12826 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:52.059185  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:52.060309  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 93961.2us
I20260808 01:50:52.071201  9479 raft_consensus.cc:2410] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260808 01:50:51.983975 10363 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260808 01:50:52.042876 10365 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 9: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:52.073388  9599 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:52.071163  9631 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:50:52.059543  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:52.003587 13102 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.838s	user 0.021s	sys 0.008s
I20260808 01:50:52.079180  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:52.010273 13113 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.651s	user 0.015s	sys 0.001s
I20260808 01:50:51.966486 12995 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:51.992277 13145 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:51.971046 12809 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:51.996549 13082 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:52.002506 13073 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:52.099097  9633 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.004s	sys 0.025s
I20260808 01:50:52.013394 13115 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:50:52.127692  9630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.015s	sys 0.023s
I20260808 01:50:52.063339 13086 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:52.129256  9559 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:52.128780  9633 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.004s	sys 0.025s
W20260808 01:50:52.133635  9630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.015s	sys 0.024s
I20260808 01:50:52.133917  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1946) took 2247 ms. Trace:
I20260808 01:50:52.137338  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 454) took 1349 ms. Trace:
I20260808 01:50:52.134296  9635 rpcz_store.cc:276] 0808 01:50:49.886211 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:50.778037 (+891826us) service_pool.cc:224] Handling call
0808 01:50:50.813720 (+ 35683us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:50.813979 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:50.814002 (+    23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:50.817708 (+  3706us) log.cc:844] Serialized 26 byte log entry
0808 01:50:51.447922 (+630214us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:51.447994 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:51.448107 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:52.133668 (+685561us) raft_consensus.cc:1685] finished
0808 01:50:52.133693 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:52.133848 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":618192,"threads_started":1,"wal-append.queue_time_us":949990}
I20260808 01:50:52.144253  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:52.147459  9600 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.017s	sys 0.020s
W20260808 01:50:52.147895  9600 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.017s	sys 0.020s
W20260808 01:50:52.148573  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 561299us
I20260808 01:50:52.137801  9473 rpcz_store.cc:276] 0808 01:50:50.787740 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:50.793320 (+  5580us) service_pool.cc:224] Handling call
0808 01:50:50.793709 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:50.823151 (+ 29442us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:50.823189 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:50.897683 (+ 74494us) log.cc:844] Serialized 26 byte log entry
0808 01:50:51.301709 (+404026us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:50:51.301770 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:51.301861 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:52.137075 (+835214us) raft_consensus.cc:1685] finished
0808 01:50:52.137104 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:52.137270 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29023,"raft.queue_time_us":34942,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":147,"thread_start_us":347975,"threads_started":1,"wal-append.queue_time_us":730334}
I20260808 01:50:52.154480  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:52.155840  9554 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:52.004069 13140 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:52.008785 12997 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:52.196123  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 378) took 2127 ms. Trace:
I20260808 01:50:52.196529  9638 rpcz_store.cc:276] 0808 01:50:50.069207 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.366705 (+1297498us) service_pool.cc:224] Handling call
0808 01:50:51.368170 (+  1465us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:51.368481 (+   311us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:50:51.749807 (+381326us) log.cc:844] Serialized 14 byte log entry
0808 01:50:52.195586 (+445779us) log.cc:844] Serialized 14 byte log entry
0808 01:50:52.195762 (+   176us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:52.195810 (+    48us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:50:52.195857 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:52.195925 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:52.196062 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5147,"prepare.queue_time_us":678630,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"thread_start_us":792059,"threads_started":2}
I20260808 01:50:52.133239 13073 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:52.203329  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:52.209560  9554 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:52.067655 13138 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:52.216440  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:52.138407 13012 raft_consensus.cc:3055] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:52.131908 12856 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:52.137212 12954 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:52.147086 13089 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:52.100049 13113 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:52.251235  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:52.266589  9560 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:50:52.262105  9523 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:52.265165  9475 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:50:52.195420 13159 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:52.061554 13107 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:52.207458 12676 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:52.127149 13145 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:52.136854 12995 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.115s
W20260808 01:50:52.304067 10332 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.711s	user 0.025s	sys 0.025s
W20260808 01:50:52.310329 10332 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.025s	sys 0.025s
W20260808 01:50:52.311334 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 718821us
W20260808 01:50:52.312513  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:52.313858 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:52.314442 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:52.250437 13012 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:52.316300  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1958) took 2408 ms. Trace:
W20260808 01:50:52.336254 10361 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.023s	sys 0.014s
W20260808 01:50:52.393186 10361 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.024s	sys 0.016s
I20260808 01:50:52.394409  9637 rpcz_store.cc:276] 0808 01:50:49.908181 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.205762 (+1297581us) service_pool.cc:224] Handling call
0808 01:50:51.652140 (+446378us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:51.733389 (+ 81249us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:52.316054 (+582665us) spinlock_profiling.cc:243] Waited 446 ms on lock 0x7b54003f9b50. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:52.316220 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1025624832}
W20260808 01:50:52.383751 10367 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.629s	user 0.028s	sys 0.009s
I20260808 01:50:52.196657 12819 consensus_queue.cc:1059] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.088s
W20260808 01:50:52.396979 10367 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.029s	sys 0.010s
I20260808 01:50:52.273093 13089 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:52.404711  9637 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:52.402740  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:52.398375  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:52.403548  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:52.286511 13159 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:52.305945 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:52.335903 10360 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.014s	sys 0.023s
I20260808 01:50:52.321657 13134 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:52.397830 12661 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.108s	user 0.015s	sys 0.002s
I20260808 01:50:52.385013 12856 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:52.434230 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:52.435034 12826 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:52.436252 12937 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:52.438337 13140 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:52.441769 12745 consensus_meta.cc:287] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.113s	user 0.016s	sys 0.000s
I20260808 01:50:52.421852 13115 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:52.448071  9636 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.653s	user 0.025s	sys 0.020s
I20260808 01:50:52.464222  9629 raft_consensus.cc:2410] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260808 01:50:52.397859 12883 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:52.456979 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 63189.7us
W20260808 01:50:52.420564 10360 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.015s	sys 0.024s
I20260808 01:50:52.411888 13159 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:52.497885 10332 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:52.490414 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:52.490586  9636 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.026s	sys 0.021s
I20260808 01:50:52.494186  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:52.497402  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.688s	user 0.055s	sys 0.073s
W20260808 01:50:52.501067  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.056s	sys 0.073s
W20260808 01:50:52.501302  9557 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.706s	user 0.010s	sys 0.018s
W20260808 01:50:52.501520  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 692032us
I20260808 01:50:52.502872  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1953) took 2608 ms. Trace:
I20260808 01:50:52.480971 12937 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:52.416044 13089 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:52.486179 10366 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.007s	sys 0.027s
W20260808 01:50:52.513530 10366 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.731s	user 0.008s	sys 0.027s
W20260808 01:50:52.066023 13036 consensus_queue.cc:1175] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6e7856bed1be43acaa821290d506ee97" responder_term: 9 status { last_received { term: 8 index: 8 } last_committed_idx: 8 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 01:50:52.491914 12661 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:52.491381 12826 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:52.495527 13016 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.014s	sys 0.034s
W20260808 01:50:52.529385  9551 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.016s	sys 0.019s
W20260808 01:50:52.531316  9551 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.017s	sys 0.019s
W20260808 01:50:52.501735  9557 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.010s	sys 0.018s
I20260808 01:50:52.509866  9636 rpcz_store.cc:276] 0808 01:50:49.894709 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.170173 (+1275464us) service_pool.cc:224] Handling call
0808 01:50:51.170544 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:51.170846 (+   302us) raft_consensus.cc:1534] Early marking committed up to index 5
0808 01:50:51.793545 (+622699us) log.cc:844] Serialized 14 byte log entry
0808 01:50:52.502338 (+708793us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:52.502418 (+    80us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:52.502481 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:52.502577 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:52.502786 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":825770,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"thread_start_us":1294505,"threads_started":2}
I20260808 01:50:52.481330 12856 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:52.495365 13164 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:50:52.555027  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:52.449851 13082 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:52.556304  9552 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:52.511471 12997 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:52.509424 12954 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:52.515443 12937 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:50:52.529134 12756 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s	user 0.013s	sys 0.014s
W20260808 01:50:52.575209  9524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.775s	user 0.011s	sys 0.004s
W20260808 01:50:52.536105 13016 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.754s	user 0.015s	sys 0.034s
W20260808 01:50:52.576426  9478 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.010s	sys 0.006s
I20260808 01:50:52.498889 12883 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:52.566740  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.004s	sys 0.020s
I20260808 01:50:52.506222 13012 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:52.558823 13164 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:52.491894 12745 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:52.537593 12809 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:52.626585  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:52.650558  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.709s	user 0.005s	sys 0.022s
W20260808 01:50:52.523813 12804 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.927s	user 0.014s	sys 0.017s
W20260808 01:50:52.652782  9478 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.012s	sys 0.007s
W20260808 01:50:52.580050  9598 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.021s	user 0.003s	sys 0.012s
W20260808 01:50:52.648042  9477 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.004s	sys 0.014s
W20260808 01:50:52.713431  9598 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.155s	user 0.007s	sys 0.012s
W20260808 01:50:52.713516  9477 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.004s	sys 0.015s
W20260808 01:50:52.617790 13162 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.382s	user 0.000s	sys 0.001s
W20260808 01:50:52.662833  9631 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.009s	sys 0.025s
W20260808 01:50:52.557627 13152 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.426s	user 0.001s	sys 0.001s
W20260808 01:50:52.663811  9633 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.009s	sys 0.025s
W20260808 01:50:52.745430  9633 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.009s	sys 0.025s
I20260808 01:50:52.580504 12934 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:52.573939 10360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 509) took 1618 ms. Trace:
I20260808 01:50:52.632176 13138 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:52.722187 12804 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.125s	user 0.015s	sys 0.017s
I20260808 01:50:52.556583 12826 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:52.785557 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 479) took 1047 ms. Trace:
I20260808 01:50:52.786123 10361 rpcz_store.cc:276] 0808 01:50:51.737776 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.737992 (+   216us) service_pool.cc:224] Handling call
0808 01:50:51.738298 (+   306us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:51.738694 (+   396us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:51.738725 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:51.739220 (+   495us) log.cc:844] Serialized 26 byte log entry
0808 01:50:52.393846 (+654626us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:52.393912 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:52.394049 (+   137us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:52.785294 (+391245us) raft_consensus.cc:1685] finished
0808 01:50:52.785325 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:52.785484 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1156,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":16242,"thread_start_us":654172,"threads_started":1,"wal-append.queue_time_us":1044130}
W20260808 01:50:52.787289 13172 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260808 01:50:52.790731 13029 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:52.799382 13161 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.219s	user 0.000s	sys 0.001s
I20260808 01:50:52.802791 13012 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:50:52.556828 12738 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.007s	sys 0.020s
I20260808 01:50:52.710264 13145 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:52.696609  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 546644us
W20260808 01:50:52.580183 13112 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.966s	user 0.014s	sys 0.002s
W20260808 01:50:52.653827  9601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.006s	sys 0.011s
W20260808 01:50:52.834895  9601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.756s	user 0.007s	sys 0.011s
I20260808 01:50:52.764607 12997 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:52.835285  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 775750us
W20260808 01:50:52.574432 12778 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.000s	sys 0.018s
W20260808 01:50:52.574782 13139 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.007s	sys 0.008s
I20260808 01:50:52.568331 12995 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:50:52.720685  9631 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.010s	sys 0.025s
W20260808 01:50:52.743521  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:52.816011 12738 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.949s	user 0.008s	sys 0.021s
I20260808 01:50:52.570740 13169 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:52.629323 13113 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:52.653398 12998 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.816s	user 0.000s	sys 0.015s
I20260808 01:50:52.769825  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:52.577910 12756 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.689s	user 0.014s	sys 0.014s
I20260808 01:50:52.770947 10369 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:52.655337 13164 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:52.719659 13140 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:50:52.771196 13015 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.044s	user 0.013s	sys 0.004s
W20260808 01:50:52.825959  9444 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.194:36095 timed out after 1.590s (SENT) [suppressed 14 similar messages]
W20260808 01:50:52.578820  9524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.779s	user 0.011s	sys 0.005s
I20260808 01:50:52.834904 13134 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:52.832521  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:52.840190  9474 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260808 01:50:52.843493  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:52.744470 13175 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election won for term 9
W20260808 01:50:52.809273 13171 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.166s	user 0.000s	sys 0.000s
I20260808 01:50:52.808396 12942 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Stepping down as leader of term 7
W20260808 01:50:52.852211 13103 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.794s	user 0.012s	sys 0.015s
W20260808 01:50:52.841449 12778 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.856s	user 0.000s	sys 0.021s
I20260808 01:50:52.861106  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1934) took 1018 ms. Trace:
W20260808 01:50:52.872162 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 374348us
I20260808 01:50:52.879303  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1951) took 2985 ms. Trace:
I20260808 01:50:52.879773  9521 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:50:52.849203 13139 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.848s	user 0.007s	sys 0.009s
W20260808 01:50:52.885288  9635 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.029s	sys 0.026s
W20260808 01:50:52.887844  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 1.08831e+06us
I20260808 01:50:52.914877  9634 rpcz_store.cc:276] 0808 01:50:49.893976 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.164014 (+1270038us) service_pool.cc:224] Handling call
0808 01:50:51.164511 (+   497us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:51.165006 (+   495us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:51.165044 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:51.165720 (+   676us) log.cc:844] Serialized 26 byte log entry
0808 01:50:51.691127 (+525407us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:51.691231 (+   104us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:51.691316 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:52.878970 (+1187654us) raft_consensus.cc:1685] finished
0808 01:50:52.879005 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:52.879210 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":130,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"thread_start_us":524899,"threads_started":1,"wal-append.queue_time_us":827072}
I20260808 01:50:52.830991  9600 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:52.867871 12998 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.042s	user 0.000s	sys 0.015s
I20260808 01:50:52.899636  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:52.897277 13102 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.759s	user 0.012s	sys 0.022s
I20260808 01:50:52.908514 10366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 478) took 1171 ms. Trace:
I20260808 01:50:52.876174 13087 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:52.918277  9474 raft_consensus.cc:2376] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6. Re-sending same reply.
W20260808 01:50:52.928167  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 97572.6us
I20260808 01:50:52.929703  9600 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:52.902619  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:52.931209  9552 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:52.925858 10366 rpcz_store.cc:276] 0808 01:50:51.737442 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.737595 (+   153us) service_pool.cc:224] Handling call
0808 01:50:51.737991 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:51.738425 (+   434us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:51.738466 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:51.752612 (+ 14146us) log.cc:844] Serialized 26 byte log entry
0808 01:50:52.514232 (+761620us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:50:52.514295 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:52.514404 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:52.908170 (+393766us) raft_consensus.cc:1685] finished
0808 01:50:52.908197 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:52.908426 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5287,"raft.queue_time_us":559,"raft.run_cpu_time_us":528,"raft.run_wall_time_us":17451,"spinlock_wait_cycles":1896320,"thread_start_us":732032,"threads_started":1,"wal-append.queue_time_us":996421}
I20260808 01:50:52.897359 13134 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:52.933729  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 1933) took 1141 ms. Trace:
I20260808 01:50:52.934149  9559 rpcz_store.cc:276] 0808 01:50:51.792674 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.799145 (+  6471us) service_pool.cc:224] Handling call
0808 01:50:52.933486 (+1134341us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b5400325fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:52.933662 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":355227904}
W20260808 01:50:52.935184 13073 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.681s	user 0.009s	sys 0.003s
I20260808 01:50:52.935580 13134 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:52.936357 13176 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.171s	user 0.001s	sys 0.000s
I20260808 01:50:52.937278  9556 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260808 01:50:52.894531 13089 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:52.914876 13175 raft_consensus.cc:697] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:52.893314 13015 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:52.907802 13103 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.850s	user 0.012s	sys 0.016s
I20260808 01:50:52.921567  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:52.848421 13112 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:52.935468  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:52.896970 12661 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:52.944649  9555 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:52.850297 12997 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:52.913123  9551 rpcz_store.cc:276] 0808 01:50:51.842451 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.842637 (+   186us) service_pool.cc:224] Handling call
0808 01:50:51.843084 (+   447us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:51.843387 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 4
0808 01:50:52.533586 (+690199us) log.cc:844] Serialized 14 byte log entry
0808 01:50:52.860588 (+327002us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:52.860674 (+    86us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:52.860739 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:52.860838 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:52.861020 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":281978,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":54,"raft.queue_time_us":131,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":19267,"thread_start_us":926808,"threads_started":2}
I20260808 01:50:52.869558 12995 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:52.950240  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:52.964246 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:52.966238 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:52.971117 10361 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:52.901621 12942 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:52.943056 10369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 480) took 1027 ms. Trace:
W20260808 01:50:52.968089  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:52.972929  9445 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:52.979544 10369 rpcz_store.cc:276] 0808 01:50:51.915686 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.915890 (+   204us) service_pool.cc:224] Handling call
0808 01:50:51.916370 (+   480us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:52.868491 (+952121us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:52.942790 (+ 74299us) spinlock_profiling.cc:243] Waited 854 ms on lock 0x7b5400594054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:52.942988 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1964814208}
W20260808 01:50:52.982489 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:52.982834  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:52.983527  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:52.946882  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2032) took 2194 ms. Trace:
W20260808 01:50:52.925912 13102 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.793s	user 0.012s	sys 0.022s
I20260808 01:50:52.927233 13169 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:52.938184 12917 consensus_meta.cc:287] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.065s	user 0.011s	sys 0.001s
W20260808 01:50:52.990934  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 111254us
W20260808 01:50:52.991390  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 69924.8us
I20260808 01:50:53.021466  9521 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:52.991999 12819 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:52.992121 13204 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:52.992699 13015 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:52.986904  9629 rpcz_store.cc:276] 0808 01:50:50.752587 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.117171 (+364584us) service_pool.cc:224] Handling call
0808 01:50:52.946609 (+1829438us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b54001b44d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:52.946817 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3071974272}
I20260808 01:50:52.939180 12745 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:52.939463 12826 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:52.993697 12995 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:53.052767  9476 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.851s	user 0.005s	sys 0.009s
W20260808 01:50:53.053232  9476 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.859s	user 0.006s	sys 0.009s
I20260808 01:50:52.942891 12856 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:52.945596 13180 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.068s
W20260808 01:50:52.995122 13181 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
W20260808 01:50:52.996147 13196 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260808 01:50:52.999106 13206 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:50:53.003620  9552 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
W20260808 01:50:53.105325  9599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.023s	user 0.009s	sys 0.025s
I20260808 01:50:52.944396 13192 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:50:53.010010  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1975) took 1203 ms. Trace:
I20260808 01:50:53.017649  9558 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:52.940349 12883 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:53.021343  9524 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:52.970786 12661 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:53.022794  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:53.039027 13156 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:50:53.039417 13175 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:53.040773 12738 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:53.023968 12955 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:52.979539 13034 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:53.049634  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 381) took 2392 ms. Trace:
I20260808 01:50:53.135354  9633 rpcz_store.cc:276] 0808 01:50:50.657044 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.473731 (+816687us) service_pool.cc:224] Handling call
0808 01:50:51.474086 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:51.474347 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:50:52.133809 (+659462us) log.cc:844] Serialized 14 byte log entry
0808 01:50:53.049135 (+915326us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:53.049225 (+    90us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:53.049285 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:53.049381 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:53.049558 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4944,"prepare.queue_time_us":573009,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":143119,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":6735,"thread_start_us":1573892,"threads_started":3,"wal-append.queue_time_us":827596}
I20260808 01:50:53.138023  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:52.923285 13150 consensus_queue.cc:1059] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.111s
W20260808 01:50:52.994006 12693 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.232s	user 0.015s	sys 0.001s
I20260808 01:50:53.097985  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2290) took 1161 ms. Trace:
I20260808 01:50:53.100381  9636 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260808 01:50:53.102383  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:53.179227  9637 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:50:53.179823 12997 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:53.180565  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:53.105667 10369 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:50:52.893879 13140 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:53.024001 12819 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:53.185421 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:53.186564  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2043) took 2383 ms. Trace:
I20260808 01:50:53.188467  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:53.124413 13012 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:53.187567  9630 rpcz_store.cc:276] 0808 01:50:50.803022 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.572149 (+769127us) service_pool.cc:224] Handling call
0808 01:50:51.592068 (+ 19919us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:52.134143 (+542075us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:52.134176 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:52.147358 (+ 13182us) log.cc:844] Serialized 26 byte log entry
0808 01:50:52.654520 (+507162us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:52.654600 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:52.654777 (+   177us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:53.186207 (+531430us) raft_consensus.cc:1685] finished
0808 01:50:53.186243 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:53.186463 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":193696,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":12367,"thread_start_us":892680,"threads_started":2,"wal-append.queue_time_us":595714}
I20260808 01:50:53.107209  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2054) took 2300 ms. Trace:
I20260808 01:50:53.042373 13177 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:53.115294  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:52.935712 13073 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:53.117170  9599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.035s	user 0.009s	sys 0.026s
I20260808 01:50:53.119940  9638 rpcz_store.cc:276] 0808 01:50:51.806396 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:52.198652 (+392256us) service_pool.cc:224] Handling call
0808 01:50:52.199032 (+   380us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:52.199297 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:52.199333 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:52.243390 (+ 44057us) log.cc:844] Serialized 26 byte log entry
0808 01:50:52.655206 (+411816us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:50:52.800340 (+145134us) log.cc:844] Serialized 14 byte log entry
0808 01:50:52.816897 (+ 16557us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:52.816983 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:53.009691 (+192708us) raft_consensus.cc:1685] finished
0808 01:50:53.009721 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:53.009924 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16285,"prepare.queue_time_us":144949,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":43,"raft.queue_time_us":117264,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":15364,"thread_start_us":313624,"threads_started":1,"wal-append.queue_time_us":619014}
I20260808 01:50:53.197290  9629 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:53.198854  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:53.055673 13195 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:53.198036  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 1.12473e+06us
W20260808 01:50:53.111927 13078 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.065s	user 0.009s	sys 0.005s
I20260808 01:50:53.129233 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:53.129608  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 106941us
I20260808 01:50:53.203151  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:53.203814  9559 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:53.042325 12917 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.139858  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:53.156108 10331 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.691s	user 0.018s	sys 0.018s
W20260808 01:50:53.205113 10331 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.018s	sys 0.018s
I20260808 01:50:53.156303  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:53.175488  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:53.205641  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:53.206132  9551 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:53.206450  9638 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:50:53.200091  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:53.027262 13169 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:53.134505 12826 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:50:53.203642  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:53.128943  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 107716us
I20260808 01:50:53.205771  9552 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:53.207737  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:53.186594 10364 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:53.225795  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:53.230460  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:53.157289 13205 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:53.176226  9478 rpcz_store.cc:276] 0808 01:50:51.936979 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:52.044040 (+107061us) service_pool.cc:224] Handling call
0808 01:50:52.044437 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:52.044702 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:50:52.712749 (+668047us) log.cc:844] Serialized 14 byte log entry
0808 01:50:53.097390 (+384641us) log.cc:844] Serialized 14 byte log entry
0808 01:50:53.097586 (+   196us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:53.097640 (+    54us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:53.097688 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:53.097765 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:53.097913 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51142,"prepare.queue_time_us":445948,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":86,"raft.queue_time_us":63081,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":7478,"spinlock_wait_cycles":20785408,"thread_start_us":1000228,"threads_started":2}
I20260808 01:50:52.769779 10360 rpcz_store.cc:276] 0808 01:50:50.951883 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:50.964081 (+ 12198us) service_pool.cc:224] Handling call
0808 01:50:51.042532 (+ 78451us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:51.050164 (+  7632us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:50:51.719647 (+669483us) log.cc:844] Serialized 14 byte log entry
0808 01:50:52.506455 (+786808us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:52.512193 (+  5738us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:50:52.518647 (+  6454us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:52.527407 (+  8760us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:52.570208 (+ 42801us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":950614,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":29872,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":22461,"thread_start_us":1426713,"threads_started":2,"wal-append.queue_time_us":852795}
I20260808 01:50:53.211575  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:53.208223 10361 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:53.208087 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:53.191598  9631 rpcz_store.cc:276] 0808 01:50:50.806352 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:52.079186 (+1272834us) service_pool.cc:224] Handling call
0808 01:50:52.079589 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:52.079905 (+   316us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:50:52.122306 (+ 42401us) log.cc:844] Serialized 14 byte log entry
0808 01:50:53.106764 (+984458us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:53.106839 (+    75us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:53.106905 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:53.106992 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:53.107136 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7661,"prepare.queue_time_us":57711,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":7854,"raft.queue_time_us":77318,"spinlock_wait_cycles":27373952,"thread_start_us":983747,"threads_started":2,"wal-append.queue_time_us":842952}
I20260808 01:50:53.196329 12819 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:53.219620  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:53.219820  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:53.220685  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:53.134851 12955 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:53.224578  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:53.232800 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:53.232949  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:53.209447  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:53.234934  9636 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:53.246810  9556 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 9: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:53.202354 13102 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260808 01:50:53.196759 13177 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:53.214138  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:53.240456  9472 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:53.258505  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:53.191956 13150 consensus_queue.cc:1243] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260808 01:50:53.236968  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:53.202647 13034 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:53.255491  9629 raft_consensus.cc:2376] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7. Re-sending same reply.
I20260808 01:50:53.157646 12738 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.257880 10367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 510) took 2286 ms. Trace:
W20260808 01:50:53.259454  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45452.2us
I20260808 01:50:53.191151 12693 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.221637 12997 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:53.257112  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:53.270160 10367 rpcz_store.cc:276] 0808 01:50:50.956941 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:50.957148 (+   207us) service_pool.cc:224] Handling call
0808 01:50:51.015037 (+ 57889us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:51.015355 (+   318us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:50:51.749360 (+734005us) log.cc:844] Serialized 14 byte log entry
0808 01:50:53.212963 (+1463603us) log.cc:844] Serialized 14 byte log entry
0808 01:50:53.242028 (+ 29065us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:53.242178 (+   150us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:50:53.242317 (+   139us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:53.242581 (+   264us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:53.242724 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":87379,"prepare.queue_time_us":951846,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":3744,"raft.queue_time_us":91670,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":6000,"spinlock_wait_cycles":10961664,"thread_start_us":2119664,"threads_started":4,"wal-append.queue_time_us":877986}
I20260808 01:50:53.284368  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:53.233842 13205 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:53.278424  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 452493us
I20260808 01:50:53.219816 13078 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:53.261680 12955 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:53.288111 13140 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:53.288977  9550 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 8: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:53.288750 12997 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:53.291163 13193 log.cc:927] Time spent T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.240s	user 0.000s	sys 0.001s
W20260808 01:50:53.291590 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 376398us
I20260808 01:50:53.260869 13103 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:53.264788 13177 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:53.281680  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:53.276739  9600 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:53.261943 13211 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:53.290259 12693 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:53.226444 13195 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.268450 13180 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:50:53.230742 13192 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260808 01:50:53.207266 13012 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:53.306273  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 73417.4us
I20260808 01:50:53.368799  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:53.303530 10360 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:53.303709  9634 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260808 01:50:53.336511  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:53.336040  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:53.337141 10332 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:53.267226 13034 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.313035 12864 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:53.374636  9633 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 7: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:53.298697 12856 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:53.307044 13112 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:53.335513 13033 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.612s	user 0.015s	sys 0.004s
I20260808 01:50:53.302521 13205 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:53.399066  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:53.316045 12756 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:53.303350 13164 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.415211  9631 raft_consensus.cc:2376] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9. Re-sending same reply.
I20260808 01:50:53.361919 13180 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:53.313843 13219 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:53.421669  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:53.422241  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:53.422658  9630 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:53.423120  9633 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:53.406019  9554 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:53.399474  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:53.411604  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 116303us
W20260808 01:50:53.411963 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 74939.4us
W20260808 01:50:53.411767 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:53.362630 13029 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.571s	user 0.015s	sys 0.003s
W20260808 01:50:53.407851 13187 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.303s	user 0.000s	sys 0.001s
W20260808 01:50:53.398077 13181 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
W20260808 01:50:53.361443 13194 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.316s	user 0.000s	sys 0.001s
I20260808 01:50:53.300760 13215 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.128s
W20260808 01:50:53.418351 12973 consensus_meta.cc:287] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.379s	user 0.011s	sys 0.005s
W20260808 01:50:53.469244  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 69864.8us
I20260808 01:50:53.465656  9632 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:53.470574  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:53.471765  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 212555us
W20260808 01:50:53.451603 13200 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.336s	user 0.001s	sys 0.000s
W20260808 01:50:53.457662 13170 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260808 01:50:53.452829 10332 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:53.403342 13145 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:50:53.338596 13113 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.077s
I20260808 01:50:53.408452 13033 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:53.363911 13210 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.124s
I20260808 01:50:53.456694  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:53.519344  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 62722.6us
I20260808 01:50:53.522583  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:53.527181  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:53.414299 12756 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:53.500015 12864 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:53.501673 13205 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.502490 12942 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.503335 13139 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:53.450150 13156 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:53.423771 13180 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:53.483136 12973 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:53.405426 12856 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:53.510787  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:53.468963 12998 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:53.398895 13218 consensus_queue.cc:1059] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.118s
I20260808 01:50:53.455492 13219 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:53.619547 13214 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.247s	user 0.001s	sys 0.000s
I20260808 01:50:53.619760 12998 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.620842  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2293) took 1558 ms. Trace:
I20260808 01:50:53.621038 13219 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:53.621259  9476 rpcz_store.cc:276] 0808 01:50:52.062998 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:52.063182 (+   184us) service_pool.cc:224] Handling call
0808 01:50:52.065816 (+  2634us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:52.066374 (+   558us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:50:52.066419 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:52.085404 (+ 18985us) log.cc:844] Serialized 26 byte log entry
0808 01:50:53.098216 (+1012812us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:53.098264 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:53.098335 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:53.620603 (+522268us) raft_consensus.cc:1685] finished
0808 01:50:53.620631 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:53.620773 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":53256,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":4680,"thread_start_us":862932,"threads_started":1,"wal-append.queue_time_us":1125127}
I20260808 01:50:53.622949 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:53.622480 12998 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:53.497008  9630 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
W20260808 01:50:53.539438  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 262493us
I20260808 01:50:53.479611 13212 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:53.624650  9600 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:53.624485 13212 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.482568 13107 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:53.459841 13221 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260808 01:50:53.510838 13207 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.314s	user 0.001s	sys 0.001s
W20260808 01:50:53.560696  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 126345us
I20260808 01:50:53.486562 12661 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:53.460258  9633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0ea307d9adc442539683547d0176a05b"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:50:53.509759 13112 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.063s
I20260808 01:50:53.626279  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:53.609728  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:53.609498  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 127605us
I20260808 01:50:53.626888  9556 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 8: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:53.627648  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 100589us
I20260808 01:50:53.572842  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2284) took 1638 ms. Trace:
W20260808 01:50:53.510799 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 58101.6us
I20260808 01:50:53.633237  9633 raft_consensus.cc:606] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:53.546741 12917 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:53.618251 13225 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260808 01:50:53.638624  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:53.638983  9630 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:53.641562  9636 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:53.651037  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:53.567996 13114 consensus_meta.cc:287] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.700s	user 0.013s	sys 0.005s
I20260808 01:50:53.665570  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:53.565589 13169 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:53.617077 13197 log.cc:927] Time spent T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.342s	user 0.001s	sys 0.000s
I20260808 01:50:53.671806 13164 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:53.672178  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1976) took 1865 ms. Trace:
I20260808 01:50:53.645360 13078 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:53.668781 13134 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:53.672974 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:53.674147 10364 raft_consensus.cc:2410] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260808 01:50:53.634728  9477 rpcz_store.cc:276] 0808 01:50:51.934668 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.934894 (+   226us) service_pool.cc:224] Handling call
0808 01:50:51.935315 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:51.935703 (+   388us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:51.935735 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:51.936303 (+   568us) log.cc:844] Serialized 26 byte log entry
0808 01:50:52.714057 (+777754us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:52.714106 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:52.714184 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:53.572578 (+858394us) raft_consensus.cc:1685] finished
0808 01:50:53.572601 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:53.572765 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":71768,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":56241,"thread_start_us":720005,"threads_started":1,"wal-append.queue_time_us":960038}
W20260808 01:50:53.618551 13199 log.cc:927] Time spent T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.348s	user 0.002s	sys 0.000s
W20260808 01:50:53.675125 13209 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.334s	user 0.001s	sys 0.000s
I20260808 01:50:53.675063 13235 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:50:53.677296 12973 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:53.666824 13210 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:53.678434 13210 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.636873 10332 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:53.648525  9481 raft_consensus.cc:2393] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 8: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:50:53.680434 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45029.3us
I20260808 01:50:53.672618  9635 rpcz_store.cc:276] 0808 01:50:51.806932 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:52.199110 (+392178us) service_pool.cc:224] Handling call
0808 01:50:52.263790 (+ 64680us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:52.264284 (+   494us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:52.264333 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:52.265005 (+   672us) log.cc:844] Serialized 26 byte log entry
0808 01:50:52.916069 (+651064us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:52.973847 (+ 57778us) log.cc:844] Serialized 14 byte log entry
0808 01:50:52.990604 (+ 16757us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:52.990714 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:53.671953 (+681239us) raft_consensus.cc:1685] finished
0808 01:50:53.671975 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:53.672109 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":57312,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":63,"raft.queue_time_us":331972,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":7343,"thread_start_us":574605,"threads_started":1,"wal-append.queue_time_us":823744}
I20260808 01:50:53.565220 12756 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:53.652979 13208 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.351s	user 0.001s	sys 0.001s
W20260808 01:50:53.642541  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:53.653733 13218 raft_consensus.cc:3055] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:50:53.653861  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:53.654259 13174 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260808 01:50:53.692643 13078 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20260808 01:50:53.692407 13140 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.007s
I20260808 01:50:53.692592 13218 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:53.690217 12745 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:53.700742 12745 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:53.701262 12745 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:53.701700 13195 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:53.656198  9550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 1929) took 1879 ms (client timeout 1646 ms). Trace:
W20260808 01:50:53.702642  9550 rpcz_store.cc:269] 0808 01:50:51.776962 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.777217 (+   255us) service_pool.cc:224] Handling call
0808 01:50:53.655928 (+1878711us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b54003264d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:53.656101 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3409067392}
I20260808 01:50:53.704376  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:53.706388  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:53.709299 13205 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:53.709434  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:53.710912  9635 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:53.659919  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:53.711531  9599 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:53.712275  9552 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:53.712579 13033 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:53.658066 12661 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:53.658725 13237 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:53.713321 13033 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:53.713794  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:53.660895 12997 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:53.659236 13238 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.048s
I20260808 01:50:53.666234  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:53.715181 10365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 482) took 1204 ms. Trace:
I20260808 01:50:53.601588 13061 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:50:53.716069 13061 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:53.636857 10362 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:53.669950  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:53.717485 10362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 514) took 1458 ms. Trace:
I20260808 01:50:53.717278 13061 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.717926 10362 rpcz_store.cc:276] 0808 01:50:52.259337 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:52.265146 (+  5809us) service_pool.cc:224] Handling call
0808 01:50:53.622752 (+1357606us) spinlock_profiling.cc:243] Waited 21.9 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:50:53.717386 (+ 94634us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50263424}
I20260808 01:50:53.730187 13210 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:53.563604 13200 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260808 01:50:53.715618 10365 rpcz_store.cc:276] 0808 01:50:52.510725 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:52.510895 (+   170us) service_pool.cc:224] Handling call
0808 01:50:52.511312 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:52.511724 (+   412us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:52.511754 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:52.512298 (+   544us) log.cc:844] Serialized 26 byte log entry
0808 01:50:52.974791 (+462493us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:52.974871 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:52.974974 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:53.714942 (+739968us) raft_consensus.cc:1685] finished
0808 01:50:53.714966 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:53.715114 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":137379,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":3907,"thread_start_us":402131,"threads_started":1,"wal-append.queue_time_us":692247}
W20260808 01:50:53.669505 13159 consensus_meta.cc:287] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.162s	user 0.012s	sys 0.004s
I20260808 01:50:53.732376 13159 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.732758  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:53.732786  9477 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:53.735102  9637 raft_consensus.cc:2368] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8.
I20260808 01:50:53.632705 13156 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.690546  9521 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:53.623402 12856 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:53.690599 12937 consensus_meta.cc:287] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.073s	user 0.014s	sys 0.001s
W20260808 01:50:53.696480 13086 consensus_meta.cc:287] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.550s	user 0.017s	sys 0.000s
W20260808 01:50:53.696623 13216 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.214s	user 0.001s	sys 0.000s
I20260808 01:50:53.690274 13221 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.007s
I20260808 01:50:53.697898  9633 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 7: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:53.700896 13218 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:53.691476 13235 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:53.692548 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:53.715557  9630 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:53.653859 12676 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:53.668740 12738 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:53.673740 13134 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:53.718514 13238 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 01:50:53.718045  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 48156.6us
I20260808 01:50:53.670254 13082 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:53.652472 12942 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:53.664852  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:53.718436 13061 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:53.753424  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:53.788740  9551 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:53.754969  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1927) took 1985 ms. Trace:
I20260808 01:50:53.752867 10366 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:50:53.686949 13114 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:53.617331 13222 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:53.693040 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:53.719235  9552 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:53.841162  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:53.841562  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:53.842221  9479 raft_consensus.cc:2410] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260808 01:50:53.748435  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:53.842350  9473 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:53.843087  9634 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 8: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:53.789779  9557 rpcz_store.cc:276] 0808 01:50:51.775196 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.775389 (+   193us) service_pool.cc:224] Handling call
0808 01:50:51.795225 (+ 19836us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:52.534409 (+739184us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:52.534461 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:52.649339 (+114878us) log.cc:844] Serialized 26 byte log entry
0808 01:50:53.239313 (+589974us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:53.239370 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:53.239469 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:53.754705 (+515236us) raft_consensus.cc:1685] finished
0808 01:50:53.754730 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:53.754902 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40356,"raft.queue_time_us":302761,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":30941,"thread_start_us":1203750,"threads_started":2,"wal-append.queue_time_us":691490}
I20260808 01:50:53.755702  9633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "244e080726394210b5467bf688813cf0"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:50:53.761015 10364 raft_consensus.cc:3055] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:53.844398 10364 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:53.844585  9635 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:53.763063 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:53.772516  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:53.845727  9630 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: []
W20260808 01:50:53.846266  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 73819.4us
I20260808 01:50:53.769833 12676 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:53.742144  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:53.772622 10334 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:50:53.849963 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 77438.9us
I20260808 01:50:53.783663 13016 raft_consensus.cc:3055] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:50:53.806361  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1923) took 2044 ms. Trace:
I20260808 01:50:53.746924  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:53.852069  9555 rpcz_store.cc:276] 0808 01:50:51.762209 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.762400 (+   191us) service_pool.cc:224] Handling call
0808 01:50:51.762834 (+   434us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:52.963307 (+1200473us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:53.805909 (+842602us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54001aca54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:53.806237 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2717436672}
I20260808 01:50:53.853466  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:53.760746  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 423) took 1759 ms. Trace:
I20260808 01:50:53.854898  9558 rpcz_store.cc:276] 0808 01:50:52.001204 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:52.001769 (+   565us) service_pool.cc:224] Handling call
0808 01:50:52.009797 (+  8028us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:53.120300 (+1110503us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:53.760376 (+640076us) spinlock_profiling.cc:243] Waited 999 ms on lock 0x7b540002c1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:53.760646 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2297949056}
I20260808 01:50:53.772289  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 432) took 1566 ms. Trace:
I20260808 01:50:53.856380  9554 rpcz_store.cc:276] 0808 01:50:52.239082 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:52.239327 (+   245us) service_pool.cc:224] Handling call
0808 01:50:52.243970 (+  4643us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:53.487802 (+1243832us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:53.772039 (+284237us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b540008a7d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:53.772219 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2672000896}
W20260808 01:50:53.824177  9472 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2280) took 1977 ms (client timeout 1744 ms). Trace:
I20260808 01:50:53.825493  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2288) took 1889 ms. Trace:
I20260808 01:50:53.857988  9634 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260808 01:50:53.857826  9472 rpcz_store.cc:269] 0808 01:50:51.846562 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.851007 (+  4445us) service_pool.cc:224] Handling call
0808 01:50:53.823830 (+1972823us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b540021e654. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:53.824061 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2911859072}
I20260808 01:50:53.854454  9556 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:50:53.825754  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2285) took 1890 ms. Trace:
I20260808 01:50:53.829511  9474 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260808 01:50:53.862020  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:53.834343  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:53.837035  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:53.768469 10363 raft_consensus.cc:2410] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260808 01:50:53.752434 12937 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.744256 12864 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:53.777316 13082 raft_consensus.cc:697] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:53.741056 13205 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:53.739717 13029 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260808 01:50:53.840425  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:53.738250  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:53.767520 13211 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Stepping down as leader of term 9
W20260808 01:50:53.761565  9632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1973) took 2013 ms (client timeout 1837 ms). Trace:
I20260808 01:50:53.894424  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:53.895768  9632 rpcz_store.cc:269] 0808 01:50:51.748182 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.829101 (+ 80919us) service_pool.cc:224] Handling call
0808 01:50:53.761328 (+1932227us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54003fa054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:53.761472 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3747577728}
I20260808 01:50:53.775827  9631 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:50:53.773615 12778 raft_consensus.cc:3055] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:50:53.751196 13034 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:53.740818  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 50400.9us
I20260808 01:50:53.740760 13156 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:53.900231  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:53.900890  9560 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:50:53.900978  9479 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:50:53.887146 13172 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.649s	user 0.005s	sys 0.000s
I20260808 01:50:53.839277  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:53.858294  9481 rpcz_store.cc:276] 0808 01:50:51.936234 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:52.003366 (+ 67132us) service_pool.cc:224] Handling call
0808 01:50:53.825180 (+1821814us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b5400093dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:53.825404 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3606614912}
I20260808 01:50:53.861444  9475 rpcz_store.cc:276] 0808 01:50:51.935139 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:52.018636 (+ 83497us) service_pool.cc:224] Handling call
0808 01:50:52.019079 (+   443us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:52.268127 (+249048us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:53.825393 (+1557266us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b54002bbbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:53.825664 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":565258496}
I20260808 01:50:53.773025 12954 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:53.847625  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:53.763599 13012 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:53.754971 13159 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:53.863145  9556 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:50:53.792176 12738 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:53.876106  9634 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260808 01:50:53.864266  9476 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:50:53.913666  9634 pending_rounds.cc:85] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 3
I20260808 01:50:53.877975  9551 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:53.887413  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:53.741938 13212 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:53.887993  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 243876us
I20260808 01:50:53.894634  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:53.914841  9478 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 8: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:50:53.842782 13115 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.850s	user 0.009s	sys 0.021s
I20260808 01:50:53.915207  9557 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 8: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:53.915330 13115 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.923s	user 0.010s	sys 0.021s
I20260808 01:50:53.916183  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:53.917420  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:53.917991  9477 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: []
I20260808 01:50:53.918201  9557 raft_consensus.cc:2410] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260808 01:50:53.919134  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:53.919526  9476 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260808 01:50:53.920261  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:53.920454  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:53.921037  9477 raft_consensus.cc:2393] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 7: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:53.921155  9476 raft_consensus.cc:2393] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:53.896437  9552 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 8: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:53.922345  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:53.897457  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:53.738327 12955 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:53.926034 12955 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:53.860112 10364 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:53.900117  9633 raft_consensus.cc:606] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:53.926959  9633 raft_consensus.cc:614] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:50:53.927071 10364 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:53.898772  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:53.928531  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:53.929316 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:53.929980 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:53.762172 12819 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.930540 13205 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:53.931337 13205 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:53.931703 13205 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:53.932399 12819 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:53.847971  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:53.932905 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:50:53.933348  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 85466.6us
I20260808 01:50:53.933625 10365 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:53.933893  9521 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:53.932133 13205 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:53.773097 12745 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:53.909664  9479 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:50:53.874485 13016 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:53.936592 13016 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.912349  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:53.937376 12745 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:53.904019  9638 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:53.937954  9556 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:50:53.911854  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:53.913995  9634 pending_rounds.cc:107] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260808 01:50:53.938848  9480 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 7: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:53.939118  9634 raft_consensus.cc:2889] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:53.939842 12809 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:50:53.940450 12809 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:53.932075 13201 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:53.941425  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:53.942085 13201 raft_consensus.cc:3055] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:50:53.942482 13201 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:53.939127  9475 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 8
I20260808 01:50:53.943217 13145 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:53.943521 13201 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.944042 13204 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:50:53.944917 13201 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:53.935025 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:50:53.911177  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 63631.4us
I20260808 01:50:53.935006 13012 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:53.946084 13012 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:53.946187  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:53.946538 13012 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:53.941479 12809 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.935092  9599 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:53.951061  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:53.951746 13033 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:50:53.935284  9559 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:53.959390  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:53.965372  9475 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:50:53.967216  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:53.966042  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:53.774650 13177 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:53.895634 13211 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:53.970414 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 278178us
I20260808 01:50:53.884390 13210 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:53.767783 13235 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:53.901943 13206 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:53.909284 12954 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:54.110067  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:53.969892  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:53.976141  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:53.900980 12778 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:53.901413 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:54.133312  9524 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:54.133319  9477 raft_consensus.cc:2393] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:54.135030  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:53.951824  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:53.989101  9635 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260808 01:50:53.989266  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:54.107334  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 117246us
I20260808 01:50:54.137470  9601 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:54.106652 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 115036us
I20260808 01:50:54.112437  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:54.138832  9475 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:54.141139 10365 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:54.109383  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:54.143049  9481 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:54.144176  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:54.145381  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:54.146443  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:54.147727  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:53.952265 13033 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:53.975440 13175 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:53.978025 12937 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:54.194582  9475 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
W20260808 01:50:54.199810  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 248025us
I20260808 01:50:54.207428  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:54.198987  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:54.208600  9477 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:50:54.198494  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 218244us
W20260808 01:50:54.200158  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66942.3us
I20260808 01:50:53.970907 13238 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:54.109009 12856 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:54.200368  9481 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 7: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:50:54.200158  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 62793.3us
I20260808 01:50:54.212972  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:54.213477  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 224288us
I20260808 01:50:54.220228  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:53.970335 13221 consensus_queue.cc:1059] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.233s
I20260808 01:50:53.991838 13086 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:54.275483  9551 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260808 01:50:54.276228  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:53.979933 12738 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:54.222586 12997 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:54.294239 13183 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:50:54.294073 12778 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:54.294363 12997 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:54.295383 12778 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:54.264482 13175 raft_consensus.cc:697] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:54.297087 12997 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:54.297598 13175 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:54.300103 13016 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:54.203390 13033 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:54.306520  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:54.307983  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:54.255519  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:53.991195 12804 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:54.261498  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:54.112185 13114 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:54.255009  9477 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 8: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:54.315447  9481 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 8: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:50:54.304926  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 97561.4us
I20260808 01:50:54.317312  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:54.148128 13034 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.133s
I20260808 01:50:54.259706 10364 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260808 01:50:54.318251 10364 pending_rounds.cc:85] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting all ops after (but not including) 6
I20260808 01:50:54.318537 10364 pending_rounds.cc:107] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260808 01:50:54.214464 13029 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:50:54.309293  9601 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:54.261370 12917 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:54.327742 12917 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:54.328311 12917 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:54.214640 13233 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:54.318842 10364 raft_consensus.cc:2889] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:54.321105  9477 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260808 01:50:54.334080  9477 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: []
W20260808 01:50:54.310145  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 54726.8us
I20260808 01:50:54.262300 13086 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:54.336186 13086 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:54.339265  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:54.340301  9635 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:50:54.341735  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:54.347597  9477 raft_consensus.cc:1217] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Deduplicated request from leader. Original: 2.2->[4.3-8.4]   Dedup: 4.3->[8.4-8.4]
I20260808 01:50:54.221205 12693 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.094s
I20260808 01:50:54.290376 13238 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:54.349013  9477 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9. Ops: [4.3-8.4]
I20260808 01:50:54.349993  9635 raft_consensus.cc:2410] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260808 01:50:54.394196  9477 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260808 01:50:54.395251  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:54.396027  9635 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:54.135421 13210 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:54.358018 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 219112us
I20260808 01:50:54.402133 10332 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:54.403719 10332 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:54.357585  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 48345us
W20260808 01:50:54.380287  9444 connection.cc:256] RPC call timeout handler was delayed by -1.6197848760057241s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3308s for a response
I20260808 01:50:53.979182 13082 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:54.393527  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2420) took 1086 ms. Trace:
I20260808 01:50:54.316352 13183 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:54.324389 13195 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:54.321787 13029 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:54.410105  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:54.657150  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:54.325327 13244 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:54.417089  9556 rpcz_store.cc:276] 0808 01:50:53.307508 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:53.948445 (+640937us) service_pool.cc:224] Handling call
0808 01:50:53.948792 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:53.949149 (+   357us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:50:54.205787 (+256638us) log.cc:844] Serialized 14 byte log entry
0808 01:50:54.393170 (+187383us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:54.393230 (+    60us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:54.393275 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:54.393342 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:54.393464 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40344,"prepare.queue_time_us":460161,"thread_start_us":402191,"threads_started":2}
I20260808 01:50:54.283870 12856 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:54.411332 13159 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:54.674525  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:54.422132 13183 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:54.440341 13247 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:54.693413  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:54.446868 13240 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:50:54.452968  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1979) took 1254 ms. Trace:
I20260808 01:50:54.455654  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:54.458256  9560 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260808 01:50:54.461854  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:54.466794  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 244e080726394210b5467bf688813cf0 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:50:54.694242  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:54.473264  9555 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:54.475577 13086 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:54.696237 10364 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260808 01:50:54.478045  9552 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
W20260808 01:50:54.478091 13250 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.199s	user 0.001s	sys 0.001s
I20260808 01:50:54.446081 12917 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:54.479918  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2062) took 1375 ms. Trace:
W20260808 01:50:54.482327  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:54.490612  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 3b8e0e8fda6744ad877514b11658b9d9: Illegal state: Leader has not yet committed an operation in its own term
I20260808 01:50:54.497887 13167 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:54.717873  9630 rpcz_store.cc:276] 0808 01:50:53.104410 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:53.769109 (+664699us) service_pool.cc:224] Handling call
0808 01:50:53.845540 (+ 76431us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:53.846097 (+   557us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:54.479638 (+633541us) spinlock_profiling.cc:243] Waited 76 ms on lock 0x7b540008cd50. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:54.479845 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":174851840}
I20260808 01:50:54.500242 13073 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:54.502632  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:54.514954  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:54.726244  9476 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 8: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:50:54.518046 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:54.522560 10366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 483) took 1561 ms. Trace:
I20260808 01:50:54.529070 13177 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:54.543757  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 1949) took 1405 ms. Trace:
I20260808 01:50:54.732028  9559 rpcz_store.cc:276] 0808 01:50:53.138008 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:53.219291 (+ 81283us) service_pool.cc:224] Handling call
0808 01:50:54.543481 (+1324190us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b54002a8354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:54.543661 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1589675392}
I20260808 01:50:54.597672 12738 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:54.601770 13210 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:54.619614 13238 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:54.324267 13114 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:54.289327 13211 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:54.353057 12937 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 01:50:54.639163  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:54.421530 12934 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260808 01:50:54.777637  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:54.650286  9635 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:50:54.319829 13243 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:54.662168  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:54.666646 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 127387us
I20260808 01:50:54.682999  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:54.424189 13221 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:54.785353 13034 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:54.695072  9638 rpcz_store.cc:276] 0808 01:50:53.198778 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:53.225475 (+ 26697us) service_pool.cc:224] Handling call
0808 01:50:54.452634 (+1227159us) spinlock_profiling.cc:243] Waited 674 ms on lock 0x7b54000dfa54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:54.452871 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1551244160}
I20260808 01:50:54.695446  9478 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:50:54.786532 13034 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:54.787607  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:54.788688  9478 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 5: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:54.695969  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2428) took 1334 ms. Trace:
I20260808 01:50:54.790908  9560 rpcz_store.cc:276] 0808 01:50:53.362038 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:54.457349 (+1095311us) service_pool.cc:224] Handling call
0808 01:50:54.457808 (+   459us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:54.695629 (+237821us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:54.695897 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:54.795136  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:54.795641 13254 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260808 01:50:54.795931  9638 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:54.797420 10362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 519) took 1039 ms. Trace:
I20260808 01:50:54.799324  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:54.800491  9560 raft_consensus.cc:2368] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6.
I20260808 01:50:54.802014  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:54.703126  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2423) took 1394 ms. Trace:
I20260808 01:50:54.659273 13029 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:54.803254  9552 rpcz_store.cc:276] 0808 01:50:53.308285 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:54.477315 (+1169030us) service_pool.cc:224] Handling call
0808 01:50:54.477798 (+   483us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:54.702757 (+224959us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:54.703055 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:54.716270  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2433) took 1037 ms. Trace:
I20260808 01:50:54.719426  9601 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:54.805729  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 86400.5us
I20260808 01:50:54.806066 13222 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:50:54.807008  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:54.807168 12937 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:54.808396 12937 raft_consensus.cc:697] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:54.708642 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:54.809885 10365 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:54.813005  9560 raft_consensus.cc:2410] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260808 01:50:54.720095  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:54.815380  9630 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:50:54.804982  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:54.817337  9552 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:54.790308  9478 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:54.821389  9478 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:54.821496 13086 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:54.707475  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:54.805193  9555 rpcz_store.cc:276] 0808 01:50:53.679198 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:54.472571 (+793373us) service_pool.cc:224] Handling call
0808 01:50:54.473002 (+   431us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:54.706269 (+233267us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:54.716193 (+  9924us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:54.728086 10366 rpcz_store.cc:276] 0808 01:50:52.961588 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:52.963919 (+  2331us) service_pool.cc:224] Handling call
0808 01:50:54.522272 (+1558353us) spinlock_profiling.cc:243] Waited 780 ms on lock 0x7b54000fda54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:54.522442 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1794914688}
I20260808 01:50:54.680800 13159 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:54.729869  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:54.837193  9558 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:54.682287 13145 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:54.680703 12745 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:54.736428  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:54.802289 13114 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:54.669085 13251 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:54.694183 13244 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:54.700949 13247 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:54.922710 13247 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:50:54.870780 12809 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.920s	user 0.017s	sys 0.074s
W20260808 01:50:54.927139 12809 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.976s	user 0.017s	sys 0.074s
I20260808 01:50:54.681418 13113 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.061s
I20260808 01:50:54.798254 10362 rpcz_store.cc:276] 0808 01:50:53.757883 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:53.758369 (+   486us) service_pool.cc:224] Handling call
0808 01:50:53.758857 (+   488us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:53.759337 (+   480us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:53.759386 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:53.759932 (+   546us) log.cc:844] Serialized 26 byte log entry
0808 01:50:53.979512 (+219580us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:53.991475 (+ 11963us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:54.134347 (+142872us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:54.796827 (+662480us) raft_consensus.cc:1685] finished
0808 01:50:54.796862 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:54.797295 (+   433us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1892,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":95,"thread_start_us":150725,"threads_started":1,"wal-append.queue_time_us":945973}
I20260808 01:50:54.734921 13167 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:54.785701 13221 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:50:54.828759 13012 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.881s	user 0.009s	sys 0.006s
I20260808 01:50:54.692987 13183 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:54.827402  9556 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:50:54.827721  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:54.986272  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:54.986829  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:54.987305  9556 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
W20260808 01:50:54.830479  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 168421us
I20260808 01:50:54.830947 10365 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:54.814756  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:54.836257  9557 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.015s	sys 0.067s
I20260808 01:50:54.988790  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:54.989523  9557 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.015s	sys 0.068s
I20260808 01:50:54.988736  9556 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:54.840478  9630 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 4.
I20260808 01:50:54.991242  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:54.760663  9559 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260808 01:50:54.993468  9559 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260808 01:50:54.871639  9552 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
W20260808 01:50:54.891139  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:54.895560  9638 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
W20260808 01:50:54.898970  9599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.036s	sys 0.062s
I20260808 01:50:54.805019 12917 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:54.908072  9524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.017s	sys 0.076s
W20260808 01:50:54.997816  9524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.017s	sys 0.076s
I20260808 01:50:54.998732  9560 raft_consensus.cc:2410] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260808 01:50:54.910322  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:54.745571 13240 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:54.799141 12856 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:54.822638 13086 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:54.907797 13145 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:54.906493 12745 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:54.835328 13210 consensus_queue.cc:1059] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.096s
I20260808 01:50:54.652189 13195 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:54.956007  9472 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.019s	sys 0.020s
W20260808 01:50:55.021692  9472 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.019s	sys 0.020s
I20260808 01:50:55.023164  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2078) took 1100 ms. Trace:
W20260808 01:50:55.011689  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:55.023646  9472 rpcz_store.cc:276] 0808 01:50:53.922627 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:54.284241 (+361614us) service_pool.cc:224] Handling call
0808 01:50:54.284612 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:54.284847 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 4
0808 01:50:54.315347 (+ 30500us) log.cc:844] Serialized 14 byte log entry
0808 01:50:55.022501 (+707154us) log.cc:844] Serialized 14 byte log entry
0808 01:50:55.022698 (+   197us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:55.022751 (+    53us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:55.022802 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:55.022882 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:55.023020 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":30319,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"thread_start_us":706521,"threads_started":1}
I20260808 01:50:54.810094 12937 consensus_queue.cc:237] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:54.698478 13180 consensus_meta.cc:287] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.061s	user 0.013s	sys 0.000s
I20260808 01:50:54.896768 13243 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:54.997637 13138 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:54.943763 13113 consensus_queue.cc:1243] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260808 01:50:55.004793  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:55.005332  9551 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.010s	sys 0.013s
W20260808 01:50:55.039620  9551 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.010s	sys 0.013s
I20260808 01:50:54.866302 13159 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:54.961963 13221 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:54.982924 13218 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:50:55.059715 13205 consensus_meta.cc:287] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.124s	user 0.010s	sys 0.003s
I20260808 01:50:55.060388 13205 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:54.899259 13211 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:50:55.065923 12942 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.266s	user 0.018s	sys 0.008s
I20260808 01:50:55.067088  9550 raft_consensus.cc:2410] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20260808 01:50:55.068535 13061 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.269s	user 0.020s	sys 0.010s
W20260808 01:50:55.073271 13201 consensus_meta.cc:287] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.128s	user 0.014s	sys 0.000s
I20260808 01:50:55.075909  9554 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
W20260808 01:50:55.012904  9631 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.006s	sys 0.019s
W20260808 01:50:55.076923  9631 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.006s	sys 0.019s
I20260808 01:50:55.076068 13243 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:54.994904 13012 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:54.772887 13233 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:55.006600 13114 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:50:54.995960  9599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.669s	user 0.036s	sys 0.062s
W20260808 01:50:55.080384  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 763108us
W20260808 01:50:55.041329  9481 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.004s	sys 0.046s
I20260808 01:50:55.009418 13086 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:55.048312  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:55.037050 12756 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.247s	user 0.014s	sys 0.000s
I20260808 01:50:55.025543 13034 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:55.082186  9481 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.731s	user 0.004s	sys 0.047s
I20260808 01:50:55.025146 13195 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:55.083009  9599 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:55.080925  9523 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:50:55.008791 12804 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.009s	sys 0.014s
I20260808 01:50:55.022768 12856 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.067s
I20260808 01:50:55.095647  9599 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:55.092720 13150 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.009s	sys 0.025s
I20260808 01:50:55.098031 13034 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 01:50:55.098915 12804 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.010s	sys 0.014s
I20260808 01:50:55.041558 12864 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20260808 01:50:55.045277 13180 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:55.097232 13012 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:55.101480 13033 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.778s	user 0.009s	sys 0.006s
I20260808 01:50:55.106549 13061 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260808 01:50:55.116041 13150 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.693s	user 0.009s	sys 0.025s
I20260808 01:50:55.101442 13167 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:55.106902 13145 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:55.090381 12942 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.033s
I20260808 01:50:55.106608 13212 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:55.117296 13016 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election won for term 8
W20260808 01:50:55.138557 13253 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.357s	user 0.000s	sys 0.002s
I20260808 01:50:55.038391 13244 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:55.128901 13205 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:55.135789 13061 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:55.124039 13034 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20260808 01:50:55.144340  9558 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:50:55.138096 13086 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:55.091022 12756 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:55.118782 12804 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:55.167713 12745 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:55.152154 13033 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:55.189632 10331 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.019s	sys 0.070s
W20260808 01:50:55.191157 10331 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.020s	sys 0.070s
I20260808 01:50:55.161723 13233 consensus_queue.cc:1059] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.020s
I20260808 01:50:55.196771 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:55.198792 10366 raft_consensus.cc:2393] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 4: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:55.172881 13073 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:50:55.172958 13016 raft_consensus.cc:697] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:55.052706 13240 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:55.163787 12954 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.962s	user 0.009s	sys 0.005s
W20260808 01:50:55.167224 12778 consensus_meta.cc:287] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.870s	user 0.014s	sys 0.004s
W20260808 01:50:55.233884 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:55.180337  9555 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20260808 01:50:55.169382 13212 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:55.177897 13061 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:55.203012  9636 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 7: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:50:55.167613  9634 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.847s	user 0.005s	sys 0.027s
I20260808 01:50:55.217202 12756 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:55.219336 12804 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:55.220252 13180 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:55.223116 13086 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:55.225231 13029 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:55.225581 13221 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:55.236505  9476 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:55.187544 13177 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:55.237547  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:55.241156  9634 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.923s	user 0.005s	sys 0.027s
I20260808 01:50:55.241335  9633 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 10: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:50:55.155296  9629 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.375s	user 0.006s	sys 0.018s
I20260808 01:50:55.261215  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:55.285014  9629 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.504s	user 0.006s	sys 0.018s
I20260808 01:50:55.261574 13212 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:55.202358 13205 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:55.238086 13175 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.937s	user 0.025s	sys 0.007s
W20260808 01:50:55.276607  9521 leader_election.cc:336] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.924s (SENT)
W20260808 01:50:55.244170 12676 consensus_meta.cc:287] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.353s	user 0.017s	sys 0.001s
I20260808 01:50:55.284045  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1997) took 1391 ms. Trace:
I20260808 01:50:55.285431  9632 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:50:55.243108 13251 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:55.293663 13270 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:55.294216  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:55.302027 10332 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:55.303637 12738 raft_consensus.cc:3055] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Stepping down as leader of term 6
W20260808 01:50:55.307094  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:55.307129  9634 rpcz_store.cc:276] 0808 01:50:53.892196 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:53.940159 (+ 47963us) service_pool.cc:224] Handling call
0808 01:50:53.940524 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:53.940738 (+   214us) raft_consensus.cc:1534] Early marking committed up to index 5
0808 01:50:54.317860 (+377122us) log.cc:844] Serialized 14 byte log entry
0808 01:50:55.283344 (+965484us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:55.283580 (+   236us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:55.283671 (+    91us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:55.283779 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:55.283961 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":544257,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":137,"thread_start_us":1341466,"threads_started":2}
W20260808 01:50:55.284017 13206 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.086s	user 0.008s	sys 0.015s
W20260808 01:50:55.315241  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:55.235224 12954 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:55.338052  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:55.340379  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:55.343104  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:55.313583 13175 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.012s	user 0.027s	sys 0.007s
I20260808 01:50:55.319191 12676 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:55.377868  9634 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 8: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:55.385355  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:55.386519  9637 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:55.383278 10332 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:55.317174 12738 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:55.399824  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:55.324541 13206 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.127s	user 0.008s	sys 0.015s
I20260808 01:50:55.391263  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:55.401067  9634 raft_consensus.cc:2393] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 9: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:55.401683  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2086) took 1026 ms. Trace:
I20260808 01:50:55.289839 13247 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:55.287140 13177 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:55.328262 12745 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:55.401880 10332 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:55.315874 13205 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:55.317195 13270 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:55.385959 13180 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:55.449949 10364 pending_rounds.cc:85] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting all ops after (but not including) 6
I20260808 01:50:55.450496 10364 pending_rounds.cc:107] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260808 01:50:55.450985 10364 raft_consensus.cc:2889] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:50:55.402871  9634 rpcz_store.cc:276] 0808 01:50:54.375466 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:55.390753 (+1015287us) service_pool.cc:224] Handling call
0808 01:50:55.401596 (+ 10843us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:55.455974  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:55.457015  9634 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:50:55.457767  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 391) took 1040 ms. Trace:
I20260808 01:50:55.458243  9634 rpcz_store.cc:276] 0808 01:50:54.417288 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:55.455424 (+1038136us) service_pool.cc:224] Handling call
0808 01:50:55.457688 (+  2264us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:55.460253  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:55.472750  9477 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:55.376763 13159 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:55.404544 13206 raft_consensus.cc:697] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:55.413165 13034 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:55.322553 13251 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:55.445811 12756 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:55.462962 13212 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:55.471868 13180 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:55.481025 13195 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:55.489529 12738 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:55.524694  9634 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:55.403713 13016 consensus_queue.cc:237] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:55.563714  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:55.536123 12745 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:55.396056 13183 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:55.561616  9521 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:55.562034  9630 raft_consensus.cc:2368] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8.
I20260808 01:50:55.572921 13201 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:55.539688 13033 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:55.543656  9637 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:55.577006  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2079) took 1232 ms. Trace:
I20260808 01:50:55.577581  9637 rpcz_store.cc:276] 0808 01:50:54.344390 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:55.384960 (+1040570us) service_pool.cc:224] Handling call
0808 01:50:55.576916 (+191956us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:55.579030  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:55.572701 13206 consensus_queue.cc:237] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:55.587922  9557 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.009s	sys 0.041s
W20260808 01:50:55.588434  9557 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.010s	sys 0.042s
I20260808 01:50:55.589722  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2424) took 2281 ms. Trace:
W20260808 01:50:55.598791  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.006s	sys 0.044s
W20260808 01:50:55.600454  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.006s	sys 0.044s
I20260808 01:50:55.539155 12756 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:55.609300  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 615572us
I20260808 01:50:55.610198  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:55.611421 12778 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:50:55.612923  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:55.603767  9472 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260808 01:50:55.553987 13034 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:55.585534 13251 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:55.573340 13201 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:55.601181  9560 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.016s	sys 0.030s
W20260808 01:50:55.640488  9560 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.017s	sys 0.030s
W20260808 01:50:55.615806  9631 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.000s	sys 0.018s
I20260808 01:50:55.559093 13212 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:55.618609  9599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.000s	sys 0.042s
I20260808 01:50:55.590967 13204 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:55.591928 12745 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:50:55.636057 12937 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.023s	sys 0.035s
I20260808 01:50:55.594131 13134 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:55.631234  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:55.633051 12778 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:55.601446 13260 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Leader election won for term 4
W20260808 01:50:55.643178  9599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.000s	sys 0.043s
W20260808 01:50:55.669669  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 563795us
W20260808 01:50:55.672632  9524 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.006s	sys 0.020s
W20260808 01:50:55.673202  9524 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.006s	sys 0.020s
W20260808 01:50:55.673542  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 520043us
W20260808 01:50:55.643646  9631 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.000s	sys 0.018s
I20260808 01:50:55.614828 13221 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:55.590397  9557 rpcz_store.cc:276] 0808 01:50:53.308595 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:54.052077 (+743482us) service_pool.cc:224] Handling call
0808 01:50:54.198455 (+146378us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:54.198670 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 4
0808 01:50:54.990073 (+791403us) log.cc:844] Serialized 14 byte log entry
0808 01:50:55.589070 (+598997us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:55.589131 (+    61us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:55.589251 (+   120us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:55.589357 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:55.589654 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":428709,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":147,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":3377,"thread_start_us":1365868,"threads_started":2}
I20260808 01:50:55.631644  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:55.651086 12937 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.023s	sys 0.036s
I20260808 01:50:55.601766 13276 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.127s
I20260808 01:50:55.641038 13205 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:55.642772 13201 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:55.676532  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 45019.6us
I20260808 01:50:55.642742 12676 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:55.690501  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:55.692641  9551 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.006s	sys 0.012s
W20260808 01:50:55.693086  9551 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.006s	sys 0.012s
I20260808 01:50:55.695866  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:55.740968  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:55.688191  9481 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.000s	sys 0.017s
W20260808 01:50:55.761915  9481 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.000s	sys 0.017s
I20260808 01:50:55.666268 13270 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:55.782447  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 80421.8us
I20260808 01:50:55.670218 13247 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:55.681236 13138 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:55.689052 13145 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.004s	sys 0.024s
W20260808 01:50:55.941533 13061 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.673s	user 0.012s	sys 0.000s
W20260808 01:50:55.942759 13264 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.318s	user 0.000s	sys 0.000s
I20260808 01:50:55.953701  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1962) took 1159 ms. Trace:
I20260808 01:50:55.962108 13212 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:55.962539  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2321) took 1243 ms. Trace:
W20260808 01:50:55.963474 13279 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.362s	user 0.003s	sys 0.001s
I20260808 01:50:55.963737  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 435) took 1288 ms. Trace:
I20260808 01:50:56.022423  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:55.966535 13012 raft_consensus.cc:3055] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:55.967811  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:55.972504 13273 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.432s	user 0.000s	sys 0.001s
I20260808 01:50:55.680907 13134 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:50:55.782114 13082 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:50:55.973522 12745 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:55.976322 13034 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:56.044615 10366 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.751s	user 0.029s	sys 0.082s
W20260808 01:50:56.045135 10366 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.751s	user 0.029s	sys 0.082s
I20260808 01:50:56.010146 13180 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:55.979921 13218 consensus_meta.cc:287] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.923s	user 0.014s	sys 0.000s
W20260808 01:50:55.983022  9632 consensus_meta.cc:287] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.582s	user 0.013s	sys 0.013s
W20260808 01:50:55.980609  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:55.616142 13238 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.013s	sys 0.006s
I20260808 01:50:55.989208  9634 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:50:56.051242  9632 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:56.051735  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2083) took 1686 ms. Trace:
I20260808 01:50:56.052193  9634 rpcz_store.cc:276] 0808 01:50:54.365802 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:55.562704 (+1196902us) service_pool.cc:224] Handling call
0808 01:50:56.051667 (+488963us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:55.980567 13243 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.757s	user 0.027s	sys 0.067s
I20260808 01:50:55.799270 13177 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:55.872862 13260 raft_consensus.cc:697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:56.030375 13012 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:55.981171 13033 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:55.937696 13089 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:56.009660  9555 rpcz_store.cc:276] 0808 01:50:54.825795 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:54.999489 (+173694us) service_pool.cc:224] Handling call
0808 01:50:55.179968 (+180479us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:55.235765 (+ 55797us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:55.953346 (+717581us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b540029efd0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:55.953598 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":413843840}
I20260808 01:50:56.015116  9472 rpcz_store.cc:276] 0808 01:50:54.718961 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:55.025191 (+306230us) service_pool.cc:224] Handling call
0808 01:50:55.025622 (+   431us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:55.628653 (+603031us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:55.962162 (+333509us) spinlock_profiling.cc:243] Waited 578 ms on lock 0x7b540038ccd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:55.962452 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1328832384}
I20260808 01:50:56.024435  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1963) took 1088 ms. Trace:
I20260808 01:50:55.997938 13277 raft_consensus.cc:3055] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:50:56.027172  9554 rpcz_store.cc:276] 0808 01:50:54.675824 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:54.675988 (+   164us) service_pool.cc:224] Handling call
0808 01:50:55.963495 (+1287507us) spinlock_profiling.cc:243] Waited 291 ms on lock 0x7b540002e254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:55.963651 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":670309632}
W20260808 01:50:56.052371 13238 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.980s	user 0.013s	sys 0.006s
I20260808 01:50:56.029489  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:56.032213  9479 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:56.106925  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:56.107977  9472 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:56.015100 13061 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:56.011157 13145 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.828s	user 0.004s	sys 0.034s
I20260808 01:50:55.999629 13159 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:56.051798  9632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2003) took 1632 ms (client timeout 1947 ms). Trace:
I20260808 01:50:56.054234  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:56.145730  9632 rpcz_store.cc:269] 0808 01:50:54.419660 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:55.328055 (+908395us) service_pool.cc:224] Handling call
0808 01:50:56.051742 (+723687us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:56.146029  9634 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:56.146834  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:56.145385  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2096) took 2170 ms. Trace:
I20260808 01:50:56.147553  9632 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:55.974406  9637 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:50:55.979055 13271 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:56.029831 13195 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:55.965162  9636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 1994) took 2488 ms (client timeout 1744 ms). Trace:
W20260808 01:50:56.072682 10364 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.004s	sys 0.016s
I20260808 01:50:56.077661  9560 rpcz_store.cc:276] 0808 01:50:54.935765 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:54.999498 (+ 63733us) service_pool.cc:224] Handling call
0808 01:50:55.020662 (+ 21164us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:55.021310 (+   648us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:55.055132 (+ 33822us) log.cc:844] Serialized 14 byte log entry
0808 01:50:55.641400 (+586268us) log.cc:844] Serialized 14 byte log entry
0808 01:50:55.641618 (+   218us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:55.645852 (+  4234us) log.cc:844] Serialized 26 byte log entry
0808 01:50:55.646038 (+   186us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:55.646079 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:55.646157 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:56.024058 (+377901us) raft_consensus.cc:1685] finished
0808 01:50:56.024109 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:56.024342 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":484,"prepare.queue_time_us":58925,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":89,"raft.queue_time_us":5662,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":17889,"spinlock_wait_cycles":24537472,"thread_start_us":585403,"threads_started":1,"wal-append.queue_time_us":945412}
W20260808 01:50:56.063223 13243 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.840s	user 0.028s	sys 0.068s
I20260808 01:50:55.984687 12937 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:56.073338 13244 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.008s	sys 0.012s
I20260808 01:50:56.134083  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1983) took 2934 ms. Trace:
I20260808 01:50:56.176234  9629 rpcz_store.cc:276] 0808 01:50:53.199824 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:53.778454 (+578630us) service_pool.cc:224] Handling call
0808 01:50:53.778986 (+   532us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:53.779557 (+   571us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:53.779603 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:53.780221 (+   618us) log.cc:844] Serialized 26 byte log entry
0808 01:50:55.285676 (+1505455us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:55.285745 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:55.285862 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:56.133780 (+847918us) raft_consensus.cc:1685] finished
0808 01:50:56.133814 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:56.134002 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":148,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":100,"thread_start_us":1504856,"threads_started":1,"wal-append.queue_time_us":1690699}
W20260808 01:50:56.133595  9479 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2077) took 2183 ms (client timeout 1544 ms). Trace:
W20260808 01:50:56.122077  9521 leader_election.cc:336] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.837s (SENT)
I20260808 01:50:56.179118  9521 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:56.178948  9479 rpcz_store.cc:269] 0808 01:50:53.922180 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:53.936263 (+ 14083us) service_pool.cc:224] Handling call
0808 01:50:55.967583 (+2031320us) spinlock_profiling.cc:243] Waited 33 ms on lock 0x7b5800005498. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:50:56.105797 (+138214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":75968256}
I20260808 01:50:56.146925  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:56.180540  9634 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:50:56.180922  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:56.181790  9479 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 8: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:56.181649  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:56.182533  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2001) took 1763 ms. Trace:
I20260808 01:50:56.183048  9634 rpcz_store.cc:276] 0808 01:50:54.419048 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.181379 (+1762331us) service_pool.cc:224] Handling call
0808 01:50:56.182470 (+  1091us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:56.184336  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:56.185170  9634 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
W20260808 01:50:56.185418 13292 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
W20260808 01:50:55.992785 10332 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.087s	sys 0.089s
W20260808 01:50:56.186961 10332 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.761s	user 0.087s	sys 0.089s
W20260808 01:50:56.187754 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 785898us
I20260808 01:50:56.188647 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:56.013108 13138 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:56.068966 13205 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:56.068926 13210 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:56.147610  9480 rpcz_store.cc:276] 0808 01:50:53.975182 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:54.468604 (+493422us) service_pool.cc:224] Handling call
0808 01:50:54.469090 (+   486us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:54.469603 (+   513us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:54.469642 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:54.814412 (+344770us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b2c00134ff8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:50:54.819391 (+  4979us) log.cc:844] Serialized 26 byte log entry
0808 01:50:55.170426 (+351035us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:55.170482 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:55.191437 (+ 20955us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:55.950343 (+758906us) spinlock_profiling.cc:243] Waited 9.21 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:50:56.135579 (+185236us) raft_consensus.cc:1685] finished
0808 01:50:56.135617 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:56.145285 (+  9668us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":644,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":134,"spinlock_wait_cycles":438345728,"thread_start_us":350370,"threads_started":1,"wal-append.queue_time_us":724588}
W20260808 01:50:56.155038  9636 rpcz_store.cc:269] 0808 01:50:53.477033 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:53.650716 (+173683us) service_pool.cc:224] Handling call
0808 01:50:55.964876 (+2314160us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b540003acd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:55.965065 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3567328000}
W20260808 01:50:56.155189 10364 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.004s	sys 0.016s
I20260808 01:50:56.158157  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:56.079541 13277 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:50:56.154320 13235 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.029s	sys 0.011s
W20260808 01:50:56.143508 10360 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.071s	sys 0.187s
I20260808 01:50:56.190325 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:56.198840 10360 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.071s	sys 0.188s
I20260808 01:50:56.199599 13210 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:56.201766 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:56.194919 13222 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:50:56.203006 13222 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:56.203843 13222 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:56.202988 13205 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:56.205050  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2076) took 1865 ms. Trace:
I20260808 01:50:56.205629  9637 rpcz_store.cc:276] 0808 01:50:54.339825 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.154366 (+1814541us) service_pool.cc:224] Handling call
0808 01:50:56.204945 (+ 50579us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:56.206866  9637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "439040d168824496acc076b1c280bde3"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
W20260808 01:50:56.207381 13291 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260808 01:50:56.153142 12756 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:56.109762 13033 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:56.208509 13276 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:56.208962 13276 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:56.209045  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:56.209937 13033 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:56.186331  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:56.211612  9634 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:56.211932  9636 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260808 01:50:56.212901  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2082) took 1848 ms. Trace:
I20260808 01:50:56.213358  9636 rpcz_store.cc:276] 0808 01:50:54.364755 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.211025 (+1846270us) service_pool.cc:224] Handling call
0808 01:50:56.211475 (+   450us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:56.212559 (+  1084us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.212829 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:56.207958 13205 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:56.212118  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2081) took 1864 ms. Trace:
I20260808 01:50:56.214887 13012 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:56.216166 13012 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:56.217444 13138 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:56.218860 13235 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.634s	user 0.029s	sys 0.011s
I20260808 01:50:56.218945 13260 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:56.163233 13271 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:56.195781  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2427) took 2855 ms. Trace:
I20260808 01:50:56.214854  9634 rpcz_store.cc:276] 0808 01:50:54.347432 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.186009 (+1838577us) service_pool.cc:224] Handling call
0808 01:50:56.212055 (+ 26046us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:56.224963  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 102982us
I20260808 01:50:56.209379 13276 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:56.226794  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:56.072297 13183 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:56.222887  9551 rpcz_store.cc:276] 0808 01:50:53.340652 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:54.290243 (+949591us) service_pool.cc:224] Handling call
0808 01:50:54.323064 (+ 32821us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:55.040173 (+717109us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:55.040223 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:55.045933 (+  5710us) log.cc:844] Serialized 52 byte log entry
0808 01:50:55.693532 (+647599us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:50:55.693575 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:55.693645 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:56.195432 (+501787us) raft_consensus.cc:1685] finished
0808 01:50:56.195461 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:56.195695 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29300,"raft.queue_time_us":260282,"spinlock_wait_cycles":120366464,"thread_start_us":1300587,"threads_started":2,"wal-append.queue_time_us":969440}
I20260808 01:50:56.224223 10365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 528) took 1075 ms. Trace:
I20260808 01:50:56.243832 10365 rpcz_store.cc:276] 0808 01:50:55.149005 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:55.149168 (+   163us) service_pool.cc:224] Handling call
0808 01:50:55.149534 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:55.150010 (+   476us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:55.150065 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:55.152733 (+  2668us) log.cc:844] Serialized 26 byte log entry
0808 01:50:55.640649 (+487916us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:55.640726 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:55.640822 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:56.223990 (+583168us) raft_consensus.cc:1685] finished
0808 01:50:56.224012 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:56.224159 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":114,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":7349,"thread_start_us":487516,"threads_started":1,"wal-append.queue_time_us":982379}
I20260808 01:50:56.229718  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2324) took 1148 ms. Trace:
I20260808 01:50:56.231281  9634 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260808 01:50:56.231314  9636 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: []
I20260808 01:50:56.246937  9479 rpcz_store.cc:276] 0808 01:50:55.081386 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.180620 (+1099234us) service_pool.cc:224] Handling call
0808 01:50:56.229656 (+ 49036us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:56.237730  9472 raft_consensus.cc:2410] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260808 01:50:56.227244  9474 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260808 01:50:56.247387  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 393) took 1829 ms. Trace:
I20260808 01:50:56.247696  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2087) took 1869 ms. Trace:
I20260808 01:50:56.248453  9634 rpcz_store.cc:276] 0808 01:50:54.417823 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.224946 (+1807123us) service_pool.cc:224] Handling call
0808 01:50:56.230999 (+  6053us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:56.247081 (+ 16082us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.247327 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":66560}
I20260808 01:50:56.248780  9636 rpcz_store.cc:276] 0808 01:50:54.378526 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.214455 (+1835929us) service_pool.cc:224] Handling call
0808 01:50:56.231024 (+ 16569us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:56.247335 (+ 16311us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.247618 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:56.248805  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:56.261076  9632 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:50:56.109591 13089 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:56.228127 13244 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.008s	sys 0.020s
W20260808 01:50:56.192366 13086 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.913s	user 0.018s	sys 0.004s
I20260808 01:50:56.265297  9629 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260808 01:50:56.288125  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 394) took 1870 ms. Trace:
I20260808 01:50:56.218881 13295 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:56.288856  9601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.010s	sys 0.021s
I20260808 01:50:56.286777  9638 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:56.289202  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:56.290880  9636 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260808 01:50:56.291862  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2004) took 1871 ms. Trace:
I20260808 01:50:56.292407  9636 rpcz_store.cc:276] 0808 01:50:54.423309 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.289953 (+1866644us) service_pool.cc:224] Handling call
0808 01:50:56.290511 (+   558us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:56.291511 (+  1000us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.291788 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:56.294149  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1999) took 1842 ms. Trace:
I20260808 01:50:56.241725 13205 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:56.298818  9634 rpcz_store.cc:276] 0808 01:50:54.418455 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.260262 (+1841807us) service_pool.cc:224] Handling call
0808 01:50:56.260633 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:56.260829 (+   196us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:56.260864 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:56.260910 (+    46us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:50:56.260941 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.261000 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:56.261146 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:56.294268  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 476) took 1340 ms. Trace:
I20260808 01:50:56.264325  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 475) took 1457 ms. Trace:
I20260808 01:50:56.300647  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2006) took 1859 ms. Trace:
W20260808 01:50:56.289608  9601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.010s	sys 0.021s
I20260808 01:50:56.300987  9480 rpcz_store.cc:276] 0808 01:50:54.806717 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.210661 (+1403944us) service_pool.cc:224] Handling call
0808 01:50:56.263603 (+ 52942us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:56.263868 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:56.263908 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:56.263967 (+    59us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:56.264015 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.264097 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:56.264243 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:56.288240  9632 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9. Ops: []
W20260808 01:50:56.286375  9598 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.000s	sys 0.031s
I20260808 01:50:56.288892  9629 rpcz_store.cc:276] 0808 01:50:54.418137 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.229453 (+1811316us) service_pool.cc:224] Handling call
0808 01:50:56.265044 (+ 35591us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:56.287789 (+ 22745us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.288024 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:56.296815  9638 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9. Ops: [4.3-8.4]
I20260808 01:50:56.239588 13206 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:56.264997 13280 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.233s	user 0.000s	sys 0.001s
W20260808 01:50:56.293437 13240 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.638s	user 0.020s	sys 0.010s
I20260808 01:50:56.301640  9636 rpcz_store.cc:276] 0808 01:50:54.441475 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.299075 (+1857600us) service_pool.cc:224] Handling call
0808 01:50:56.299851 (+   776us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:56.300164 (+   313us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:56.300203 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:56.300266 (+    63us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:56.300314 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.300396 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:56.300563 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:56.303419  9599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.016s	sys 0.015s
I20260808 01:50:56.304039  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2005) took 1863 ms. Trace:
W20260808 01:50:56.311610  9599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.016s	sys 0.015s
I20260808 01:50:56.312005  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:56.311764  9638 rpcz_store.cc:276] 0808 01:50:54.441147 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.295231 (+1854084us) service_pool.cc:224] Handling call
0808 01:50:56.296430 (+  1199us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:56.303768 (+  7338us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.303987 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:56.312885  9636 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:56.302820  9598 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.000s	sys 0.031s
I20260808 01:50:56.304184  9472 rpcz_store.cc:276] 0808 01:50:54.954344 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.292963 (+1338619us) service_pool.cc:224] Handling call
0808 01:50:56.293383 (+   420us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:56.293637 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:56.293667 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:56.293722 (+    55us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:56.294008 (+   286us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.294076 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:56.294209 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:56.303052  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2002) took 1883 ms. Trace:
W20260808 01:50:56.315502  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:56.305265  9634 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:56.241813 13036 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:56.175668 12937 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:56.317130  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:56.315420  9632 rpcz_store.cc:276] 0808 01:50:54.419376 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.287482 (+1868106us) service_pool.cc:224] Handling call
0808 01:50:56.287952 (+   470us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:56.302763 (+ 14811us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.302982 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:56.318542  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:56.320353  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 395) took 1085 ms. Trace:
I20260808 01:50:56.320775  9634 rpcz_store.cc:276] 0808 01:50:55.235452 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.304755 (+1069303us) service_pool.cc:224] Handling call
0808 01:50:56.305078 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:56.317574 (+ 12496us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.320288 (+  2714us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:56.321942  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:56.322649  9634 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:50:56.327739  9634 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
W20260808 01:50:56.315172 13240 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.021s	sys 0.011s
I20260808 01:50:56.337116  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2010) took 1036 ms. Trace:
I20260808 01:50:56.337745  9634 rpcz_store.cc:276] 0808 01:50:55.295372 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.323625 (+1028253us) service_pool.cc:224] Handling call
0808 01:50:56.327387 (+  3762us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:56.331172 (+  3785us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.331500 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:50:56.339130  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:56.340960  9636 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:50:56.342453  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:56.344108  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:56.345208  9634 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 7: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:50:56.321892 13211 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.259s	user 0.017s	sys 0.008s
I20260808 01:50:56.344367  9480 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:56.346634  9632 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: [9.4-9.4]
I20260808 01:50:56.352816 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:50:56.354367  9633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2084) took 1986 ms (client timeout 1681 ms). Trace:
W20260808 01:50:56.356017  9633 rpcz_store.cc:269] 0808 01:50:54.368073 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:54.461488 (+ 93415us) service_pool.cc:224] Handling call
0808 01:50:56.354082 (+1892594us) spinlock_profiling.cc:243] Waited 542 ms on lock 0x7b540027e554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:56.354278 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1246870784}
I20260808 01:50:56.357147  9634 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: []
I20260808 01:50:56.357003  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:56.358820  9633 raft_consensus.cc:2376] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6. Re-sending same reply.
I20260808 01:50:56.360301  9633 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:56.361136  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2014) took 1064 ms. Trace:
I20260808 01:50:56.361147  9636 raft_consensus.cc:2410] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260808 01:50:56.402596  9637 raft_consensus.cc:606] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:56.403155  9637 raft_consensus.cc:614] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:50:56.403594 10366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 530) took 1135 ms. Trace:
I20260808 01:50:56.343518 13012 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:56.404099 10366 rpcz_store.cc:276] 0808 01:50:55.268669 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:55.270862 (+  2193us) service_pool.cc:224] Handling call
0808 01:50:55.287741 (+ 16879us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:55.288038 (+   297us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:50:56.045764 (+757726us) log.cc:844] Serialized 14 byte log entry
0808 01:50:56.403216 (+357452us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:56.403289 (+    73us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:56.403344 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.403412 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:56.403535 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1044311,"prepare.run_cpu_time_us":134,"prepare.run_wall_time_us":2625,"thread_start_us":1108083,"threads_started":2}
W20260808 01:50:56.406337  9630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2000) took 1987 ms (client timeout 1837 ms). Trace:
W20260808 01:50:56.407094  9630 rpcz_store.cc:269] 0808 01:50:54.418746 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:54.990926 (+572180us) service_pool.cc:224] Handling call
0808 01:50:56.364865 (+1373939us) spinlock_profiling.cc:243] Waited 570 ms on lock 0x7b54003fa054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:56.406236 (+ 41371us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1310534400}
I20260808 01:50:56.332940 13277 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:56.417343 13277 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:56.434352 13180 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:56.435316 13180 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:56.310177 13206 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:56.401445  9476 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2323) took 1682 ms (client timeout 1863 ms). Trace:
W20260808 01:50:56.481141  9476 rpcz_store.cc:269] 0808 01:50:54.719502 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:54.736046 (+ 16544us) service_pool.cc:224] Handling call
0808 01:50:56.365382 (+1629336us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b54002bacd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:56.401345 (+ 35963us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":856401280}
I20260808 01:50:56.361563  9633 rpcz_store.cc:276] 0808 01:50:55.297194 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.359612 (+1062418us) service_pool.cc:224] Handling call
0808 01:50:56.360044 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:56.360793 (+   749us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.361065 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:50:56.401793  9550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 2430) took 2765 ms (client timeout 1628 ms). Trace:
W20260808 01:50:56.482429  9550 rpcz_store.cc:269] 0808 01:50:53.636246 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:53.706101 (+ 69855us) service_pool.cc:224] Handling call
0808 01:50:56.367437 (+2661336us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b540029f4d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:56.401691 (+ 34254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3122354432}
I20260808 01:50:56.330694 13036 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:56.402786 13140 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:56.370711 13271 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:56.523013  9559 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260808 01:50:56.401275 13073 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:50:56.465054 13297 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.139s	user 0.005s	sys 0.001s
W20260808 01:50:56.554476 13201 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.760s	user 0.012s	sys 0.000s
W20260808 01:50:56.554474  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 240089us
W20260808 01:50:56.547104 13300 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.219s	user 0.001s	sys 0.000s
I20260808 01:50:56.555272 13201 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:56.495826 13260 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:56.561534 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:56.544317 12937 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:56.562668 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:56.566516  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:56.548305 13175 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:56.546113 13140 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:56.564581 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:56.572808 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:50:56.573995  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:56.551615 13073 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:56.553448 13061 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:56.483865 13012 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:56.587025 10365 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:56.588843  9600 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:56.585996 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:50:56.592916 13298 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.235s	user 0.000s	sys 0.000s
I20260808 01:50:56.577656 13260 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:56.593313 13302 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:56.593132 12937 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election won for term 7
W20260808 01:50:56.598004 13301 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260808 01:50:56.581308 13175 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:56.587872 13180 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:56.595861 13089 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:56.619405 13305 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260808 01:50:56.607427 13302 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:56.603489 13210 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:56.622874 13302 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:56.592921 13140 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:56.641494 13036 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:56.642602 13206 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:56.648172 13260 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:56.649803 13175 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:56.650367 13201 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:56.658573 13309 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.058s	user 0.003s	sys 0.000s
I20260808 01:50:56.674800 13295 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:56.677995 13205 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:56.665609 13222 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:56.694432 13222 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:56.689743 13276 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:56.704397 13276 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:56.801086  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:56.807876  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:56.808732  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 103b913d6c014692b6dea1780af2aa20: Illegal state: Replica 259bcb3f0d254e8b9effac3277e654e0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:50:56.811851  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2007) took 1528 ms. Trace:
I20260808 01:50:56.812757 10362 raft_consensus.cc:2368] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7.
I20260808 01:50:56.812966 10360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 529) took 1578 ms. Trace:
W20260808 01:50:56.815896 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:56.813925 10360 rpcz_store.cc:276] 0808 01:50:55.234236 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:55.238577 (+  4341us) service_pool.cc:224] Handling call
0808 01:50:55.319658 (+ 81081us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:50:55.537746 (+218088us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:55.540520 (+  2774us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:50:55.583086 (+ 42566us) log.cc:844] Serialized 52 byte log entry
0808 01:50:56.199624 (+616538us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:56.199674 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.200363 (+   689us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b5400553054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:56.200419 (+    56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:56.811856 (+611437us) raft_consensus.cc:1685] finished
0808 01:50:56.811915 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:56.812594 (+   679us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":19310,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":188,"spinlock_wait_cycles":392383744,"thread_start_us":615971,"threads_started":1,"wal-append.queue_time_us":724787}
I20260808 01:50:56.825834 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:56.827771 10365 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:50:56.812717  9638 rpcz_store.cc:276] 0808 01:50:55.283707 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.313160 (+1029453us) service_pool.cc:224] Handling call
0808 01:50:56.313600 (+   440us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:56.313917 (+   317us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:56.313956 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:50:56.317629 (+  3673us) log.cc:844] Serialized 26 byte log entry
0808 01:50:56.547014 (+229385us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:56.547072 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.547148 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:50:56.810947 (+263799us) raft_consensus.cc:1685] finished
0808 01:50:56.811036 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:56.811494 (+   458us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":194200,"threads_started":1,"wal-append.queue_time_us":250498}
I20260808 01:50:56.842312  9600 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:56.858959 12937 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:56.867316 12756 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:56.869594 12756 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:56.870811 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 51778.4us
I20260808 01:50:56.875982 12756 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:56.861896 12937 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:56.880863 13033 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:50:56.882025 13033 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:56.882637 13033 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:50:56.887099  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 52567.9us
I20260808 01:50:56.889472  9635 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:56.880671  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:56.878427  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:56.878350 13036 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:56.897414 13206 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:56.898931 13206 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:56.901335 13073 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:56.880954  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 1996) took 3399 ms. Trace:
I20260808 01:50:56.909439 13036 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:56.915571  9638 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:50:56.918342 13036 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:56.905629  9631 rpcz_store.cc:276] 0808 01:50:53.481587 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:53.923388 (+441801us) service_pool.cc:224] Handling call
0808 01:50:53.923806 (+   418us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:54.323046 (+399240us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:50:55.077456 (+754410us) log.cc:844] Serialized 14 byte log entry
0808 01:50:55.678594 (+601138us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:55.678658 (+    64us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:50:55.678719 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.880242 (+1201523us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b540008d254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:56.880323 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:56.880697 (+   374us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9819,"prepare.queue_time_us":162202,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":37,"raft.queue_time_us":3901,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":18430,"spinlock_wait_cycles":917513984,"thread_start_us":1280277,"threads_started":2,"wal-append.queue_time_us":937305}
I20260808 01:50:56.917807 13073 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:56.939970  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:56.918803 13206 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:56.954324  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2316) took 2538 ms. Trace:
I20260808 01:50:56.955026  9475 rpcz_store.cc:276] 0808 01:50:54.415791 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:54.536721 (+120930us) service_pool.cc:224] Handling call
0808 01:50:54.537221 (+   500us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:54.801393 (+264172us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:54.801463 (+    70us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:54.801556 (+    93us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:50:54.801713 (+   157us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.953408 (+2151695us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b54000938d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:56.953488 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:56.953950 (+   462us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":606553856}
W20260808 01:50:56.957775 13307 log.cc:927] Time spent T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.342s	user 0.001s	sys 0.014s
I20260808 01:50:56.957958 13029 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:56.964869 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:56.965585  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:56.972271 12738 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:57.004297 13033 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:57.006412  9638 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 10: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:57.008526 13033 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:56.976107  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34802 (request call id 33) took 2206 ms. Trace:
I20260808 01:50:57.009831  9637 rpcz_store.cc:276] 0808 01:50:54.769780 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.206619 (+1436839us) service_pool.cc:224] Handling call
0808 01:50:56.975546 (+768927us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b540032d054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4498dc33 00007f5d4998d41a 00007f5d42ee1934 00007f5d42ee18c2 00007f5d42ee17f1 00007f5d42ee176c 00007f5d42ee0a22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:56.975821 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":448189824}
I20260808 01:50:57.011205 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:56.980469  9371 catalog_manager.cc:5697] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 1 to 3, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 3 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:50:57.012271 10360 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:50:57.012472  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 439040d168824496acc076b1c280bde3 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:50:57.016346  9636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:50:57.020995 10363 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: []
I20260808 01:50:56.983371  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2008) took 1688 ms. Trace:
I20260808 01:50:56.990597  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1961) took 2196 ms. Trace:
I20260808 01:50:56.994153  9364 maintenance_manager.cc:419] P 3cd8e8730323483db1c5f22e90bb080c: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033390
I20260808 01:50:57.001197 12954 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:57.023815 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:57.003338 13012 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:57.004211 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:57.017699  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:57.000281  9635 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260808 01:50:57.022014  9559 rpcz_store.cc:276] 0808 01:50:54.793891 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:54.994334 (+200443us) service_pool.cc:224] Handling call
0808 01:50:56.521970 (+1527636us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:56.547423 (+ 25453us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.989853 (+442430us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b540029f4d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:56.990264 (+   411us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3235758464}
I20260808 01:50:57.018420  9636 raft_consensus.cc:606] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:57.038574  9636 raft_consensus.cc:614] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:50:57.021857  9632 rpcz_store.cc:276] 0808 01:50:55.294548 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.318453 (+1023905us) service_pool.cc:224] Handling call
0808 01:50:56.336280 (+ 17827us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:56.347317 (+ 11037us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:56.982529 (+635212us) spinlock_profiling.cc:243] Waited 9.94 ms on lock 0x7b54003f9b54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:56.983107 (+   578us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":22851840}
I20260808 01:50:57.028116  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2314) took 3046 ms. Trace:
I20260808 01:50:57.041548  9477 rpcz_store.cc:276] 0808 01:50:53.975062 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:54.408617 (+433555us) service_pool.cc:224] Handling call
0808 01:50:54.408805 (+   188us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:55.523508 (+1114703us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:57.021028 (+1497520us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54001013d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:57.021396 (+   368us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2430754048}
I20260808 01:50:57.081881 10365 raft_consensus.cc:2410] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260808 01:50:57.083531  9371 catalog_manager.cc:5697] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 3 to 4, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 4 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
W20260808 01:50:57.102885 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:57.103696  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 9bf65b9272854b479e73f2cea21e93f2 (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:50:57.107908  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:57.111167 10332 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:57.112135  9631 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:57.113906 13313 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:57.114549 13313 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:57.115129 13275 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:57.115805  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 396) took 1879 ms. Trace:
I20260808 01:50:57.116130  9636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:50:57.115165 13313 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:57.116334  9629 rpcz_store.cc:276] 0808 01:50:55.236166 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.303629 (+1067463us) service_pool.cc:224] Handling call
0808 01:50:56.320657 (+ 17028us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:56.324454 (+  3797us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:50:56.324510 (+    56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:56.324572 (+    62us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:50:56.324618 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:57.115228 (+790610us) spinlock_profiling.cc:243] Waited 3.52 ms on lock 0x7b540003c0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:57.115286 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:57.115603 (+   317us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8100096}
I20260808 01:50:57.120654  9636 raft_consensus.cc:606] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:57.123513  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 2
W20260808 01:50:57.130589  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:57.133538 13295 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:50:57.133960  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:57.134465 13295 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:57.103760  9371 catalog_manager.cc:5697] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 5 to 8, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 8 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:57.114358 13016 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:57.147979 13016 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:57.149822 13299 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.824s	user 0.024s	sys 0.027s
I20260808 01:50:57.116046 13275 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:57.164723  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:50:57.166239  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
W20260808 01:50:57.169587  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:57.175954 13204 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:50:57.178520  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:57.175778  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:57.181295  9550 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:57.179446  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2309) took 3277 ms. Trace:
I20260808 01:50:57.178906 13204 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:57.182551  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2085) took 3256 ms. Trace:
I20260808 01:50:57.185475 13204 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:57.182035  9474 rpcz_store.cc:276] 0808 01:50:53.901719 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:53.902825 (+  1106us) service_pool.cc:224] Handling call
0808 01:50:53.913174 (+ 10349us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:50:56.248205 (+2335031us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:57.178865 (+930660us) spinlock_profiling.cc:243] Waited 2.31 s on lock 0x7b540023aad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:57.179258 (+   393us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5321246976}
I20260808 01:50:57.191231 13016 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:57.171306 13295 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:57.193329 12856 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:57.194274 12856 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:57.196223  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:57.196806 13140 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:57.197523  9474 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:50:57.197714  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:57.198031 13082 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:57.198748  9477 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:50:57.197537 12856 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:57.201409  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2011) took 1905 ms. Trace:
I20260808 01:50:57.166944 13275 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:57.192389 12778 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:50:57.201915  9634 rpcz_store.cc:276] 0808 01:50:55.295699 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:56.348085 (+1052386us) service_pool.cc:224] Handling call
0808 01:50:56.355326 (+  7241us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:56.357611 (+  2285us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:57.200634 (+843023us) spinlock_profiling.cc:243] Waited 6.81 ms on lock 0x7b540027e550. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:57.201002 (+   368us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15673600}
I20260808 01:50:57.173436  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:57.207543  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 55613.2us
I20260808 01:50:57.206878  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2320) took 2488 ms. Trace:
I20260808 01:50:57.215849  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:57.225184  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:57.207007 12778 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:57.228904  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:57.183208  9481 rpcz_store.cc:276] 0808 01:50:53.925751 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:54.320013 (+394262us) service_pool.cc:224] Handling call
0808 01:50:54.320372 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:54.320622 (+   250us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:50:55.089344 (+768722us) log.cc:844] Serialized 14 byte log entry
0808 01:50:55.763121 (+673777us) log.cc:844] Serialized 14 byte log entry
0808 01:50:57.164327 (+1401206us) spinlock_profiling.cc:243] Waited 9.92 ms on lock 0x7b1800289960. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:50:57.181338 (+ 17011us) spinlock_profiling.cc:243] Waited 2.63 ms on lock 0x7b1800289960. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667 00007f5d449ae5e8
0808 01:50:57.181710 (+   372us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:57.181822 (+   112us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:50:57.182001 (+   179us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:57.182167 (+   166us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:57.182342 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27314,"prepare.queue_time_us":108836,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":106,"raft.queue_time_us":139,"raft.run_cpu_time_us":14671,"raft.run_wall_time_us":186193,"spinlock_wait_cycles":28849920,"thread_start_us":1403778,"threads_started":2,"wal-append.queue_time_us":934041,"wal-append.run_cpu_time_us":1951,"wal-append.run_wall_time_us":1006807}
I20260808 01:50:57.214426 13016 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:57.239930  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:57.240049  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:57.244797 12778 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:57.245434  9477 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:50:57.246703  9474 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260808 01:50:57.246829  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:57.212723 13204 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:57.251137  9559 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:50:57.255362  9631 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:50:57.265311  9479 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:50:57.268450  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:57.268200 13204 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:57.270150 12804 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:57.270869 12804 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:57.271275 12804 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:57.271342  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:57.272089 13204 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:57.233868 10362 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 527) took 2098 ms (client timeout 1958 ms). Trace:
W20260808 01:50:57.273097 10362 rpcz_store.cc:269] 0808 01:50:55.129064 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:55.132949 (+  3885us) service_pool.cc:224] Handling call
0808 01:50:56.352610 (+1219661us) spinlock_profiling.cc:243] Waited 2.78 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:50:57.212480 (+859870us) spinlock_profiling.cc:243] Waited 458 ms on lock 0x7b5400594cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:57.225308 (+ 12828us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1059847296}
I20260808 01:50:57.276551  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:57.278318  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:57.278904  9445 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:57.279269  9559 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:57.279928 10365 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:57.280021 12778 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:57.280682  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:57.281041 12778 raft_consensus.cc:697] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:57.282200 12778 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:57.283839  9554 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260808 01:50:57.282361 13313 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:57.287078  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:57.288343 12804 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:57.289613  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:57.280371  9445 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:57.301172 13012 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:57.301551  9551 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:57.314400 12804 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:57.289647 13313 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:57.320920 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:57.326810  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:57.323602  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50765.5us
I20260808 01:50:57.259850  9371 catalog_manager.cc:5697] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:57.340143  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:57.302187 13012 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:57.355037  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:57.360877  9559 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:50:57.325380 13016 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:57.334589 13204 consensus_queue.cc:1059] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.030s
W20260808 01:50:57.370591 10368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 492) took 3420 ms (client timeout 1747 ms). Trace:
I20260808 01:50:57.375633  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2070) took 3458 ms. Trace:
I20260808 01:50:57.375941 13244 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:57.376690 13244 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:57.376207  9473 rpcz_store.cc:276] 0808 01:50:53.917417 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:53.922432 (+  5015us) service_pool.cc:224] Handling call
0808 01:50:56.334541 (+2412109us) spinlock_profiling.cc:243] Waited 47 ms on lock 0x7b5800005498. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:50:56.334965 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:56.335250 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 5
0808 01:50:56.341907 (+  6657us) log.cc:844] Serialized 14 byte log entry
0808 01:50:57.351601 (+1009694us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:57.363637 (+ 12036us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:50:57.370939 (+  7302us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:57.373350 (+  2411us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:57.375371 (+  2021us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":648535,"prepare.queue_time_us":7058,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":655074,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":226,"spinlock_wait_cycles":108153600,"thread_start_us":334137,"threads_started":2,"wal-append.queue_time_us":270831}
I20260808 01:50:57.378868  9550 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:57.379132 13316 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20260808 01:50:57.342327 12804 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
W20260808 01:50:57.373927 10368 rpcz_store.cc:269] 0808 01:50:53.934535 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:53.934730 (+   195us) service_pool.cc:224] Handling call
0808 01:50:57.333547 (+3398817us) spinlock_profiling.cc:243] Waited 2.1 s on lock 0x7b540056fed4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:50:57.354627 (+ 21080us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4825426688}
I20260808 01:50:57.384232  9551 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 8: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:57.388301  9371 catalog_manager.cc:5697] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 1 to 4, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 4 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:57.370533 13134 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:50:57.395644 13134 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:57.399556  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:57.396131 13134 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:57.390583  9371 catalog_manager.cc:5697] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 2 to 6, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 6 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:57.382566 13244 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:57.400975 13204 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:57.404626 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:57.405964 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:57.406241  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:57.407018  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:57.407004 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:57.409780  9521 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:57.412135 13204 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:57.412711  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:57.413398 10368 raft_consensus.cc:2368] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7.
I20260808 01:50:57.416497  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:57.421872  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 1.72581e+06us
I20260808 01:50:57.423273 10332 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:57.430600 13016 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:57.432950 13012 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:57.433776 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:57.209240  9478 rpcz_store.cc:276] 0808 01:50:54.718587 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:54.832236 (+113649us) service_pool.cc:224] Handling call
0808 01:50:54.832742 (+   506us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:54.839244 (+  6502us) raft_consensus.cc:1534] Early marking committed up to index 4
0808 01:50:54.913816 (+ 74572us) log.cc:844] Serialized 14 byte log entry
0808 01:50:57.085629 (+2171813us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b18003af120. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d4453242a 00007f5d4499de0a
0808 01:50:57.206226 (+120597us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:50:57.206303 (+    77us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:50:57.206376 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:57.206494 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:50:57.206625 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":75204,"prepare.queue_time_us":74286,"prepare.run_cpu_time_us":122,"prepare.run_wall_time_us":124,"raft.queue_time_us":78613,"spinlock_wait_cycles":39947648,"thread_start_us":19795,"threads_started":1,"wal-append.queue_time_us":2244248}
I20260808 01:50:57.441444 13082 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:57.445587 10360 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:50:57.447263 13295 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:50:57.458462 13295 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:57.458998 13295 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:57.447448 13206 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:50:57.456012  9550 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
W20260808 01:50:57.448319 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:57.466825  9481 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:57.467451  9481 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:50:57.467885  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 84591.6us
W20260808 01:50:57.469386  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:57.448477 13313 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:57.450554  9371 catalog_manager.cc:5697] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 1 to 2, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 2 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:57.471212  9481 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:57.453536  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 50582us
I20260808 01:50:57.441148 13204 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:57.474968 12804 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:57.478667  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:57.479605  9523 leader_election.cc:400] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7.
I20260808 01:50:57.484707  9523 leader_election.cc:403] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260808 01:50:57.480123  9600 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:57.488095 10368 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 3. Current term is 4. Ops: []
I20260808 01:50:57.484126  9371 catalog_manager.cc:5697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 2 to 5, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 5 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:57.494231  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:57.497784 10365 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:57.484376 12804 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:57.479338 13313 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:57.518335  9557 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:57.495164 13012 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:50:57.520876  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:57.524549 13325 log.cc:927] Time spent T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260808 01:50:57.525342  9481 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:57.525904  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:57.516928 12804 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:57.536340  9478 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:50:57.544222 10367 raft_consensus.cc:2410] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260808 01:50:57.506287 13016 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7.
I20260808 01:50:57.515764  9371 catalog_manager.cc:5697] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:57.549576 10366 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260808 01:50:57.554872  9473 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:50:57.561410  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:57.563032 13087 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:57.565238  9479 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:57.567319 10334 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:57.568207  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:57.569123 10334 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:57.570438 13082 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:57.575104 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:57.575685 13082 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:57.576586 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:57.578922  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:57.579099 13016 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:50:57.592480  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:57.593137  9521 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:57.594005  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:57.544835 13012 raft_consensus.cc:697] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:57.625355  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:57.626318  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:57.606389 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:57.640509  9481 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260808 01:50:57.611326 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:50:57.688485  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:57.609074  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:57.742250 10363 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260808 01:50:57.746979 10363 pending_rounds.cc:85] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting all ops after (but not including) 7
I20260808 01:50:57.747326 10363 pending_rounds.cc:107] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260808 01:50:57.747681 10363 raft_consensus.cc:2889] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:50:57.743099 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 146560us
I20260808 01:50:57.607298 13145 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:50:57.578526 13082 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:57.610055 12693 raft_consensus.cc:3055] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:50:57.594527 13319 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:57.655303 12745 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:57.764341  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:57.763437  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:57.763506 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 194401us
I20260808 01:50:57.749037  9371 catalog_manager.cc:5697] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 2 to 5, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 5 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:57.771625  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:57.611652 13087 consensus_queue.cc:1059] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.146s
I20260808 01:50:57.748989 13244 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:57.607708 13295 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:57.777935 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:57.750876 13140 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:57.757337 13145 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:57.783730  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:57.625366 13134 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:57.595080 13204 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.177s
W20260808 01:50:57.793215  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:57.794741  9477 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:57.795490  9474 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:50:57.795992 13328 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260808 01:50:57.796363 12819 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.244s
I20260808 01:50:57.776247 12745 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:57.770135 12693 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:57.606082 13313 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:57.799860 13222 consensus_queue.cc:1059] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260808 01:50:57.799463  9371 catalog_manager.cc:5697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 3 to 7, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 7 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:50:57.800987  9555 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:57.803335  9473 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:57.806960 13233 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:57.807169  9550 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:50:57.763356 13036 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:57.808490 13140 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:57.809399  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:57.810370  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:57.819224 13145 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:57.750465 13206 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260808 01:50:57.774240 13319 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:57.799129  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 205285us
I20260808 01:50:57.843601 10332 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:57.860692  9481 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:50:57.779816 13246 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:57.861721  9550 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:57.858598  9479 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:57.808530 13295 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:57.843291 12738 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:57.883165  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:57.864655 13140 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:50:57.860658  9445 connection.cc:256] RPC call timeout handler was delayed by -1.8471723679977003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17470000000000002s for a response
I20260808 01:50:57.843521 12745 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:57.874768 13233 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:57.874987 13244 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:57.877436  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:57.874856 13303 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:57.878779 13183 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:57.878983  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:57.879556 13012 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:57.871940 13319 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:57.886206  9478 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:57.870914  9475 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:57.878360  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:57.902952  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:57.815862  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:57.917004  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:57.864058 13082 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:57.913981  9551 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:57.923388  9560 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260808 01:50:57.924015  9555 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:50:57.869062 13036 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:57.927506  9371 catalog_manager.cc:5697] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260808 01:50:57.928472  9524 leader_election.cc:336] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.544s (SENT)
I20260808 01:50:57.897938 13246 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:57.932838  9524 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:57.936518  9556 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:50:57.937861  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:57.899885 12804 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:57.949882  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:57.950138  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:57.935755 10362 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: []
I20260808 01:50:57.961680  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:57.962220 13112 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260808 01:50:57.962615  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:57.966144 13332 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260808 01:50:57.929913 13183 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:57.970858  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:57.972699 13036 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:57.973117 13244 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:57.973395  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:57.976298 13233 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:57.976998 12693 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:57.977715 13134 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:57.959766 13082 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:57.970901  9555 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:57.924401 12738 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:57.958489  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:57.914980 13303 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:57.991328  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 76422us
I20260808 01:50:57.996757  9550 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.001366  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.002404  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:58.004000  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.035935  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:57.945609 13295 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:58.044193  9551 raft_consensus.cc:2410] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 3.
W20260808 01:50:58.044662 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 267016us
I20260808 01:50:58.046641  9479 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:50:58.051281  9480 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:58.080541  9473 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:58.053705  9558 raft_consensus.cc:2393] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 9: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:50:58.054991  9557 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.513s	user 0.011s	sys 0.003s
I20260808 01:50:58.055557  9474 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:50:58.081465  9557 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:50:58.069758  9556 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:50:58.053188  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.040400 13016 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:58.085130  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.083303 10334 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:58.086119  9473 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
W20260808 01:50:58.090588  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 272052us
I20260808 01:50:58.091292 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:58.109560  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:58.110575  9479 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:58.111935  9472 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:58.118021  9552 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260808 01:50:58.121676  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.122895  9474 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:50:58.078306 13295 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:58.082123 13036 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:58.127225  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:58.129565  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:58.087625 13244 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:58.130092  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:58.131217  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:58.139865  9558 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:58.141808  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.149523  9550 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:58.150218  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.079953 13233 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20260808 01:50:58.149403  9476 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
W20260808 01:50:58.156208  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:58.134117 13295 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:58.132642 12693 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:58.161398  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.144497 12738 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:58.174026  9444 leader_election.cc:400] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8.
I20260808 01:50:58.174552  9444 leader_election.cc:403] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:50:58.171226  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.177152  9481 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
W20260808 01:50:58.178494  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 242695us
I20260808 01:50:58.182276  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:58.182473  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 573478us
W20260808 01:50:58.183074  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:58.185825 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 92967.9us
I20260808 01:50:58.181766  9478 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
W20260808 01:50:58.189577 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 412329us
I20260808 01:50:58.186899  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:58.191290  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:58.191768  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:58.160005 13134 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20260808 01:50:58.196977  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:58.203013 13343 raft_consensus.cc:2764] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260808 01:50:58.203996 13343 consensus_queue.cc:1059] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
W20260808 01:50:58.205132  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 290114us
W20260808 01:50:58.207119  9524 leader_election.cc:336] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.566s (SENT)
I20260808 01:50:58.207615  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:58.196446  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:58.176734 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:58.211663  9473 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 7: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:58.213171  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:58.150588 13036 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:58.216022 12826 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.032s	sys 0.063s
I20260808 01:50:58.159487 13244 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.217134  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.218408 13140 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:58.219600  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:58.220288  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 45926.8us
W20260808 01:50:58.224746  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:58.266117  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:58.189819  9599 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.194:36095 timed out after 1.540s (SENT) [suppressed 31 similar messages]
I20260808 01:50:58.231376  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:58.232519  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.210897 13342 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:58.271572 13082 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:58.274139 13341 raft_consensus.cc:3055] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:58.274644 13341 raft_consensus.cc:740] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:58.262336  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.229275 13320 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:58.185101 13340 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:50:58.166654 13246 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.190778  9560 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:50:58.196416 12778 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8.
I20260808 01:50:58.215405 13336 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader election won for term 8
W20260808 01:50:58.245829 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50261.6us
I20260808 01:50:58.197849 12738 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260808 01:50:58.251772  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:58.296409  9479 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:50:58.297652  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:58.299252  9479 raft_consensus.cc:2393] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:58.253846  9555 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:50:58.265201  9477 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:50:58.266700  9521 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:50:58.263082  9554 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
W20260808 01:50:58.269989 13334 log.cc:927] Time spent T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.067s	user 0.001s	sys 0.001s
I20260808 01:50:58.270159  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:58.277194  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.279636  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.283666  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.286036  9481 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:50:58.286283  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:58.265815 13337 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:50:58.287388  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 104946us
W20260808 01:50:58.262995 12826 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.693s	user 0.032s	sys 0.064s
I20260808 01:50:58.289038  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:50:58.280162  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 118798us
I20260808 01:50:58.302318  9474 raft_consensus.cc:2410] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 8 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260808 01:50:58.302548  9558 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:58.305142  9557 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:58.301769  9476 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:50:58.307206  9480 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:58.309692  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.318826  9473 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: []
W20260808 01:50:58.319537  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 112470us
W20260808 01:50:58.320320  9524 leader_election.cc:336] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.617s (SENT)
I20260808 01:50:58.320670  9524 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:58.320964  9473 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: [4.3-9.4]
I20260808 01:50:58.322288 13352 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:58.324091 13320 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.324855 13036 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:58.325248 12819 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:50:58.325747 12819 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:58.326113 13244 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:58.318185 13342 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.291625 13341 consensus_queue.cc:260] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:58.291189 13340 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:58.331038 13342 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:58.331489 13340 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:58.334257 13206 raft_consensus.cc:3055] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:58.347535 13206 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:58.276959 12745 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:58.335075 12826 raft_consensus.cc:3055] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:58.337702 13145 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.294651 12804 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:58.369865  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:58.308400 13336 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:50:58.339195  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:58.343261 12819 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.344525  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:58.345196 13244 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:58.289932 13347 consensus_queue.cc:1059] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.082s
W20260808 01:50:58.354362  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:58.348937 13206 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:58.354472 13319 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.355636  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.356452  9475 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:58.356760  9474 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260808 01:50:58.357697  9477 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:58.358453 13320 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:50:58.364802  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 118932us
I20260808 01:50:58.366601  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:58.366883 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 109336us
I20260808 01:50:58.256940 13012 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:58.338480 13205 raft_consensus.cc:3055] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:50:58.368053  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:58.373157  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 100062us
I20260808 01:50:58.395139 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:58.325562 13036 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.049s
I20260808 01:50:58.389745  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.363842 12826 raft_consensus.cc:740] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:58.400002  9478 raft_consensus.cc:2368] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7.
I20260808 01:50:58.401342 13145 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:58.401989 12778 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:50:58.382818 13303 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:58.388011 12738 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:58.415027  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:58.413494  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.414268  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.412613  9558 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:50:58.441378  9475 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:58.397330 13183 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:58.412938 13082 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260808 01:50:58.446645 13082 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:58.447294 13082 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:58.447446 10332 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:58.444693  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.396080 13205 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:58.418845  9473 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:50:58.450716  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:58.403144 13346 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:50:58.459601  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:58.404979 13206 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:58.456013  9481 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:50:58.464172  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:58.421597 12819 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:58.419404 10362 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260808 01:50:58.413014 10363 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:58.471076  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.473584  9550 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:50:58.466459 12738 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.420794 12826 consensus_queue.cc:260] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:58.476091 13140 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:58.471982  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.491776  9476 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 7: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:58.483175  9479 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:58.482477  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.493052  9473 raft_consensus.cc:2410] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 9 because replica has last-logged OpId of term: 7 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260808 01:50:58.411453 13343 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260808 01:50:58.413537 13348 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:50:58.487010  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:58.393297 13244 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:58.488651 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:58.489451  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:58.503845 12745 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:58.504416 12738 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:58.497951 13145 consensus_queue.cc:1175] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "259bcb3f0d254e8b9effac3277e654e0" responder_term: 8 status { last_received { term: 5 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 01:50:58.505172 12738 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:50:58.506863 13319 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:58.507124 10361 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:58.508332  9472 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:58.491195 13036 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:50:58.509323 13036 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:58.510403 13036 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:58.511149  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:58.512243 13348 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:58.512727  9476 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: []
I20260808 01:50:58.513376 13348 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.513960 13350 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:50:58.496215  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:58.497844  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:58.466992 13342 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:50:58.494335  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:58.484894  9524 leader_election.cc:336] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.947s (SENT)
I20260808 01:50:58.497575 12826 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:58.411444 13134 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.076s
I20260808 01:50:58.496833 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:58.516876  9479 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260808 01:50:58.523905 13244 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:50:58.525188  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 76035.9us
I20260808 01:50:58.448338 13303 consensus_queue.cc:1059] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.047s
I20260808 01:50:58.526530 13134 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:50:58.526472  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:58.527298 13354 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:58.527659 13183 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:58.527842 13354 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:58.528256 13183 raft_consensus.cc:697] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:58.529335 13183 consensus_queue.cc:237] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:58.532418 13335 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.268s	user 0.005s	sys 0.000s
I20260808 01:50:58.415259 13012 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:58.554922  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 210565us
W20260808 01:50:58.555601  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 70771us
W20260808 01:50:58.570011  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:58.574378  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:58.527300 13303 raft_consensus.cc:3055] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Stepping down as leader of term 8
W20260808 01:50:58.588989  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 53120us
W20260808 01:50:58.615620  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 138092us
I20260808 01:50:58.616042  9599 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:58.528282 13354 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:58.534749 13204 raft_consensus.cc:3055] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:50:58.527314 13134 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:58.619935  9601 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:58.614372 13303 raft_consensus.cc:740] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:58.622099 13342 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:58.622570 13342 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:58.621802 13177 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
I20260808 01:50:58.623013 13303 consensus_queue.cc:260] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.623008 13342 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:58.624704 13303 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:58.633136 13342 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.635615 13342 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:58.636318 13342 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:58.636309  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:58.636996 13205 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.637739 13342 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.639858 13365 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:50:58.642570  9601 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:58.643739 13358 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260808 01:50:58.645437 13342 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:58.646664 13365 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:58.647370 13365 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:58.647925 13365 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:50:58.617908 13204 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:58.649855 13204 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.651305 13204 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:58.615168 13082 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.653272 13082 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:58.654417 10360 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:58.415410 13336 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:58.664765  9524 leader_election.cc:336] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 2.066s (SENT)
I20260808 01:50:58.665329  9524 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:58.552297 13340 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:58.676743  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:58.677423  9601 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:58.682504  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:58.683214  9480 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:58.647058 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:58.638551 13205 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:58.676594 10334 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:50:58.689831 13134 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:58.692580 13012 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:58.686420 13177 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:58.694202 13363 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election won for term 9
W20260808 01:50:58.679505 13368 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.095s	user 0.003s	sys 0.000s
I20260808 01:50:58.682703 12826 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
W20260808 01:50:58.707085  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 91131us
W20260808 01:50:58.707623  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 86921.4us
W20260808 01:50:58.708398 13295 consensus_meta.cc:287] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.524s	user 0.008s	sys 0.006s
I20260808 01:50:58.711402  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:58.713920  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:58.718538  9558 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260808 01:50:58.710613 13340 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:58.680233 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:58.723354  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:58.699455 13342 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:58.723778  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:50:58.736985 10332 leader_election.cc:336] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.725s (SENT)
W20260808 01:50:58.749652  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 292740us
I20260808 01:50:58.722162  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:58.750272  9473 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:50:58.766090  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 80982.3us
I20260808 01:50:58.769619 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:50:58.771374  9599 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:58.772456  9601 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:58.754629 13363 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:58.774739  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 60504.6us
I20260808 01:50:58.778604 10364 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: []
W20260808 01:50:58.780764  9524 leader_election.cc:336] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 2.131s (SENT)
W20260808 01:50:58.790405 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:58.769299 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:58.780083 13320 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:58.796309 13342 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:58.796470 13320 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:58.797137 13342 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.799207 13367 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:50:58.799520 10363 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 9: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:58.800068 13356 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:50:58.800596 13356 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:58.800983 13356 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:58.802054  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:58.799796 13342 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:58.755184 13295 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:58.810946  9480 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:50:58.813279  9524 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:58.778587 12738 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:50:58.821300 10334 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:58.823235  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:58.827730  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:58.829418 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 92493.5us
W20260808 01:50:58.831074 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:58.833205 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:58.824333 13345 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:50:58.839251 13295 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:58.840420 10369 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:58.844766 13346 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260808 01:50:58.855158 13347 raft_consensus.cc:2764] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260808 01:50:58.808113 13367 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260808 01:50:58.843082  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 71867.5us
I20260808 01:50:58.803591 13012 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:50:58.816831 13355 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:50:58.849359  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:58.842571 13357 raft_consensus.cc:3055] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:50:58.851356  9601 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:58.852628  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:58.864295  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:58.877173  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 148002us
I20260808 01:50:58.845119 12738 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:58.798444 13320 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.879776  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:58.880664  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:58.848340 13348 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:50:58.863627 13367 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:58.863286 13012 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:58.864297 13345 raft_consensus.cc:697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:58.870491 13357 raft_consensus.cc:740] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:58.895766  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 130674us
W20260808 01:50:58.903551  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:58.916399  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:58.918455  9524 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:50:58.887817 13206 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:50:58.938900 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:58.940093  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 51761.7us
I20260808 01:50:58.940596 12738 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.944075 12819 raft_consensus.cc:3055] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:58.892163 13367 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:58.948434 13357 consensus_queue.cc:260] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.950361 10366 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:58.957751 12826 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:50:59.027266 12826 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:50:58.958782  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 51878.6us
I20260808 01:50:58.924047 13347 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260808 01:50:58.960263 13366 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.165s	user 0.000s	sys 0.001s
W20260808 01:50:58.963865  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:58.876343 13355 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:58.995478 13345 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:58.991050 13375 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260808 01:50:58.900983 13012 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:50:58.963274 13363 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:58.987149  9601 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:58.997972 13365 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:59.034293 12826 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:50:59.022086 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 99636.6us
I20260808 01:50:59.008509 13342 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:50:59.036792 13357 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:59.041355  9473 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:50:59.016800 12819 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:58.954290 13341 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:59.033450 13295 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:50:59.062094  9524 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:59.076462 10334 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:50:59.083318  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 167047us
I20260808 01:50:59.081562  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:59.083787 13313 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:50:59.090070 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:50:59.090443 13313 raft_consensus.cc:697] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:50:59.091637 13313 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:59.095472 13319 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:59.095331 12819 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:59.044709 13244 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.040s
I20260808 01:50:59.097730 12819 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:59.064026 13205 consensus_queue.cc:1059] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260808 01:50:59.099680  9599 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:59.106910 13320 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:50:59.108789 12826 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:59.086628 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:59.096077 13319 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:59.089672 13353 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:50:59.069706 13342 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:50:59.127769 13244 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:59.128841 13244 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260808 01:50:59.129685 13379 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
I20260808 01:50:59.131471 13012 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:59.134312 13012 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:59.135099 13205 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:50:59.072804 13354 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:59.138224 13347 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:59.157387  9476 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260808 01:50:59.121884 13319 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:59.141894 13183 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:50:59.179963  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:59.180904  9480 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7. Ops: []
I20260808 01:50:59.142964 13342 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:59.138616 13177 raft_consensus.cc:3055] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:50:59.080163 13381 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.079s
I20260808 01:50:59.180935  9473 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:59.141536 13365 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:59.084221 13134 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.077s
I20260808 01:50:59.138875 13353 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:50:59.159560  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 220795us
I20260808 01:50:59.185104 10365 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260808 01:50:59.149751 12745 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:59.191604  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 96626us
I20260808 01:50:59.151618 13347 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:50:59.193738  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:50:59.207890  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 151842us
W20260808 01:50:59.207912  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 106439us
I20260808 01:50:59.209609  9599 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:59.176693 13183 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:59.201634 13347 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:59.185679 13177 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:59.227373  9473 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:50:59.228381  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.230139  9634 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 5. Current term is 6. Ops: []
W20260808 01:50:59.230389 13303 consensus_meta.cc:287] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.605s	user 0.016s	sys 0.000s
I20260808 01:50:59.175720 13356 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:59.231119  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.231925  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:50:59.223366 13382 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.160s	user 0.000s	sys 0.001s
I20260808 01:50:59.233548 13244 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:50:59.234189 13394 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:50:59.237049  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:59.211596 13082 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:50:59.243184  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:59.249491  9558 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:59.249965  9550 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:59.233444  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:59.245604  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:59.207026 12745 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:59.260139  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:59.260399  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:59.191756 13391 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260808 01:50:59.262952  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.263401  9551 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:59.249064 13394 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:59.255054  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.255793  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.256583  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:59.257895  9630 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: [5.4-9.6]
I20260808 01:50:59.254655  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:59.244266 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 58575us
I20260808 01:50:59.269153  9629 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:59.263744  9636 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:59.263976  9635 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
I20260808 01:50:59.270911  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:59.276346  9634 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:59.277704  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:59.278017 10368 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: []
I20260808 01:50:59.270306 13394 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:50:59.281286 13388 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.164s	user 0.001s	sys 0.000s
W20260808 01:50:59.282895 13378 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.333s	user 0.002s	sys 0.003s
I20260808 01:50:59.285629 10363 raft_consensus.cc:2410] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260808 01:50:59.287761 12745 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:59.287953 12826 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:59.291105 13353 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:59.293239 13356 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:59.296305 12826 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:50:59.276041 13355 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:59.301034 13183 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:59.298282  9524 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:59.305250 13367 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:59.307822 13367 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:50:59.308817 13205 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:50:59.318825 13205 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260808 01:50:59.322472  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:59.298539 13389 log.cc:927] Time spent T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Append to log took a long time: real 0.296s	user 0.000s	sys 0.003s
I20260808 01:50:59.302374 13183 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:50:59.333051  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 87575.9us
W20260808 01:50:59.334772  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:59.354511  9444 connection.cc:256] RPC call timeout handler was delayed by -4.1860056899895426s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.259s for a response
I20260808 01:50:59.267056 13082 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:50:59.337190  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:59.334048  9524 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:59.339675  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
W20260808 01:50:59.359686 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:59.339951  9631 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:59.366133  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:59.368896  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.368976  9630 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260808 01:50:59.335361  9550 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:50:59.371289  9637 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:50:59.371374 13397 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:50:59.372335  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:59.372838 13205 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:50:59.364133 13244 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:59.376699 13354 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:59.376219 13082 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:59.377799 13347 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:59.380249  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:59.380079 13319 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:59.381083  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2140) took 1007 ms. Trace:
W20260808 01:50:59.382552 13383 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.431s	user 0.001s	sys 0.001s
I20260808 01:50:59.360313 13391 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:59.384656  9638 raft_consensus.cc:2410] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260808 01:50:59.385685  9599 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:59.458091  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.458871 12826 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:59.387205  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:59.388302 13396 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:59.459506  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.389271  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:50:59.388829  9444 connection.cc:256] RPC call timeout handler was delayed by -3.4603209949855227s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15900000000000003s for a response
I20260808 01:50:59.395018  9524 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:59.396461 13246 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:50:59.396669  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:59.465159  9444 connection.cc:256] RPC call timeout handler was delayed by -3.3784209949953947s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16810000000000003s for a response
I20260808 01:50:59.396669  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:59.397603  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.398361 12819 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:59.467803  9635 raft_consensus.cc:2410] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
W20260808 01:50:59.469029  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 74097.8us
I20260808 01:50:59.466491  9636 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:59.427472 13205 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:50:59.465256  9638 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
W20260808 01:50:59.465807  9444 connection.cc:256] RPC call timeout handler was delayed by -3.2353209950088058s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18400000000000002s for a response
I20260808 01:50:59.397294 12778 consensus_queue.cc:1059] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260808 01:50:59.466531  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:59.452418  9476 rpcz_store.cc:276] 0808 01:50:58.374282 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:58.513579 (+139297us) service_pool.cc:224] Handling call
0808 01:50:58.513888 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:50:59.159139 (+645251us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:50:59.380818 (+221679us) spinlock_profiling.cc:243] Waited 634 ms on lock 0x7b54002bb6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:50:59.381018 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1458822912}
I20260808 01:50:59.459527 12826 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:59.469849  9524 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:59.481630  9633 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: []
W20260808 01:50:59.535504  9444 connection.cc:256] RPC call timeout handler was delayed by -2.8933209950046148s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22200000000000003s for a response
W20260808 01:50:59.441128  9521 leader_election.cc:336] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.958s (SENT)
I20260808 01:50:59.464102 13082 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:59.540647  9634 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
W20260808 01:50:59.495651  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 110104us
I20260808 01:50:59.542902  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:50:59.543533  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:59.548144  9444 connection.cc:256] RPC call timeout handler was delayed by -2.8870209949964192s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2227s for a response
I20260808 01:50:59.491362  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.471540  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.549393  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:59.460887 13357 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:59.550894 13381 raft_consensus.cc:3055] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:50:59.548301 13363 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260808 01:50:59.559850  9444 connection.cc:256] RPC call timeout handler was delayed by -2.6872209949942771s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2449s for a response
W20260808 01:50:59.563071  9444 connection.cc:256] RPC call timeout handler was delayed by -2.6143209949950688s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.253s for a response
W20260808 01:50:59.563372  9444 connection.cc:256] RPC call timeout handler was delayed by -2.4181209949892946s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27480000000000004s for a response
W20260808 01:50:59.563695  9444 connection.cc:256] RPC call timeout handler was delayed by -2.0680209950078279s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31370000000000003s for a response
I20260808 01:50:59.498200 13303 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:59.560813  9550 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:59.494607 13246 consensus_queue.cc:1243] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 7201605bb2f54c949a4ebe6e9b3b4b2a log is divergent from this leader: its last log entry 8.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260808 01:50:59.471897 13348 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:50:59.508924 13319 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:59.569674  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:50:59.569442  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 139318us
I20260808 01:50:59.561941  9637 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 8: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:59.565426  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.538899 13205 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:50:59.559494  9636 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:59.564061  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.539397 12819 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:59.459820 13396 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:59.516662 13347 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:50:59.578943  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 109133us
I20260808 01:50:59.568809  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:59.588886  9634 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:59.462833 13244 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:59.602154  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:59.585219  9632 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:50:59.654886  9599 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:59.655267  9631 raft_consensus.cc:2376] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8. Re-sending same reply.
I20260808 01:50:59.656121  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:59.611699  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:59.657058  9637 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 6
I20260808 01:50:59.567966 13303 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:50:59.659220  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.660174  9632 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 8: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:50:59.659871 13303 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:59.660990  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 90901.3us
I20260808 01:50:59.661198  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.661679  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:59.561741 13381 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:50:59.571723 13348 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:59.662323  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:59.630751  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.648841  9558 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:59.552582 13342 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:59.652729  9550 raft_consensus.cc:2410] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260808 01:50:59.564118 13357 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:59.573827  9521 leader_election.cc:336] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.040s (SENT)
W20260808 01:50:59.701814  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 128057us
I20260808 01:50:59.662040  9632 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:50:59.703022  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:59.704087  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:59.675201 13303 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:59.677042  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.711020  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:59.677695 13396 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:59.661361 12819 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:59.679306  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:59.711274  9633 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:59.724129  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.680060  9558 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:50:59.683102  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:59.734061  9477 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:59.683935  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:59.684653  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:59.677695  9635 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:50:59.684562  9444 leader_election.cc:400] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6.
I20260808 01:50:59.686574  9481 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:50:59.736930  9444 leader_election.cc:403] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260808 01:50:59.737036  9481 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:50:59.689991 13395 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.429s	user 0.000s	sys 0.014s
I20260808 01:50:59.692623  9637 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:50:59.566218 13204 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260808 01:50:59.756971 13399 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.446s	user 0.010s	sys 0.004s
I20260808 01:50:59.757551  9601 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:59.759425 13353 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:59.759923 13353 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:59.760402 13353 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:59.483389 13177 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:59.762946 13177 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:59.763890  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:59.764662  9630 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:59.704553  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:59.767160 13357 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:59.705363  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:59.768162  9472 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:59.767967  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:59.768617  9474 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:50:59.768392  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:59.769007  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:59.770296  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:59.714741 13244 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:59.722288  9476 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:50:59.792874  9476 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260808 01:50:59.729185  9632 raft_consensus.cc:2368] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7.
I20260808 01:50:59.694257 13348 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:59.691381  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:59.795857  9638 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:59.738020  9475 raft_consensus.cc:2393] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 7: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:50:59.739653  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 77406.3us
I20260808 01:50:59.798229  9601 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:50:59.730878 13365 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 01:50:59.752166  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:59.760265  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:50:59.695113  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:50:59.714346 13406 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.138s	user 0.006s	sys 0.006s
I20260808 01:50:59.813584  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:59.814312  9632 raft_consensus.cc:2376] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10. Re-sending same reply.
I20260808 01:50:59.771770 13381 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:59.821552  9474 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
W20260808 01:50:59.822821  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 138425us
I20260808 01:50:59.771826  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:59.812585 13397 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:50:59.791131  9630 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
W20260808 01:50:59.740640 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:59.799678  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:59.893586 13204 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:50:59.894449 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:59.804975  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:59.895316 10367 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:59.896200  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.896469 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:50:59.896883  9633 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:50:59.807991  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:59.897795  9551 raft_consensus.cc:2393] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 10: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:50:59.771260 12826 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:50:59.898870 12826 raft_consensus.cc:3055] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:50:59.899251 12826 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:50:59.824326  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.900321 12826 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:50:59.901245  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:59.901605  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.828935  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:59.902503  9600 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:59.903241  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.903868  9634 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:50:59.897797  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:59.904917  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.905165  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:59.906188  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:59.906330  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:50:59.830289  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:50:59.908595  9637 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:50:59.831185  9477 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260808 01:50:59.833096  9480 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:50:59.911386  9377 catalog_manager.cc:5697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:50:59.914089  9377 catalog_manager.cc:5697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:50:59.916093 13140 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:50:59.917834  9377 catalog_manager.cc:5697] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 4 to 9, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 9 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:50:59.918936 13354 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:50:59.781913  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:50:59.794582 13396 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:50:59.880751  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:59.928032  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:59.928390  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:59.901968  9631 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:50:59.907063  9633 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:50:59.923338  9634 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:50:59.930436  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:59.920680  9377 catalog_manager.cc:5697] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:50:59.939797  9481 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:50:59.923214  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 90329.3us
I20260808 01:50:59.726780 13394 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:50:59.826709  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 178054us
W20260808 01:50:59.944465  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 146995us
I20260808 01:50:59.952188  9637 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:50:59.953655 10369 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: []
I20260808 01:50:59.931547  9551 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
W20260808 01:50:59.955541  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:59.831245 10363 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260808 01:50:59.957684  9524 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:50:59.957805  9599 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:50:59.959486  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:50:59.964105  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:59.962152 13313 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:50:59.965847 13313 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:50:59.891492 13397 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:50:59.823959 12819 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:50:59.956936  9481 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:50:59.972289  9444 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:50:59.973420  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:50:59.977481  9636 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:50:59.982764  9636 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260808 01:50:59.986527  9551 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:50:59.963050 10363 pending_rounds.cc:85] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting all ops after (but not including) 7
I20260808 01:50:59.988148  9633 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260808 01:50:59.933689 13365 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.058s
I20260808 01:50:59.999572  9524 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:00.002424  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:50:59.985983 12819 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.014s
I20260808 01:51:00.001451  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:51:00.004719  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 301912us
I20260808 01:50:59.977571 13397 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:00.013177  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:50:59.987089 13347 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:50:59.990173 13183 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:50:59.952958 13396 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:00.027541  9636 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:00.022571  9631 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:50:59.995244 10363 pending_rounds.cc:107] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260808 01:51:00.031889  9630 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:00.006117  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:00.031158 13205 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:00.039036 13205 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:00.039690 13205 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:00.040462 13205 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260808 01:51:00.043035  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 93651.1us
I20260808 01:51:00.044138  9636 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:00.044176  9599 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:00.046955 13313 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:00.048698  9599 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:00.002391 13244 raft_consensus.cc:2764] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260808 01:51:00.063627  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 257425us
I20260808 01:51:00.066475  9638 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:00.068216  9638 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: []
I20260808 01:51:00.059356  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:00.035035 10363 raft_consensus.cc:2889] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:51:00.036012 13391 consensus_meta.cc:287] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.581s	user 0.012s	sys 0.000s
I20260808 01:51:00.075073 13391 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260808 01:51:00.076263  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 76852.4us
I20260808 01:51:00.076910  9481 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260808 01:51:00.077123 13342 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:00.078840 13134 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260808 01:51:00.080947 13183 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:51:00.083381 13313 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:00.087523  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:00.088088  9635 raft_consensus.cc:2410] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260808 01:51:00.069855  9634 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:51:00.055308 13410 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:00.093641  9638 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260808 01:51:00.095498  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:00.097422  9638 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: []
I20260808 01:51:00.098773  9638 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7. Ops: []
I20260808 01:51:00.069607 13367 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:00.078341 13342 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election won for term 10
W20260808 01:51:00.118371  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 112479us
I20260808 01:51:00.127663  9634 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 7. Current term is 8. Ops: []
I20260808 01:51:00.130986  9558 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: []
I20260808 01:51:00.070580 13363 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20260808 01:51:00.093428 13410 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:00.153860  9558 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260808 01:51:00.104739 13357 raft_consensus.cc:2764] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260808 01:51:00.130262 13411 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:00.166777  9521 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:00.167501  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 166172us
I20260808 01:51:00.168612 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:00.170276 10365 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:00.159555 13177 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:00.148095 13246 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 5. Current term is 6.
W20260808 01:51:00.179396 13418 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260808 01:51:00.147038 13417 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:51:00.158310 13394 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:00.164551 13410 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:00.237531 13417 raft_consensus.cc:697] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:00.243247  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:00.215474 13082 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.651s	user 0.013s	sys 0.000s
W20260808 01:51:00.245528  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 158096us
I20260808 01:51:00.203629 13411 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:00.244390  9481 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:00.215797 13177 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:00.257907  9478 raft_consensus.cc:2410] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260808 01:51:00.203219 13134 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.048s
W20260808 01:51:00.264752  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:00.267629  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:00.270853 13342 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:00.303748 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:00.347082  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 180419us
I20260808 01:51:00.350040  9481 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:51:00.352603  9521 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:00.354142  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 86628.8us
I20260808 01:51:00.242151 12819 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.023s
I20260808 01:51:00.251734 13421 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:00.239091 13417 consensus_queue.cc:237] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:00.280043 13396 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:00.391996  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:00.400966  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:00.408301  9475 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7. Ops: []
I20260808 01:51:00.280632 13134 consensus_queue.cc:1243] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260808 01:51:00.409072  9480 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
W20260808 01:51:00.367053 13348 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.547s	user 0.012s	sys 0.000s
I20260808 01:51:00.364298 13319 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:00.410763  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:00.413429  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:00.350324 13140 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:00.414183  9481 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:51:00.355441 13036 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:00.411170  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:00.362630 12778 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:00.421882  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:00.379063 13414 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260808 01:51:00.426208  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 73688.8us
W20260808 01:51:00.432545  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 75870.6us
I20260808 01:51:00.433411  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:00.412129 13353 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.651s	user 0.009s	sys 0.005s
I20260808 01:51:00.376485 13430 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:00.423132  9477 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
W20260808 01:51:00.448669 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 89104us
I20260808 01:51:00.458315  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:00.419097 13396 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:00.461606  9377 catalog_manager.cc:5697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:00.266362 13411 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:00.463642  9377 catalog_manager.cc:5697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 01:51:00.478168 13422 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260808 01:51:00.479008 13319 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:00.431730 13036 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:00.480762 13419 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260808 01:51:00.432092 13140 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:00.454182 13353 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:00.497641 13381 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:00.429656 12778 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:00.506242  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:00.456614 13428 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.057s
I20260808 01:51:00.507349 13428 consensus_queue.cc:1243] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20260808 01:51:00.513870 13420 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260808 01:51:00.514994  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2103) took 1015 ms. Trace:
I20260808 01:51:00.515692  9637 rpcz_store.cc:276] 0808 01:50:59.499292 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:59.971836 (+472544us) service_pool.cc:224] Handling call
0808 01:50:59.974562 (+  2726us) raft_consensus.cc:1471] Updating replica for 3 ops
0808 01:51:00.003205 (+ 28643us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:00.003243 (+    38us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0808 01:51:00.033302 (+ 30059us) log.cc:844] Serialized 78 byte log entry
0808 01:51:00.257976 (+224674us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:51:00.258025 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:00.258096 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:00.514690 (+256594us) raft_consensus.cc:1685] finished
0808 01:51:00.514715 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:00.514920 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12458,"raft.queue_time_us":35022,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":10206,"thread_start_us":174475,"threads_started":1,"wal-append.queue_time_us":440379}
I20260808 01:51:00.516734 13411 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:00.404685 13421 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:00.519359 13421 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:00.519260 13397 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:00.521133  9632 raft_consensus.cc:2368] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8.
I20260808 01:51:00.522372  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2102) took 1023 ms. Trace:
I20260808 01:51:00.522761  9632 rpcz_store.cc:276] 0808 01:50:59.498966 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:59.824067 (+325101us) service_pool.cc:224] Handling call
0808 01:51:00.522181 (+698114us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b540025b7d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:00.522309 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1427867392}
I20260808 01:51:00.425521 13348 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:00.524412  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2098) took 1112 ms. Trace:
I20260808 01:51:00.525513 13348 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:00.448980 13177 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:00.463563 13430 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:00.524969  9630 rpcz_store.cc:276] 0808 01:50:59.411977 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:59.950031 (+538054us) service_pool.cc:224] Handling call
0808 01:51:00.031668 (+ 81637us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:00.032889 (+  1221us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:00.524129 (+491240us) spinlock_profiling.cc:243] Waited 81.1 ms on lock 0x7b54003faf50. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:00.524323 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":186637696}
I20260808 01:51:00.534603  9377 catalog_manager.cc:5697] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 4 to 6, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 6 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:51:00.536643  9377 catalog_manager.cc:5697] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 5 to 9, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 9 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:51:00.539629  9377 catalog_manager.cc:5697] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:00.541738  9377 catalog_manager.cc:5697] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 4 to 5, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 5 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:00.553623 13036 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:00.516515  9633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:00.557011  9633 raft_consensus.cc:606] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:00.557480  9633 raft_consensus.cc:614] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:00.548633  9377 catalog_manager.cc:5697] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:51:00.560148  9629 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:00.563227  9481 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
W20260808 01:51:00.563905  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 74849.8us
I20260808 01:51:00.565984  9478 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260808 01:51:00.573935  9635 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260808 01:51:00.578761  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:00.589241  9634 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260808 01:51:00.594412  9479 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260808 01:51:00.597222  9473 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260808 01:51:00.601928 13036 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:00.602943 13036 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:00.605643  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:00.602262 10332 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:00.609037 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:00.613905 13140 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:00.614840  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 03b17f21031f4ff4828c85b7332fcc2d (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:00.627753 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:00.623135  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:00.631695  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:00.632555  9475 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:00.626511  9601 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:00.635298  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:00.635881  9476 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:00.600558 12819 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:00.639544  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:00.663873 13433 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:51:00.669612 12819 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:00.672544  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:00.657889 13438 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:51:00.664191 13295 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:00.657616  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:00.674901  9633 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:51:00.676014 13433 raft_consensus.cc:697] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:00.677493 13433 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:00.667596  9634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:00.680492  9634 raft_consensus.cc:606] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:00.680599 13036 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:00.680872 10363 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260808 01:51:00.683115 13303 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 01:51:00.684739  9634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "439040d168824496acc076b1c280bde3"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:00.685792  9634 raft_consensus.cc:606] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:00.686331  9634 raft_consensus.cc:614] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:51:00.688002  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 439040d168824496acc076b1c280bde3 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:00.672580 12819 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:00.690203 12819 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:00.599412 13421 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.093s
I20260808 01:51:00.692090 13394 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:00.696029 13431 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:00.696964  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:00.696830 13431 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:00.697289  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:00.697846  9558 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 9: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:00.698057  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:00.698498 13204 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:00.698969 13438 raft_consensus.cc:697] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:00.699416  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:00.700086 13438 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:00.700861  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:00.701500  9555 raft_consensus.cc:2376] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8. Re-sending same reply.
I20260808 01:51:00.699137 13204 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:00.702936 13414 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:51:00.696065 13348 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:00.703882 13204 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:00.703974 13414 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:51:00.704550 13414 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:00.704180 13348 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:00.705407 13414 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:00.705953  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:00.696197 13177 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:00.707222  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:00.708011  9557 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:51:00.696910  9634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:00.708523  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:00.709846  9634 raft_consensus.cc:606] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:00.710884  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:00.711190  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:00.711537  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:00.712153  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:00.712437  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:00.712642  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 73247.3us
W20260808 01:51:00.718370  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:00.718530  9475 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
W20260808 01:51:00.717546  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 54178.3us
I20260808 01:51:00.721733  9550 raft_consensus.cc:2410] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260808 01:51:00.722905  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:00.707129 13177 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:00.747342  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 314076us
I20260808 01:51:00.616945 13313 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:51:00.758307  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:00.748682  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:00.752638  9521 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:00.716912 13414 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:00.752576  9601 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:00.639344 13140 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:00.718010 13428 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:00.776679  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:00.776959  9550 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260808 01:51:00.787362  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:00.787946  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:00.786303  9559 raft_consensus.cc:2410] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260808 01:51:00.780164  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:00.790455  9556 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:00.784782  9551 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:00.788025  9550 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:00.797900  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 75057.5us
I20260808 01:51:00.799306  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:00.804347  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:00.805107  9557 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:00.725097 13036 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.059s
I20260808 01:51:00.600804 13342 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:00.812235  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:00.812302  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59795.8us
I20260808 01:51:00.814379  9521 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:00.783238 13140 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:00.778798 13348 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:00.775118 13204 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:00.751952 13411 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.049s
I20260808 01:51:00.760187 13134 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.042s
I20260808 01:51:00.784637 13183 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:00.824894  9634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:00.841943  9634 raft_consensus.cc:606] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:00.842471  9634 raft_consensus.cc:614] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:00.843155  9550 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:00.829210 13295 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:51:00.825912 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 165274us
I20260808 01:51:00.814468 13396 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:51:00.860819 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:00.830770 13140 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:51:00.869735  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet f358d8403bf440059a6753dc4ed0be38 (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:00.889864  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 01:51:00.795799 13428 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:00.859014 13348 raft_consensus.cc:2764] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260808 01:51:00.893708  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:00.865680 13431 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:00.842613 13204 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:00.924654  9557 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
W20260808 01:51:00.928469  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 95852.6us
I20260808 01:51:00.869414 13313 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20260808 01:51:00.934376  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:00.946301  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:00.995739  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 49567.4us
I20260808 01:51:00.996860  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:00.935869 13205 raft_consensus.cc:3055] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:01.010177  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:00.935424 13423 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:00.936146 13453 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:01.008423 13420 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260808 01:51:00.992682 13431 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:00.992218 13183 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:00.998939 13442 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:00.999603 13429 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:00.999428 13415 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:51:00.994828 13348 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:00.936635 13450 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260808 01:51:01.012868  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 198569us
I20260808 01:51:00.998757 13448 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:01.000907 13396 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:51:01.060426 13348 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:00.997641 13456 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:01.068423 13456 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:01.023965 13453 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:01.019896  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:01.071487 13445 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.288s	user 0.001s	sys 0.000s
W20260808 01:51:01.071884  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 52219.2us
I20260808 01:51:01.018451 13423 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:01.014217 13246 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:01.004915 13454 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:00.994103 13441 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.279s
I20260808 01:51:01.051510 13442 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:00.926857 13295 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:00.997659 13424 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.288s
I20260808 01:51:01.079401 13424 consensus_queue.cc:1243] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20260808 01:51:01.084926 13410 consensus_meta.cc:287] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.841s	user 0.010s	sys 0.000s
I20260808 01:51:01.055107  9521 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:01.057928 13415 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:01.091673  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 2f740cd3515641a8b2c4add8f809dd82: Illegal state: Replica 6e7856bed1be43acaa821290d506ee97 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:01.093001 13415 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:01.058640 13448 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:01.029273 13177 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:01.064846 13437 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:01.066475 13348 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:01.055958 13450 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:01.053755 13429 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:01.072960 13183 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:51:01.075179 13246 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:51:01.013011 13205 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:51:01.101830  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:01.102022  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:01.098241  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:01.149991  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:01.150489  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 52351.4us
I20260808 01:51:01.151655  9599 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:01.122750  9476 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:01.039597 13431 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:01.085827 13410 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:01.129653  9632 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260808 01:51:01.130250  9637 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260808 01:51:01.134155 13461 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.190s	user 0.000s	sys 0.001s
I20260808 01:51:01.135058 13450 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:01.135174  9638 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:01.136211  9630 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: []
I20260808 01:51:01.137167 13448 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:01.138365 13353 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:01.134953  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 79968.2us
I20260808 01:51:01.134953  9633 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: [7.5-8.6]
I20260808 01:51:01.134192 12826 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:51:01.140069 13454 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:01.166821  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:01.181773  9632 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 8: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:01.183135  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:01.163074  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:01.184056  9632 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
W20260808 01:51:01.167459  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 242478us
I20260808 01:51:01.144956 13183 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:01.145697 13396 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:01.176816  9480 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:01.177783  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:01.190021  9630 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:51:01.179076  9635 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260808 01:51:01.176738  9521 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:01.192864  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:01.156055 13134 raft_consensus.cc:2764] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260808 01:51:01.193809 13429 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:01.187002 13246 raft_consensus.cc:3055] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:51:01.195390 13246 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:01.196614 13246 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:01.197867 13246 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:01.198915 13205 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:01.161471 13244 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:01.200774 13438 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:01.201434 13438 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:01.201910 13438 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:01.200094 13432 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:51:01.203471 13432 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:01.204974 13432 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:01.170013 13394 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:01.200088 13205 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:01.213641  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:01.188807  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:01.215539  9556 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:01.238801  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:01.238870  9632 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:51:01.240607  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:01.241042  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:01.241598  9550 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:01.241921  9560 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:01.231688  9558 pending_rounds.cc:85] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 3
I20260808 01:51:01.243680  9558 pending_rounds.cc:107] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 9.4
I20260808 01:51:01.244067  9558 raft_consensus.cc:2889] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:01.244603  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:01.245464  9560 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:01.161540 13464 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:01.185961 13183 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:01.251935 13456 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:01.252111  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:01.256004 13458 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.199s	user 0.000s	sys 0.001s
W20260808 01:51:01.259613  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 68950.2us
I20260808 01:51:01.279317  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:01.279606  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 412281us
I20260808 01:51:01.261256  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:01.280547  9637 raft_consensus.cc:2376] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10. Re-sending same reply.
W20260808 01:51:01.282155  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 68620.7us
I20260808 01:51:01.283342  9521 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:01.285976  9475 pending_rounds.cc:85] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 3
I20260808 01:51:01.286407  9475 pending_rounds.cc:107] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260808 01:51:01.286707  9475 raft_consensus.cc:2889] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:01.274385  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:01.276343  9633 raft_consensus.cc:2410] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260808 01:51:01.283607  9601 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:01.276674  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 82043e5428784a1eb53a98811ecbbbb4: Illegal state: Replica 259bcb3f0d254e8b9effac3277e654e0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:01.295680  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:01.291373  9632 raft_consensus.cc:2376] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9. Re-sending same reply.
I20260808 01:51:01.296608  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:01.296806  9474 raft_consensus.cc:2410] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 11 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260808 01:51:01.302527  9599 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:01.237979 13453 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:01.216735 13442 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:01.314453  9550 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:01.321815  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:01.277766 13411 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:51:01.323784  9474 raft_consensus.cc:2410] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260808 01:51:01.326884  9599 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:01.329069  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 45864.8us
I20260808 01:51:01.329630 13446 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:01.329488  9523 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:01.217675 13396 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:01.318246 13367 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260808 01:51:01.355592  9477 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.012s	sys 0.027s
W20260808 01:51:01.356339  9477 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.013s	sys 0.027s
I20260808 01:51:01.347565  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:01.357421  9552 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:01.347709  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:01.358963  9551 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:51:01.330467 13446 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:01.366331  9551 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260808 01:51:01.367908  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:01.368688  9632 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:01.369741  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:01.371279  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:01.372807  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:01.373543  9633 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:51:01.379789  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:01.374451  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:01.382546  9633 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:01.326834 13428 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:01.331460 13353 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:51:01.298863 13454 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.065s
I20260808 01:51:01.395824  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:01.396147  9552 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
W20260808 01:51:01.375555 13414 consensus_meta.cc:287] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.593s	user 0.008s	sys 0.008s
I20260808 01:51:01.480933  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:01.481889  9633 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:01.400998  9599 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:01.484021  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 83181.4us
I20260808 01:51:01.485183  9599 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:01.401144  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:01.486440  9632 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:51:01.414440  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:01.417136  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 120659us
I20260808 01:51:01.423400  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:01.479264  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:01.489035  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:01.489936  9559 raft_consensus.cc:2410] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260808 01:51:01.479020  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:01.490314  9550 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:51:01.510788 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:01.510965  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:01.512298 10365 raft_consensus.cc:2410] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260808 01:51:01.514436  9599 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:01.379546 13453 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:01.520141  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:01.378801 13464 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:51:01.519630  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:01.526158  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:01.529143  9551 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:01.529505  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:01.530431  9632 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:51:01.365561 13411 consensus_queue.cc:1059] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.151s
W20260808 01:51:01.535878  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:01.538506  9550 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: []
W20260808 01:51:01.533769  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 110462us
I20260808 01:51:01.547254  9521 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:01.519764 13418 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
I20260808 01:51:01.542115  9552 raft_consensus.cc:3055] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:01.535292 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:01.549583  9552 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:01.479576 13367 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:01.509783 13448 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:01.560043  9551 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260808 01:51:01.550568  9552 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:01.383917 13432 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.150s
I20260808 01:51:01.482633 13437 raft_consensus.cc:697] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:01.556968 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:01.557935  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:01.529343 13381 consensus_meta.cc:287] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.031s	user 0.008s	sys 0.007s
I20260808 01:51:01.521049 13453 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:01.520119 13394 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:01.510211 13303 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:01.510028 13429 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:01.561806 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:01.559245 13465 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260808 01:51:01.524544 13424 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:01.567361 13411 raft_consensus.cc:3055] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:51:01.535458 13353 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:01.559092 13452 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.357s	user 0.002s	sys 0.000s
I20260808 01:51:01.555169 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:01.572155  9521 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:01.533687 13428 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:01.575069  9552 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:51:01.580876  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 66568us
I20260808 01:51:01.585428  9560 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:01.621951 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:01.624998  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:01.548914 13313 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.025s
I20260808 01:51:01.628333  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:01.582801 12819 consensus_meta.cc:287] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.891s	user 0.013s	sys 0.001s
I20260808 01:51:01.574796 13448 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:01.585680 13453 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:01.630694  9550 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:01.598287 13411 raft_consensus.cc:740] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:01.683393  9633 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:01.595346 13467 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 01:51:01.597792 13456 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 01:51:01.675539  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 103461us
I20260808 01:51:01.696900  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:01.624743 13454 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Stepping down as leader of term 7
W20260808 01:51:01.667847 13419 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.097s	user 0.003s	sys 0.000s
I20260808 01:51:01.694263  9599 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:01.572152 13410 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:01.727088  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:01.726224  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:01.668128 13485 raft_consensus.cc:2764] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260808 01:51:01.727609  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:01.733884  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:01.668303 13473 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
I20260808 01:51:01.732084  9552 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260808 01:51:01.599301 13303 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:01.571635 13367 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:01.686092 12819 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:01.677922 13429 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:01.702484 13442 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:01.632563 13477 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:01.704154  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:01.746444 13474 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:01.747635 13469 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:01.750955 13467 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260808 01:51:01.751385 13246 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.553s	user 0.016s	sys 0.000s
I20260808 01:51:01.724679 13454 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:51:01.752713 13468 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260808 01:51:01.810616  9551 raft_consensus.cc:3055] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Stepping down as leader of term 6
W20260808 01:51:01.753242 13476 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.226s	user 0.001s	sys 0.001s
I20260808 01:51:01.753829  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:01.754066 12826 consensus_meta.cc:287] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.566s	user 0.011s	sys 0.000s
I20260808 01:51:01.689849 13428 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:01.760116 13381 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:01.755245 13471 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.370s	user 0.000s	sys 0.001s
I20260808 01:51:01.751040 13478 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:01.764729  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 240637us
I20260808 01:51:01.631752 13205 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:01.582351 13446 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:01.808283 13442 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:01.741150 13424 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:51:01.739845 13415 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election won for term 11
W20260808 01:51:01.767200 13475 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.428s	user 0.001s	sys 0.001s
I20260808 01:51:01.755066  9550 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 4. (term mismatch)
I20260808 01:51:01.915262  9550 pending_rounds.cc:85] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 3
W20260808 01:51:01.837358  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 111223us
I20260808 01:51:01.765944 13347 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:01.830082 13441 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:01.915779  9550 pending_rounds.cc:107] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20260808 01:51:01.919185  9550 raft_consensus.cc:2889] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:01.812932 13177 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:01.922613  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:01.836555  9555 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:01.804782 13474 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:01.668665 13437 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:01.740900 13353 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:01.811146  9551 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:01.703199 13464 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.117s
I20260808 01:51:01.916014  9556 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260808 01:51:01.815490 13452 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260808 01:51:01.811137 13467 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:01.815102 13432 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:01.808043 13429 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:01.818394 12819 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:01.839249 13396 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:02.022281  9473 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260808 01:51:02.024281  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:01.893246 13414 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260808 01:51:01.999442  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 76932.1us
I20260808 01:51:01.980650  9551 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:01.815488 13477 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:01.863808 13381 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:01.979163 13140 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.075s	user 0.015s	sys 0.000s
W20260808 01:51:02.023507 13481 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.111s	user 0.000s	sys 0.001s
I20260808 01:51:02.053920  9551 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:02.056290  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:02.054569  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:02.023576  9554 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:51:01.929025 13490 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:01.971691 13432 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:51:02.066236  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 106263us
W20260808 01:51:02.033567 13450 consensus_meta.cc:287] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.847s	user 0.014s	sys 0.000s
I20260808 01:51:02.029738 13082 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:01.957895 13177 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:01.972050 13467 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:51:02.059860 13466 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.442s	user 0.000s	sys 0.002s
I20260808 01:51:01.955749 13474 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:02.019490 13363 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:02.021721 12778 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:01.823195 13411 consensus_queue.cc:260] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:01.816195 13469 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:01.948391 13347 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:02.074095 13177 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:51:02.116277  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:02.073599 13082 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:02.077565  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:02.059919 13348 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:01.932276 13441 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:02.060739 13396 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:51:02.092159 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 94327us
I20260808 01:51:02.070040 13432 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:51:02.076789 13438 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.874s	user 0.014s	sys 0.003s
W20260808 01:51:02.155282  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 130944us
I20260808 01:51:02.018836 13484 consensus_queue.cc:1059] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.316s
I20260808 01:51:02.072182 13415 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:02.057477 12819 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:02.077257  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:02.059723 13140 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:02.067080 13490 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:01.957719 13478 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:01.956743 13454 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:02.073182 13450 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:02.201869  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:02.205159  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 140686us
I20260808 01:51:02.206375  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:02.130743 13367 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:02.119026 13511 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:02.210858 13500 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260808 01:51:02.203842  9479 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:51:02.212301 12778 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:51:02.099561 13363 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:02.195736 13484 consensus_queue.cc:1243] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260808 01:51:02.221704 10365 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:02.244426 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:02.123432 13303 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:02.228936 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 73387.7us
I20260808 01:51:02.221800 10363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 658) took 1536 ms. Trace:
I20260808 01:51:02.231109  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:02.201099 13411 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:51:02.236413  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 161178us
I20260808 01:51:02.137445 13348 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:02.216181 13489 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:51:02.184491 13438 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:02.255899  9444 leader_election.cc:400] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8.
I20260808 01:51:02.272497  9444 leader_election.cc:403] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:51:02.222026 13428 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:02.261768  9600 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:02.221550 13495 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:02.251714 10363 rpcz_store.cc:276] 0808 01:51:00.685509 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:00.685788 (+   279us) service_pool.cc:224] Handling call
0808 01:51:00.686168 (+   380us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:00.759283 (+ 73115us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:00.763220 (+  3937us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:00.897540 (+134320us) log.cc:844] Serialized 52 byte log entry
0808 01:51:01.133145 (+235605us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:01.142381 (+  9236us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:01.144421 (+  2040us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:02.221521 (+1077100us) raft_consensus.cc:1685] finished
0808 01:51:02.221552 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:02.221725 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28701,"raft.queue_time_us":117180,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":14498,"thread_start_us":132409,"threads_started":1,"wal-append.queue_time_us":249639}
I20260808 01:51:02.241268 13511 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:02.278469 13482 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.184s	user 0.001s	sys 0.000s
W20260808 01:51:02.290450  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 117267us
I20260808 01:51:02.264699 13484 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election won for term 7
W20260808 01:51:02.309921  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 54165.3us
I20260808 01:51:02.298992  9631 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:02.231070 13246 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260808 01:51:02.292695 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:02.295534  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:02.312418  9551 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:02.321312  9559 raft_consensus.cc:1217] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[4.3-9.5]   Dedup: 4.3->[7.4-9.5]
I20260808 01:51:02.242743 13410 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20260808 01:51:02.322001  9559 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: [4.3-9.5]
I20260808 01:51:02.271813 13489 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:02.323338  9629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2096) took 2906 ms (client timeout 1522 ms). Trace:
I20260808 01:51:02.231500 13082 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:02.344885  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:02.355700  9556 raft_consensus.cc:2410] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 3.
I20260808 01:51:02.322314 12778 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8.
W20260808 01:51:02.322105 13506 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.252s	user 0.001s	sys 0.000s
W20260808 01:51:02.327183  9629 rpcz_store.cc:269] 0808 01:50:59.392135 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:59.392277 (+   142us) service_pool.cc:224] Handling call
0808 01:51:02.298309 (+2906032us) spinlock_profiling.cc:243] Waited 619 ms on lock 0x7b5400307fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:02.298489 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1424358144}
I20260808 01:51:02.289539 13303 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.056s
W20260808 01:51:02.382030  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 63330.5us
I20260808 01:51:02.419747  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2389) took 1099 ms. Trace:
W20260808 01:51:02.343485 13433 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.479s	user 0.020s	sys 0.047s
I20260808 01:51:02.420251  9475 rpcz_store.cc:276] 0808 01:51:01.285197 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:01.285362 (+   165us) service_pool.cc:224] Handling call
0808 01:51:01.285741 (+   379us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:01.398577 (+112836us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:01.398615 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:01.399467 (+   852us) log.cc:844] Serialized 52 byte log entry
0808 01:51:01.830844 (+431377us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:51:01.830891 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:01.830968 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:02.384093 (+553125us) raft_consensus.cc:1685] finished
0808 01:51:02.384116 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:02.384287 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":361851,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":149,"thread_start_us":374024,"threads_started":2,"wal-append.queue_time_us":558064}
I20260808 01:51:02.357414  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:02.322486 13381 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:02.419479  9599 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:02.429354  9629 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260808 01:51:02.430891  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2071) took 1048 ms. Trace:
I20260808 01:51:02.431624  9629 rpcz_store.cc:276] 0808 01:51:01.382613 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:02.420716 (+1038103us) service_pool.cc:224] Handling call
0808 01:51:02.421203 (+   487us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:02.430498 (+  9295us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:02.430813 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:02.344408 13484 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:02.441846  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 124243us
I20260808 01:51:02.346724 13489 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:51:02.359011 13492 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:51:02.357139 13410 consensus_queue.cc:1243] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 9.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260808 01:51:02.459668  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:02.429307 13433 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.567s	user 0.021s	sys 0.062s
I20260808 01:51:02.418363 13442 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:02.458256  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2508) took 1227 ms. Trace:
W20260808 01:51:02.452242  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:02.343684 13429 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:02.462417  9629 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:51:02.465854 13410 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:51:02.464677  9558 rpcz_store.cc:276] 0808 01:51:01.230637 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:01.230856 (+   219us) service_pool.cc:224] Handling call
0808 01:51:01.231410 (+   554us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:01.244841 (+ 13431us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:01.244894 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:01.245526 (+   632us) log.cc:844] Serialized 26 byte log entry
0808 01:51:01.322751 (+ 77225us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:51:01.322787 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:01.322872 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:02.277620 (+954748us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:02.453068 (+175448us) raft_consensus.cc:1685] finished
0808 01:51:02.453091 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:02.458166 (+  5075us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24145152,"thread_start_us":76792,"threads_started":1,"wal-append.queue_time_us":813100}
I20260808 01:51:02.467095 13410 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:51:02.465628 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:02.472245 10363 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:02.322902 13415 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:02.447893 13456 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.691s	user 0.014s	sys 0.024s
I20260808 01:51:02.354038 13495 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.116s
I20260808 01:51:02.472203  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2041) took 2960 ms. Trace:
I20260808 01:51:02.448707 13397 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:02.474431 13391 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:02.477386 13478 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:02.491487 13478 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:02.490203  9636 rpcz_store.cc:276] 0808 01:50:59.511502 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:00.045348 (+533846us) service_pool.cc:224] Handling call
0808 01:51:00.045819 (+   471us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:00.271539 (+225720us) raft_consensus.cc:1534] Early marking committed up to index 5
0808 01:51:00.599040 (+327501us) log.cc:844] Serialized 14 byte log entry
0808 01:51:01.004664 (+405624us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:01.014862 (+ 10198us) log.cc:844] Serialized 52 byte log entry
0808 01:51:01.015306 (+   444us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:01.017231 (+  1925us) log.cc:844] Serialized 14 byte log entry
0808 01:51:02.271323 (+1254092us) spinlock_profiling.cc:243] Waited 4.01 ms on lock 0x7b18002e7d40. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667 00007f5d449ae5e8
0808 01:51:02.458552 (+187229us) log.cc:844] Serialized 14 byte log entry
0808 01:51:02.468135 (+  9583us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:02.471481 (+  3346us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:02.471535 (+    54us) raft_consensus.cc:1685] finished
0808 01:51:02.471554 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:02.472134 (+   580us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24183,"prepare.queue_time_us":415498,"prepare.run_cpu_time_us":129,"prepare.run_wall_time_us":4278,"raft.queue_time_us":121326,"raft.run_cpu_time_us":364,"raft.run_wall_time_us":15791,"spinlock_wait_cycles":9222400,"thread_start_us":923625,"threads_started":7,"wal-append.queue_time_us":226018}
I20260808 01:51:02.456879 13347 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:02.477710 13502 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.124s
I20260808 01:51:02.491979 13478 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:02.492659 13424 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.004s	sys 0.001s
W20260808 01:51:02.493120 13424 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.004s	sys 0.001s
I20260808 01:51:02.493405 13424 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:02.458933 13492 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:02.494273 13492 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:02.494418 13424 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:02.494846 13478 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:02.498036 13484 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:02.469347 12819 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:02.456535 12778 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.029s
I20260808 01:51:02.503561 13514 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260808 01:51:02.485358 13486 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.131s	user 0.001s	sys 0.000s
W20260808 01:51:02.488014  9599 leader_election.cc:336] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Timed out: RequestConsensusVote RPC to 127.7.165.194:36095 timed out after 2.169s (SENT)
I20260808 01:51:02.460810 13521 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:02.469115 13140 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:02.513972  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2509) took 1278 ms. Trace:
I20260808 01:51:02.507225  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:02.473184 13485 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260808 01:51:02.485787 13456 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.015s	sys 0.024s
I20260808 01:51:02.514612  9557 rpcz_store.cc:276] 0808 01:51:01.235757 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:01.235964 (+   207us) service_pool.cc:224] Handling call
0808 01:51:01.236416 (+   452us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:01.237562 (+  1146us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:01.237603 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:01.276709 (+ 39106us) log.cc:844] Serialized 26 byte log entry
0808 01:51:01.411133 (+134424us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:01.575089 (+163956us) log.cc:844] Serialized 14 byte log entry
0808 01:51:01.599946 (+ 24857us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:01.600037 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:02.290169 (+690132us) spinlock_profiling.cc:243] Waited 22.7 ms on lock 0x7b1800464920. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:51:02.508704 (+218535us) raft_consensus.cc:1685] finished
0808 01:51:02.513603 (+  4899us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:02.513843 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54050,"prepare.queue_time_us":505874,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":57,"raft.queue_time_us":29563,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":6851,"spinlock_wait_cycles":52235520,"thread_start_us":137415,"threads_started":2,"wal-append.queue_time_us":326273}
I20260808 01:51:02.470978 13381 consensus_queue.cc:1059] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.047s
I20260808 01:51:02.488006 13450 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:02.479988 13469 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:02.542606 13479 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.187s
W20260808 01:51:02.545452 13524 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.164s	user 0.002s	sys 0.000s
W20260808 01:51:02.553790  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:02.516719 13490 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:02.557686  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 73060.2us
I20260808 01:51:02.520907 13347 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:02.523944 13485 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:02.572717 13347 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:02.572880 13485 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:02.573247 13347 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:02.471896 13501 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.303s
W20260808 01:51:02.591203  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 103233us
I20260808 01:51:02.592221 13501 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:02.592835 13501 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:02.582845 13183 consensus_meta.cc:287] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.291s	user 0.009s	sys 0.009s
W20260808 01:51:02.597366  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 277118us
I20260808 01:51:02.583385 13490 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:02.593410 13177 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:02.596560 13429 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 01:51:02.607406  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:02.608348 13511 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:02.608666  9550 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.008s	sys 0.015s
I20260808 01:51:02.613812  9629 raft_consensus.cc:2393] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 6: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:02.608110  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:02.616408  9550 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.009s	sys 0.015s
I20260808 01:51:02.605235 13527 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:02.589852 13410 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260808 01:51:02.637157  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:02.638527  9629 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:51:02.612668 13495 raft_consensus.cc:3055] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:51:02.612211 13438 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:02.639839  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 68181.3us
I20260808 01:51:02.643082  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:02.645270  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:02.641682 13495 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:02.599529 13183 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:02.650442  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:02.652896  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:02.653810  9629 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:02.635454 13527 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:02.648144 13520 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.173s	user 0.002s	sys 0.000s
I20260808 01:51:02.658699  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:02.659839 12778 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:02.662545 13532 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 01:51:02.667372 13478 raft_consensus.cc:3055] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:02.667243 13495 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:02.669085 13495 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:02.659729 13381 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:02.671350 13508 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.290s	user 0.000s	sys 0.001s
I20260808 01:51:02.671461 13381 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:02.614948 13490 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:02.675608  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2061) took 1307 ms. Trace:
W20260808 01:51:02.675828  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:02.676224  9632 rpcz_store.cc:276] 0808 01:51:01.368098 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:01.758463 (+390365us) service_pool.cc:224] Handling call
0808 01:51:01.812608 (+ 54145us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:01.813033 (+   425us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:01.813071 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:01.819707 (+  6636us) log.cc:844] Serialized 52 byte log entry
0808 01:51:02.055759 (+236052us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:51:02.055813 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:02.055891 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:02.675366 (+619475us) raft_consensus.cc:1685] finished
0808 01:51:02.675387 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:02.675544 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":206164,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":140,"thread_start_us":141584,"threads_started":1,"wal-append.queue_time_us":340153}
I20260808 01:51:02.681279  9476 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 7. Current term is 8. Ops: []
I20260808 01:51:02.681128  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:02.668027 13478 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:02.682215  9550 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:02.673846 13381 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:02.710230  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:02.711256  9557 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:02.665581 13423 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:02.720969  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:02.696370 13453 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.995s	user 0.011s	sys 0.007s
I20260808 01:51:02.722134 12778 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.067s
W20260808 01:51:02.785218 13536 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260808 01:51:02.786026 13183 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:02.791707 13177 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:02.793280 13448 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:02.793589 13490 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:02.729295 13532 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
W20260808 01:51:02.797349  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 203545us
I20260808 01:51:02.799132  9637 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: [6.4-8.5]
I20260808 01:51:02.708001 13484 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260808 01:51:02.802269  9445 connection.cc:256] RPC call timeout handler was delayed by -1.3081520950072445s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260808 01:51:02.821034  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:02.830237  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:02.831571  9632 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:02.826521  9599 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:02.801689 13502 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:02.856120  9629 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260808 01:51:02.728645 13437 consensus_queue.cc:1059] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.107s
I20260808 01:51:02.851066 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:02.862887 13539 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:02.850514 13533 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:02.864881 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:02.865190 13529 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.100s	user 0.002s	sys 0.000s
W20260808 01:51:02.866073  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:02.841498 13490 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:02.801950 13410 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.065s
I20260808 01:51:02.795982 13453 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:02.867097 13474 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.757s	user 0.010s	sys 0.005s
I20260808 01:51:02.868503 13531 raft_consensus.cc:993] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:51:02.870626 13502 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:02.871562 13515 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:02.825245 13183 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:02.875555 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:02.877204 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:02.879406 13411 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.611s	user 0.013s	sys 0.000s
I20260808 01:51:02.880106 13177 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:02.897002 10361 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:02.826846 13501 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:02.911232  9636 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
W20260808 01:51:02.892674 13460 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260808 01:51:02.896555 13533 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:02.960165 13533 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:02.908063 13490 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:02.947885  9445 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:02.963969 13500 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.709s	user 0.003s	sys 0.000s
I20260808 01:51:02.919310 13421 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:02.965471 13431 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:02.925681 13515 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:02.923739 13353 raft_consensus.cc:1081] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Signalling peer 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:51:02.966140 13515 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:02.966512  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2388) took 1681 ms. Trace:
I20260808 01:51:02.964943 13421 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:02.966974  9478 rpcz_store.cc:276] 0808 01:51:01.284783 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:01.285307 (+   524us) service_pool.cc:224] Handling call
0808 01:51:01.303617 (+ 18310us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:01.335176 (+ 31559us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:51:01.379166 (+ 43990us) log.cc:844] Serialized 14 byte log entry
0808 01:51:01.914325 (+535159us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:01.926114 (+ 11789us) log.cc:844] Serialized 26 byte log entry
0808 01:51:01.926284 (+   170us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:51:01.926319 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:01.926395 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:02.963624 (+1037229us) spinlock_profiling.cc:243] Waited 7.22 ms on lock 0x7b24000ef228. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41713594 00007f5d4171169b 00007f5d418cd6d9 00007f5d418cd365 00007f5d418cc9ad 00007f5d445162fe 00007f5d4452da7c 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411
0808 01:51:02.966139 (+  2515us) raft_consensus.cc:1685] finished
0808 01:51:02.966266 (+   127us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:02.966444 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":126945,"prepare.queue_time_us":185037,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":393449,"raft.run_cpu_time_us":603,"raft.run_wall_time_us":62073,"spinlock_wait_cycles":16615168,"thread_start_us":321508,"threads_started":2,"wal-append.queue_time_us":554389}
I20260808 01:51:02.969269 13421 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:02.969358  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:02.964016 13490 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:02.973014  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 526) took 1052 ms. Trace:
I20260808 01:51:02.959438  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:02.974290  9636 raft_consensus.cc:2376] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10. Re-sending same reply.
I20260808 01:51:02.975735 13414 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:02.976289 13414 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:02.973145 13490 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:02.980733  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2109) took 1853 ms. Trace:
I20260808 01:51:02.981199  9635 rpcz_store.cc:276] 0808 01:51:01.127540 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:01.127890 (+   350us) service_pool.cc:224] Handling call
0808 01:51:01.128255 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:01.190819 (+ 62564us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:02.980393 (+1789574us) spinlock_profiling.cc:243] Waited 45.9 ms on lock 0x7b540003b6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:02.980635 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":105483520}
W20260808 01:51:02.983392  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:02.849170 13478 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:02.973706  9472 rpcz_store.cc:276] 0808 01:51:01.920697 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:02.194468 (+273771us) service_pool.cc:224] Handling call
0808 01:51:02.194920 (+   452us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:02.213584 (+ 18664us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:02.213627 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:02.213678 (+    51us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:51:02.213709 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:02.972709 (+759000us) spinlock_profiling.cc:243] Waited 18.4 ms on lock 0x7b5400399754. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:02.972774 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:02.972944 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":42377472}
I20260808 01:51:02.999545  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:03.000888  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 53144.9us
I20260808 01:51:03.007088  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:03.008428  9635 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
W20260808 01:51:02.973104 13540 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260808 01:51:03.033875 13492 consensus_meta.cc:287] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.538s	user 0.006s	sys 0.006s
I20260808 01:51:03.012568  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:03.034997  9478 raft_consensus.cc:2410] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260808 01:51:02.922066 13474 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:02.902117 13539 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:02.955919 13177 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:03.031600 13545 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260808 01:51:03.032063 13453 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:02.956102 10362 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:03.036514  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:02.932047 13183 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:03.008392  9472 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:03.041079  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2069) took 1666 ms. Trace:
I20260808 01:51:03.059864  9638 rpcz_store.cc:276] 0808 01:51:01.374635 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:02.300517 (+925882us) service_pool.cc:224] Handling call
0808 01:51:02.301027 (+   510us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:02.301354 (+   327us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:02.301403 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:02.310372 (+  8969us) log.cc:844] Serialized 26 byte log entry
0808 01:51:02.461784 (+151412us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:02.461819 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:02.461882 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:03.040813 (+578931us) raft_consensus.cc:1685] finished
0808 01:51:03.040834 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:03.040986 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":79403,"threads_started":1,"wal-append.queue_time_us":319239}
I20260808 01:51:03.062647  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:03.053267  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:03.052968  9560 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:51:03.075891 13489 consensus_meta.cc:287] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.615s	user 0.010s	sys 0.014s
I20260808 01:51:03.076866 13489 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:03.017333 13421 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:03.017767 13484 raft_consensus.cc:3055] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:03.089102 13484 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:03.060554 13467 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.961s	user 0.018s	sys 0.000s
I20260808 01:51:03.083082  9479 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:51:03.084206  9629 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: [7.5-8.6]
I20260808 01:51:03.085134  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:03.044129 13532 consensus_queue.cc:1175] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" responder_term: 10 status { last_received { term: 10 index: 10 } last_committed_idx: 9 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 01:51:03.087911 13474 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:03.087958 13183 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:03.067883  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:03.094254  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:03.099607  9478 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:51:03.110882  9630 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:03.112558  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:03.113706  9478 raft_consensus.cc:2410] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 11 because replica has last-logged OpId of term: 10 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 4.
I20260808 01:51:03.061558 13411 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:03.060990 13460 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:03.089869 13533 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:03.080113 13495 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:03.090888 13421 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:03.034519 13492 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:03.144104  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 531) took 1166 ms. Trace:
I20260808 01:51:03.127454  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:03.152989  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:03.151647  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:03.158190  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:03.158449 13489 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:03.160917  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:03.163944 13539 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:03.166939 13453 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:03.167666  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:03.161526 13515 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:03.169149  9550 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:51:03.167827  9555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 2488) took 2470 ms (client timeout 2169 ms). Trace:
I20260808 01:51:03.089126 13177 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:03.170876  9555 rpcz_store.cc:269] 0808 01:51:00.704681 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:00.709158 (+  4477us) service_pool.cc:224] Handling call
0808 01:51:03.167580 (+2458422us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54000fee54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:03.167706 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2416953088}
W20260808 01:51:03.134755 13396 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.955s	user 0.013s	sys 0.003s
I20260808 01:51:03.134320 13460 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:03.139482 13432 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.946s	user 0.017s	sys 0.000s
I20260808 01:51:03.158489  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:03.045415 13347 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:03.131511 13183 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:03.127801  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:03.144670  9475 rpcz_store.cc:276] 0808 01:51:01.978004 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:02.439597 (+461593us) service_pool.cc:224] Handling call
0808 01:51:02.440106 (+   509us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:02.449508 (+  9402us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:02.449560 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:02.462731 (+ 13171us) log.cc:844] Serialized 52 byte log entry
0808 01:51:02.730498 (+267767us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:51:02.998828 (+268330us) log.cc:844] Serialized 14 byte log entry
0808 01:51:03.007936 (+  9108us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:03.008022 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:03.143817 (+135795us) raft_consensus.cc:1685] finished
0808 01:51:03.143844 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:03.144031 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31338,"prepare.queue_time_us":248855,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":15770,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":31326,"thread_start_us":437033,"threads_started":2,"wal-append.queue_time_us":661964}
I20260808 01:51:03.109241 13467 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:03.129351 13411 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:03.267870  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2386) took 2475 ms. Trace:
I20260808 01:51:03.843657  9556 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:51:03.201257  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:03.709803  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1991) took 2232 ms. Trace:
I20260808 01:51:03.718660  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2040) took 4207 ms. Trace:
I20260808 01:51:03.844949  9559 rpcz_store.cc:276] 0808 01:51:01.477588 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:01.530069 (+ 52481us) service_pool.cc:224] Handling call
0808 01:51:01.530492 (+   423us) raft_consensus.cc:1471] Updating replica for 3 ops
0808 01:51:02.325037 (+794545us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:03.709270 (+1384233us) spinlock_profiling.cc:243] Waited 778 ms on lock 0x7b54001dfe54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:03.709653 (+   383us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1790529280}
W20260808 01:51:03.244344  9523 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.196:33455 timed out after 1.816s (SENT) [suppressed 27 similar messages]
I20260808 01:51:03.704447  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:03.399662  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:03.848901  9557 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
W20260808 01:51:03.269018  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:03.193681  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:03.844095  9473 rpcz_store.cc:276] 0808 01:51:00.792662 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:00.792876 (+   214us) service_pool.cc:224] Handling call
0808 01:51:00.810121 (+ 17245us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:02.031108 (+1220987us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:03.267420 (+1236312us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54002dc3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:03.267775 (+   355us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2778346240}
W20260808 01:51:03.861248  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 616917us
I20260808 01:51:03.872762  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:03.150705 13421 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:51:03.174062 13485 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.600s	user 0.017s	sys 0.000s
W20260808 01:51:03.756404 13537 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.753s	user 0.003s	sys 0.003s
I20260808 01:51:03.899111  9638 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:03.899912  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:03.900620  9473 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 8
I20260808 01:51:03.901002  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:03.855715 13543 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:03.150404 13495 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:03.902377  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:03.902289 13543 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:03.903134  9637 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:03.845706  9631 rpcz_store.cc:276] 0808 01:50:59.511144 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:00.032194 (+521050us) service_pool.cc:224] Handling call
0808 01:51:02.298626 (+2266432us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:02.311514 (+ 12888us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:03.718294 (+1406780us) spinlock_profiling.cc:243] Waited 2.27 s on lock 0x7b5400307fd0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:03.718557 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5211265792}
I20260808 01:51:03.220142 13396 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:51:03.904490 13495 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:03.889091  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2154) took 1082 ms. Trace:
I20260808 01:51:03.904666 13543 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:03.905124 13495 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:03.905644 13495 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:03.905736  9475 rpcz_store.cc:276] 0808 01:51:02.807146 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:03.872361 (+1065215us) service_pool.cc:224] Handling call
0808 01:51:03.888995 (+ 16634us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:03.907469  9635 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260808 01:51:03.907608  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:03.908452  9475 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 8: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:03.908823  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2158) took 1100 ms. Trace:
I20260808 01:51:03.908892 13495 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:03.909381  9475 rpcz_store.cc:276] 0808 01:51:02.808477 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:03.907201 (+1098724us) service_pool.cc:224] Handling call
0808 01:51:03.908766 (+  1565us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:03.909799  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:03.913516  9635 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:03.910945  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:03.202298 13492 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:03.232195  9550 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:03.711385 13515 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:03.914443  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 536) took 1131 ms. Trace:
I20260808 01:51:03.658061 13533 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:03.875931 13490 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.896s	user 0.013s	sys 0.009s
I20260808 01:51:03.919025  9637 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:51:03.918439  9475 rpcz_store.cc:276] 0808 01:51:02.783626 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:03.913596 (+1129970us) service_pool.cc:224] Handling call
0808 01:51:03.913938 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:03.914111 (+   173us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:03.914134 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:03.914179 (+    45us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:03.914214 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:03.914267 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:03.914388 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:03.924880  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:03.925351  9475 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: []
I20260808 01:51:03.925967  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2149) took 1120 ms. Trace:
I20260808 01:51:03.665884 13414 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:03.926283  9475 rpcz_store.cc:276] 0808 01:51:02.805513 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:03.924884 (+1119371us) service_pool.cc:224] Handling call
0808 01:51:03.925174 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:03.925748 (+   574us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:03.925919 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:03.921948  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:03.927403  9478 raft_consensus.cc:2410] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260808 01:51:03.871781 13432 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:03.872095 13183 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:03.912461 13437 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.806s	user 0.018s	sys 0.009s
I20260808 01:51:03.935348  9473 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:51:03.935753  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:03.936663  9473 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 4. Current term is 5. Ops: []
I20260808 01:51:03.937579  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2152) took 1131 ms. Trace:
I20260808 01:51:03.938009  9473 rpcz_store.cc:276] 0808 01:51:02.806472 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:03.936215 (+1129743us) service_pool.cc:224] Handling call
0808 01:51:03.936484 (+   269us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:03.937297 (+   813us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:03.937510 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:03.898310 13485 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:03.935318 13437 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.843s	user 0.019s	sys 0.009s
I20260808 01:51:04.006779  9556 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260808 01:51:04.007581  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2532) took 1023 ms. Trace:
I20260808 01:51:04.007889  9556 rpcz_store.cc:276] 0808 01:51:02.984382 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:03.940959 (+956577us) service_pool.cc:224] Handling call
0808 01:51:04.005075 (+ 64116us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:04.007278 (+  2203us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:04.007504 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:04.012143  9475 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9. Ops: [6.4-8.5]
I20260808 01:51:04.031337  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2382) took 3404 ms. Trace:
I20260808 01:51:04.032135 13183 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:04.033463  9636 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: [5.4-9.6]
I20260808 01:51:04.031790  9480 rpcz_store.cc:276] 0808 01:51:00.626766 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:00.627355 (+   589us) service_pool.cc:224] Handling call
0808 01:51:00.627800 (+   445us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:01.198935 (+571135us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:04.031056 (+2832121us) spinlock_profiling.cc:243] Waited 549 ms on lock 0x7b540019b9d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:04.031268 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1261873024}
I20260808 01:51:04.034308 13414 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:04.036202  9635 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:51:04.040328  9480 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:04.040899  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2156) took 1233 ms. Trace:
I20260808 01:51:04.041574  9473 rpcz_store.cc:276] 0808 01:51:02.807727 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:03.938997 (+1131270us) service_pool.cc:224] Handling call
0808 01:51:04.040367 (+101370us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:04.040567 (+   200us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:04.040591 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:04.040632 (+    41us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:04.040660 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:04.040719 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:04.040839 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:51:04.029963 13567 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.213s	user 0.000s	sys 0.001s
I20260808 01:51:04.031612 13432 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:04.035049 13568 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
I20260808 01:51:04.047102  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2157) took 1239 ms. Trace:
I20260808 01:51:04.036949  9556 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:53208
I20260808 01:51:04.053644  9480 rpcz_store.cc:276] 0808 01:51:02.808179 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:04.039732 (+1231553us) service_pool.cc:224] Handling call
0808 01:51:04.040137 (+   405us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:04.046783 (+  6646us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:04.047037 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:04.054885  9556 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:51:04.055616  9556 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:04.039930 13460 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:04.057035  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 537) took 1153 ms. Trace:
I20260808 01:51:04.058957  9480 rpcz_store.cc:276] 0808 01:51:02.903461 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:04.055719 (+1152258us) service_pool.cc:224] Handling call
0808 01:51:04.056283 (+   564us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:04.056539 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:04.056576 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:04.056637 (+    61us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:04.056693 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:04.056778 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:04.056933 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:04.063205  9473 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7. Ops: []
I20260808 01:51:04.060515 13411 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:04.071336 13566 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.215s	user 0.002s	sys 0.000s
I20260808 01:51:04.091216  9556 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:04.068516 13347 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:04.094394 13441 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:04.074570  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2159) took 1265 ms. Trace:
I20260808 01:51:04.134371 13204 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:04.163430 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 259563us
W20260808 01:51:04.166189 10332 connection.cc:256] RPC call timeout handler was delayed by -2.6105185869964771s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22280000000000003s for a response
I20260808 01:51:04.171221 13479 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:04.172034 13479 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:04.138636 13502 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:04.138979 13430 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:04.178934 10332 connection.cc:256] RPC call timeout handler was delayed by -2.4378126290102955s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16490000000000002s for a response
I20260808 01:51:04.142143  9473 rpcz_store.cc:276] 0808 01:51:02.808800 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:04.042722 (+1233922us) service_pool.cc:224] Handling call
0808 01:51:04.062925 (+ 20203us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:04.074143 (+ 11218us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:04.074478 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:04.181345  9556 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:04.183538 10332 connection.cc:256] RPC call timeout handler was delayed by -1.7756569210032467s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31070000000000003s for a response
W20260808 01:51:04.191367 10332 connection.cc:256] RPC call timeout handler was delayed by -1.2812058959971182s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1894s for a response
W20260808 01:51:04.191856 10332 connection.cc:256] RPC call timeout handler was delayed by -1.2812058959971182s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1894s for a response
W20260808 01:51:04.192251 10332 connection.cc:256] RPC call timeout handler was delayed by -1.1930058959987946s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19920000000000002s for a response
W20260808 01:51:04.192623 10332 connection.cc:256] RPC call timeout handler was delayed by -1.1390058959950693s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20520000000000002s for a response
W20260808 01:51:04.193014 10332 connection.cc:256] RPC call timeout handler was delayed by -1.0931058959977236s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21030000000000004s for a response
I20260808 01:51:04.176018 13479 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:04.241729  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:53208 (request call id 2533) took 1199 ms. Trace:
W20260808 01:51:04.216149  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:04.249342  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:04.248848  9556 rpcz_store.cc:276] 0808 01:51:02.998179 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:04.011564 (+1013385us) service_pool.cc:224] Handling call
0808 01:51:04.184206 (+172642us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:51:04.277642  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:04.273532 13455 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 3.216s	user 0.010s	sys 0.010s
I20260808 01:51:04.216217 13204 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.073s
I20260808 01:51:04.291014 13574 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:04.292214  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2534) took 1293 ms. Trace:
I20260808 01:51:04.292783  9556 rpcz_store.cc:276] 0808 01:51:02.998803 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:04.291166 (+1292363us) service_pool.cc:224] Handling call
0808 01:51:04.291568 (+   402us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:04.291780 (+   212us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:04.291827 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:04.291882 (+    55us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:51:04.291927 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:04.292000 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:04.292142 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:04.241966 13465 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:04.295966 13465 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:04.306903 13465 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:04.291913 13574 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:04.316663  9556 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 7. Current term is 8. Ops: []
I20260808 01:51:04.364084  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2535) took 1322 ms. Trace:
I20260808 01:51:04.364614  9556 rpcz_store.cc:276] 0808 01:51:03.041576 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:04.294014 (+1252438us) service_pool.cc:224] Handling call
0808 01:51:04.294451 (+   437us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:04.363723 (+ 69272us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:04.364007 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:04.288179  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:04.376864  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:04.378222  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:04.379423  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:04.386037  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:04.389202 13574 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:04.398903 13492 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:04.399786 13492 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:04.412714 13492 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:04.445003 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 177961us
I20260808 01:51:04.446327 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:04.179586 13183 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:04.448107 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 254406us
I20260808 01:51:04.448627 13183 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:04.455091  9473 raft_consensus.cc:2376] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6. Re-sending same reply.
I20260808 01:51:04.471827  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:51:04.461220 13574 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:04.472503  9377 catalog_manager.cc:5697] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 5 to 6, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 6 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:04.461416 13572 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:04.184588 13490 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:04.480538  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:04.187741 13485 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:04.468019 13313 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.013s	sys 0.032s
I20260808 01:51:04.491919  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:04.493844  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:04.495395  9473 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 4.
I20260808 01:51:04.496937  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:04.497866 13313 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.014s	sys 0.032s
I20260808 01:51:04.484882 13574 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:04.484977 13430 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:04.498602  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:04.486635 13569 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:04.511371 13578 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
W20260808 01:51:04.516363  9560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 2514) took 2763 ms (client timeout 1960 ms). Trace:
I20260808 01:51:04.520787  9473 raft_consensus.cc:2410] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
W20260808 01:51:04.523910  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 279669us
W20260808 01:51:04.524469 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 62370.6us
I20260808 01:51:04.536068  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:04.536412  9524 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:04.536739  9473 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:04.537690  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:04.538486  9473 raft_consensus.cc:2410] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
W20260808 01:51:04.534399  9560 rpcz_store.cc:269] 0808 01:51:01.752592 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:01.753551 (+   959us) service_pool.cc:224] Handling call
0808 01:51:04.515987 (+2762436us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b540008acd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:04.516255 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2841517952}
I20260808 01:51:04.539937  9524 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:04.524915 13590 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:51:04.486948 13533 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:04.508814 13183 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:04.530236 13411 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:51:04.532039 13574 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:04.532534 13430 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:04.545496  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2536) took 1503 ms. Trace:
I20260808 01:51:04.560504  9556 rpcz_store.cc:276] 0808 01:51:03.041912 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:04.365671 (+1323759us) service_pool.cc:224] Handling call
0808 01:51:04.366155 (+   484us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:04.366426 (+   271us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:51:04.448168 (+ 81742us) log.cc:844] Serialized 14 byte log entry
0808 01:51:04.545137 (+ 96969us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:04.545198 (+    61us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:04.545247 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:04.545313 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:04.545436 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1315,"prepare.queue_time_us":79545,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":2429,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":5953,"thread_start_us":174153,"threads_started":2}
I20260808 01:51:04.518575 13490 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:04.563870  9556 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 9. Ops: []
I20260808 01:51:04.564668  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1999) took 1508 ms. Trace:
I20260808 01:51:04.564996  9556 rpcz_store.cc:276] 0808 01:51:03.055784 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:04.563250 (+1507466us) service_pool.cc:224] Handling call
0808 01:51:04.563660 (+   410us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:04.564390 (+   730us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:04.564607 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:04.566773  9556 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260808 01:51:04.567476  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2003) took 1510 ms. Trace:
I20260808 01:51:04.490697 13572 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:04.554087 10365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 559) took 3032 ms (client timeout 1816 ms). Trace:
I20260808 01:51:04.541115 13460 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260808 01:51:04.551785 13421 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.640s	user 0.011s	sys 0.004s
I20260808 01:51:04.558297  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2151) took 1752 ms. Trace:
W20260808 01:51:04.559434  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:04.541479 13569 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:04.551658 13434 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:04.569445  9377 catalog_manager.cc:5697] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 5 to 9, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 9 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:51:04.517810  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:51:04.567531 13411 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:04.547196  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:04.567968  9556 rpcz_store.cc:276] 0808 01:51:03.057120 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:04.566071 (+1508951us) service_pool.cc:224] Handling call
0808 01:51:04.566481 (+   410us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:04.567181 (+   700us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:04.567410 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:51:04.574349 10365 rpcz_store.cc:269] 0808 01:51:01.522622 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:01.528233 (+  5611us) service_pool.cc:224] Handling call
0808 01:51:04.553787 (+3025554us) spinlock_profiling.cc:243] Waited 642 ms on lock 0x7b5400785ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:04.553990 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1475921664}
I20260808 01:51:04.556537  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 524) took 2642 ms. Trace:
I20260808 01:51:04.562412 13430 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:04.583264  9478 rpcz_store.cc:276] 0808 01:51:02.806188 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:03.127419 (+321231us) service_pool.cc:224] Handling call
0808 01:51:04.557975 (+1430556us) spinlock_profiling.cc:243] Waited 80.8 ms on lock 0x7b540022b5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:04.558179 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":185760640}
I20260808 01:51:04.620245  9473 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
W20260808 01:51:04.590929 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 139869us
I20260808 01:51:04.610821  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2531) took 1723 ms. Trace:
I20260808 01:51:04.625406  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:04.565346 13490 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:04.628443  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:04.630189 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:04.618849  9477 rpcz_store.cc:276] 0808 01:51:01.913686 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:01.949516 (+ 35830us) service_pool.cc:224] Handling call
0808 01:51:01.981312 (+ 31796us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:02.213561 (+232249us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:02.213617 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:02.226933 (+ 13316us) log.cc:844] Serialized 26 byte log entry
0808 01:51:02.320779 (+ 93846us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:02.320831 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:04.555964 (+2235133us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b54002a6cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:04.556038 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:04.556109 (+    71us) raft_consensus.cc:1685] finished
0808 01:51:04.556125 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:04.556455 (+   330us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":769,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":9664,"spinlock_wait_cycles":532713728,"thread_start_us":52981,"threads_started":1,"wal-append.queue_time_us":70607,"wal-append.run_cpu_time_us":4920,"wal-append.run_wall_time_us":1291421}
I20260808 01:51:04.626787  9550 rpcz_store.cc:276] 0808 01:51:02.887313 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:03.936725 (+1049412us) service_pool.cc:224] Handling call
0808 01:51:03.939538 (+  2813us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:03.939809 (+   271us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:03.939834 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:03.942123 (+  2289us) log.cc:844] Serialized 26 byte log entry
0808 01:51:04.214270 (+272147us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:51:04.214314 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:04.214406 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:04.610520 (+396114us) raft_consensus.cc:1685] finished
0808 01:51:04.610543 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:04.610738 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":270774,"threads_started":1,"wal-append.queue_time_us":464796}
I20260808 01:51:04.651012  9473 raft_consensus.cc:2393] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 8: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:51:04.603498 13580 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260808 01:51:04.653941  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:04.653975  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:04.655679  9550 raft_consensus.cc:2368] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11.
I20260808 01:51:04.590152 13204 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:04.596386 13411 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:51:04.663862  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:04.664849  9550 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:04.667213 13313 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:51:04.667886 13313 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:04.668910 13313 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:04.635583 13460 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:04.670802 13204 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:04.672462  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:04.673470  9477 raft_consensus.cc:2410] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 10 because replica has last-logged OpId of term: 9 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260808 01:51:04.673868 13204 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:04.673643 13484 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:04.590029 13244 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.073s
I20260808 01:51:04.674744 13484 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:04.657481  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:04.675778  9478 raft_consensus.cc:2410] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260808 01:51:04.675673  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:04.676333  9555 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:04.676749  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:04.568383 13533 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:04.677403  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2150) took 1871 ms. Trace:
I20260808 01:51:04.677856  9475 rpcz_store.cc:276] 0808 01:51:02.805893 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:03.927135 (+1121242us) service_pool.cc:224] Handling call
0808 01:51:03.934803 (+  7668us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:04.012704 (+ 77901us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:04.677081 (+664377us) spinlock_profiling.cc:243] Waited 76.8 ms on lock 0x7b54002bb6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:04.677327 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":176625920}
I20260808 01:51:04.589816 13590 raft_consensus.cc:697] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:04.679150 13533 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:04.680009 13513 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 2.371s	user 0.019s	sys 0.014s
I20260808 01:51:04.679637 13590 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:04.656944  9473 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:04.596541 13434 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:04.590365 13414 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:04.671185 13460 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:04.573726 13584 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.085s
W20260808 01:51:04.659554  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 112632us
I20260808 01:51:04.691598  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:04.692557  9556 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:04.693678  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:04.694303  9556 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:04.695240  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:04.684154  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:04.696708  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:04.697753  9475 raft_consensus.cc:2410] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260808 01:51:04.699108  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:04.700549  9475 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:04.668967 13448 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:51:04.677400  9478 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:04.653074 12826 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:04.595134 13441 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.068s
I20260808 01:51:04.695591  9555 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:04.674427  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:04.711236  9477 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:04.711092  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:04.590754 13421 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:04.688766  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 522) took 2775 ms. Trace:
I20260808 01:51:04.714205  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:04.704543  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 111471us
I20260808 01:51:04.714510  9474 rpcz_store.cc:276] 0808 01:51:01.912944 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:01.914942 (+  1998us) service_pool.cc:224] Handling call
0808 01:51:01.917100 (+  2158us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:01.953949 (+ 36849us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:01.953994 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:01.980417 (+ 26423us) log.cc:844] Serialized 26 byte log entry
0808 01:51:02.114886 (+134469us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:02.115002 (+   116us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:02.115194 (+   192us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:04.679288 (+2564094us) spinlock_profiling.cc:243] Waited 5.62 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:04.681208 (+  1920us) raft_consensus.cc:1685] finished
0808 01:51:04.681231 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:04.688658 (+  7427us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26152,"raft.queue_time_us":11573,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":59187,"spinlock_wait_cycles":12928384,"thread_start_us":51376,"threads_started":1,"wal-append.queue_time_us":197206}
I20260808 01:51:04.707074  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:04.719646  9475 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20260808 01:51:04.720080  9475 pending_rounds.cc:85] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 5
I20260808 01:51:04.720307  9475 pending_rounds.cc:107] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20260808 01:51:04.722011  9475 raft_consensus.cc:2889] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:04.712070  9478 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:51:04.723690  9473 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:04.688410 13434 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:04.724027  9478 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:51:04.727655  9524 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:04.727905 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 88037.3us
I20260808 01:51:04.709985 13204 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:04.725057  9474 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260808 01:51:04.713209  9550 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:04.729120  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:04.731079 10332 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:04.732743  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:04.733840  9550 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:04.724897 13424 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:51:04.736182  9599 leader_election.cc:400] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11.
I20260808 01:51:04.724750 13470 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:51:04.704046 13414 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:04.727511 13588 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
I20260808 01:51:04.753786 13572 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:04.754292 13363 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:04.756419 13569 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:04.751608  9599 leader_election.cc:403] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260808 01:51:04.720636 12826 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:04.765244  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:04.771363  9555 raft_consensus.cc:2410] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 10 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260808 01:51:04.773368 13460 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:04.774109 12826 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:04.775050 13477 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:04.776638  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:04.779208  9556 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:04.759167 13204 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:04.748214 13424 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:04.782325 13441 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:04.782455  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:04.798943  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 71357.3us
I20260808 01:51:04.735055 13596 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
W20260808 01:51:04.797991  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 69828.4us
I20260808 01:51:04.799448  9555 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:51:04.810417  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:04.770080 13587 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.083s	user 0.001s	sys 0.001s
I20260808 01:51:04.800228  9477 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:04.800354  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:04.776995 13490 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:04.812561  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:04.815965  9524 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:04.797572 13204 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:04.816180  9556 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:04.816196  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:04.820150  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:04.818943 13204 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:04.826313 13569 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:04.828140 13527 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:04.829324 13527 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:04.819445 13391 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:04.831092 13391 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:04.831523 13391 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:04.831703 13504 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:04.832165 13504 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:04.833235 13504 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:04.817795 13596 consensus_queue.cc:1243] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260808 01:51:04.835620 10367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 659) took 2605 ms. Trace:
I20260808 01:51:04.836103 10367 rpcz_store.cc:276] 0808 01:51:02.230186 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:02.239584 (+  9398us) service_pool.cc:224] Handling call
0808 01:51:02.261419 (+ 21835us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:02.343484 (+ 82065us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:02.343523 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:02.345031 (+  1508us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:02.345100 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:04.835288 (+2490188us) spinlock_profiling.cc:243] Waited 65.2 ms on lock 0x7b5400690fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:04.835340 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:04.835527 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":149881600}
I20260808 01:51:04.821420 13460 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:04.837433 13460 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:04.838513 13460 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:04.839366 12826 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:04.839860 12826 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:04.840667  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2387) took 3556 ms. Trace:
I20260808 01:51:04.841121  9476 rpcz_store.cc:276] 0808 01:51:01.284346 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:01.284549 (+   203us) service_pool.cc:224] Handling call
0808 01:51:01.285036 (+   487us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:02.681791 (+1396755us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:04.840389 (+2158598us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54002c0454. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:04.840594 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3210513792}
I20260808 01:51:04.842211 13434 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:04.824576 13595 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:04.843459 13595 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:04.843014 13424 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:04.843897 13595 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:04.835677 13596 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20260808 01:51:04.817576 13490 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:04.820802 13470 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.006s
I20260808 01:51:04.853849  9599 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:04.881912 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 145751us
W20260808 01:51:04.882266  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:04.883764  9599 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:04.883634 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:04.819147 13599 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260808 01:51:05.065914  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:05.062517 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 179009us
W20260808 01:51:04.887010 13598 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
W20260808 01:51:04.957800  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 139447us
W20260808 01:51:04.907269  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:05.075088  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:05.071480 10332 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:04.968501  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 120359us
I20260808 01:51:04.877089 13421 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:04.876569 13424 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:04.882799 13569 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:05.062464 13504 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:05.035895 13441 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11.
I20260808 01:51:04.933490 13533 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:51:04.887579 13381 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:04.920805 13590 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:04.892352 13348 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:05.069984 13599 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:05.086592  9524 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:05.071731 13454 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:05.086423 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:05.095574  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:05.003268 13470 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:05.088541  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:05.091079 13604 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.155s	user 0.001s	sys 0.000s
I20260808 01:51:04.982468 13391 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:05.092002 13381 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:05.092636  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:05.088641 13588 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.006s
I20260808 01:51:04.912531 13526 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election won for term 10
W20260808 01:51:05.100559  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:05.061024 13204 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:05.101194 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:05.103628  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:05.127292  9559 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:51:05.119771  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:05.130326  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:05.132659  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:05.137365 13590 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:05.138988  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2131) took 2283 ms. Trace:
I20260808 01:51:05.139223  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2138) took 2103 ms. Trace:
I20260808 01:51:05.139469  9629 rpcz_store.cc:276] 0808 01:51:02.856083 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:02.857360 (+  1277us) service_pool.cc:224] Handling call
0808 01:51:02.879818 (+ 22458us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:03.099050 (+219232us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:05.138613 (+2039563us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b54001b3fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:05.138910 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":469306112}
I20260808 01:51:05.141130 13414 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:05.141877 13414 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:05.142481 13414 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:05.143656 13414 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:05.122646 13526 raft_consensus.cc:697] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:05.123538 13391 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:05.139635  9638 rpcz_store.cc:276] 0808 01:51:03.036185 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:03.064007 (+ 27822us) service_pool.cc:224] Handling call
0808 01:51:03.064407 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:03.899567 (+835160us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:05.138986 (+1239419us) spinlock_profiling.cc:243] Waited 824 ms on lock 0x7b5400420ed4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:05.139161 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1894651648}
I20260808 01:51:05.125873 13467 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:05.148837 13526 consensus_queue.cc:237] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:05.152165  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 75520.2us
I20260808 01:51:05.158857  9633 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7. Ops: []
I20260808 01:51:05.138329 13590 raft_consensus.cc:697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:05.160678 13590 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:05.101931 13348 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260808 01:51:05.163197  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:05.166572  9476 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:51:05.167614  9559 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:51:05.167868  9524 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:05.098091 13599 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:05.087713 13605 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:05.100044 13460 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:05.164108  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:05.164404  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:05.186635  9478 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:05.187105  9599 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:05.165192  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:05.102662 13597 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:05.168702  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:05.188930  9559 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 8: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:04.966135 12826 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:05.172524  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2132) took 2258 ms. Trace:
I20260808 01:51:05.193528  9636 rpcz_store.cc:276] 0808 01:51:02.914549 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:02.975221 (+ 60672us) service_pool.cc:224] Handling call
0808 01:51:02.975688 (+   467us) raft_consensus.cc:1471] Updating replica for 3 ops
0808 01:51:04.033945 (+1058257us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:05.172068 (+1138123us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540003c5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:05.172427 (+   359us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2431826304}
I20260808 01:51:05.185063  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:05.188019  9474 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:51:05.190634  9599 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:05.195242  9550 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:05.192471  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:05.196444  9477 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 8: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:05.192385  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:05.176394  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 529) took 3213 ms. Trace:
I20260808 01:51:05.196496 13460 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:05.202090  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:05.198748  9479 rpcz_store.cc:276] 0808 01:51:01.963421 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:02.277323 (+313902us) service_pool.cc:224] Handling call
0808 01:51:02.277649 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:03.105970 (+828321us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:05.176011 (+2070041us) spinlock_profiling.cc:243] Waited 802 ms on lock 0x7b54002bacd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:05.176300 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1843923712}
I20260808 01:51:05.202934  9550 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 9: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:05.203061 13381 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:05.196118 13504 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:05.203857  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:05.205159  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:05.205286 13460 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:05.205799  9550 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:51:05.206734  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:05.207453  9550 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:05.208042  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:05.208647  9559 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:05.207484 13460 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:05.212153 13460 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:05.217892 13460 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:05.218978 13433 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:51:05.219700 13433 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:05.220207 13433 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:51:05.101269 13470 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:05.253805  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:05.256695  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:05.258984  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:05.259913  9474 raft_consensus.cc:2410] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 3.
I20260808 01:51:05.274947  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:05.276052  9473 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:05.278795  9473 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260808 01:51:05.279456  9473 pending_rounds.cc:85] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 5
I20260808 01:51:05.280045  9473 pending_rounds.cc:107] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260808 01:51:05.280543  9473 raft_consensus.cc:2889] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:05.282635  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:05.288708  9550 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260808 01:51:05.289239  9550 pending_rounds.cc:85] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 3
I20260808 01:51:05.289489  9550 pending_rounds.cc:107] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260808 01:51:05.289760  9550 raft_consensus.cc:2889] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:05.255657 13528 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:05.298744  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:05.195354 13569 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:05.311882  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:05.312676  9559 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:51:05.323799 13204 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:05.327178 13411 consensus_meta.cc:287] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.637s	user 0.014s	sys 0.000s
I20260808 01:51:05.327709 13411 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:05.330540 13411 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:05.333628  9478 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:05.315469  9559 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:05.256875 13595 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:05.361600  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:05.362666  9474 raft_consensus.cc:2410] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260808 01:51:05.362716  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:05.363499  9473 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:05.363730  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:05.364404  9474 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:05.367688 13460 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:05.371131  9554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 2497) took 4659 ms (client timeout 2274 ms). Trace:
I20260808 01:51:05.371480 13365 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260808 01:51:05.371614  9554 rpcz_store.cc:269] 0808 01:51:00.711724 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:00.712192 (+   468us) service_pool.cc:224] Handling call
0808 01:51:05.370911 (+4658719us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54000de154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:05.371022 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2551560960}
I20260808 01:51:05.372380 13365 consensus_queue.cc:1243] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 9.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260808 01:51:05.372684  9559 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:05.374260 13470 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:05.375283 13319 raft_consensus.cc:3055] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:05.375758 13319 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:05.375926 13502 consensus_queue.cc:1059] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:51:05.377110 13502 raft_consensus.cc:3055] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:05.365355  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:05.377629 13502 raft_consensus.cc:740] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:05.331691 13528 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:05.378650 13502 consensus_queue.cc:260] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:05.383407 12826 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:05.347158 13347 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:05.379142  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:05.391068 13347 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:05.392014 13082 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:05.392457  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:05.380229 13528 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:05.383828 13502 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:05.376832 13319 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:05.387913  9557 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:05.396873 13434 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:05.397012 13319 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:05.389127  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:05.398205  9472 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:05.315578 13490 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:05.399509  9599 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:05.401024  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 540) took 1135 ms. Trace:
I20260808 01:51:05.403081 13246 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:05.403606 13246 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:05.404042 13467 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:05.404668 13467 raft_consensus.cc:697] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:05.404554 13246 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:05.405874 13246 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:05.389308 13595 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:05.401503  9480 rpcz_store.cc:276] 0808 01:51:04.265784 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:04.266008 (+   224us) service_pool.cc:224] Handling call
0808 01:51:04.266462 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:04.566515 (+300053us) raft_consensus.cc:1534] Early marking committed up to index 10
0808 01:51:04.566835 (+   320us) log.cc:844] Serialized 14 byte log entry
0808 01:51:04.651803 (+ 84968us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:04.651889 (+    86us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:51:04.651960 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:05.400676 (+748716us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b54000217d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:05.400761 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:05.400945 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":583,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":124,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"spinlock_wait_cycles":689459968,"thread_start_us":84170,"threads_started":1,"wal-append.queue_time_us":84525}
I20260808 01:51:05.399814  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2009) took 2243 ms. Trace:
I20260808 01:51:05.408746  9550 rpcz_store.cc:276] 0808 01:51:03.156134 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:05.208933 (+2052799us) service_pool.cc:224] Handling call
0808 01:51:05.209217 (+   284us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:05.290184 (+ 80967us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:05.399533 (+109349us) spinlock_profiling.cc:243] Waited 77.7 ms on lock 0x7b54002a8354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:05.399747 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":178810112}
I20260808 01:51:05.379145  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:05.410714  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:05.413723  9550 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:05.405738 13467 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:05.419330  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:05.425948  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:05.370420 13460 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:05.432612  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 264846us
I20260808 01:51:05.417788  9473 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
W20260808 01:51:05.452018  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:05.459834  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:05.459926  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:05.372922 13365 raft_consensus.cc:3055] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Stepping down as leader of term 8
W20260808 01:51:05.415270 13417 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.456s	user 0.023s	sys 0.023s
I20260808 01:51:05.492491  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:05.487525 10368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 513) took 2612 ms. Trace:
I20260808 01:51:05.393698 13347 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:05.499545  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:05.500682  9474 raft_consensus.cc:2410] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260808 01:51:05.503458  9555 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:05.504714  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:05.505475  9524 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:05.505594  9472 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:05.506737 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 374312us
W20260808 01:51:05.510881  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:05.419329 13490 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:05.515533  9524 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:05.519912  9550 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:05.521780  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:05.522015  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:05.499137 13417 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.533s	user 0.024s	sys 0.023s
I20260808 01:51:05.576833  9550 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
W20260808 01:51:05.577760  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 62291.5us
I20260808 01:51:05.577915  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:05.578923  9474 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:05.488998 13365 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:05.492331 13492 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260808 01:51:05.591130  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:05.592213  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:05.592231  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:05.496423 10368 rpcz_store.cc:276] 0808 01:51:02.874941 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:02.876161 (+  1220us) service_pool.cc:224] Handling call
0808 01:51:02.876509 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:02.876755 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:51:02.921202 (+ 44447us) log.cc:844] Serialized 14 byte log entry
0808 01:51:03.587143 (+665941us) log.cc:844] Serialized 14 byte log entry
0808 01:51:04.668586 (+1081443us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b1800484fc0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:51:05.174882 (+506296us) spinlock_profiling.cc:243] Waited 78.9 ms on lock 0x7b1800484fc0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667 00007f5d449ae5e8
0808 01:51:05.487057 (+312175us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:05.487123 (+    66us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:05.487178 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:05.487261 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:05.487450 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":343125,"prepare.run_cpu_time_us":121,"prepare.run_wall_time_us":127,"raft.queue_time_us":160966,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":252,"spinlock_wait_cycles":213258496,"thread_start_us":846678,"threads_started":3,"wal-append.queue_time_us":1127214,"wal-append.run_cpu_time_us":1778,"wal-append.run_wall_time_us":1001801}
I20260808 01:51:05.515844 13411 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:05.514629 13432 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:05.518033 13470 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:05.517696 13454 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:05.639181  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:05.640733  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:05.583009 13347 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:05.596292 13460 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:51:05.597342 13204 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:05.588587 13526 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:05.694612  9474 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
W20260808 01:51:05.694886  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:05.685436  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:05.543119 13490 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:05.725198  9524 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:05.672277  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:05.648322 13610 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:05.741787  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2006) took 2682 ms. Trace:
I20260808 01:51:05.742244  9557 rpcz_store.cc:276] 0808 01:51:03.059514 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:04.815332 (+1755818us) service_pool.cc:224] Handling call
0808 01:51:05.387715 (+572383us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:05.396925 (+  9210us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:05.741577 (+344652us) spinlock_profiling.cc:243] Waited 563 ms on lock 0x7b54000b45d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:05.741723 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1295660288}
W20260808 01:51:05.668874 13424 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.582s	user 0.012s	sys 0.000s
W20260808 01:51:05.765383  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:05.763492  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:05.769095 13134 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:05.769469  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:05.769817 13134 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:05.670527 13460 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:05.739225 13204 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:05.771250 13532 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:05.771798 13532 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:05.772511 13483 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:05.771908 13204 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:05.770776 13460 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:05.773103 13483 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:05.765085 13603 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:05.775485 13603 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:05.775998 13483 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:05.776684 13603 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:05.778030  9444 connection.cc:256] RPC call timeout handler was delayed by -1.8412856009672396s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1626s for a response
W20260808 01:51:05.778617  9444 connection.cc:256] RPC call timeout handler was delayed by -1.8412856009672396s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1626s for a response
W20260808 01:51:05.779264  9444 connection.cc:256] RPC call timeout handler was delayed by -1.7899856009753421s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1683s for a response
W20260808 01:51:05.779811  9444 connection.cc:256] RPC call timeout handler was delayed by -1.6864856009779032s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17980000000000002s for a response
W20260808 01:51:05.780316  9444 connection.cc:256] RPC call timeout handler was delayed by -1.6807798380032182s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1522s for a response
W20260808 01:51:05.780725  9444 connection.cc:256] RPC call timeout handler was delayed by -1.5532856009667739s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1946s for a response
W20260808 01:51:05.781061  9444 connection.cc:256] RPC call timeout handler was delayed by -1.5532856009667739s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1946s for a response
W20260808 01:51:05.781451  9444 connection.cc:256] RPC call timeout handler was delayed by -1.4071798379882239s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1826s for a response
I20260808 01:51:05.784749  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 2546) took 1195 ms. Trace:
W20260808 01:51:05.781805  9444 connection.cc:256] RPC call timeout handler was delayed by -1.4047856009856332s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21110000000000004s for a response
I20260808 01:51:05.785250  9555 rpcz_store.cc:276] 0808 01:51:04.588932 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:04.809832 (+220900us) service_pool.cc:224] Handling call
0808 01:51:05.784484 (+974652us) spinlock_profiling.cc:243] Waited 692 ms on lock 0x7b540002c6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:05.784669 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1591565568}
W20260808 01:51:05.785990  9444 connection.cc:256] RPC call timeout handler was delayed by -1.4047856009856332s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21110000000000004s for a response
W20260808 01:51:05.786460  9444 connection.cc:256] RPC call timeout handler was delayed by -1.2598274269839749s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1507s for a response
I20260808 01:51:05.788631  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2065) took 4419 ms. Trace:
I20260808 01:51:05.791414  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:05.645394 13432 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:05.644673 13411 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:05.718780 13347 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:05.652082 13454 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:05.786933  9444 connection.cc:256] RPC call timeout handler was delayed by -1.2598274269839749s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1507s for a response
I20260808 01:51:05.789237  9633 rpcz_store.cc:276] 0808 01:51:01.369205 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:01.765872 (+396667us) service_pool.cc:224] Handling call
0808 01:51:01.766226 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:05.159482 (+3393256us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:05.788297 (+628815us) spinlock_profiling.cc:243] Waited 3.39 s on lock 0x7b5400421dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:05.788545 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7792577920}
I20260808 01:51:05.667318 13417 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:05.799201  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:05.852581  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:05.855584  9444 connection.cc:256] RPC call timeout handler was delayed by -1.2463274270121474s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1522s for a response
W20260808 01:51:05.856931  9444 connection.cc:256] RPC call timeout handler was delayed by -1.2463274270121474s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1522s for a response
I20260808 01:51:05.651628 13470 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:05.836347  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:05.872309  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:05.857376  9444 connection.cc:256] RPC call timeout handler was delayed by -1.2380501539737452s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1927s for a response
W20260808 01:51:05.874516  9444 connection.cc:256] RPC call timeout handler was delayed by -1.2380501539737452s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1927s for a response
W20260808 01:51:05.875118  9444 connection.cc:256] RPC call timeout handler was delayed by -1.132927427010145s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1648s for a response
W20260808 01:51:05.875587  9444 connection.cc:256] RPC call timeout handler was delayed by -1.132927427010145s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1648s for a response
I20260808 01:51:05.854406  9475 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:51:05.804518 13507 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:05.885082  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:05.821943 13303 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:05.876068  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1309501539799385s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2046s for a response
I20260808 01:51:05.822402 13397 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:05.830830 13424 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:05.902201  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:05.904196  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:05.912741  9555 raft_consensus.cc:2393] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 5: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:05.740175 13526 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:05.903316  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1309501539799385s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2046s for a response
I20260808 01:51:05.908124  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:05.913959  9557 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
W20260808 01:51:05.915292  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0906856009678449s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.246s for a response
W20260808 01:51:05.915915  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0906856009678449s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.246s for a response
W20260808 01:51:05.917146  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0705798379785847s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22000000000000003s for a response
I20260808 01:51:05.920491  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:05.929564  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:05.944789  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:05.953459  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:05.739940 13365 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:05.904696 13414 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.020s	sys 0.030s
I20260808 01:51:05.978746  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:05.913456 13446 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.734s	user 0.038s	sys 0.014s
W20260808 01:51:05.885396 13605 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.678s	user 0.021s	sys 0.032s
I20260808 01:51:05.765820 13610 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:05.907634 13363 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:06.013887 13363 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:05.987144  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:05.975692 13082 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:05.984599  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:05.986717  9558 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.014s	sys 0.043s
W20260808 01:51:06.025725  9558 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.015s	sys 0.043s
I20260808 01:51:05.985487 13532 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:06.004936  9474 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260808 01:51:06.011289  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:06.017238  9555 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:06.029342  9557 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:51:06.033504  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.868s	user 0.013s	sys 0.029s
W20260808 01:51:06.033941  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.868s	user 0.013s	sys 0.029s
W20260808 01:51:05.992081 13446 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.810s	user 0.039s	sys 0.014s
W20260808 01:51:06.022217  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:06.042237  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:05.990579 13584 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.027s	sys 0.012s
I20260808 01:51:05.868330 13417 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:05.912374 13303 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:06.070216  9560 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: [6.4-8.5]
I20260808 01:51:06.070425  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:06.071184  9555 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:51:06.040024  9474 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
W20260808 01:51:06.040421  9521 connection.cc:256] RPC call timeout handler was delayed by -1.1400141449994408s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2643s for a response
W20260808 01:51:05.983012 13414 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.021s	sys 0.030s
I20260808 01:51:05.909240 13507 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:05.935021 13397 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:06.072731  9521 connection.cc:256] RPC call timeout handler was delayed by -1.0725141449947841s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2718s for a response
I20260808 01:51:06.074945  9521 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:06.003211 13605 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.796s	user 0.026s	sys 0.032s
I20260808 01:51:06.033984 13319 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:06.025514 13082 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:06.032395 13365 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:51:06.064980 13584 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.744s	user 0.028s	sys 0.012s
W20260808 01:51:05.937259 13588 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.707s	user 0.004s	sys 0.033s
W20260808 01:51:06.068691 13504 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.864s	user 0.011s	sys 0.003s
I20260808 01:51:06.085628  9475 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
W20260808 01:51:06.108636  9479 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.015s	sys 0.015s
I20260808 01:51:06.076875  9521 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:06.018055 13597 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.768s	user 0.014s	sys 0.028s
W20260808 01:51:06.032934 13569 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.719s	user 0.009s	sys 0.032s
W20260808 01:51:06.103467  9377 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.734s	user 0.018s	sys 0.024s
I20260808 01:51:05.984840 13424 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:06.077822 13533 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:06.080145 13446 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260808 01:51:06.109092 13588 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.879s	user 0.005s	sys 0.034s
I20260808 01:51:06.096472 13082 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:51:06.111748  9472 consensus_meta.cc:287] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.604s	user 0.012s	sys 0.000s
I20260808 01:51:06.113173  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:06.109422  9554 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.727s	user 0.006s	sys 0.035s
I20260808 01:51:06.079691 13605 raft_consensus.cc:697] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:51:06.117285  9377 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.747s	user 0.018s	sys 0.024s
I20260808 01:51:06.118777  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:06.039384 13347 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:06.137933  9472 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
W20260808 01:51:06.116560 13597 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.868s	user 0.014s	sys 0.029s
I20260808 01:51:06.140579  9475 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260808 01:51:06.151329  9521 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:06.109072  9479 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.015s	sys 0.015s
I20260808 01:51:06.109838 13504 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260808 01:51:06.150967  9478 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.011s	sys 0.026s
W20260808 01:51:06.116719 13569 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.803s	user 0.010s	sys 0.033s
W20260808 01:51:06.212483  9559 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.707s	user 0.008s	sys 0.033s
W20260808 01:51:06.217492  9478 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.012s	sys 0.027s
W20260808 01:51:06.173815  9473 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.659s	user 0.010s	sys 0.026s
W20260808 01:51:06.218158  9559 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.008s	sys 0.033s
W20260808 01:51:06.161186  9556 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.775s	user 0.010s	sys 0.031s
W20260808 01:51:06.116626 13313 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.725s	user 0.008s	sys 0.021s
W20260808 01:51:06.230298  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 79095.9us
I20260808 01:51:06.133980  9599 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:06.222777  9554 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.849s	user 0.013s	sys 0.035s
I20260808 01:51:06.232801  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:06.219272  9473 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.705s	user 0.010s	sys 0.032s
W20260808 01:51:06.136938 13599 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.940s	user 0.010s	sys 0.004s
W20260808 01:51:06.222078  9556 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.836s	user 0.012s	sys 0.037s
I20260808 01:51:06.151895 13575 consensus_queue.cc:1059] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.080s
I20260808 01:51:06.235137  9557 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
W20260808 01:51:06.234334  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 100475us
I20260808 01:51:06.148950 13411 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:06.118970 13414 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:06.129022 13533 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:06.146136 13490 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:06.139979 13612 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
W20260808 01:51:06.228695 13313 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.838s	user 0.009s	sys 0.025s
I20260808 01:51:06.146596 13616 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:06.178318 13446 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 01:51:06.243949  9551 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.011s	sys 0.006s
I20260808 01:51:06.146307 13363 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:06.223951 13597 raft_consensus.cc:697] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:06.259670  9475 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260808 01:51:06.233640 13470 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:06.263847  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:06.245494 13504 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:06.152457 12778 raft_consensus.cc:2764] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election decision vote started in defunct term 10: won
I20260808 01:51:06.230207 13623 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:06.286051  9551 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.012s	sys 0.006s
I20260808 01:51:06.258632 13569 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:06.304857  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 479481us
W20260808 01:51:06.347054  9477 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.016s	sys 0.034s
W20260808 01:51:06.347456  9477 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.016s	sys 0.034s
I20260808 01:51:06.302839 13526 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:06.280975 13613 log.cc:927] Time spent T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.211s	user 0.001s	sys 0.001s
I20260808 01:51:06.325258 13363 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:06.326431  9557 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:06.246543 13599 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:06.139406 13618 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.276s
I20260808 01:51:06.334401 13533 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:06.352044 13614 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.249s	user 0.002s	sys 0.000s
W20260808 01:51:06.351658  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 406972us
I20260808 01:51:06.278035 13607 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20260808 01:51:06.316776 13414 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:51:06.378091 13624 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.263s	user 0.000s	sys 0.002s
W20260808 01:51:06.409814 10332 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.749s	user 0.009s	sys 0.017s
W20260808 01:51:06.410277 10332 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.749s	user 0.009s	sys 0.017s
W20260808 01:51:06.311962  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 49078.2us
W20260808 01:51:06.315843  9600 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.659s	user 0.011s	sys 0.023s
W20260808 01:51:06.416913  9600 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.760s	user 0.011s	sys 0.024s
W20260808 01:51:06.417413  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 760897us
I20260808 01:51:06.329257 13504 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:51:06.384241  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:06.283929 13631 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.086s
I20260808 01:51:06.341467 13507 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:06.346292  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:06.349496 10331 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.689s	user 0.007s	sys 0.025s
W20260808 01:51:06.458936 10331 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.008s	sys 0.026s
W20260808 01:51:06.459029  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:06.400398 10333 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.001s	sys 0.020s
W20260808 01:51:06.459668 10333 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.001s	sys 0.020s
W20260808 01:51:06.459851  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:06.402889  9523 leader_election.cc:336] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Timed out: RequestConsensusVote RPC to 127.7.165.196:33455 timed out after 1.816s (SENT)
W20260808 01:51:06.461848  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 59028.7us
I20260808 01:51:06.404145  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:06.463272  9557 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:06.281041 13616 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:06.278512 13605 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:06.405901 13533 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:06.409348 13424 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:06.428809  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:06.400822 13611 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.271s	user 0.000s	sys 0.002s
I20260808 01:51:06.379532 13526 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:06.357209 13610 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:06.351528 13569 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:06.468462 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:06.496536 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:06.469878 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:06.472433 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:06.433759 13504 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:06.479974  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:06.480242  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:06.482251  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:06.390059 13363 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:06.491205 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:06.492980 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:06.494251 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:06.511929  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:06.513412  9638 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260808 01:51:06.537129  9634 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:06.540005 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:06.541536 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:06.545146  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:06.546774  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2553) took 1605 ms. Trace:
I20260808 01:51:06.547216  9556 rpcz_store.cc:276] 0808 01:51:04.941517 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:05.385464 (+443947us) service_pool.cc:224] Handling call
0808 01:51:05.385824 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:05.386042 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:51:06.238525 (+852483us) log.cc:844] Serialized 14 byte log entry
0808 01:51:06.544572 (+306047us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:06.544638 (+    66us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:06.544690 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:06.545969 (+  1279us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:06.546706 (+   737us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":931369,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":20501,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":6425,"thread_start_us":1157573,"threads_started":3}
I20260808 01:51:06.493441  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:06.517148  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:06.517853  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:06.349819 13623 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:06.563016 13246 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.156s	user 0.015s	sys 0.001s
I20260808 01:51:06.518859 13575 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:06.520303  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:06.519987  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:06.492182 13533 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:06.513639 10368 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:06.524253  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:06.556356  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 63017.9us
I20260808 01:51:06.556617  9636 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:51:06.517168 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:06.436108 13597 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:06.596257  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2403) took 1247 ms. Trace:
W20260808 01:51:06.599006  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 170339us
I20260808 01:51:06.573328  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:06.675045  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:06.689193  9629 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:51:06.688553  9638 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:06.578528  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2524) took 3901 ms. Trace:
I20260808 01:51:06.564077 10369 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:06.690587 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:06.682937  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2557) took 1622 ms. Trace:
I20260808 01:51:06.677392 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:06.597023  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2398) took 1332 ms. Trace:
I20260808 01:51:06.593705  9550 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
W20260808 01:51:06.539021 13626 log.cc:927] Time spent T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.291s	user 0.001s	sys 0.000s
W20260808 01:51:06.666347  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:06.687800  9632 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:51:06.692745  9551 rpcz_store.cc:276] 0808 01:51:05.060247 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:05.597966 (+537719us) service_pool.cc:224] Handling call
0808 01:51:05.663042 (+ 65076us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:05.663279 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 5
0808 01:51:06.342388 (+679109us) log.cc:844] Serialized 14 byte log entry
0808 01:51:06.682432 (+340044us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:06.682531 (+    99us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:51:06.682585 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:06.682698 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:06.682859 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63906,"prepare.queue_time_us":684927,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":3852,"raft.queue_time_us":4852,"thread_start_us":1013379,"threads_started":2}
I20260808 01:51:06.698129  9472 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260808 01:51:06.694239  9479 rpcz_store.cc:276] 0808 01:51:05.272334 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:05.273687 (+  1353us) service_pool.cc:224] Handling call
0808 01:51:05.274138 (+   451us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:05.283857 (+  9719us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:51:05.332256 (+ 48399us) log.cc:844] Serialized 14 byte log entry
0808 01:51:06.596661 (+1264405us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:06.596717 (+    56us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:51:06.596763 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:06.596830 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:06.596961 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":410291,"prepare.queue_time_us":17752,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":385457,"raft.run_cpu_time_us":422,"raft.run_wall_time_us":19854,"thread_start_us":760939,"threads_started":1}
W20260808 01:51:06.704306  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:06.691308  9558 rpcz_store.cc:276] 0808 01:51:02.677471 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:02.677587 (+   116us) service_pool.cc:224] Handling call
0808 01:51:05.323258 (+2645671us) spinlock_profiling.cc:243] Waited 3.42 ms on lock 0x7b5800003c98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d4185589e 000000000038d209 00007f5d49914940 00007f5d499dced4 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:51:05.323543 (+   285us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:05.324109 (+   566us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:51:06.031759 (+707650us) log.cc:844] Serialized 14 byte log entry
0808 01:51:06.404788 (+373029us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:06.405521 (+   733us) log.cc:844] Serialized 26 byte log entry
0808 01:51:06.405675 (+   154us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:51:06.405717 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:06.405798 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:06.578042 (+172244us) raft_consensus.cc:1685] finished
0808 01:51:06.578202 (+   160us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:06.578431 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":905319,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":1028,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":23528,"spinlock_wait_cycles":7858944,"thread_start_us":1025021,"threads_started":2,"wal-append.queue_time_us":456971}
I20260808 01:51:06.695640  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2558) took 1620 ms. Trace:
W20260808 01:51:06.697813 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 225455us
I20260808 01:51:06.680619  9478 rpcz_store.cc:276] 0808 01:51:05.348706 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:05.348878 (+   172us) service_pool.cc:224] Handling call
0808 01:51:05.349717 (+   839us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:05.349945 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 9
0808 01:51:05.409354 (+ 59409us) log.cc:844] Serialized 14 byte log entry
0808 01:51:06.219530 (+810176us) log.cc:844] Serialized 14 byte log entry
0808 01:51:06.595720 (+376190us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b180032ed80. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667 00007f5d449ae5e8
0808 01:51:06.595888 (+   168us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:06.595942 (+    54us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:51:06.596006 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:06.596077 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:06.596193 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":354065,"prepare.queue_time_us":59195,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":382607,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":11470,"spinlock_wait_cycles":45132928,"thread_start_us":808860,"threads_started":1,"wal-append.queue_time_us":980524}
W20260808 01:51:06.574282 13642 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
W20260808 01:51:06.575098 13619 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.234s	user 0.001s	sys 0.000s
W20260808 01:51:06.574187 13638 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
W20260808 01:51:06.591032 13611 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
W20260808 01:51:06.666847 12826 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.277s	user 0.013s	sys 0.001s
I20260808 01:51:06.707129 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:06.713361  9634 raft_consensus.cc:2410] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260808 01:51:06.708662 10332 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:06.584991 13629 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.301s	user 0.000s	sys 0.001s
I20260808 01:51:06.563223 13569 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:06.590560 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:06.514501 13526 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:06.558223 13424 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.260s
W20260808 01:51:06.697505 13433 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.477s	user 0.011s	sys 0.012s
I20260808 01:51:06.710749 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:51:06.962646 13365 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.858s	user 0.011s	sys 0.003s
W20260808 01:51:06.962809  9363 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.702s	user 0.009s	sys 0.011s
I20260808 01:51:06.724289  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2550) took 2007 ms. Trace:
W20260808 01:51:06.749701  9523 leader_election.cc:336] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Timed out: RequestConsensusVote RPC to 127.7.165.196:33455 timed out after 1.551s (SENT)
I20260808 01:51:06.972289  9554 rpcz_store.cc:276] 0808 01:51:04.717228 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:05.372629 (+655401us) service_pool.cc:224] Handling call
0808 01:51:05.372913 (+   284us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:05.373126 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:51:06.234208 (+861082us) log.cc:844] Serialized 14 byte log entry
0808 01:51:06.723706 (+489498us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:06.723806 (+   100us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:06.723889 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:06.724003 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:06.724203 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14049,"prepare.queue_time_us":906932,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":56,"raft.queue_time_us":17797,"spinlock_wait_cycles":23280384,"thread_start_us":1349619,"threads_started":3,"wal-append.queue_time_us":572585}
W20260808 01:51:06.973091  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:06.725864 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:51:06.720466  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 244e080726394210b5467bf688813cf0: Illegal state: Leader has not yet committed an operation in its own term
W20260808 01:51:06.978754  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 229133us
I20260808 01:51:06.673624 13594 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:06.689385  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:06.784945  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:06.803681  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 227217us
W20260808 01:51:06.973502  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 288055us
I20260808 01:51:06.947582 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:06.982743  9629 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:06.983239  9552 raft_consensus.cc:2410] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260808 01:51:06.976166  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2195) took 1534 ms. Trace:
I20260808 01:51:06.972653  9636 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
W20260808 01:51:06.983772  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 294443us
I20260808 01:51:06.598791 13627 raft_consensus.cc:3055] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:51:06.787495  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:06.755201  9445 leader_election.cc:336] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Timed out: RequestConsensusVote RPC to 127.7.165.196:33455 timed out after 1.683s (SENT)
I20260808 01:51:06.983861  9477 rpcz_store.cc:276] 0808 01:51:05.442143 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:05.442353 (+   210us) service_pool.cc:224] Handling call
0808 01:51:05.443027 (+   674us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:05.771623 (+328596us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:51:05.771947 (+   324us) log.cc:844] Serialized 14 byte log entry
0808 01:51:06.597932 (+825985us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:06.597989 (+    57us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:51:06.598034 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:06.975796 (+377762us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b540021d754. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:06.975851 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:06.976099 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":250330,"prepare.queue_time_us":2239,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":255847,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":167,"spinlock_wait_cycles":754016000,"thread_start_us":575528,"threads_started":1,"wal-append.queue_time_us":1036493}
I20260808 01:51:06.979188  9524 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:06.666590 13527 raft_consensus.cc:2764] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260808 01:51:06.588810 13134 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:06.974717 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 266086us
W20260808 01:51:06.985512  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 230379us
I20260808 01:51:06.555891 13204 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:06.596184 13363 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:07.014081 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:06.707258  9550 rpcz_store.cc:276] 0808 01:51:05.075583 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:06.593172 (+1517589us) service_pool.cc:224] Handling call
0808 01:51:06.593517 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:06.695360 (+101843us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:06.695579 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:07.018561  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:07.019100  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:07.019594  9479 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:07.020215  9478 raft_consensus.cc:2410] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260808 01:51:07.020107  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:06.987239  9523 leader_election.cc:336] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Timed out: RequestConsensusVote RPC to 127.7.165.196:33455 timed out after 1.563s (SENT)
I20260808 01:51:06.574059 13649 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:06.812340  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:06.564544 13414 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.032231  9477 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260808 01:51:07.032907  9477 pending_rounds.cc:85] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 4
I20260808 01:51:07.033365  9477 pending_rounds.cc:107] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260808 01:51:06.573896 13623 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:06.590401 13397 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:06.942152 13660 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.240s	user 0.000s	sys 0.000s
W20260808 01:51:06.941644 13082 consensus_meta.cc:287] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.790s	user 0.012s	sys 0.002s
I20260808 01:51:07.009510  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:07.047449  9554 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 7. Current term is 8. Ops: []
I20260808 01:51:07.048350  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 492) took 1883 ms. Trace:
I20260808 01:51:07.048811  9554 rpcz_store.cc:276] 0808 01:51:05.165141 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:07.046732 (+1881591us) service_pool.cc:224] Handling call
0808 01:51:07.047174 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:07.048026 (+   852us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.048273 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:07.049798 13454 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:06.981194 13433 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.045077 13082 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:07.053035 13433 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:07.034696  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 51127us
I20260808 01:51:06.736866 13246 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:06.710566 13502 consensus_meta.cc:287] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.312s	user 0.013s	sys 0.002s
W20260808 01:51:06.942562 13460 consensus_meta.cc:287] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.169s	user 0.012s	sys 0.001s
I20260808 01:51:06.784399 13581 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:51:06.700742 13641 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.140s	user 0.000s	sys 0.001s
I20260808 01:51:07.024516 13446 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:07.035382 13204 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:06.981225 13411 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:07.030421 13649 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:07.024133 13532 consensus_queue.cc:1059] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260808 01:51:06.992322 13470 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:07.030138 12826 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260808 01:51:07.040117  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:07.030618 13313 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.022s	sys 0.030s
I20260808 01:51:06.800228 13441 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.241s
I20260808 01:51:06.705273 13622 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:07.092907 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.093550 13204 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:07.083336 13446 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:07.041993  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:06.995796 13483 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:06.804620 13599 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.041837 13504 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.043725 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:06.580108 13616 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.058516  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:07.067378 13246 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:06.737517 13652 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:06.992033 13627 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:07.063205  9629 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:06.981841 13631 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:07.065996  9524 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:07.112154 13483 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:06.995994 13647 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:07.061774 10362 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:07.049814 13533 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:07.077442 13478 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:07.077728 13532 raft_consensus.cc:3055] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:07.168777 13669 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:07.169621 13669 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:07.170120 13669 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:07.078775 13470 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.074399 13411 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:07.175148 13662 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.183s	user 0.001s	sys 0.000s
I20260808 01:51:07.177081 13411 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.074610 13649 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:07.091536 13441 raft_consensus.cc:3055] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:51:07.178936 13441 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:07.142467 13483 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:07.180104 13441 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.091780 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.181375 13432 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.182737 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.183966 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.185101 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:07.068917 13607 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:07.096129 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.188601 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.188988 13417 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:07.189764 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:51:07.095875 13313 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.022s	sys 0.030s
I20260808 01:51:07.068146 13460 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.192792 13460 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:07.124866  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66502.5us
I20260808 01:51:07.193851 13303 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:07.168780 13532 raft_consensus.cc:740] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:07.196892 13319 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:07.095892 13446 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.197741 13319 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:07.129647 13502 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:07.057948 13082 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:07.133045  9633 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 9: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:07.186520 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:07.150096 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.197707 13532 consensus_queue.cc:260] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.213390 13526 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:07.214397 13532 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:07.216157 13584 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260808 01:51:07.255403 13625 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 1.208s	user 0.009s	sys 0.000s
I20260808 01:51:07.256605 13504 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:07.256438 13574 raft_consensus.cc:3055] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:07.264792 13581 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:07.265015 13574 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:07.266567 13574 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.270967 13574 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:51:07.081655  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:07.322623  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 327014us
W20260808 01:51:07.323691  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:07.324988  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:07.325542  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:07.080994  9475 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:07.067373  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2420) took 1263 ms. Trace:
I20260808 01:51:07.034060  9477 raft_consensus.cc:2889] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:07.073112  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2198) took 1622 ms. Trace:
I20260808 01:51:07.096446  9478 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:07.335592  9473 rpcz_store.cc:276] 0808 01:51:05.450847 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:05.455427 (+  4580us) service_pool.cc:224] Handling call
0808 01:51:05.455857 (+   430us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:05.456385 (+   528us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:05.456416 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:05.457235 (+   819us) log.cc:844] Serialized 26 byte log entry
0808 01:51:06.235255 (+778020us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:51:06.235299 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:06.235387 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:06.591661 (+356274us) spinlock_profiling.cc:243] Waited 1.76 ms on lock 0x7b4800211158. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:51:07.069729 (+478068us) raft_consensus.cc:1685] finished
0808 01:51:07.069754 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:07.073021 (+  3267us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42978,"raft.queue_time_us":161,"raft.run_cpu_time_us":3659,"raft.run_wall_time_us":1131860,"spinlock_wait_cycles":4050816,"thread_start_us":720617,"threads_started":1,"wal-append.queue_time_us":943602}
I20260808 01:51:07.082767  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2196) took 1639 ms. Trace:
W20260808 01:51:07.132231  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66334.4us
I20260808 01:51:07.334383  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2193) took 2169 ms. Trace:
I20260808 01:51:07.355268  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2424) took 1534 ms. Trace:
I20260808 01:51:06.997319 13594 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:07.361493  9472 rpcz_store.cc:276] 0808 01:51:05.807651 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:06.149992 (+342341us) service_pool.cc:224] Handling call
0808 01:51:06.150450 (+   458us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:06.703138 (+552688us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.341650 (+638512us) spinlock_profiling.cc:243] Waited 547 ms on lock 0x7b540023aad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:07.342123 (+   473us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1258653568}
W20260808 01:51:07.354887 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 159848us
I20260808 01:51:06.701664 13575 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:07.346410  9599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.638s	user 0.020s	sys 0.035s
W20260808 01:51:07.369948  9599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.662s	user 0.021s	sys 0.035s
W20260808 01:51:07.352573  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 150019us
I20260808 01:51:07.163287 13631 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.372460  9601 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:07.187134 13607 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.196463 13627 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:07.327857  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:07.262868 13649 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:07.374431 13570 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260808 01:51:07.348743  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:07.351533  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 491) took 2189 ms. Trace:
I20260808 01:51:07.370007  9558 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:07.344514  9481 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 10. (term mismatch)
I20260808 01:51:07.163661 13652 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:07.391871 13652 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:07.399179  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2560) took 2028 ms. Trace:
I20260808 01:51:07.393987  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2551) took 2630 ms. Trace:
W20260808 01:51:07.345896  9523 leader_election.cc:336] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Timed out: RequestConsensusVote RPC to 127.7.165.196:33455 timed out after 1.638s (SENT)
I20260808 01:51:07.397444 13574 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.004s
I20260808 01:51:07.385881  9481 pending_rounds.cc:85] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 9
I20260808 01:51:07.331688  9474 rpcz_store.cc:276] 0808 01:51:05.803887 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:06.072889 (+269002us) service_pool.cc:224] Handling call
0808 01:51:06.076106 (+  3217us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:06.076387 (+   281us) raft_consensus.cc:1534] Early marking committed up to index 5
0808 01:51:06.097700 (+ 21313us) log.cc:844] Serialized 14 byte log entry
0808 01:51:07.062809 (+965109us) log.cc:844] Serialized 14 byte log entry
0808 01:51:07.065174 (+  2365us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:07.065230 (+    56us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:51:07.067073 (+  1843us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.067157 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:07.067300 (+   143us) inbound_call.cc:177] Queueing success response
0808 01:51:07.283400 (+216100us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
Metrics: {"mutex_wait_us":151734,"prepare.queue_time_us":486222,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":95,"raft.queue_time_us":190893,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":10658,"spinlock_wait_cycles":36235520,"thread_start_us":808288,"threads_started":2,"wal-append.queue_time_us":567312}
I20260808 01:51:07.400753  9558 rpcz_store.cc:276] 0808 01:51:05.367128 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:07.368481 (+2001353us) service_pool.cc:224] Handling call
0808 01:51:07.369390 (+   909us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:07.393680 (+ 24290us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.395899 (+  2219us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:51:07.284220 13656 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.413s	user 0.017s	sys 0.000s
I20260808 01:51:07.381284  9551 rpcz_store.cc:276] 0808 01:51:05.162306 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:07.040950 (+1878644us) service_pool.cc:224] Handling call
0808 01:51:07.041344 (+   394us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:07.056666 (+ 15322us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:07.056707 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:07.057202 (+   495us) log.cc:844] Serialized 26 byte log entry
0808 01:51:07.350287 (+293085us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:07.350468 (+   181us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.350652 (+   184us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:07.350799 (+   147us) raft_consensus.cc:1685] finished
0808 01:51:07.350825 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:07.351110 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14914,"raft.queue_time_us":23092,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":7022,"thread_start_us":265171,"threads_started":1,"wal-append.queue_time_us":179812}
I20260808 01:51:07.414569  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:07.415350  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:07.415735  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 43727.9us
W20260808 01:51:07.417275  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 71838.3us
I20260808 01:51:07.417619  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:07.419189 13607 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.419298 13446 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:07.420722  9557 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:51:07.421588 13649 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:07.421761  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 494) took 2224 ms. Trace:
I20260808 01:51:07.417373 13652 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:07.423364  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 98307.6us
I20260808 01:51:07.423877 13652 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.424973  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:07.429625 13478 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 01:51:07.430527  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:07.431607  9558 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:51:07.431991 13543 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:51:07.434538 13617 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.356s	user 0.011s	sys 0.003s
I20260808 01:51:07.438939  9631 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:07.413656  9481 pending_rounds.cc:107] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 9.10
I20260808 01:51:07.441080  9481 raft_consensus.cc:2889] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:07.359268  9476 rpcz_store.cc:276] 0808 01:51:05.164347 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:05.168473 (+  4126us) service_pool.cc:224] Handling call
0808 01:51:05.168783 (+   310us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:05.191168 (+ 22385us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:51:05.237446 (+ 46278us) log.cc:844] Serialized 14 byte log entry
0808 01:51:05.464380 (+226934us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:05.507271 (+ 42891us) log.cc:844] Serialized 26 byte log entry
0808 01:51:05.507570 (+   299us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:51:05.507621 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:05.507722 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:07.200202 (+1692480us) spinlock_profiling.cc:243] Waited 4.3 ms on lock 0x7b4800211158. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:51:07.246125 (+ 45923us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:07.333556 (+ 87431us) raft_consensus.cc:1685] finished
0808 01:51:07.333683 (+   127us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:07.334097 (+   414us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":46441,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":14448,"raft.run_cpu_time_us":18768,"raft.run_wall_time_us":1717181,"spinlock_wait_cycles":54251264,"thread_start_us":211965,"threads_started":1,"wal-append.queue_time_us":720034}
I20260808 01:51:07.418277 13532 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:07.444036 13631 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.401643 13575 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:07.444715 13532 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.447211  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:07.447575  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:07.449594  9551 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 8: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:07.450779 13594 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:07.450937 13574 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:07.451679 13381 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:07.452234 13574 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:07.452651 13381 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:07.454399  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:07.455207 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.455456  9474 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260808 01:51:07.455766  9472 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260808 01:51:07.456751 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.342633  9480 rpcz_store.cc:276] 0808 01:51:05.443912 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:05.444110 (+   198us) service_pool.cc:224] Handling call
0808 01:51:05.444545 (+   435us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:05.445208 (+   663us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:05.445252 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:05.486052 (+ 40800us) log.cc:844] Serialized 26 byte log entry
0808 01:51:06.029128 (+543076us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:51:06.029196 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:06.029291 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:07.082479 (+1053188us) raft_consensus.cc:1685] finished
0808 01:51:07.082501 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:07.082701 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":296,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":28043,"spinlock_wait_cycles":55680,"thread_start_us":436320,"threads_started":1,"wal-append.queue_time_us":833925}
I20260808 01:51:07.401157  9559 rpcz_store.cc:276] 0808 01:51:04.721462 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:05.373632 (+652170us) service_pool.cc:224] Handling call
0808 01:51:05.373969 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:05.433830 (+ 59861us) raft_consensus.cc:1534] Early marking committed up to index 4
0808 01:51:06.221688 (+787858us) log.cc:844] Serialized 14 byte log entry
0808 01:51:06.475774 (+254086us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:06.494557 (+ 18783us) log.cc:844] Serialized 26 byte log entry
0808 01:51:06.495106 (+   549us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:51:06.513373 (+ 18267us) log.cc:844] Serialized 14 byte log entry
0808 01:51:06.524870 (+ 11497us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:06.525020 (+   150us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:07.351463 (+826443us) raft_consensus.cc:1685] finished
0808 01:51:07.351518 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:07.351873 (+   355us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":72451,"prepare.queue_time_us":910076,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":1715,"raft.queue_time_us":96203,"raft.run_cpu_time_us":489,"raft.run_wall_time_us":24785,"thread_start_us":924524,"threads_started":2,"wal-append.queue_time_us":481033}
I20260808 01:51:07.249138 13669 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.455518  9476 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:07.422344 13246 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:07.430110 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:51:07.460767  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:07.464828  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:07.463893  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:07.422295  9557 rpcz_store.cc:276] 0808 01:51:05.197149 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:07.353198 (+2156049us) service_pool.cc:224] Handling call
0808 01:51:07.409782 (+ 56584us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:07.421351 (+ 11569us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.421655 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:07.447678 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.467907  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 490) took 2305 ms. Trace:
I20260808 01:51:07.477561  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:07.478366  9556 rpcz_store.cc:276] 0808 01:51:05.161920 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:06.623757 (+1461837us) service_pool.cc:224] Handling call
0808 01:51:06.624269 (+   512us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:07.082272 (+458003us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:07.082307 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:07.082356 (+    49us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:07.082386 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.467322 (+384936us) spinlock_profiling.cc:243] Waited 458 ms on lock 0x7b54000ff854. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:07.467424 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:07.467680 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1052789120}
I20260808 01:51:07.481518  9555 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: []
W20260808 01:51:07.418521  9523 leader_election.cc:336] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Timed out: RequestConsensusVote RPC to 127.7.165.196:33455 timed out after 1.824s (SENT)
I20260808 01:51:07.482399  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2010) took 1733 ms. Trace:
I20260808 01:51:07.487984  9550 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:51:07.488281  9555 rpcz_store.cc:276] 0808 01:51:05.748989 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:07.480436 (+1731447us) service_pool.cc:224] Handling call
0808 01:51:07.481047 (+   611us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:07.482052 (+  1005us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.482306 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:07.492053 13492 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:07.493317 13492 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:07.493496  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2147) took 1012 ms. Trace:
I20260808 01:51:07.494120  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:07.495523  9558 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 12: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:07.482604  9556 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:07.497427 12778 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:07.499207 13478 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:07.500068 13478 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.502185 13441 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:07.478434 13649 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:07.503479 13421 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:07.504012 10369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 517) took 1034 ms. Trace:
I20260808 01:51:07.504581 13246 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:07.506047  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 1996) took 4451 ms. Trace:
I20260808 01:51:07.508649 13627 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:51:07.498562 12778 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.510407 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.531791 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:07.532992 10361 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:07.532398 12778 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.465591 13470 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.510860 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.510836  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:07.536142 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.536767  9557 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:07.537855 13381 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:07.511487  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2011) took 1762 ms. Trace:
I20260808 01:51:07.512089 10365 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:07.510412  9472 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:51:07.497577 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:07.514026  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 89370.2us
I20260808 01:51:07.497509  9558 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
W20260808 01:51:07.542168  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:07.490547  9555 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260808 01:51:07.493995  9631 rpcz_store.cc:276] 0808 01:51:06.481277 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:06.481536 (+   259us) service_pool.cc:224] Handling call
0808 01:51:07.492898 (+1011362us) spinlock_profiling.cc:243] Waited 821 ms on lock 0x7b540003bbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:07.493421 (+   523us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1888879872}
I20260808 01:51:07.522182  9560 rpcz_store.cc:276] 0808 01:51:03.054792 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:04.542860 (+1488068us) service_pool.cc:224] Handling call
0808 01:51:04.543246 (+   386us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:06.071039 (+1527793us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.505072 (+1434033us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b54000b4ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:07.505737 (+   665us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3510736128}
W20260808 01:51:07.502236 13617 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260808 01:51:07.525064 13478 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.527001 13421 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.551637 13532 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.520637 10369 rpcz_store.cc:276] 0808 01:51:06.469394 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:06.469564 (+   170us) service_pool.cc:224] Handling call
0808 01:51:07.503388 (+1033824us) spinlock_profiling.cc:243] Waited 51.1 ms on lock 0x7b5400646c54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:07.503678 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":117458304}
I20260808 01:51:07.553953 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:07.556149  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 138966us
I20260808 01:51:07.557724  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 493) took 2368 ms. Trace:
I20260808 01:51:07.563898  9559 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:51:07.564800  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2564) took 1448 ms. Trace:
I20260808 01:51:07.565173  9559 rpcz_store.cc:276] 0808 01:51:06.116087 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:07.544215 (+1428128us) service_pool.cc:224] Handling call
0808 01:51:07.563364 (+ 19149us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:07.564458 (+  1094us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.564730 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:51:07.566787  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 60595.5us
I20260808 01:51:07.572280 13483 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:07.539484  9556 rpcz_store.cc:276] 0808 01:51:05.749341 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:07.481645 (+1732304us) service_pool.cc:224] Handling call
0808 01:51:07.482195 (+   550us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:07.511165 (+ 28970us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.511425 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:07.534211 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.536229 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.539832  9523 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:07.467931 13669 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.539318 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.545759  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2013) took 1794 ms. Trace:
I20260808 01:51:07.581892  9555 rpcz_store.cc:276] 0808 01:51:05.749892 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:07.489418 (+1739526us) service_pool.cc:224] Handling call
0808 01:51:07.490050 (+   632us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:07.544401 (+ 54351us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.544685 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:07.546617 13246 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.544947 10360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 663) took 1026 ms. Trace:
I20260808 01:51:07.548064  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2565) took 1427 ms. Trace:
I20260808 01:51:07.548534  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:07.545420  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2563) took 1431 ms. Trace:
I20260808 01:51:07.585966  9560 rpcz_store.cc:276] 0808 01:51:06.120860 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:07.546663 (+1425803us) service_pool.cc:224] Handling call
0808 01:51:07.547234 (+   571us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:07.547686 (+   452us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:07.547717 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:07.547786 (+    69us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:51:07.547813 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.547883 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:07.548001 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:07.549726 13590 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:07.590426 13319 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.591079 13590 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:07.592236  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 173881us
I20260808 01:51:07.592482  9476 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:07.593289 13483 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:07.594041  9523 leader_election.cc:336] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Timed out: RequestConsensusVote RPC to 127.7.165.196:33455 timed out after 2.003s (SENT)
W20260808 01:51:07.550874  9552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2017) took 1753 ms (client timeout 1507 ms). Trace:
I20260808 01:51:07.585796 10360 rpcz_store.cc:276] 0808 01:51:06.518218 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:06.524968 (+  6750us) service_pool.cc:224] Handling call
0808 01:51:07.497189 (+972221us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:51:07.543919 (+ 46730us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26308864}
W20260808 01:51:07.595435  9552 rpcz_store.cc:269] 0808 01:51:05.788138 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:05.791040 (+  2902us) service_pool.cc:224] Handling call
0808 01:51:07.539127 (+1748087us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54001aca54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:07.541562 (+  2435us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2738239360}
I20260808 01:51:07.598067 13246 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.525154 13574 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:07.583585 10365 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:07.589432 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.591378 13421 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.600792 13319 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.593791 13590 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.602895 13470 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:07.545224 13652 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.603571  9555 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:07.603760  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:07.604355 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:07.606060 13483 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260808 01:51:07.606350  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 58121.3us
I20260808 01:51:07.608879 13607 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:07.611603 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:07.586339  9558 rpcz_store.cc:276] 0808 01:51:06.111112 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:07.496454 (+1385342us) service_pool.cc:224] Handling call
0808 01:51:07.496967 (+   513us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:07.542237 (+ 45270us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.542548 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:07.549886  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 534) took 5357 ms. Trace:
I20260808 01:51:07.538149 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.613495 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.614753  9557 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:07.614743  9523 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:07.616678  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 70986.8us
I20260808 01:51:07.621258 10369 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:51:07.590420 13692 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260808 01:51:07.634156  9481 rpcz_store.cc:276] 0808 01:51:02.192117 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:02.801119 (+609002us) service_pool.cc:224] Handling call
0808 01:51:02.824352 (+ 23233us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:07.451193 (+4626841us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.549510 (+ 98317us) spinlock_profiling.cc:243] Waited 4.52 s on lock 0x7b540038d1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:07.549782 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10391417088}
I20260808 01:51:07.620956 10365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 521) took 1096 ms. Trace:
I20260808 01:51:07.627058  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 496) took 1159 ms. Trace:
I20260808 01:51:07.558382  9554 rpcz_store.cc:276] 0808 01:51:05.189072 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:07.049907 (+1860835us) service_pool.cc:224] Handling call
0808 01:51:07.050386 (+   479us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:07.063748 (+ 13362us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:07.063792 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:07.064326 (+   534us) log.cc:844] Serialized 26 byte log entry
0808 01:51:07.344017 (+279691us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:51:07.406541 (+ 62524us) log.cc:844] Serialized 14 byte log entry
0808 01:51:07.479043 (+ 72502us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.541069 (+ 62026us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:07.556792 (+ 15723us) raft_consensus.cc:1685] finished
0808 01:51:07.556820 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:07.557093 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8020,"prepare.queue_time_us":128232,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":89,"raft.queue_time_us":13650,"raft.run_cpu_time_us":402,"raft.run_wall_time_us":18961,"thread_start_us":331879,"threads_started":2,"wal-append.queue_time_us":179463}
I20260808 01:51:07.629606 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:07.644238  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:07.651513  9552 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260808 01:51:07.644783  9558 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260808 01:51:07.645561  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:07.645686 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.631910 13483 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:07.645707  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:07.633514  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:07.652479  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 58612us
I20260808 01:51:07.658957  9555 rpcz_store.cc:276] 0808 01:51:06.467299 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:07.602860 (+1135561us) service_pool.cc:224] Handling call
0808 01:51:07.603358 (+   498us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:07.626736 (+ 23378us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.626989 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:07.658890 10365 rpcz_store.cc:276] 0808 01:51:06.521349 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:07.429161 (+907812us) service_pool.cc:224] Handling call
0808 01:51:07.617928 (+188767us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:07.650664  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:07.635494  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:07.654984  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:07.668686  9551 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260808 01:51:07.682737  9476 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:51:07.689163  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2012) took 1930 ms. Trace:
I20260808 01:51:07.689740  9551 rpcz_store.cc:276] 0808 01:51:05.749620 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:07.519924 (+1770304us) service_pool.cc:224] Handling call
0808 01:51:07.585598 (+ 65674us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:07.679238 (+ 93640us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.679526 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:07.684502 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.665105 13483 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:07.695683  9472 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:51:07.700847 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:51:07.698792  9371 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50068 (request call id 237) took 42156 ms (client timeout 15000 ms). Trace:
I20260808 01:51:07.652981 13414 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:07.668429 13470 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:07.683787 10362 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:07.708312  9478 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
W20260808 01:51:07.709582  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 121725us
I20260808 01:51:07.731839 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.730978 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.719624  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:07.741150  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:07.741994 10368 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:51:07.742596  9473 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:07.652535 13467 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20260808 01:51:07.750762  9599 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:07.750775 13651 consensus_queue.cc:1059] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:51:07.753062  9481 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:07.727234 13470 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.754746 13434 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:07.755972  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:07.756960 13319 consensus_queue.cc:1059] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:51:07.753917 13584 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:07.754493 10365 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:07.743000  9480 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:51:07.766886 10366 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:51:07.785305 10365 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:07.767499  9472 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:07.768702  9475 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:07.724332 13446 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260808 01:51:07.705376  9371 rpcz_store.cc:269] 0808 01:50:25.542192 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:25.542643 (+   451us) service_pool.cc:224] Handling call
0808 01:51:07.698352 (+42155709us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:50:58.476453 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:50:58.477219 (+   766us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:50:58.477276 (+    57us) write_op.cc:435] Acquired schema lock
0808 01:50:58.477341 (+    65us) tablet.cc:662] Decoding operations
0808 01:50:58.541946 (+ 64605us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:50:58.614669 (+ 72723us) write_op.cc:641] Partition lock acquired for write op
0808 01:50:58.617216 (+  2547us) tablet.cc:685] Acquiring locks for 12 operations
0808 01:50:58.689604 (+ 72388us) tablet.cc:701] Row locks acquired
0808 01:50:58.689651 (+    47us) write_op.cc:260] PREPARE: finished
0808 01:50:58.689872 (+   221us) write_op.cc:270] Start()
0808 01:50:58.690053 (+   181us) write_op.cc:276] Timestamp: P: 1786153858689794 usec, L: 0
0808 01:50:58.690093 (+    40us) op_driver.cc:348] REPLICATION: starting
0808 01:50:58.773396 (+ 83303us) log.cc:844] Serialized 4801 byte log entry
0808 01:50:59.503365 (+729969us) op_driver.cc:464] REPLICATION: finished
0808 01:50:59.702929 (+199564us) write_op.cc:301] APPLY: starting
0808 01:50:59.703042 (+   113us) tablet.cc:1367] starting BulkCheckPresence
0808 01:50:59.704019 (+   977us) tablet.cc:1370] finished BulkCheckPresence
0808 01:50:59.704145 (+   126us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:50:59.763372 (+ 59227us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:50:59.763748 (+   376us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=24
0808 01:50:59.763828 (+    80us) write_op.cc:312] APPLY: finished
0808 01:50:59.764240 (+   412us) log.cc:844] Serialized 89 byte log entry
0808 01:50:59.764460 (+   220us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:59.764617 (+   157us) write_op.cc:454] Released schema lock
0808 01:50:59.764689 (+    72us) write_op.cc:341] FINISH: Updating metrics
0808 01:50:59.765218 (+   529us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:50:59.765266 (+    48us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":59876,"reported_tablets":41,"child_traces":[["op",{"apply.queue_time_us":199159,"apply.run_cpu_time_us":4814,"apply.run_wall_time_us":62251,"num_ops":12,"prepare.queue_time_us":203079,"prepare.run_cpu_time_us":11683,"prepare.run_wall_time_us":414879,"raft.queue_time_us":174840,"raft.run_cpu_time_us":3578,"raft.run_wall_time_us":75678,"replication_time_us":813073,"thread_start_us":256020,"threads_started":4,"wal-append.queue_time_us":193281,"wal-append.run_cpu_time_us":43973,"wal-append.run_wall_time_us":1779014}]]}
I20260808 01:51:07.701687 13607 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.789469 10365 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:07.791494 10365 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:51:07.795892  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 238571us
I20260808 01:51:07.813230  9475 pending_rounds.cc:85] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 3
I20260808 01:51:07.813706 13631 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:51:07.815150 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:07.816288  9472 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260808 01:51:07.795894 13446 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:07.817631 13581 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:07.818054 10365 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260808 01:51:07.818747 10365 pending_rounds.cc:85] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting all ops after (but not including) 7
I20260808 01:51:07.819072 10365 pending_rounds.cc:107] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260808 01:51:07.819360 10365 raft_consensus.cc:2889] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:07.821396 13319 consensus_queue.cc:1243] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260808 01:51:07.822250 13319 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:07.800885 13467 consensus_queue.cc:1243] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260808 01:51:07.823071 13319 raft_consensus.cc:3055] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:07.823691 13319 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:07.818013 13446 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:07.812719 13483 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:07.813726  9475 pending_rounds.cc:107] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260808 01:51:07.832912  9475 raft_consensus.cc:2889] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:07.825186  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:07.833837  9550 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:51:07.837779 13470 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.842759 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.839677  9475 pending_rounds.cc:107] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260808 01:51:07.846014  9475 raft_consensus.cc:2889] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:51:07.847375  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 96809us
I20260808 01:51:07.843930 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:07.854209  9480 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: []
W20260808 01:51:07.858131  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 102469us
I20260808 01:51:07.865306  9481 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 12
I20260808 01:51:07.803802 13434 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.892654 13441 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:51:07.893361 13441 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:07.893774 13441 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:51:07.894517 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:07.896483 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:07.843019 13492 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:07.907516 13693 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.065s	user 0.000s	sys 0.002s
I20260808 01:51:07.887140 13543 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:51:07.924489 13319 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.942767 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:07.952034  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:07.941627  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2570) took 1466 ms. Trace:
I20260808 01:51:07.960909  9560 rpcz_store.cc:276] 0808 01:51:06.475287 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:07.588066 (+1112779us) service_pool.cc:224] Handling call
0808 01:51:07.613192 (+ 25126us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:07.852281 (+239089us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:07.852333 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:07.852435 (+   102us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:51:07.852506 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.941214 (+ 88708us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b5400383954. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:07.941297 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:07.941533 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":537901952}
I20260808 01:51:07.926641 13319 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:07.966284  9477 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
W20260808 01:51:07.963374  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 173244us
I20260808 01:51:07.944468 13434 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.978492 13434 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:07.979393 13434 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:07.941955 13574 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.031s
W20260808 01:51:07.982813  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 130842us
W20260808 01:51:07.983083  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 99453.5us
I20260808 01:51:07.983665 13574 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:51:07.984119  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:07.985320  9600 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:07.986927 13483 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:07.987692  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 78956.3us
I20260808 01:51:07.987993 13483 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:07.988363 13607 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.988648  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:07.989600 13421 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:07.990700 13575 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:07.991298  9599 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:07.991704 13575 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:51:07.992998 13434 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:07.993105  9599 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:07.990499  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:07.994592 13543 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:07.995230 13543 raft_consensus.cc:697] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:07.996105 13492 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:07.996377 13543 consensus_queue.cc:237] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:07.999917 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:07.986325 13574 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:08.009424 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:08.016232 13574 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:51:08.017202 13574 raft_consensus.cc:697] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:08.019838  9481 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:08.020844 13417 raft_consensus.cc:3055] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:08.021382 13417 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:08.021142 13574 consensus_queue.cc:237] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:08.023492 13391 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:08.024159 13391 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:08.025475 13391 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:08.027369 13652 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:51:08.028447 13653 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:08.028995 13653 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:08.029448  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:07.961722 13649 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:08.031286 13649 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:08.032464  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet ed168d59c43d4bfaa07d104032ad8977: Illegal state: Replica 52ccf4dcca1843f7bb29e9172e2d4ab4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:08.036497  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:08.037760  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:08.030445  9560 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:08.039312  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:08.039906 13441 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:08.039366  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:08.046176  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:08.040539  9550 raft_consensus.cc:2410] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 10 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260808 01:51:08.030087 13653 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:08.049500 13653 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:08.053874  9558 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:08.060508 13391 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:08.063879  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2566) took 1939 ms. Trace:
I20260808 01:51:08.066658  9479 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:08.064880 13417 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:08.064472  9559 rpcz_store.cc:276] 0808 01:51:06.124293 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:07.588301 (+1464008us) service_pool.cc:224] Handling call
0808 01:51:07.668616 (+ 80315us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:07.803064 (+134448us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:07.803094 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:07.803168 (+    74us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:51:07.803212 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:08.063513 (+260301us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b5400325fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:08.063580 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:08.063776 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3385,"raft.queue_time_us":51146,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":24307,"spinlock_wait_cycles":291405056}
I20260808 01:51:08.068125 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:08.070137 10366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 573) took 1396 ms. Trace:
I20260808 01:51:08.075610  9477 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
W20260808 01:51:08.071960  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 51832.1us
I20260808 01:51:07.988493 13483 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:08.075591 10366 rpcz_store.cc:276] 0808 01:51:06.674035 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:07.189508 (+515473us) service_pool.cc:224] Handling call
0808 01:51:08.069883 (+880375us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b5400690fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:08.070058 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":381646208}
I20260808 01:51:08.083487 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:07.966284 13651 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.106s
I20260808 01:51:08.034386 13649 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:08.095392 13575 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:08.075196 13441 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:08.099905 13575 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:08.086606 13483 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:08.104816  9560 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:08.101445  9524 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:08.089516 13434 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260808 01:51:08.116528 13417 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:08.156077  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:08.127394 13669 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:08.162396  9524 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:08.163395 13699 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:51:08.163821  9524 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:08.165020  9524 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:08.166232 13697 raft_consensus.cc:2764] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260808 01:51:08.163985 13699 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:08.159584 13627 consensus_meta.cc:287] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.629s	user 0.017s	sys 0.000s
I20260808 01:51:08.168334  9363 maintenance_manager.cc:643] P 3cd8e8730323483db1c5f22e90bb080c: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 11.141s	user 0.219s	sys 0.112s Metrics: {"bytes_written":30274,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":335593,"dirs.run_cpu_time_us":632,"dirs.run_wall_time_us":34572,"drs_written":1,"lbm_read_time_us":587,"lbm_reads_lt_1ms":4,"lbm_write_time_us":302369,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":8,"lbm_writes_lt_1ms":24,"peak_mem_usage":0,"raft.queue_time_us":68845,"raft.run_cpu_time_us":3290,"raft.run_wall_time_us":13347,"reinsert_count":0,"rows_written":63,"thread_start_us":538686,"threads_started":3,"update_count":12,"wal-append.queue_time_us":528914}
I20260808 01:51:08.126909 13414 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.051s
I20260808 01:51:08.172200 13669 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:08.190138 10361 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:51:08.192402  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:08.180262 13699 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:51:08.194469 13434 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:08.195060 13434 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:08.195562 13434 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:08.196096 13446 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:08.190764 13669 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:08.187659 13492 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260808 01:51:08.208415 13319 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260808 01:51:08.236517  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 250754us
I20260808 01:51:08.225832 13446 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:08.246492 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:08.255523 13700 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260808 01:51:08.256278 13460 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260808 01:51:08.264251  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:08.226081 13319 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:08.268467  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:08.264920 13460 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:08.221871 13575 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:08.293581  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 135312us
I20260808 01:51:08.295047  9523 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:08.297356 13446 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260808 01:51:08.299222 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 260637us
W20260808 01:51:08.303664 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:08.304004  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 318730us
I20260808 01:51:08.308234 13708 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 8.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:51:08.297463  9523 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:08.310029 13711 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Stepping down as leader of term 8
W20260808 01:51:08.313757  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 53615.9us
W20260808 01:51:08.319507  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 154587us
I20260808 01:51:08.320888  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:08.325832  9479 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:08.330813 13702 consensus_queue.cc:1059] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
W20260808 01:51:08.331348  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 56821.8us
I20260808 01:51:08.331856 13702 consensus_queue.cc:1243] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 8.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260808 01:51:08.332470 13702 raft_consensus.cc:3055] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:08.332890 13702 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:08.284988 13319 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:08.287199 13460 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:08.310518 13711 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:08.324038 13421 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:08.372165  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:08.373972  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 53342.4us
I20260808 01:51:08.376420  9524 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:08.380312  9600 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:08.383023  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 85637.5us
I20260808 01:51:08.366930 13421 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:08.396704  9364 maintenance_manager.cc:419] P 3cd8e8730323483db1c5f22e90bb080c: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7452 bytes on disk
I20260808 01:51:08.379000 13414 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:08.394935 13599 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:51:08.402269  9559 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 9. Current term is 10. Ops: []
W20260808 01:51:08.397416  9521 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.822s (SENT) [suppressed 43 similar messages]
I20260808 01:51:08.404151 13574 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:08.404531  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:08.404772  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 495) took 1942 ms. Trace:
I20260808 01:51:08.405349 13597 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:08.406247  9554 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260808 01:51:08.406659 13705 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260808 01:51:08.411128 13711 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:08.405396 13718 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:08.425987 13718 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:08.427361 13711 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:08.398973 13421 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:51:08.429219 13695 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.190s	user 0.002s	sys 0.000s
I20260808 01:51:08.404840  9560 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260808 01:51:08.410243  9601 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:08.422151  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:08.423383  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 157322us
I20260808 01:51:08.416200  9524 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:08.426537 13718 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:08.439116  9478 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:08.369131 13702 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:08.447856 13702 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:08.420639  9556 rpcz_store.cc:276] 0808 01:51:06.466467 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:07.594739 (+1128272us) service_pool.cc:224] Handling call
0808 01:51:07.595292 (+   553us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:07.632910 (+ 37618us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:07.632954 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:07.728181 (+ 95227us) log.cc:844] Serialized 26 byte log entry
0808 01:51:07.889973 (+161792us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:51:07.922752 (+ 32779us) log.cc:844] Serialized 14 byte log entry
0808 01:51:07.923062 (+   310us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:07.923179 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:08.404517 (+481338us) raft_consensus.cc:1685] finished
0808 01:51:08.404545 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:08.404705 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18464,"prepare.queue_time_us":33162,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":84,"raft.queue_time_us":90700,"raft.run_cpu_time_us":386,"raft.run_wall_time_us":32637,"thread_start_us":109056,"threads_started":1,"wal-append.queue_time_us":362434}
I20260808 01:51:08.453003 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:08.453992 10363 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:51:08.419813 13543 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:08.423841 13414 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:08.458736 13543 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:08.467173 13391 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 01:51:08.474262  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 70698.1us
W20260808 01:51:08.486088  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:08.498236 13391 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:51:08.498901 13391 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:08.499361 13391 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:08.503365 13421 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:08.506191  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:08.506824 13421 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:08.471515 13543 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:08.508793  9363 maintenance_manager.cc:643] P 3cd8e8730323483db1c5f22e90bb080c: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.094s	user 0.003s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":578,"lbm_reads_lt_1ms":4}
I20260808 01:51:08.509826  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:08.541450  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 439040d168824496acc076b1c280bde3: Illegal state: Replica 52ccf4dcca1843f7bb29e9172e2d4ab4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 11 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:08.520968  9445 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:08.512415  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:08.537696 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:08.581538  9560 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:51:08.578619  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 168500us
W20260808 01:51:08.583774  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 211771us
I20260808 01:51:08.524626 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 672) took 1012 ms. Trace:
W20260808 01:51:08.545627  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:08.586298  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 164228us
W20260808 01:51:08.514151 13710 log.cc:927] Time spent T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260808 01:51:08.585986 10361 rpcz_store.cc:276] 0808 01:51:07.512072 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:07.533841 (+ 21769us) service_pool.cc:224] Handling call
0808 01:51:08.524391 (+990550us) spinlock_profiling.cc:243] Waited 602 ms on lock 0x7b5400594054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:08.524541 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1385464192}
W20260808 01:51:08.593786  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:08.483978 13607 raft_consensus.cc:2764] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260808 01:51:08.599538  9523 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:08.609011  9556 raft_consensus.cc:2393] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:08.492765 13590 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.105s
I20260808 01:51:08.527940 13653 consensus_queue.cc:1059] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.087s
I20260808 01:51:08.510195 13414 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:08.615739 13715 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20260808 01:51:08.616492 13704 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:08.616752  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:08.611294 13701 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.017s
I20260808 01:51:08.604625  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:08.622141  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:08.529336 13581 consensus_queue.cc:1059] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.082s
I20260808 01:51:08.537235 13584 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.077s
I20260808 01:51:08.608261 13599 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260808 01:51:08.637612  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 116734us
I20260808 01:51:08.633360 13417 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:08.634011 13391 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:08.635787  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:08.620092  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:51:08.640311 13651 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:08.641649 13709 raft_consensus.cc:3055] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:08.618841 13652 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:08.651077 13704 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:08.652648  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 43683.6us
W20260808 01:51:08.655752 13722 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260808 01:51:08.658444  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:08.659677 13599 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:08.660161 13421 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:08.621587 13597 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:08.678615 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:08.689859 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:08.683964  9560 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
W20260808 01:51:08.694046  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 58366.8us
I20260808 01:51:08.669095 13709 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:51:08.710772 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:08.683792 13599 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:08.713555  9445 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:08.687541 13421 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:08.706512 13699 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.510s	user 0.008s	sys 0.007s
I20260808 01:51:08.718262  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:08.716763  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 88351.4us
I20260808 01:51:08.694233 13597 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:08.730226 13709 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:08.730643 13391 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:08.731922 13709 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:08.704926 13669 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:08.730822 13724 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:08.732839 13724 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:08.733273 13724 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:08.721446 13421 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:08.725806 13652 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:08.726253 13699 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:08.731551 13704 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:08.734601 13669 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:08.756398 13699 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:08.734917 13651 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:08.758368 13651 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:08.758841 13651 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:08.723672 13574 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:08.761509  9557 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 5. Current term is 6. Ops: []
I20260808 01:51:08.735978 13492 raft_consensus.cc:3055] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:51:08.763541 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:51:08.742108 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:08.751395 13590 raft_consensus.cc:3055] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Stepping down as leader of term 4
W20260808 01:51:08.754869  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:08.765126 13590 raft_consensus.cc:740] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:08.705843 13434 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:08.781236  9560 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 7. Current term is 8. Ops: []
W20260808 01:51:08.742945 13483 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.611s	user 0.011s	sys 0.005s
I20260808 01:51:08.763546 13492 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:08.788426 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:08.794041  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 58075.4us
I20260808 01:51:08.740135 13652 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:08.804268 13703 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:08.804628 13652 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:51:08.784489 13729 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:51:08.813490 13434 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:08.784206 13590 consensus_queue.cc:260] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:08.816946 13590 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:51:08.816704 13492 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:08.817950  9601 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:08.818557  9550 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:51:08.822808 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:08.833205 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:08.755256 13725 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:08.837968 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:08.794363 13483 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:08.850999  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:08.818007 13492 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:51:08.856668  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 127492us
W20260808 01:51:08.861021  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 108751us
I20260808 01:51:08.818825 13467 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260808 01:51:08.850438 13434 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:08.866372  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 149850us
I20260808 01:51:08.817305 13543 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:08.829427 13708 consensus_queue.cc:1059] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
W20260808 01:51:08.869010  9377 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55058 (request call id 127) took 11752 ms (client timeout 15000 ms). Trace:
W20260808 01:51:08.875543  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 57716.2us
I20260808 01:51:08.877503 13704 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:08.880847 13434 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:08.883315 13651 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:08.886103 13651 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:08.861660 13319 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:08.888530 13718 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:08.896431 13725 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:08.897116 13725 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:08.904081  9372 catalog_manager.cc:5697] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 2 to 8, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 8 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:51:08.872025  9377 rpcz_store.cc:269] 0808 01:50:57.081044 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:57.085325 (+  4281us) service_pool.cc:224] Handling call
0808 01:51:08.819804 (+11734479us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:51:06.431066 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:51:06.589553 (+158487us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:51:06.589616 (+    63us) write_op.cc:435] Acquired schema lock
0808 01:51:06.589645 (+    29us) tablet.cc:662] Decoding operations
0808 01:51:07.084811 (+495166us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:51:07.114208 (+ 29397us) write_op.cc:641] Partition lock acquired for write op
0808 01:51:07.115290 (+  1082us) tablet.cc:685] Acquiring locks for 13 operations
0808 01:51:07.184216 (+ 68926us) tablet.cc:701] Row locks acquired
0808 01:51:07.184253 (+    37us) write_op.cc:260] PREPARE: finished
0808 01:51:07.184514 (+   261us) write_op.cc:270] Start()
0808 01:51:07.184687 (+   173us) write_op.cc:276] Timestamp: P: 1786153867184446 usec, L: 0
0808 01:51:07.184728 (+    41us) op_driver.cc:348] REPLICATION: starting
0808 01:51:07.185900 (+  1172us) log.cc:844] Serialized 5129 byte log entry
0808 01:51:07.266390 (+ 80490us) op_driver.cc:464] REPLICATION: finished
0808 01:51:07.272246 (+  5856us) write_op.cc:301] APPLY: starting
0808 01:51:07.272319 (+    73us) tablet.cc:1367] starting BulkCheckPresence
0808 01:51:07.273035 (+   716us) tablet.cc:1370] finished BulkCheckPresence
0808 01:51:07.273125 (+    90us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:51:07.573229 (+300104us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:51:07.621436 (+ 48207us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=13,key_file_lookups=13,delta_file_lookups=13,mrs_lookups=26
0808 01:51:07.621536 (+   100us) write_op.cc:312] APPLY: finished
0808 01:51:07.696857 (+ 75321us) log.cc:844] Serialized 176 byte log entry
0808 01:51:07.699896 (+  3039us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:51:07.784183 (+ 84287us) write_op.cc:454] Released schema lock
0808 01:51:07.805439 (+ 21256us) write_op.cc:341] FINISH: Updating metrics
0808 01:51:07.895331 (+ 89892us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:51:07.895367 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5336017,"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":5324,"apply.run_cpu_time_us":27401,"apply.run_wall_time_us":622945,"cfile_cache_hit":24,"cfile_cache_hit_bytes":25116,"cfile_cache_miss":4,"cfile_cache_miss_bytes":6220,"cfile_init":2,"lbm_read_time_us":1393,"lbm_reads_lt_1ms":12,"num_ops":13,"prepare.queue_time_us":1005674,"prepare.run_cpu_time_us":10055,"prepare.run_wall_time_us":791005,"replication_time_us":73546,"thread_start_us":781796,"threads_started":2}]]}
W20260808 01:51:08.907812  9521 leader_election.cc:336] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.551s (SENT)
I20260808 01:51:08.908514  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:08.908823  9474 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:08.909795 13699 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:51:08.861824 13703 raft_consensus.cc:697] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:08.911494 13703 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:08.914809  9521 leader_election.cc:336] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.563s (SENT)
W20260808 01:51:08.915390  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 45266.6us
I20260808 01:51:08.915658  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:08.915597 13653 raft_consensus.cc:3055] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:51:08.917132 13653 raft_consensus.cc:740] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:08.898391 13725 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:08.891098 13704 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:08.918622 13725 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:08.892275 13391 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:08.919217 13704 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:08.919967 13391 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:08.874426 13575 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:08.892638  9600 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:08.921399 13575 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:08.922186  9600 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:08.891669 13483 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:08.892812 13701 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:08.925236 13701 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:08.889222 13319 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:08.918207 13708 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:51:08.906354  9372 catalog_manager.cc:5697] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 3 to 4, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 4 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:08.932312 13699 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:08.932727 13701 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:08.932399 13543 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:08.936340  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 57656.9us
W20260808 01:51:08.936640  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 65971.7us
I20260808 01:51:08.933792 13434 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:08.935353  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 67502.2us
I20260808 01:51:08.945726  9599 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:08.948173 13391 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:08.950167 13575 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:08.943388 13574 raft_consensus.cc:2764] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260808 01:51:08.934715 13460 consensus_meta.cc:287] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.566s	user 0.014s	sys 0.000s
W20260808 01:51:08.963326  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 53368.5us
I20260808 01:51:08.964408  9600 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:08.943049 13651 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:51:08.966849 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:08.935084 13718 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:08.943701 13708 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:08.940529 13724 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:08.978896 10364 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:08.980366  9524 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:08.987489  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:08.991004  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:09.007576 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:08.918231 13653 consensus_queue.cc:260] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:09.015851 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:09.018278  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:09.021009 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:09.023170 10365 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
W20260808 01:51:09.026407  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46111.4us
I20260808 01:51:09.026476  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:08.975656 13708 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:09.026993 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:08.954280 13597 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:09.030587  9478 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260808 01:51:09.031481  9474 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:09.029580 13729 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260808 01:51:09.033959 13733 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260808 01:51:09.034011  9481 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:51:09.033687  9372 catalog_manager.cc:5697] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 1 to 6, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 6 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:08.966313 13460 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:09.056862 13421 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:09.056766 13708 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:09.041630  9477 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:09.043380 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:08.980113 13391 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.050s
W20260808 01:51:09.040717  9521 leader_election.cc:336] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.694s (SENT)
I20260808 01:51:09.072297  9521 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:09.049602  9481 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:09.058831 13703 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:51:09.058426 13460 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:09.042078 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:09.090413 10366 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:09.069571 10368 raft_consensus.cc:2410] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260808 01:51:09.092862  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:09.070132  9479 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:09.057165  9372 catalog_manager.cc:5697] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:51:09.076121 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:09.076485  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 130816us
W20260808 01:51:09.082753  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 64257.3us
W20260808 01:51:09.083544  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:09.040403 13704 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:09.042964 13697 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:09.096891  9372 catalog_manager.cc:5697] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 5 to 6, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 6 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:51:09.074355 13391 consensus_queue.cc:1243] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 10.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260808 01:51:09.075290 13417 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 01:51:09.104056 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 66942.9us
I20260808 01:51:09.107403  9550 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:51:09.111110  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:09.091462 13653 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:51:09.113333 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:09.092123 13724 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:09.127465 10332 leader_election.cc:336] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.756s (SENT)
I20260808 01:51:09.130542 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:09.090730 13434 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:09.105283 13703 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:09.135131  9372 catalog_manager.cc:5697] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 5 to 8, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 8 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:09.129982 13708 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:09.166050  9560 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260808 01:51:09.160714  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 120047us
W20260808 01:51:09.168218  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 215433us
I20260808 01:51:09.170732  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:09.135710 13697 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:09.172581  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:09.174192 10366 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:09.149327 13421 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:09.194617 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:09.187475  9474 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:51:09.240304  9521 leader_election.cc:336] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.856s (SENT)
I20260808 01:51:09.240880  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:09.241871  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:09.246294 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:09.247109  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:09.249537  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:09.247476  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 78525.3us
I20260808 01:51:09.252516  9554 raft_consensus.cc:2393] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:51:09.254742 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 127325us
I20260808 01:51:09.255764  9372 catalog_manager.cc:5697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:51:09.230593 13753 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:09.272809  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 72586.6us
W20260808 01:51:09.259013 13734 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.319s	user 0.000s	sys 0.002s
I20260808 01:51:09.268018  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:09.279554  9474 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:09.281299  9480 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260808 01:51:09.281840  9480 pending_rounds.cc:85] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 4
I20260808 01:51:09.287514  9372 catalog_manager.cc:5697] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:09.265849 13699 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:09.288427  9480 pending_rounds.cc:107] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260808 01:51:09.299401  9480 raft_consensus.cc:2889] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:09.275723 13754 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:09.295138 13743 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.204s	user 0.001s	sys 0.000s
I20260808 01:51:09.295487  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:09.295982 13745 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260808 01:51:09.296584  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:09.313217  9557 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:51:09.296823  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:09.297385  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:09.315026  9555 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:09.297634  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:09.302327 13599 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.556s	user 0.014s	sys 0.000s
I20260808 01:51:09.318527  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:09.319007 13599 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:09.319499  9552 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:09.320890 13754 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:09.322278 13599 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:09.333928  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:09.334731  9551 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:09.309010 13597 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:09.336067  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:09.310596  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:09.336876  9631 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:51:09.309087  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:09.323050  9554 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 9: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:09.335855  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:09.303057  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 3759209338e14437aff5f92a2fff36da: Illegal state: Replica 52ccf4dcca1843f7bb29e9172e2d4ab4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:09.324182  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:09.330773  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:09.331003  9629 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:09.331890  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:09.329766  9638 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:09.323414 13754 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:09.339753 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:09.363808 10366 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:09.364076  9474 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
W20260808 01:51:09.349032  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 47771.8us
I20260808 01:51:09.365504 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:09.349431 10363 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:09.337098  9633 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
W20260808 01:51:09.368664  9521 leader_election.cc:336] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.932s (SENT)
W20260808 01:51:09.369879  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:09.372865  9636 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:09.375023  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:09.376504  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:09.378270  9631 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 8: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:09.378890  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:09.379177  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:09.379967  9552 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 10: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:09.375676  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:09.382452 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:09.382606  9599 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:09.375489  9521 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:09.384248  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:09.379611  9555 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:09.389829  9479 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:09.395044  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:09.395809  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:09.394443  9633 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:51:09.394059  9372 catalog_manager.cc:5697] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 4 to 5, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 5 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:51:09.372165 13597 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:09.406973  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:09.408396  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:09.429296  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:09.483919  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:09.288539 13703 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:09.368404 13492 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260808 01:51:09.405228 13704 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:09.518299  9372 catalog_manager.cc:5697] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 4 to 7, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 7 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:51:09.524176  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:09.525763  9636 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260808 01:51:09.405369 13599 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:51:09.528100  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 153122us
I20260808 01:51:09.547669  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:09.548555  9558 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:09.367965 13391 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260808 01:51:09.532524  9560 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 7. Current term is 8. Ops: []
W20260808 01:51:09.515012 13714 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260808 01:51:09.550926  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:09.536557  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:09.554157  9554 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 9: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:09.555665  9372 catalog_manager.cc:5697] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 5 to 7, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 7 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:09.554388  9474 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:09.515034  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:09.532464  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 94874.1us
W20260808 01:51:09.549573 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:09.557415  9372 catalog_manager.cc:5697] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 3 to 6, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 6 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:51:09.554076  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:09.305581 13697 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:09.559684 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:09.559638  9560 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:09.561276  9555 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260808 01:51:09.560860  9372 catalog_manager.cc:5697] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 2 to 6, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 6 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:09.516336 13597 raft_consensus.cc:697] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:09.563228  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:09.568185  9552 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260808 01:51:09.525542 13651 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:09.569839  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:09.573566  9638 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260808 01:51:09.574024  9480 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:09.536263 13704 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:09.575685  9629 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:09.549593 13747 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:09.579310  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 210584us
W20260808 01:51:09.582441 13760 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
W20260808 01:51:09.616503  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 50306.2us
I20260808 01:51:09.583245  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:09.580413 13651 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:09.599148  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:09.599238 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:09.574985 13752 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:09.613032  9632 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 7. Current term is 8. Ops: []
I20260808 01:51:09.614459  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:09.652745  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:09.653169  9372 catalog_manager.cc:5697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 5 to 6, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 6 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:51:09.654179  9445 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:09.663939  9638 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:09.663983  9637 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:09.665154  9372 catalog_manager.cc:5697] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 8 to 9, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 9 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:51:09.669041  9372 catalog_manager.cc:5697] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:51:09.617834  9475 raft_consensus.cc:2376] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7. Re-sending same reply.
I20260808 01:51:09.580134 13704 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:09.653311 13708 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:09.658907 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:09.676532 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:09.678157  9372 catalog_manager.cc:5697] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 2 to 4, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 4 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:09.682024  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:09.660326  9476 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:51:09.653813 13714 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260808 01:51:09.658625  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:09.652626  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:09.655055 13599 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:09.697278 13421 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election won for term 9
W20260808 01:51:09.728111 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:09.664636 13752 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:09.743384  9474 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:09.762915 13752 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:51:09.765962 13725 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.846s	user 0.005s	sys 0.009s
I20260808 01:51:09.746099  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:09.767577 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 216508us
I20260808 01:51:09.767747  9480 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:09.747555  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:09.743898  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:09.621992 13747 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:09.749200 13597 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:09.776852 13747 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:09.759598 13765 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260808 01:51:09.700073 13708 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:09.783396  9637 raft_consensus.cc:2410] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260808 01:51:09.790632  9632 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:09.795905  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:09.809168  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:09.829734  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:09.830186  9600 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:09.795822 13421 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:09.834733  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:09.839526  9637 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:51:09.840545  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 216878us
I20260808 01:51:09.841912  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:09.843591  9638 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:09.861320 10363 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: []
I20260808 01:51:09.862241  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:09.839068 13754 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 01:51:09.867324  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:09.875644  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:09.878794  9632 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:09.843923 13751 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:09.848237 13709 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:09.883078 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:09.883960  9629 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:09.895452 13725 consensus_queue.cc:1059] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:09.896792 13725 raft_consensus.cc:3055] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:51:09.897205 13725 raft_consensus.cc:740] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:09.897449 13708 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:09.898181 13725 consensus_queue.cc:260] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:09.900826 13751 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:09.863103 13417 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:09.905221  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:09.909413  9476 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
W20260808 01:51:09.916079  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:09.927280  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:09.916893 13709 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:09.920451  9444 connection.cc:256] RPC call timeout handler was delayed by -2.9688452660047915s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3244s for a response
I20260808 01:51:09.921725 10363 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260808 01:51:09.933667 10363 pending_rounds.cc:85] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting all ops after (but not including) 8
I20260808 01:51:09.934027 10363 pending_rounds.cc:107] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260808 01:51:09.934297 10363 raft_consensus.cc:2889] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:51:09.924682 13759 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.398s	user 0.004s	sys 0.002s
I20260808 01:51:09.930706 13417 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:09.900431 13651 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:09.952426  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:09.954238 13779 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:51:09.955891 13729 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:09.956444 13729 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:51:09.956712 13763 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.170s	user 0.001s	sys 0.000s
I20260808 01:51:09.956804 13729 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:09.958668  9637 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
W20260808 01:51:09.959229 13775 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.066s	user 0.002s	sys 0.000s
W20260808 01:51:09.969415 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:09.972838  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:09.974143  9473 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 10. Current term is 11. Ops: []
W20260808 01:51:09.976138  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:09.972838  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:09.988090  9478 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:51:09.991025  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:09.991864  9472 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:09.993911 13711 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:51:09.968473 13725 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:09.999189  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:09.970134 13779 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:10.001734 13653 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:10.008073 10334 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:10.009414 13711 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:10.009824 13711 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:10.008178 13653 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:10.010267 13711 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:10.011377 13751 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:10.009944 10334 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:10.014394 13653 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:10.015350 13627 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:10.004779  9632 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:10.010171  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.016832  9634 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:51:10.020015 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:09.999142 13421 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:10.021729 13709 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:10.023046  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:10.023418  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.023888  9481 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:51:10.024289  9636 raft_consensus.cc:2410] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260808 01:51:10.025297  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.026055 10334 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:10.028012  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.033390 13581 raft_consensus.cc:3055] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:10.033912 13581 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:10.034015  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.015864 13627 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:10.035274 13627 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:10.035516  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.039149 10367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 686) took 1020 ms. Trace:
I20260808 01:51:10.039650 10367 rpcz_store.cc:276] 0808 01:51:09.018948 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:09.019852 (+   904us) service_pool.cc:224] Handling call
0808 01:51:09.035164 (+ 15312us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:09.048903 (+ 13739us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:09.048935 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:09.073894 (+ 24959us) log.cc:844] Serialized 52 byte log entry
0808 01:51:09.266828 (+192934us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:09.266877 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:09.266973 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:09.923701 (+656728us) spinlock_profiling.cc:243] Waited 6.21 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:10.016641 (+ 92940us) raft_consensus.cc:1685] finished
0808 01:51:10.016664 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:10.039060 (+ 22396us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8168,"raft.queue_time_us":13662,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":7991,"spinlock_wait_cycles":14283520,"thread_start_us":160321,"threads_started":1,"wal-append.queue_time_us":287433}
I20260808 01:51:10.041014 13779 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:10.041303 13581 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:10.043581 13779 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:10.049743  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:10.052947  9521 leader_election.cc:336] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.011s (SENT)
W20260808 01:51:10.051856  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 453980us
I20260808 01:51:10.052879 13631 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.009s
I20260808 01:51:10.062290  9524 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:10.063570 13631 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:51:10.064097 13631 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:10.064782 13631 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:10.065407 13391 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260808 01:51:10.067812  9521 leader_election.cc:336] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.072s (SENT)
I20260808 01:51:10.068825 13709 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:51:10.069350 13709 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:10.070135 13709 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:51:10.072538 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:10.035760 13417 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:10.026476  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.064013  9524 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:10.059852  9558 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260808 01:51:10.069314  9521 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:10.079365  9636 raft_consensus.cc:2410] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260808 01:51:10.081255  9629 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:10.082342  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.088341  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:10.089649 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:10.092406  9637 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:51:10.096966  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:10.098353  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.099210  9629 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:10.102334  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.052613 13762 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:10.088126 13417 consensus_queue.cc:1059] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260808 01:51:10.107466  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.092744 13724 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:10.109503 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:10.083783 13574 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:10.111982  9557 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:10.123250  9632 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:51:10.124272  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.129143  9629 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:10.129266  9550 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
W20260808 01:51:10.119088 13753 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.828s	user 0.004s	sys 0.010s
I20260808 01:51:10.141870 13574 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:10.142948 13753 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:10.131873  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:10.134583  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.127282 13599 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.013s
I20260808 01:51:10.136307  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.148000  9629 raft_consensus.cc:2410] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20260808 01:51:10.148568 13761 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.399s	user 0.003s	sys 0.002s
I20260808 01:51:10.148994  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.124071 13762 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:10.149817  9629 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
W20260808 01:51:10.136907  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 69198us
I20260808 01:51:10.125286 13701 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260808 01:51:10.110453 13434 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:10.133864 13724 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:10.141010 13669 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:10.165262  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:10.142494 13778 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.249s	user 0.001s	sys 0.001s
I20260808 01:51:10.144932  9524 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:10.145833  9636 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:10.087766 13718 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.044s
W20260808 01:51:10.152788  9521 leader_election.cc:336] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.111s (SENT)
W20260808 01:51:10.160744 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:10.174000  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.151611 13460 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260808 01:51:10.195343  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 243028us
I20260808 01:51:10.197386  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.185030  9634 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:10.213085  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.214200  9634 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:10.126894 13752 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:10.218035  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.219251  9636 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:51:10.220609  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:10.224187  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:10.232168  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 79427us
W20260808 01:51:10.234666  9521 leader_election.cc:336] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.114s (SENT)
I20260808 01:51:10.235215  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:10.240978  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.253471  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.256273 13699 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:10.260916 13762 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:10.264269 13704 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.522s	user 0.011s	sys 0.001s
I20260808 01:51:10.264953 13704 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:10.229489 13460 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:51:10.269098 13711 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:10.307277 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:10.269929 13574 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:10.239617 13717 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.653s	user 0.005s	sys 0.011s
I20260808 01:51:10.286427 13699 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:10.271852 13762 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 01:51:10.271894 13724 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:10.273137 13729 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:10.190096 13669 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:10.283680  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.287675  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.288626  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.318125  9633 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:10.318414  9629 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:10.289881  9638 raft_consensus.cc:2368] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9.
I20260808 01:51:10.240625  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:10.230146 13779 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:10.312495  9635 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:10.322191  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.292063 13421 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:10.323585  9630 raft_consensus.cc:2368] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 11. Current term is 12.
I20260808 01:51:10.313755  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:10.313289 13717 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.006s	sys 0.012s
I20260808 01:51:10.282320 13627 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:10.332161  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.310124 13708 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:10.314714 13699 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:10.338054  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.337193  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.313908 13460 raft_consensus.cc:697] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:51:10.341974  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 197116us
I20260808 01:51:10.342222  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.341112  9632 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:10.363600  9524 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:10.365825  9524 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:10.370270  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.371640  9634 raft_consensus.cc:2376] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8. Re-sending same reply.
W20260808 01:51:10.372918 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:10.378942  9629 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:51:10.381415  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.381734  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2157) took 1002 ms. Trace:
I20260808 01:51:10.382141  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.386132  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.386919  9633 raft_consensus.cc:2410] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260808 01:51:10.382604  9632 raft_consensus.cc:2410] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260808 01:51:10.336539 13724 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:51:10.388576  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:10.392431  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 160443us
I20260808 01:51:10.398993  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:10.396775  9634 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260808 01:51:10.417616  9629 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260808 01:51:10.395723  9524 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:10.424314  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:10.382151  9631 rpcz_store.cc:276] 0808 01:51:09.379125 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:09.389064 (+  9939us) service_pool.cc:224] Handling call
0808 01:51:09.402326 (+ 13262us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:09.402617 (+   291us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:51:09.514619 (+112002us) log.cc:844] Serialized 14 byte log entry
0808 01:51:10.043006 (+528387us) spinlock_profiling.cc:243] Waited 9.08 ms on lock 0x7b1800346d20. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d4453242a 00007f5d4499de0a
0808 01:51:10.339583 (+296577us) log.cc:844] Serialized 14 byte log entry
0808 01:51:10.343730 (+  4147us) log.cc:844] Serialized 14 byte log entry
0808 01:51:10.343946 (+   216us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:10.344006 (+    60us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:10.344071 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:10.344160 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:10.381662 (+ 37502us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17582,"prepare.queue_time_us":109284,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":47,"raft.queue_time_us":20175,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":24104,"spinlock_wait_cycles":162761984,"thread_start_us":281827,"threads_started":1}
I20260808 01:51:10.384245 13434 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:10.444329  9637 raft_consensus.cc:2368] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9.
I20260808 01:51:10.446247  9633 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260808 01:51:10.384053 13762 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:10.326084 13779 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:10.456569  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:10.457386  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 335145us
W20260808 01:51:10.395367 13759 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260808 01:51:10.454128  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.454303  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.335950 13421 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:10.446892  9633 pending_rounds.cc:85] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 4
I20260808 01:51:10.470094  9633 pending_rounds.cc:107] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260808 01:51:10.470799  9633 raft_consensus.cc:2889] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:10.459167  9635 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260808 01:51:10.445083 13753 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:10.485160  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.486387  9632 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:10.463826  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:10.347836 13708 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:10.455649 13725 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:51:10.459787 13434 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:10.490896  9631 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260808 01:51:10.492198 13717 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:51:10.493180 13708 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:10.489127 13753 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:10.499217  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.506273  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.500393  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.499204  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:10.501761  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.507951  9476 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:51:10.512828  9635 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260808 01:51:10.513178  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 444) took 1177 ms. Trace:
I20260808 01:51:10.508126  9638 raft_consensus.cc:2376] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9. Re-sending same reply.
W20260808 01:51:10.451002  9521 leader_election.cc:336] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.143s (SENT)
I20260808 01:51:10.518639  9521 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:10.520404  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 56699.5us
I20260808 01:51:10.521987  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.532138  9481 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260808 01:51:10.532401  9632 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:10.522665  9636 raft_consensus.cc:2368] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9.
I20260808 01:51:10.523483  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.513805  9630 rpcz_store.cc:276] 0808 01:51:09.335766 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:09.533544 (+197778us) service_pool.cc:224] Handling call
0808 01:51:10.502249 (+968705us) spinlock_profiling.cc:243] Waited 615 ms on lock 0x7b54003fa554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:10.513077 (+ 10828us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1414478592}
I20260808 01:51:10.532617 10334 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:10.536490  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.537434  9637 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:10.539870  9637 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: []
W20260808 01:51:10.546851  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:10.538003  9631 raft_consensus.cc:2376] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10. Re-sending same reply.
I20260808 01:51:10.613133  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.612260  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:10.565426 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:10.615831  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.556653  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:10.617067  9638 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:10.617633  9472 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:51:10.569262  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:10.575572  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.620864  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.622961  9637 raft_consensus.cc:2376] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7. Re-sending same reply.
I20260808 01:51:10.527194 13752 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:10.635037  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.527515 13717 raft_consensus.cc:697] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:51:10.636023  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 104127us
I20260808 01:51:10.639657  9632 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260808 01:51:10.555594 13725 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260808 01:51:10.620349 13631 consensus_meta.cc:287] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.546s	user 0.012s	sys 0.001s
W20260808 01:51:10.645293 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:10.555971 13581 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:10.659850  9635 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260808 01:51:10.655768  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.666096  9637 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:10.661427  9630 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260808 01:51:10.667279  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:10.673825 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:10.661695  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 225481us
I20260808 01:51:10.634406 13724 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:10.681324  9552 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260808 01:51:10.683703  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:10.687096  9559 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:10.681525  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:10.688450 13788 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.161s	user 0.000s	sys 0.001s
W20260808 01:51:10.679852 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50716.4us
W20260808 01:51:10.720731 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:10.688572  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.644863 13704 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:10.692055  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:10.723995  9556 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:10.655472 13597 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:51:10.655966 13725 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:10.660652 13762 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:10.727409  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.523001 13460 consensus_queue.cc:237] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:10.686744 13724 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:10.665820 13631 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:10.685245 13669 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:10.724468  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 44958.4us
I20260808 01:51:10.712566 13653 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:51:10.724929  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 273963us
I20260808 01:51:10.681072 13711 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:10.654068 13703 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.058s
I20260808 01:51:10.689721 13729 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:10.708882 13708 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:10.769078 13708 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:10.769960 13709 consensus_meta.cc:287] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.699s	user 0.013s	sys 0.000s
I20260808 01:51:10.719436 13807 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
I20260808 01:51:10.771984 13817 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:10.772821 13799 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260808 01:51:10.772642 13817 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:10.773665 13729 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:10.740792 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:10.721550  9636 raft_consensus.cc:2376] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9. Re-sending same reply.
I20260808 01:51:10.726383 13798 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:10.757519  9632 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260808 01:51:10.726130 13597 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:10.730824 13725 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:51:10.761047  9521 connection.cc:256] RPC call timeout handler was delayed by -1.1370546900143381s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2231s for a response
I20260808 01:51:10.708959 13434 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:10.767765 13796 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
I20260808 01:51:10.784055  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:10.804975  9481 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:10.751477 13724 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:51:10.784427  9521 connection.cc:256] RPC call timeout handler was delayed by -1.1170798480161466s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1666s for a response
I20260808 01:51:10.775017  9558 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260808 01:51:10.809942  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:10.793646 13786 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.319s	user 0.000s	sys 0.002s
I20260808 01:51:10.777241 13597 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:10.799052  9556 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:10.775632 13631 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:10.801071 13599 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:10.803109  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.786716  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.745491 13793 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:10.820465  9633 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:51:10.809150  9521 connection.cc:256] RPC call timeout handler was delayed by -1.0442691839998588s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16510000000000002s for a response
I20260808 01:51:10.816309  9632 raft_consensus.cc:2410] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260808 01:51:10.753057 13811 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:10.763733 13747 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:10.793646 13798 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:10.770489 13709 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:10.815778 13599 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:10.878815  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:10.833987 13421 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:10.890951 13651 raft_consensus.cc:3055] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:51:10.891335 13797 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:10.891754 13651 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:10.892046 13797 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:10.867694  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 46120.4us
I20260808 01:51:10.879685  9556 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
W20260808 01:51:10.866972 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:10.764523 13653 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:10.881884 13806 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
W20260808 01:51:10.882431 13795 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260808 01:51:10.882513 13804 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:10.894026  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 437549us
I20260808 01:51:10.928730  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:10.907727  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:10.887897 13754 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:10.929481 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:10.883015 13736 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:10.929551  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:10.926467  9481 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:51:10.929503  9636 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:10.930724  9632 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:10.934077  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:10.774366 13729 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:10.935716  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:10.935125  9521 leader_election.cc:336] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.182s (SENT)
I20260808 01:51:10.935813 10334 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:10.943333  9559 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:10.947973  9557 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:51:10.945644 13736 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:10.956828  9473 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:51:10.951833 13421 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:10.794955 13717 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:10.965862  9559 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
W20260808 01:51:11.030906  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 75455.9us
I20260808 01:51:11.032130  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:11.033908 13810 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
I20260808 01:51:10.893199 13651 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:10.847415 13747 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:10.959406 13599 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:10.939625 13754 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:10.938493 13762 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:10.908593 13798 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:10.875546 13811 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:11.076187  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:11.037809  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:11.354053  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 433425us
I20260808 01:51:11.348193  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:11.349165 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:11.114480  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:11.347636  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:11.036842 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:11.359170 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:11.359937 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:11.144516  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:11.355770  9521 leader_election.cc:336] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.190s (SENT)
W20260808 01:51:11.365283 13724 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.532s	user 0.009s	sys 0.000s
I20260808 01:51:11.365778 13724 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:11.355132  9633 raft_consensus.cc:2376] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7. Re-sending same reply.
W20260808 01:51:11.357424 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:11.372807  9634 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260808 01:51:11.376125 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:11.377610 10367 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:11.378561  9634 pending_rounds.cc:85] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 4
I20260808 01:51:11.379283  9634 pending_rounds.cc:107] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260808 01:51:11.385056 13627 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:11.383282  9634 raft_consensus.cc:2889] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:10.960362 13708 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:11.328351 13818 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:11.443234  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:11.443234  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:11.067595 13421 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:11.069177 13653 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:11.034308 13709 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:11.452716  9476 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:51:11.453080  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 97341.2us
W20260808 01:51:11.460067  9638 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.823s	user 0.010s	sys 0.003s
I20260808 01:51:11.478253  9638 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:11.355885 13729 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:11.035446 13797 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:11.481395  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:11.473646  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:11.483937  9633 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:51:11.478420  9521 leader_election.cc:336] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.215s (SENT)
W20260808 01:51:11.459993  9632 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.525s	user 0.014s	sys 0.000s
I20260808 01:51:11.485510  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:11.485718  9632 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
W20260808 01:51:11.488556  9521 leader_election.cc:336] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.231s (SENT)
I20260808 01:51:11.489288 13434 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260808 01:51:11.492832  9521 leader_election.cc:336] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.363s (SENT)
I20260808 01:51:11.493872 13813 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:11.382295 13651 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:11.504544 10361 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260808 01:51:10.959793 13804 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:11.381486 13724 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:11.496387  9521 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:11.498471  9472 consensus_meta.cc:287] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.875s	user 0.011s	sys 0.004s
I20260808 01:51:11.520334  9472 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
W20260808 01:51:11.527926  9551 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.034s	sys 0.055s
W20260808 01:51:11.528614  9551 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.035s	sys 0.056s
I20260808 01:51:11.444480 13823 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:11.475618 13599 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:11.472774 13421 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:11.536772 13421 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:11.538439 13752 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:51:11.539745 13827 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:11.540405 13827 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:11.543931  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:11.544106 10334 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:11.512987 13729 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election won for term 10
W20260808 01:51:11.560056 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:11.551589 13827 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:11.577564  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 84950.9us
W20260808 01:51:11.579373  9521 leader_election.cc:336] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.521s (SENT)
W20260808 01:51:11.580629  9521 leader_election.cc:336] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.666s (SENT)
W20260808 01:51:11.581568  9521 leader_election.cc:336] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.750s (SENT)
I20260808 01:51:11.582288 13421 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:11.530578 13818 raft_consensus.cc:697] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:51:11.582410  9521 leader_election.cc:336] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.822s (SENT)
I20260808 01:51:11.546017 13797 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:11.623512 13827 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:11.559422 13807 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:11.632160 13827 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:11.636284 13421 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:11.636650  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:11.640194  9476 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
W20260808 01:51:11.626788 13581 consensus_meta.cc:287] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.944s	user 0.011s	sys 0.003s
W20260808 01:51:11.657548  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 75150.6us
I20260808 01:51:11.660856 10334 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:11.657657 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:11.624255 13729 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:11.661393  9521 leader_election.cc:336] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.869s (SENT)
I20260808 01:51:11.532564 13813 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:11.629801 13736 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:11.658576 13697 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.826s	user 0.015s	sys 0.051s
I20260808 01:51:11.668445  9521 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:11.673861  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:11.681267  9633 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:11.679348  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:11.660418 13807 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:11.666877 13627 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:11.667260 13708 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election won for term 6
I20260808 01:51:11.573875 13823 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:11.682960 13697 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.850s	user 0.016s	sys 0.052s
I20260808 01:51:11.723112  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:11.725446  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:11.726390  9476 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:51:11.691362 13736 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:11.738297 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:11.739354  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:11.741375 13701 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:51:11.742476  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:11.743355  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:11.752584  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 676460us
I20260808 01:51:11.746392 13817 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:11.749869 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:11.756903  9599 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:11.753871  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:11.757284 13725 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.980s	user 0.014s	sys 0.000s
I20260808 01:51:11.762720 13421 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:11.762399 13703 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:11.719967 13797 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:11.721846 13708 raft_consensus.cc:697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:11.773536 13421 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:11.786816 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:11.787786  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 126404us
I20260808 01:51:11.753340 13798 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:11.763161 13725 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:11.761992 13811 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:11.718226 13627 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:11.785799  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:11.808590  9633 raft_consensus.cc:2376] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12. Re-sending same reply.
I20260808 01:51:11.797479  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:11.699966 13818 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:11.791330  9521 leader_election.cc:336] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.900s (SENT)
I20260808 01:51:11.762975 13752 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20260808 01:51:11.765304 13754 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:11.804867 10367 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:11.805295  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:11.810015  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 53230.1us
I20260808 01:51:11.775317 13747 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:11.758453 13736 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:11.852413  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 61145.2us
W20260808 01:51:11.853282  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 48701.2us
W20260808 01:51:11.858358  9521 leader_election.cc:336] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.132s (SENT)
W20260808 01:51:11.859403  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 61987.1us
W20260808 01:51:11.864590  9521 leader_election.cc:336] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.175s (SENT)
W20260808 01:51:11.870188 13832 log.cc:927] Time spent T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260808 01:51:11.797287 13807 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:11.871565 13829 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.242s	user 0.001s	sys 0.000s
I20260808 01:51:11.872994  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 591) took 1056 ms. Trace:
I20260808 01:51:11.874056 13708 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:11.873543  9478 rpcz_store.cc:276] 0808 01:51:10.817035 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:10.817205 (+   170us) service_pool.cc:224] Handling call
0808 01:51:10.870582 (+ 53377us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:10.907002 (+ 36420us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:10.907054 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:10.951451 (+ 44397us) log.cc:844] Serialized 26 byte log entry
0808 01:51:11.350833 (+399382us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:51:11.350895 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:11.351003 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:11.872641 (+521638us) raft_consensus.cc:1685] finished
0808 01:51:11.872681 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:11.872900 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":36058,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":20199,"thread_start_us":313929,"threads_started":1,"wal-append.queue_time_us":577442}
I20260808 01:51:11.878737 13421 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:11.882225 13753 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:11.883613 13725 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:11.862002 13811 raft_consensus.cc:2764] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election decision vote started in defunct term 10: won
W20260808 01:51:11.885088 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:11.890125 13597 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.058s	user 0.012s	sys 0.000s
I20260808 01:51:11.905506  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:11.910665  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:11.922037  9521 leader_election.cc:336] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.715s (SENT)
W20260808 01:51:11.922936 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:11.928526  9521 leader_election.cc:336] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.715s (SENT)
W20260808 01:51:11.928270  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 67697.2us
W20260808 01:51:11.929433  9521 leader_election.cc:336] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.923s (SENT)
I20260808 01:51:11.850416 13804 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:11.934113  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:11.934974 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:11.771360 13729 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:11.934929  9634 raft_consensus.cc:2376] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10. Re-sending same reply.
I20260808 01:51:11.861095 13754 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:11.937915  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:11.931205  9521 leader_election.cc:336] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.171s (SENT)
I20260808 01:51:11.938576  9634 raft_consensus.cc:2376] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5. Re-sending same reply.
W20260808 01:51:11.941051  9521 leader_election.cc:336] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 3.150s (SENT)
I20260808 01:51:11.941140  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:11.942538  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:11.944033  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:11.944815  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:11.929086 13703 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:11.946852 13754 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:11.950018  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:11.962966 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:11.939610 13467 consensus_queue.cc:1059] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260808 01:51:11.951211 13823 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:11.967770 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:11.967137  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:11.968964  9632 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:11.969161  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:11.970082  9477 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:11.970444  9476 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260808 01:51:11.931485 13597 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:11.953851 10367 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 9 index: 11. (index mismatch)
I20260808 01:51:11.954077 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:11.979642 10366 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:51:11.956689  9629 raft_consensus.cc:2368] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12.
I20260808 01:51:11.957003  9633 raft_consensus.cc:2368] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10.
I20260808 01:51:11.958302  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:11.958535  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:11.982373  9474 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
W20260808 01:51:11.981398  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 120615us
I20260808 01:51:11.950152  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:11.964794  9481 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260808 01:51:11.967042  9478 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260808 01:51:11.973322 13703 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:11.975025 13754 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:11.976593  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:11.961398 13699 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.592s	user 0.010s	sys 0.008s
I20260808 01:51:11.954181 13827 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:11.986217  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:11.992178  9472 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:51:11.988680  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:11.936299 13811 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260808 01:51:11.993862  9477 raft_consensus.cc:2410] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260808 01:51:11.995236  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:11.969609 13752 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:11.996455  9474 raft_consensus.cc:2393] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:11.958920  9634 raft_consensus.cc:2410] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260808 01:51:11.977376 13725 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:12.007848  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:12.009778  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:11.991609 13823 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:12.015075  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:12.028857  9629 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:11.989794 13836 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:12.032158  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 91140.8us
I20260808 01:51:12.040355  9554 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 7. Current term is 8. Ops: []
I20260808 01:51:12.042999  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:12.047899  9478 raft_consensus.cc:2393] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:12.007781 13421 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:12.054307  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:12.057657 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:12.008756 13752 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:11.997712 13699 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:12.031870 13725 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:12.069675  9629 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
W20260808 01:51:12.454607 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:12.455668 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:12.245617  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:12.310051 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:12.188958  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.028358  9472 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:13.029640  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:12.327330  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:12.666606 13850 log.cc:927] Time spent T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.363s	user 0.001s	sys 0.000s
I20260808 01:51:13.030973  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:13.031378 13651 consensus_meta.cc:287] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.506s	user 0.011s	sys 0.011s
I20260808 01:51:12.168816 13811 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:13.032740  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.033571  9472 raft_consensus.cc:2368] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10.
W20260808 01:51:12.184096 13835 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.280s	user 0.000s	sys 0.002s
I20260808 01:51:12.067682  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.034734  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.035169  9474 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:51:13.035593  9472 raft_consensus.cc:2368] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10.
I20260808 01:51:13.036217  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.036448  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:13.037048  9474 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 12: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:13.037413  9472 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:13.037942  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:13.038306  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:12.480311 13597 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:13.039223  9472 raft_consensus.cc:2393] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:12.348718  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:12.299185 13847 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.179s	user 0.003s	sys 0.000s
W20260808 01:51:12.190178 13846 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260808 01:51:13.042131 13752 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:12.275247  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2611) took 2226 ms. Trace:
W20260808 01:51:12.851030  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:13.066316  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:12.545598 13855 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:12.182638  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.071116  9479 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:51:13.070776 13855 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:13.072300  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.073571 13855 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:12.854310  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:12.047570 13836 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:12.474557 13599 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.925s	user 0.014s	sys 0.003s
W20260808 01:51:12.197542 13834 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.268s	user 0.000s	sys 0.001s
W20260808 01:51:12.832926 13669 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.002s	sys 0.010s
W20260808 01:51:13.031445  9481 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.004s	sys 0.019s
W20260808 01:51:13.082343  9481 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.005s	sys 0.019s
I20260808 01:51:13.083431  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2260) took 1088 ms. Trace:
I20260808 01:51:13.046082  9637 raft_consensus.cc:2376] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10. Re-sending same reply.
I20260808 01:51:13.086400  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2184) took 1138 ms. Trace:
I20260808 01:51:13.043097  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:13.046973  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:13.047165  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:13.087738  9630 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:13.087898  9629 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 9: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:13.050104  9631 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:51:13.088160  9634 raft_consensus.cc:2410] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260808 01:51:13.088384  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2186) took 1139 ms. Trace:
W20260808 01:51:12.299042  9444 connection.cc:256] RPC call timeout handler was delayed by -1.5268021539959591s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1513s for a response
I20260808 01:51:13.088713  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2168) took 1115 ms. Trace:
I20260808 01:51:12.456483  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.052218  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:13.089066  9444 connection.cc:256] RPC call timeout handler was delayed by -1.4890021540049929s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1555s for a response
I20260808 01:51:13.052435  9556 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:12.198518  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:12.183756 13838 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.254s	user 0.000s	sys 0.001s
W20260808 01:51:13.089933  9444 connection.cc:256] RPC call timeout handler was delayed by -1.4602021540049464s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1587s for a response
I20260808 01:51:13.074528 13855 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:12.289618  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:13.056239  9636 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
W20260808 01:51:13.092132  9444 connection.cc:256] RPC call timeout handler was delayed by -1.4395021540112793s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16100000000000003s for a response
W20260808 01:51:13.092578  9444 connection.cc:256] RPC call timeout handler was delayed by -1.408002153999405s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1645s for a response
I20260808 01:51:13.092679  9635 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:13.092929  9444 connection.cc:256] RPC call timeout handler was delayed by -1.3828021539957263s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1673s for a response
I20260808 01:51:13.089875  9632 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:51:13.093263  9444 connection.cc:256] RPC call timeout handler was delayed by -1.2883021540183108s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1778s for a response
I20260808 01:51:13.038556  9474 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:51:13.093684  9444 connection.cc:256] RPC call timeout handler was delayed by -1.2172021540172864s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1857s for a response
W20260808 01:51:13.094045  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1956021540099755s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18810000000000002s for a response
W20260808 01:51:13.094404  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1659021539962851s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19140000000000001s for a response
I20260808 01:51:12.046542 13701 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:13.098793 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 697) took 1538 ms. Trace:
W20260808 01:51:13.090106 13729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.006s	sys 0.030s
I20260808 01:51:13.067252  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2189) took 1113 ms. Trace:
I20260808 01:51:13.043411 13785 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:13.043586  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 881393us
I20260808 01:51:12.188411 13736 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:13.099234 10361 rpcz_store.cc:276] 0808 01:51:11.553462 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:11.553701 (+   239us) service_pool.cc:224] Handling call
0808 01:51:11.575994 (+ 22293us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:11.576442 (+   448us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:11.576474 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:11.577029 (+   555us) log.cc:844] Serialized 26 byte log entry
0808 01:51:11.773716 (+196687us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:11.773768 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:11.773849 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:13.091473 (+1317624us) raft_consensus.cc:1685] finished
0808 01:51:13.091506 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:13.091707 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":128,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":128,"thread_start_us":195998,"threads_started":1,"wal-append.queue_time_us":230820}
I20260808 01:51:13.079459  9633 raft_consensus.cc:2393] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 10: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:13.114593  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2187) took 1165 ms. Trace:
I20260808 01:51:13.079134 13790 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:13.088487  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2183) took 1140 ms. Trace:
I20260808 01:51:13.115688  9631 rpcz_store.cc:276] 0808 01:51:11.947901 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:12.200468 (+252567us) service_pool.cc:224] Handling call
0808 01:51:13.088429 (+887961us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:12.513726  9475 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:12.919652  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.089313  9634 rpcz_store.cc:276] 0808 01:51:11.973281 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:13.046887 (+1073606us) service_pool.cc:224] Handling call
0808 01:51:13.088650 (+ 41763us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:13.117304  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:13.091104  9478 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:13.120131  9632 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:13.121739  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2190) took 1167 ms. Trace:
I20260808 01:51:13.122040  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.033180 13811 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:13.053926 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:13.123282 13861 log.cc:927] Time spent T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.284s	user 0.000s	sys 0.001s
W20260808 01:51:13.123932 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:13.124847 13811 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:13.127161  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2633) took 2330 ms. Trace:
I20260808 01:51:13.127642  9560 rpcz_store.cc:276] 0808 01:51:10.796914 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:10.797201 (+   287us) service_pool.cc:224] Handling call
0808 01:51:10.797674 (+   473us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:10.800502 (+  2828us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:10.800568 (+    66us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:11.365512 (+564944us) spinlock_profiling.cc:243] Waited 5.71 ms on lock 0x7b2c000f57d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:51:11.365981 (+   469us) log.cc:844] Serialized 26 byte log entry
0808 01:51:11.637661 (+271680us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:11.637702 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:11.637773 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:13.126777 (+1489004us) raft_consensus.cc:1685] finished
0808 01:51:13.126802 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:13.127086 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":110901,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":12108,"spinlock_wait_cycles":13138176,"thread_start_us":271260,"threads_started":1,"wal-append.queue_time_us":426566}
I20260808 01:51:13.130232  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:13.131323 13859 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.235s	user 0.001s	sys 0.000s
I20260808 01:51:13.131912  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.132740  9476 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:13.115064  9633 rpcz_store.cc:276] 0808 01:51:11.948990 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:12.740818 (+791828us) service_pool.cc:224] Handling call
0808 01:51:13.114521 (+373703us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:13.086916  9637 rpcz_store.cc:276] 0808 01:51:11.948199 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:12.326887 (+378688us) service_pool.cc:224] Handling call
0808 01:51:13.086334 (+759447us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:13.088953  9629 rpcz_store.cc:276] 0808 01:51:11.948708 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:12.467285 (+518577us) service_pool.cc:224] Handling call
0808 01:51:13.088322 (+621037us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:13.057507  9480 raft_consensus.cc:2368] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6.
I20260808 01:51:13.058718  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.137734  9472 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:12.058884 13421 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:13.095571  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1569021540053654s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19240000000000002s for a response
W20260808 01:51:13.141085  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1398021540080663s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19430000000000003s for a response
I20260808 01:51:13.083973  9481 rpcz_store.cc:276] 0808 01:51:11.994652 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:12.001703 (+  7051us) service_pool.cc:224] Handling call
0808 01:51:12.009171 (+  7468us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:12.009433 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:51:12.481942 (+472509us) log.cc:844] Serialized 14 byte log entry
0808 01:51:13.082924 (+600982us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:13.083007 (+    83us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:13.083077 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:13.083175 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:13.083347 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":233745,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"thread_start_us":1035381,"threads_started":2}
W20260808 01:51:13.141476  9444 connection.cc:256] RPC call timeout handler was delayed by -1.02550215399242s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.207s for a response
I20260808 01:51:13.064888  9555 rpcz_store.cc:276] 0808 01:51:10.048930 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:10.062860 (+ 13930us) service_pool.cc:224] Handling call
0808 01:51:10.063273 (+   413us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:10.151755 (+ 88482us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:51:10.173019 (+ 21264us) log.cc:844] Serialized 14 byte log entry
0808 01:51:10.300417 (+127398us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b1800214d20. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d4453242a 00007f5d4499de0a
0808 01:51:10.636855 (+336438us) log.cc:844] Serialized 14 byte log entry
0808 01:51:10.678579 (+ 41724us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:10.678637 (+    58us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:10.678759 (+   122us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:12.274840 (+1596081us) spinlock_profiling.cc:243] Waited 88.2 ms on lock 0x7b5400217fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:12.274921 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:12.275146 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":172835,"prepare.queue_time_us":22747,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":92,"raft.queue_time_us":278693,"raft.run_cpu_time_us":341,"raft.run_wall_time_us":17946,"spinlock_wait_cycles":243981056,"thread_start_us":253417,"threads_started":1,"wal-append.queue_time_us":545993,"wal-append.run_cpu_time_us":3936,"wal-append.run_wall_time_us":1317508}
W20260808 01:51:13.095100 13669 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.764s	user 0.003s	sys 0.012s
W20260808 01:51:13.066463 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 443529us
I20260808 01:51:13.104714  9638 rpcz_store.cc:276] 0808 01:51:11.953875 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:13.054362 (+1100487us) service_pool.cc:224] Handling call
0808 01:51:13.067181 (+ 12819us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:51:13.111649  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:13.139003  9635 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
W20260808 01:51:13.179023 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:13.179414  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2193) took 1224 ms. Trace:
W20260808 01:51:13.104352 13729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.006s	sys 0.031s
W20260808 01:51:13.179682 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:13.182101 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:13.182715  9637 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260808 01:51:13.182890 10334 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:13.183526  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2169) took 1209 ms. Trace:
I20260808 01:51:13.140287  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.184582  9477 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:51:13.183912  9637 rpcz_store.cc:276] 0808 01:51:11.973717 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:13.181926 (+1208209us) service_pool.cc:224] Handling call
0808 01:51:13.182309 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:13.183205 (+   896us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:13.183463 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:13.134162  9632 rpcz_store.cc:276] 0808 01:51:11.954182 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:11.975954 (+ 21772us) service_pool.cc:224] Handling call
0808 01:51:12.653579 (+677625us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x7b5800231698. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:51:13.121647 (+468068us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2653440}
I20260808 01:51:13.171348  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:13.117990  9634 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:51:13.175776 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:13.187054  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2188) took 1233 ms. Trace:
I20260808 01:51:13.176054  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:13.177145  9474 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:51:13.097657 13725 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:13.171200 13790 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:13.185719  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:13.184262  9635 rpcz_store.cc:276] 0808 01:51:11.955334 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:12.263435 (+308101us) service_pool.cc:224] Handling call
0808 01:51:13.179347 (+915912us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:13.187472  9634 rpcz_store.cc:276] 0808 01:51:11.953580 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:13.116882 (+1163302us) service_pool.cc:224] Handling call
0808 01:51:13.186987 (+ 70105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:13.187878  9476 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:13.187204 10361 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:13.183883 13597 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:13.122575 13836 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:13.204362  9481 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 7
I20260808 01:51:13.204726  9637 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260808 01:51:13.190954  9479 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:13.205554  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2171) took 1231 ms. Trace:
I20260808 01:51:13.206046  9637 rpcz_store.cc:276] 0808 01:51:11.974475 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:13.185967 (+1211492us) service_pool.cc:224] Handling call
0808 01:51:13.203336 (+ 17369us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:13.205241 (+  1905us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:13.205493 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:13.206563  9631 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260808 01:51:13.192554  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.102909 13701 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:13.207522  9474 raft_consensus.cc:2368] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11.
I20260808 01:51:13.207609 13875 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:13.191646  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.208210 13875 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:13.208550  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.209438  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.209990  9477 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:13.210141  9474 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:13.210713 13875 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:13.189285 13784 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.959s	user 0.034s	sys 0.038s
I20260808 01:51:13.104668 13855 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:13.212002  9474 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260808 01:51:13.213057  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:13.213425  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:13.214810  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:13.215523  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2185) took 1267 ms. Trace:
I20260808 01:51:13.239918  9631 rpcz_store.cc:276] 0808 01:51:11.948459 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:13.185911 (+1237452us) service_pool.cc:224] Handling call
0808 01:51:13.206303 (+ 20392us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:13.215178 (+  8875us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:13.215455 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:13.216324  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:13.240411  9632 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:13.242463  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:13.241552  9637 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:51:13.242758  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:13.243773  9635 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:13.244436  9634 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 8: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:13.247510  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:13.247510  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 76263.2us
I20260808 01:51:13.248054  9559 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:51:13.192232 13599 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:13.248819  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:13.247097  9555 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:13.190527 13752 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:13.198665 13597 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:13.251834  9557 raft_consensus.cc:2410] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260808 01:51:13.252358  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:13.251638 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:13.262225  9632 raft_consensus.cc:3055] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:13.270974  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:13.274142  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:13.304250  9632 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:13.131978 13785 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:13.323251  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:13.244174 13784 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.069s	user 0.042s	sys 0.049s
I20260808 01:51:13.304443 10366 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:51:13.211467 13701 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:13.357280  9632 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:13.355899  9637 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
W20260808 01:51:13.356982  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 86082.8us
I20260808 01:51:13.370376  9474 raft_consensus.cc:1217] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Deduplicated request from leader. Original: 8.5->[9.6-9.6]   Dedup: 9.6->[]
I20260808 01:51:13.371470  9474 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260808 01:51:13.222290 13651 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:13.355695  9478 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:13.248417 13729 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:13.304764  9629 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:51:13.377362  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 131101us
I20260808 01:51:13.382457  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:13.375314  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2641) took 1371 ms. Trace:
I20260808 01:51:13.325021 13597 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:13.384028  9552 rpcz_store.cc:276] 0808 01:51:12.004155 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:12.004335 (+   180us) service_pool.cc:224] Handling call
0808 01:51:12.014261 (+  9926us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:12.014736 (+   475us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:12.014767 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:12.047024 (+ 32257us) log.cc:844] Serialized 26 byte log entry
0808 01:51:12.179572 (+132548us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:12.179634 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:12.179741 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:13.374984 (+1195243us) raft_consensus.cc:1685] finished
0808 01:51:13.375014 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:13.375222 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":601,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":13633,"spinlock_wait_cycles":21376,"thread_start_us":61123,"threads_started":1,"wal-append.queue_time_us":576107}
I20260808 01:51:13.407918  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:13.390796  9481 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:51:13.408582  9637 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 10
I20260808 01:51:13.391861  9476 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:13.398995  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:13.410627 13811 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:13.410933  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:13.411947 13811 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:13.412497 13811 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:13.413182 13811 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:13.413362 13736 consensus_queue.cc:1059] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:51:13.414134  9476 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 5. (term mismatch)
I20260808 01:51:13.414515  9476 pending_rounds.cc:85] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 4
I20260808 01:51:13.414928  9476 pending_rounds.cc:107] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20260808 01:51:13.415283  9476 raft_consensus.cc:2889] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:13.376608 13699 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:13.383487 13781 negotiation.cc:336] Failed RPC negotiation. Trace:
0808 01:51:09.752334 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.7.165.194:36095 (local address 127.0.0.1:32912)
0808 01:51:10.025045 (+272711us) negotiation.cc:107] Waiting for socket to connect
0808 01:51:10.025124 (+    79us) client_negotiation.cc:175] Beginning negotiation
0808 01:51:10.025652 (+   528us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0808 01:51:11.693106 (+1667454us) client_negotiation.cc:281] Received NEGOTIATE NegotiatePB response
0808 01:51:11.713600 (+ 20494us) client_negotiation.cc:404] Received NEGOTIATE response from server
0808 01:51:11.760700 (+ 47100us) client_negotiation.cc:190] Negotiated authn=SASL
0808 01:51:11.931658 (+170958us) client_negotiation.cc:530] Sending TLS_HANDSHAKE message to server
0808 01:51:11.931771 (+   113us) client_negotiation.cc:262] Sending TLS_HANDSHAKE NegotiatePB request
0808 01:51:12.265367 (+333596us) client_negotiation.cc:281] Received TLS_HANDSHAKE NegotiatePB response
0808 01:51:12.287460 (+ 22093us) client_negotiation.cc:543] Received TLS_HANDSHAKE response from server
0808 01:51:12.841399 (+553939us) client_negotiation.cc:530] Sending TLS_HANDSHAKE message to server
0808 01:51:12.847687 (+  6288us) client_negotiation.cc:262] Sending TLS_HANDSHAKE NegotiatePB request
0808 01:51:13.063424 (+215737us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.7.165.194:36095: sent 0 of 16 requested bytes
Metrics: {"client-negotiator.queue_time_us":216393,"thread_start_us":92898,"threads_started":1}
I20260808 01:51:13.365489 13836 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:13.437049  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.439296  9632 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:13.417618 10334 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:13.356837 13876 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:13.421525  9474 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260808 01:51:13.434665  9629 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:51:13.412407  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:13.440351 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 223604us
W20260808 01:51:13.450356 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:13.392135 13729 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:13.271284 13599 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:13.471109  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2637) took 2518 ms. Trace:
I20260808 01:51:13.306558 13752 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:13.471410  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2439) took 1026 ms. Trace:
I20260808 01:51:13.442308  9475 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:51:13.471617  9556 rpcz_store.cc:276] 0808 01:51:10.952442 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:10.969069 (+ 16627us) service_pool.cc:224] Handling call
0808 01:51:11.353001 (+383932us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:13.091900 (+1738899us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:13.470806 (+378906us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b5400257e54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:13.471039 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3907769088}
I20260808 01:51:13.472481  9474 rpcz_store.cc:276] 0808 01:51:12.445336 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:13.391662 (+946326us) service_pool.cc:224] Handling call
0808 01:51:13.421190 (+ 29528us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:13.471086 (+ 49896us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:13.471343 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:13.475549  9476 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260808 01:51:13.467315  7831 auto_leader_rebalancer.cc:456] leader transfer for tablet 2c617cb141a34007b6c1b8af0d911923 from 52ccf4dcca1843f7bb29e9172e2d4ab4 to 6e7856bed1be43acaa821290d506ee97 failed: Timed out: Client connection negotiation failed: client connection to 127.7.165.194:36095: sent 0 of 16 requested bytes
I20260808 01:51:13.476464  9481 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260808 01:51:13.416936 13725 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260808 01:51:13.483211  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:13.443895 13878 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:13.485684 10332 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.598s (SENT) [suppressed 43 similar messages]
I20260808 01:51:13.493532  9474 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: []
W20260808 01:51:13.484066  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 71743.9us
I20260808 01:51:13.481048 13860 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:13.391893 13651 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:13.493476  9558 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
W20260808 01:51:13.471055 13805 negotiation.cc:336] Failed RPC negotiation. Trace:
0808 01:51:10.288950 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:32912 (local address 127.7.165.194:36095)
0808 01:51:11.500789 (+1211839us) server_negotiation.cc:207] Beginning negotiation
0808 01:51:11.510362 (+  9573us) server_negotiation.cc:400] Waiting for connection header
0808 01:51:11.522836 (+ 12474us) server_negotiation.cc:408] Connection header received
0808 01:51:11.547802 (+ 24966us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0808 01:51:11.572039 (+ 24237us) server_negotiation.cc:462] Received NEGOTIATE request from client
0808 01:51:11.663090 (+ 91051us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0808 01:51:11.688180 (+ 25090us) server_negotiation.cc:227] Negotiated authn=SASL
0808 01:51:11.932128 (+243948us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0808 01:51:12.216762 (+284634us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0808 01:51:13.021621 (+804859us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32912: BlockingRecv error: recv got EOF from 127.0.0.1:32912 (error 108)
Metrics: {"server-negotiator.queue_time_us":523750,"thread_start_us":337420,"threads_started":1}
I20260808 01:51:13.494745  9632 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260808 01:51:13.475406 13729 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:13.514170  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:13.516009  9637 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:13.519752 13790 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:13.520390 13599 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:13.521013 13599 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:51:13.522495  9552 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:13.524559  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:13.525843  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:13.497956 13813 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:13.538436  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:13.540128  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:13.475430 13876 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:13.547333 13889 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader election won for term 7
W20260808 01:51:13.556890 13867 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.343s	user 0.001s	sys 0.000s
I20260808 01:51:13.522660 13599 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.035s
W20260808 01:51:13.559007 13868 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
I20260808 01:51:13.525029 13701 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:13.535863 13884 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:13.570483 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:13.571715  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 191581us
I20260808 01:51:13.527299 13836 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:13.569070  9630 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
W20260808 01:51:13.585608  9630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2194) took 1630 ms (client timeout 2072 ms). Trace:
W20260808 01:51:13.586134  9630 rpcz_store.cc:269] 0808 01:51:11.955638 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:12.200342 (+244704us) service_pool.cc:224] Handling call
0808 01:51:13.585498 (+1385156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:13.524138 13421 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:13.575028  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2431) took 1136 ms. Trace:
I20260808 01:51:13.527159 13860 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260808 01:51:13.602845 13869 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.344s	user 0.002s	sys 0.000s
I20260808 01:51:13.569849  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:13.603549 13866 log.cc:927] Time spent T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.360s	user 0.000s	sys 0.002s
I20260808 01:51:13.604188  9477 rpcz_store.cc:276] 0808 01:51:12.438418 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:13.210411 (+771993us) service_pool.cc:224] Handling call
0808 01:51:13.210637 (+   226us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:13.211146 (+   509us) raft_consensus.cc:1534] Early marking committed up to index 4
0808 01:51:13.211698 (+   552us) log.cc:844] Serialized 14 byte log entry
0808 01:51:13.572617 (+360919us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:13.574447 (+  1830us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:51:13.574587 (+   140us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:13.574722 (+   135us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:13.574940 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":287,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":6410,"raft.queue_time_us":21910,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"thread_start_us":194220,"threads_started":1,"wal-append.queue_time_us":317850}
I20260808 01:51:13.579443  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:13.606113 13876 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:13.606926  9601 leader_election.cc:400] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10.
I20260808 01:51:13.607301  9601 leader_election.cc:403] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:51:13.608479  9601 leader_election.cc:400] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10.
I20260808 01:51:13.608784  9601 leader_election.cc:403] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:51:13.580910  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:13.609607  9552 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:13.504722 13878 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:13.543996 13852 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:51:13.574568 13881 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:13.593436 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:13.611310 13881 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:13.612332 13878 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:13.613818 13881 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:13.589207 13884 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:13.588106 13736 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:13.617619  9474 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260808 01:51:13.578862  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:13.568352 13874 log.cc:927] Time spent T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Append to log took a long time: real 0.123s	user 0.002s	sys 0.000s
I20260808 01:51:13.583402 13864 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:13.623353 13888 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:13.527369 13699 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.050s
I20260808 01:51:13.618892 13790 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10.
I20260808 01:51:13.548985 13752 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:13.618839 13704 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10.
I20260808 01:51:13.568399 13651 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:13.628196 13729 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:13.584928 13701 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:13.631038 13701 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:13.620280  9477 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260808 01:51:13.633113  9477 pending_rounds.cc:85] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 4
I20260808 01:51:13.635056 13785 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:13.615854 13855 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:13.637143  9472 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:13.636426 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:13.637995 10366 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:51:13.637740 13813 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:51:13.639457  9477 pending_rounds.cc:107] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 10.5
I20260808 01:51:13.640290  9477 raft_consensus.cc:2889] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:13.638616  9472 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260808 01:51:13.641090  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:13.641122  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 62382.1us
W20260808 01:51:13.643420  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 105123us
I20260808 01:51:13.644600  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:13.646755  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 288864us
I20260808 01:51:13.650661  9472 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:13.665150  9601 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:13.666330  9472 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260808 01:51:13.667279  9601 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:13.668412 13729 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:13.650641  9524 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:13.650895 13876 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.021s
I20260808 01:51:13.671904 13897 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:51:13.672796 13736 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 01:51:13.618664 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 201126us
W20260808 01:51:13.673640 10332 connection.cc:256] RPC call timeout handler was delayed by -2.1107801409962121s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1754s for a response
W20260808 01:51:13.672291  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 50125.5us
I20260808 01:51:13.643505 13878 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260808 01:51:13.663441 13785 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20260808 01:51:13.649672 13574 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:13.649155 13855 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260808 01:51:13.685074 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:13.685195 10332 connection.cc:256] RPC call timeout handler was delayed by -1.9439303690160159s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16870000000000002s for a response
I20260808 01:51:13.685091  9472 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 01:51:13.650414 13811 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:13.681602 13857 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.289s	user 0.001s	sys 0.001s
I20260808 01:51:13.673393 13699 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:13.671924 13904 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260808 01:51:13.695873 10332 connection.cc:256] RPC call timeout handler was delayed by -1.9428317410056479s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16870000000000002s for a response
I20260808 01:51:13.677256 13627 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:13.683678 13421 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:13.704532 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:13.687665 13754 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260808 01:51:13.703243 10332 connection.cc:256] RPC call timeout handler was delayed by -1.7124552509922069s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2041s for a response
I20260808 01:51:13.646499 13877 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.051s
I20260808 01:51:13.701700  9472 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260808 01:51:13.706022  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:13.708000 10332 connection.cc:256] RPC call timeout handler was delayed by -1.6495324339775834s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20040000000000002s for a response
W20260808 01:51:13.709868 10332 connection.cc:256] RPC call timeout handler was delayed by -1.6475539389939513s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21480000000000002s for a response
W20260808 01:51:13.711709 10332 connection.cc:256] RPC call timeout handler was delayed by -1.5250703549827449s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.188s for a response
W20260808 01:51:13.711992 10332 connection.cc:256] RPC call timeout handler was delayed by -1.504604289017152s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15780000000000002s for a response
W20260808 01:51:13.712229 10332 connection.cc:256] RPC call timeout handler was delayed by -1.31365393899614s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2519s for a response
W20260808 01:51:13.712474 10332 connection.cc:256] RPC call timeout handler was delayed by -1.2346804550033994s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1703s for a response
I20260808 01:51:13.712704  9601 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:13.709780  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 62331.3us
I20260808 01:51:13.674012 13894 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260808 01:51:13.692168 13893 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
W20260808 01:51:13.717310 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:13.717432 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:13.708869 13421 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:13.727643 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:13.719002 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:13.729580 10364 raft_consensus.cc:2393] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:13.720304 10366 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 5.
I20260808 01:51:13.720340 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:13.731475 10368 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:13.698477 13699 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:13.735903  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:13.727015  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2642) took 1722 ms. Trace:
I20260808 01:51:13.740139  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:13.742161  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:13.743468  9560 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:13.741920  9551 rpcz_store.cc:276] 0808 01:51:12.004572 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:12.004720 (+   148us) service_pool.cc:224] Handling call
0808 01:51:12.038458 (+ 33738us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:12.039020 (+   562us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:12.039060 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:12.045282 (+  6222us) log.cc:844] Serialized 26 byte log entry
0808 01:51:12.340021 (+294739us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:51:12.340083 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:12.340192 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:13.726601 (+1386409us) raft_consensus.cc:1685] finished
0808 01:51:13.726630 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:13.726925 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":162,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":8564,"thread_start_us":269095,"threads_started":1,"wal-append.queue_time_us":1136857}
I20260808 01:51:13.745260 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:13.746471  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:13.747023  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:13.707790 13813 raft_consensus.cc:3055] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:13.703733 13904 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:13.748466  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:13.774878  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:13.776189  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:13.750444 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:13.749063  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:13.750540  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:13.749233  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62952.5us
I20260808 01:51:13.717058 13701 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:13.744812 13699 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:13.806490  9559 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:13.811179  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:13.820923  9630 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:13.821599  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.821931 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:13.822575  9476 raft_consensus.cc:2410] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260808 01:51:13.822428 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:13.823264 13839 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.857s	user 0.007s	sys 0.024s
I20260808 01:51:13.823580  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.824371 13905 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:13.824966  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.825076 13905 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:13.826020  9476 raft_consensus.cc:2410] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260808 01:51:13.823267 10368 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:13.807184  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:13.806423  9629 raft_consensus.cc:2376] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10. Re-sending same reply.
I20260808 01:51:13.810518  9556 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:51:13.744402 13877 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.075s
I20260808 01:51:13.827521  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:13.831805  9559 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:13.747671 13736 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:13.827427  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:13.830912  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:13.834986  9557 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:13.828691  9551 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:13.820125 13904 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:13.750154 13813 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:13.841269  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:13.843839  9556 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
I20260808 01:51:13.851876 13905 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:13.853247 13421 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:13.855253  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:13.855715 13892 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260808 01:51:13.827152 13884 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:13.856813  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:13.859495 13811 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:13.862715 13627 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:51:13.864801 13908 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:51:13.858273  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:13.866329 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:13.869181  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:13.871927  9559 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:13.886230  9557 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:13.892278  9551 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:13.766360 13597 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:13.908675  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.909413  9479 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 5.
I20260808 01:51:13.895649  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.906610 13627 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.005s
I20260808 01:51:13.911392 13627 consensus_queue.cc:1243] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 8.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260808 01:51:13.911639 13906 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260808 01:51:13.896924  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:13.905540  9560 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:51:13.838024 13736 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:13.909538 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:13.919272  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 110073us
I20260808 01:51:13.920671 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:13.919456  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:13.921835 10334 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:13.907814  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 234861us
I20260808 01:51:13.923059  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:13.923352  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:13.923749  9479 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:51:13.924453 10361 raft_consensus.cc:2368] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12.
I20260808 01:51:13.925201  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:13.925824  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 97220us
I20260808 01:51:13.925943  9632 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:51:13.926630  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:13.927610  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:13.929059  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:13.918809 13736 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:13.930742  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:13.930159  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:13.927886  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:13.930768  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:13.931954  9560 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 9: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:13.932067  9630 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:51:13.933619  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:13.934317  9629 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:51:13.934551  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:13.953449  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:13.962908  9552 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:13.964071  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:13.965122  9524 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:13.967026  9638 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:13.851024 13878 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.084s
I20260808 01:51:13.928908 13627 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:51:13.932843 10332 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:13.958038  9635 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260808 01:51:13.955574  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:13.969830 13877 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:13.971721  9630 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:51:13.974865  9560 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260808 01:51:13.979584  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:13.983152  9636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2154) took 3022 ms (client timeout 2152 ms). Trace:
I20260808 01:51:13.984091 13886 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:13.984362  9637 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260808 01:51:13.985014 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 52245.7us
W20260808 01:51:13.988535  9636 rpcz_store.cc:269] 0808 01:51:10.961056 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:10.961214 (+   158us) service_pool.cc:224] Handling call
0808 01:51:13.982877 (+3021663us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b5400421dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:13.983029 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3851592960}
I20260808 01:51:13.975222 13910 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:13.976115 13876 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:51:13.992630 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:13.993876  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:13.990048 13877 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:13.980360 13627 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:14.003480 13876 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:14.006906 13627 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:51:14.002496 13864 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:14.008000 13864 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:14.009095  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:14.009801  9477 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:13.993597  9552 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 9 index: 11. (index mismatch)
I20260808 01:51:13.998026 13905 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:51:14.001418 13904 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:14.014034 13904 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:13.996529  9479 raft_consensus.cc:1217] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Deduplicated request from leader. Original: 3.5->[6.6-11.8]   Dedup: 6.6->[9.7-11.8]
I20260808 01:51:14.004505 13597 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:14.015002  9479 pending_rounds.cc:85] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 6
I20260808 01:51:14.015290  9479 pending_rounds.cc:107] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 8.7
I20260808 01:51:14.015590  9479 raft_consensus.cc:2889] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:13.956627 13909 raft_consensus.cc:2764] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260808 01:51:14.005734 13898 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:14.020279  9477 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:13.956694 13599 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260808 01:51:14.004083 13877 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:14.006964 13876 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:51:13.996037 13910 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:14.025352 10332 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:14.027338  9638 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:51:14.031224 13904 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260808 01:51:14.030869 13909 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:14.035717 13886 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:51:14.036569 13886 raft_consensus.cc:697] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:14.037755 13886 consensus_queue.cc:237] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:14.043035 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:14.047356 13818 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:51:14.049124 13818 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:14.049623 13818 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:14.050024 13818 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:14.051535 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:14.052935 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:14.053592 10334 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:14.054698 13844 raft_consensus.cc:3055] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:51:14.054991 10334 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:14.051524  9475 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2253) took 2093 ms (client timeout 2011 ms). Trace:
I20260808 01:51:14.055984 13597 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:14.054999  9472 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:51:14.055167 13844 raft_consensus.cc:740] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:14.052984 10365 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260808 01:51:14.028695 13910 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:14.048906 13915 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.026s
W20260808 01:51:14.070209  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:14.059415 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:14.061810  9636 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260808 01:51:14.056114  9475 rpcz_store.cc:269] 0808 01:51:11.958259 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:11.958374 (+   115us) service_pool.cc:224] Handling call
0808 01:51:14.051278 (+2092904us) spinlock_profiling.cc:243] Waited 523 ms on lock 0x7b540021e654. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:14.051411 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1202761088}
I20260808 01:51:14.072098  9601 leader_election.cc:400] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11.
I20260808 01:51:14.072475  9601 leader_election.cc:403] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:51:14.065316  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:14.066224  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:14.074159  9476 raft_consensus.cc:2368] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6.
I20260808 01:51:14.075541  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:14.076334  9476 raft_consensus.cc:2376] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9. Re-sending same reply.
I20260808 01:51:14.077648  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:14.045982 13864 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:14.067224  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:14.067343  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:14.086515  9523 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:14.081161  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:14.088641  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:14.078465  9476 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:14.089600  9601 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:14.089910  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:14.087039  9634 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:14.090926  9632 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:51:14.091502  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:14.092928  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:14.093474  9600 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:14.096663  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:14.097060  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:14.100217  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:14.101676  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:14.102573  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:14.100576  9523 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:14.103235  9631 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:14.064508 13597 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:14.106315 13599 raft_consensus.cc:2764] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260808 01:51:14.124460 13599 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260808 01:51:14.105238  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 77703.2us
I20260808 01:51:14.127656  9445 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:14.102254 13888 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:14.094780 13651 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11.
I20260808 01:51:14.096112 13704 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:51:14.104914 13790 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260808 01:51:14.155488  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 150701us
I20260808 01:51:14.172394  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:14.173770  9478 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260808 01:51:14.175509  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:14.177418  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:14.178279  9480 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:14.179395  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:14.180209  9477 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:14.137311 13597 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:14.188889  9524 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:14.119364 13864 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:14.143780 13421 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:14.138767 13599 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:14.143622 13888 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:14.189262 13790 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:14.196061  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:14.201196  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 207373us
I20260808 01:51:14.205082  9480 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:14.206213  9477 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
W20260808 01:51:14.209633  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 103687us
I20260808 01:51:14.190049 13881 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:14.142933 13893 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.055s
I20260808 01:51:14.145153 13627 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:14.193907 13651 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:14.195878 13421 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:14.199563 13790 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:14.197873 13599 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:14.197914 13888 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:51:14.234409 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 175053us
W20260808 01:51:14.241727  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 150415us
I20260808 01:51:14.214309 13881 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:14.230820 13891 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:14.162557 13844 consensus_queue.cc:260] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:14.233872 13421 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:14.249816  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:14.240365 13599 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:14.216419 13704 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260808 01:51:14.269230 13917 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:51:14.262202 13921 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:51:14.249929 13881 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:51:14.283740 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:14.285496  9477 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:51:14.260725 13915 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 01:51:14.311794  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 62414.6us
W20260808 01:51:14.309458  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 107269us
I20260808 01:51:14.266307 13891 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:51:14.320492  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 119479us
I20260808 01:51:14.319895  9521 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:14.322547  9524 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:14.287518 13921 raft_consensus.cc:697] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:14.289032 13651 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:14.313009 13844 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:14.327179  9521 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:14.333138  9524 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:14.335700 13915 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:14.336606 13915 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:14.337296 13915 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:14.339469 13627 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:14.340584 13921 consensus_queue.cc:237] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:14.341483 13914 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260808 01:51:14.344121 13926 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:51:14.336700 13704 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.011s
I20260808 01:51:14.308905 13919 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.068s
I20260808 01:51:14.357633 13888 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:14.339268 13752 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:51:14.360877 13888 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:14.338375 13915 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:14.370437 13910 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:14.371636 13920 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260808 01:51:14.373382 13925 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.044s
I20260808 01:51:14.372529 13923 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:51:14.377370 13910 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:14.373375 13915 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:14.386942 13924 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.053s
I20260808 01:51:14.392280 13818 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:14.393661 10364 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 9: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:51:14.378968 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 55748.4us
I20260808 01:51:14.379096 13886 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:51:14.396610  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 599) took 1351 ms. Trace:
I20260808 01:51:14.397028  9481 rpcz_store.cc:276] 0808 01:51:13.044781 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:13.498470 (+453689us) service_pool.cc:224] Handling call
0808 01:51:13.866600 (+368130us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:13.866912 (+   312us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:13.866958 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:13.867017 (+    59us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:51:13.867076 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:13.867154 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:14.396394 (+529240us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b540038d1d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:14.396534 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":842293504}
I20260808 01:51:14.381143 13919 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:14.400884 13919 raft_consensus.cc:697] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:14.401863 13919 consensus_queue.cc:237] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:14.403697  9479 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260808 01:51:14.381794  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 48771.1us
I20260808 01:51:14.409052 13907 raft_consensus.cc:3055] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:51:14.409521 13907 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:14.409650 13926 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260808 01:51:14.410759  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:14.381961 13421 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:14.411939 13752 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:14.412453 13752 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:14.413256  9476 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260808 01:51:14.410923 13704 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:14.414224 13704 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:14.415227 13704 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:14.416553 13704 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:14.415598  9478 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260808 01:51:14.413348 13752 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.4, Last appended by leader: 4, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:14.372548  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 45411.6us
I20260808 01:51:14.390401 10363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 703) took 1198 ms. Trace:
I20260808 01:51:14.392663  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:14.424983 10363 rpcz_store.cc:276] 0808 01:51:13.191994 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:13.192174 (+   180us) service_pool.cc:224] Handling call
0808 01:51:13.217026 (+ 24852us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:13.217268 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:51:13.400238 (+182970us) log.cc:844] Serialized 14 byte log entry
0808 01:51:14.389871 (+989633us) spinlock_profiling.cc:243] Waited 6.98 ms on lock 0x7b18004b91c0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4453a6ac 00007f5d4186fe3b 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667
0808 01:51:14.389987 (+   116us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:14.390069 (+    82us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:14.390120 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:14.390192 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:14.390328 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":249499,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":107866,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":21394,"spinlock_wait_cycles":16046336,"thread_start_us":201550,"threads_started":2}
I20260808 01:51:14.408686 13736 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:14.379127 13597 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:14.428515  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:14.438915 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 203382us
I20260808 01:51:14.442582  9481 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:51:14.427999 13699 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.595s	user 0.011s	sys 0.003s
I20260808 01:51:14.458278  9523 leader_election.cc:400] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12.
I20260808 01:51:14.458724  9523 leader_election.cc:403] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:51:14.439009 13908 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:14.462149  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 125810us
W20260808 01:51:14.469383  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:14.470367  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:14.470645  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:14.471184  9552 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 12
I20260808 01:51:14.466970  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:14.471122  9555 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:51:14.472499  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:14.457796 13699 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:14.451146 13864 raft_consensus.cc:2764] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260808 01:51:14.476912 13930 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.034s
I20260808 01:51:14.477151 13888 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:51:14.477401 13699 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:14.479262 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:14.475956 13924 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
W20260808 01:51:14.479664  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 87070.9us
I20260808 01:51:14.482031 10363 raft_consensus.cc:2393] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 6: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:14.472160  9550 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:51:14.482074  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:14.483795  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:14.479287  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:14.492532  9601 leader_election.cc:400] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6.
I20260808 01:51:14.492295  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:14.493983  9444 leader_election.cc:400] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12.
I20260808 01:51:14.494423  9444 leader_election.cc:403] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:51:14.486923 13651 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:14.475373 13752 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:14.499675  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:14.504767  9552 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:14.503441  9481 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:51:14.487429 13864 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:14.510052 13467 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:14.506282 13875 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12.
I20260808 01:51:14.497433 13785 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:14.507457 10334 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:14.467729 13881 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:14.498159 13651 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:14.490988 13888 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.015s
I20260808 01:51:14.410504 13907 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:14.503960 13889 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.947s	user 0.002s	sys 0.024s
I20260808 01:51:14.515519 13785 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:51:14.545751  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:14.523036 13824 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:14.479832 13908 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:14.550649  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:14.554910  9472 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:14.555037  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:14.521413 13818 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:14.531095 13889 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.974s	user 0.003s	sys 0.024s
I20260808 01:51:14.560281  9557 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:14.561589  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:14.568238  9556 raft_consensus.cc:2410] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260808 01:51:14.473366 13891 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:14.525508 13934 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Leader election won for term 6
W20260808 01:51:14.542034 13877 consensus_meta.cc:287] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.511s	user 0.014s	sys 0.004s
I20260808 01:51:14.564605  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:14.564724  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:14.549546 13785 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:14.550503 13898 consensus_meta.cc:287] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.531s	user 0.009s	sys 0.006s
I20260808 01:51:14.544718 13597 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:14.527810 13876 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:14.578596 13935 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:14.548605 13813 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:14.584663  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:14.596400  9472 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:14.597836 10334 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:14.612689  9560 raft_consensus.cc:2393] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:14.573904 13889 raft_consensus.cc:697] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:14.581984 13907 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:14.578819 13811 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12.
I20260808 01:51:14.582618 13898 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:14.608764 13736 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:14.584339 13934 raft_consensus.cc:697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:14.635396  9478 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260808 01:51:14.616400 13779 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:14.579365 13877 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:14.628827 13818 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:14.654184  9472 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: []
W20260808 01:51:14.641260 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 172318us
W20260808 01:51:14.666050 13849 log.cc:927] Time spent T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 2.477s	user 0.013s	sys 0.022s
W20260808 01:51:14.668529 13940 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260808 01:51:14.672099 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:14.673352 10361 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:14.621387 13651 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:14.676174  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2653) took 1453 ms. Trace:
I20260808 01:51:14.676766  9558 rpcz_store.cc:276] 0808 01:51:13.222468 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:13.246167 (+ 23699us) service_pool.cc:224] Handling call
0808 01:51:13.246603 (+   436us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:13.498770 (+252167us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:14.675801 (+1177031us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b540002cbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:14.676103 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":562556288}
W20260808 01:51:14.681977 13943 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260808 01:51:14.684698 13818 raft_consensus.cc:697] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:51:14.703258 13856 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 1.518s	user 0.004s	sys 0.003s
I20260808 01:51:14.704101 13897 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:14.704598 13897 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:14.711232 13908 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:14.714018 13876 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:14.720928 13877 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:14.732497 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:14.642599 13813 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:14.680454 13889 consensus_queue.cc:237] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:14.722743 13871 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.264s	user 0.002s	sys 0.002s
W20260808 01:51:14.776760 13790 consensus_meta.cc:287] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.533s	user 0.012s	sys 0.000s
I20260808 01:51:14.777616 13790 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:14.784720 13790 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:14.757210  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 179282us
I20260808 01:51:14.785944 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:14.786881 10367 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 9: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:14.787817  9599 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:14.788832 13754 consensus_queue.cc:1059] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:51:14.686021 13818 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:14.705889 13897 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:14.791293 13897 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:14.793216 13888 raft_consensus.cc:3055] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:14.748625 13881 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:14.779883 13785 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:14.644291 13599 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:14.687603 13931 consensus_queue.cc:1059] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.083s
I20260808 01:51:14.802549 13599 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:14.790261  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:14.796372 13930 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:14.748103 13934 consensus_queue.cc:237] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:14.805284 13909 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:14.805784 13909 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:14.806495 13785 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:14.798326 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 65960us
I20260808 01:51:14.793882 13888 raft_consensus.cc:740] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:14.806752 13909 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:14.799538 10361 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
W20260808 01:51:14.823863  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 324219us
I20260808 01:51:14.827246  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:14.828227  9601 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:14.834357 13701 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:14.837637  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:14.844143 13909 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:14.842160 13574 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:14.847173 13574 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:14.846562 13792 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:51:14.847594 13574 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:14.847667  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:14.850219  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:14.852826 13888 consensus_queue.cc:260] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:14.850858  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:14.855636  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:14.856608 13701 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:14.850283 13792 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:14.835639 13950 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:14.863978  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:14.869428  9632 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:14.864164  9600 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:14.865912  9634 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260808 01:51:14.867482  9630 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260808 01:51:14.867342  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:14.872392  9636 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 9: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:14.872537  9631 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:51:14.879488  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:14.880817  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:14.879851  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:14.883477  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:14.884120  9636 raft_consensus.cc:2376] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10. Re-sending same reply.
I20260808 01:51:14.886001  9636 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260808 01:51:14.900368  9636 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 01:51:14.898285  9632 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:51:14.907698  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:14.909677  9636 raft_consensus.cc:2376] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8. Re-sending same reply.
I20260808 01:51:14.917819  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:14.912945 13955 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:14.881645 13701 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260808 01:51:14.937064  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:14.957855  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:14.959339  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:14.969471  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:14.971741  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:15.013008  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:15.023779  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:15.029273  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:15.030448  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:15.060364  9632 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: [9.4-9.4]
I20260808 01:51:15.029281 13953 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:15.068060  9632 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: [6.4-9.5]
I20260808 01:51:15.081215  9632 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260808 01:51:15.053738 13950 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:15.065889 13953 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:15.108150  9480 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260808 01:51:15.114568 13889 consensus_queue.cc:1059] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:15.118292 13574 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:15.125370 13754 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:15.123332 13950 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:15.125522 13953 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:15.144146  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2195) took 1084 ms. Trace:
I20260808 01:51:15.144752  9633 rpcz_store.cc:276] 0808 01:51:14.059830 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:14.060035 (+   205us) service_pool.cc:224] Handling call
0808 01:51:14.098713 (+ 38678us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:14.105237 (+  6524us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:14.105274 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:14.117806 (+ 12532us) log.cc:844] Serialized 26 byte log entry
0808 01:51:14.963851 (+846045us) spinlock_profiling.cc:243] Waited 7.97 ms on lock 0x7b180015dd20. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:51:15.122126 (+158275us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:51:15.122173 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:15.122272 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:15.143721 (+ 21449us) raft_consensus.cc:1685] finished
0808 01:51:15.143748 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:15.144051 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19339,"raft.queue_time_us":145,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":4879,"spinlock_wait_cycles":18327424,"thread_start_us":158045,"threads_started":1,"wal-append.queue_time_us":1015336}
I20260808 01:51:15.163234  9638 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:15.166963  9638 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260808 01:51:15.202503 13574 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:15.208659  9629 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260808 01:51:15.210278 13880 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.746s	user 0.001s	sys 0.006s
W20260808 01:51:15.218971  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:15.219807 13917 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:15.220435  9481 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260808 01:51:15.220407 13917 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:15.223742 13886 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:51:15.226078 13651 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:15.228956 13917 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:15.234329  9473 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:51:15.240921  9473 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:15.242298  9473 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:15.243953  9473 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:15.250029 13754 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:15.250934 13754 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:51:15.275676  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:15.292836 13754 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:15.300711  9473 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260808 01:51:15.301455  9473 pending_rounds.cc:85] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 3
I20260808 01:51:15.301831  9473 pending_rounds.cc:107] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260808 01:51:15.302127  9473 raft_consensus.cc:2889] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:51:15.302951 13729 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260808 01:51:15.304883 13754 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:15.311852 13421 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:15.316412  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2459) took 2241 ms. Trace:
I20260808 01:51:15.317806  9473 rpcz_store.cc:276] 0808 01:51:13.075417 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:13.683742 (+608325us) service_pool.cc:224] Handling call
0808 01:51:13.684108 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:15.234154 (+1550046us) spinlock_profiling.cc:243] Waited 19.5 ms on lock 0x7b54001282a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490b1a0 00007f5d449955a0 00007f5d44996611 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c
0808 01:51:15.302735 (+ 68581us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:15.316328 (+ 13593us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":44757248}
W20260808 01:51:15.323185  9475 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.026s	sys 0.038s
W20260808 01:51:15.323663  9475 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.027s	sys 0.038s
I20260808 01:51:15.329777 13889 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:51:15.336413 13891 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:15.340000 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:15.342018 13891 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:15.342698 13891 raft_consensus.cc:697] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:15.348873 13923 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 01:51:15.354144 13891 consensus_queue.cc:237] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:15.389662  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.891s	user 0.007s	sys 0.005s
W20260808 01:51:15.390200  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.892s	user 0.007s	sys 0.005s
W20260808 01:51:15.391412  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 897480us
I20260808 01:51:15.402500  9444 leader_election.cc:400] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10.
I20260808 01:51:15.402964  9444 leader_election.cc:403] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260808 01:51:15.404363  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:15.404381 13421 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10.
I20260808 01:51:15.405818 13421 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:51:15.406541  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:15.407780  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:15.407608 13785 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:15.411150 13421 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:15.411679 13421 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:15.412132 13421 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:51:15.414512  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:15.414808 13785 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:15.418407 13785 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:15.429425 13421 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:15.431926  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:15.436636 13811 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:15.437271  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:15.437733 13954 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:15.438828  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:15.438858 13729 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:15.439677 13729 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:15.434919  9475 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260808 01:51:15.437297 13811 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:15.442351  9475 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260808 01:51:15.442451 13729 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:15.443063 13811 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:15.444572  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:15.445358  9481 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:15.440340 13725 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:15.440884  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:15.448975  9480 raft_consensus.cc:2393] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 6: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:15.449189  9472 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:15.450219  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:15.451303  9481 raft_consensus.cc:2368] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11.
I20260808 01:51:15.452169 13627 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:15.452809 13627 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:15.453572  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:15.444413 13955 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:51:15.448148  9477 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260808 01:51:15.456106 13421 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:15.456110 13627 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:15.456847 13785 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:15.458187  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:15.479588 13888 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:15.480859  9480 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:15.482625  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:15.484874  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:15.486966 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:15.493268  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:15.497417  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:15.508965  9601 leader_election.cc:400] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11.
I20260808 01:51:15.508865  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:15.509428  9601 leader_election.cc:403] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260808 01:51:15.509910  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:15.520222  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:15.521196  9481 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: [8.4-10.5]
W20260808 01:51:15.521359  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:15.485600 13954 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:15.500877 13725 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:15.526114  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:15.513525 13704 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:15.511557 13421 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election won for term 12
W20260808 01:51:15.554201 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 56146.7us
I20260808 01:51:15.511185 13893 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:51:15.545389 13729 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:15.561324 13729 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:15.562016 13729 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:15.510733 13917 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.049s
I20260808 01:51:15.524117 13811 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:15.563191 13917 consensus_queue.cc:1243] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260808 01:51:15.508242 13875 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.047s
I20260808 01:51:15.510579 13924 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260808 01:51:15.523654 13949 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.039s
I20260808 01:51:15.545688 13953 consensus_queue.cc:1059] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260808 01:51:15.545450 13785 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.020s
I20260808 01:51:15.555488 13993 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:15.569068 13893 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:15.589121 13993 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:15.595489 13893 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:15.602756 13993 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:15.597596 13954 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:15.617890  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2450) took 2571 ms. Trace:
I20260808 01:51:15.618388  9474 rpcz_store.cc:276] 0808 01:51:13.046988 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:13.528772 (+481784us) service_pool.cc:224] Handling call
0808 01:51:13.550538 (+ 21766us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:13.618268 (+ 67730us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:15.617604 (+1999336us) spinlock_profiling.cc:243] Waited 65 ms on lock 0x7b54002dbed4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:15.617824 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":149610240}
I20260808 01:51:15.638557 13811 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:15.684960  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 165199us
W20260808 01:51:15.690994  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:15.677789 13391 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:15.697770  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 148148us
I20260808 01:51:15.695247  9524 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:15.732534 10364 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260808 01:51:15.740459  9444 connection.cc:256] RPC call timeout handler was delayed by -2.0079689830017742s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1956s for a response
W20260808 01:51:15.741036  9444 connection.cc:256] RPC call timeout handler was delayed by -1.9764689829898998s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19910000000000003s for a response
W20260808 01:51:15.741389  9444 connection.cc:256] RPC call timeout handler was delayed by -1.9584689830080606s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20110000000000003s for a response
W20260808 01:51:15.741724  9444 connection.cc:256] RPC call timeout handler was delayed by -1.9035689829906914s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20720000000000002s for a response
W20260808 01:51:15.742039  9444 connection.cc:256] RPC call timeout handler was delayed by -1.8981689829961397s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20779999999999998s for a response
I20260808 01:51:15.743067  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:15.702577 13907 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:15.750068  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:15.747366  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:15.752877  9631 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:15.757682  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:15.698539 13875 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.047s
I20260808 01:51:15.769475  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:15.698621 13391 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:15.767285  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:15.755564 13907 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:15.810653  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:15.808861  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:15.814364  9631 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:15.814844  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:15.717928 13811 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:15.819619  9636 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:15.763124 13725 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:15.817652 13907 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:15.872270  9554 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:15.818419 13421 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:15.866976 14005 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:15.975775  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:15.861881 13893 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:15.997779  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:15.998754  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:15.932420 14011 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:15.954253 13953 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:16.068423  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:15.977355 14005 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:16.065366 13935 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:16.001479 13725 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:16.091563 14000 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
W20260808 01:51:16.081962 13999 log.cc:927] Time spent T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.345s	user 0.002s	sys 0.000s
I20260808 01:51:16.011610 14011 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:16.059006 13953 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:16.068640 13421 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:16.085006 13391 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:16.091563 14007 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.227s	user 0.000s	sys 0.001s
W20260808 01:51:16.129123 13888 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.628s	user 0.013s	sys 0.002s
W20260808 01:51:16.136878 14020 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
W20260808 01:51:16.138084 14010 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
W20260808 01:51:16.138247 13729 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.576s	user 0.008s	sys 0.004s
I20260808 01:51:16.093086 13935 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:16.147228  9475 pending_rounds.cc:85] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 3
I20260808 01:51:16.154083  9475 pending_rounds.cc:107] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260808 01:51:16.154019  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:16.154613  9475 raft_consensus.cc:2889] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:16.104483 13993 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:16.157713  9558 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:16.160820  9554 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
W20260808 01:51:16.143158 14008 log.cc:927] Time spent T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.260s	user 0.001s	sys 0.000s
I20260808 01:51:16.171327  9558 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:16.169761  9634 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:16.144243 14011 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:16.180801  9479 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: []
W20260808 01:51:16.194288  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 451744us
W20260808 01:51:16.195127  9444 connection.cc:256] RPC call timeout handler was delayed by -1.8810689829988405s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2097s for a response
W20260808 01:51:16.195561  9444 connection.cc:256] RPC call timeout handler was delayed by -1.7793689829995856s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.221s for a response
I20260808 01:51:16.197259  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:16.138654 13729 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:16.160118 13888 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
W20260808 01:51:16.208586  9444 connection.cc:256] RPC call timeout handler was delayed by -1.4463689830154181s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.258s for a response
I20260808 01:51:16.211560  9560 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:16.211786 14018 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:16.197144 13897 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:16.191988 13886 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:16.200707 13878 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:16.220003  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:16.207667 13945 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:16.242486  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:16.251448  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:16.252185  9560 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:16.218240 14005 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:16.219408 13811 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:16.262036  9550 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
W20260808 01:51:16.268958  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 54438.6us
I20260808 01:51:16.243336 13878 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:16.285591  9560 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:16.288877  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:16.289198  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:16.267887 13704 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.710s	user 0.017s	sys 0.000s
I20260808 01:51:16.214277 14018 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:16.246551 13897 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:16.295398  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:16.225692 13886 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:16.298286 14030 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:16.304435 13878 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:16.304759 14030 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:16.306869 14005 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:16.308077 13935 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:16.308208 14030 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:16.292474 14027 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20260808 01:51:16.295367 13704 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11.
I20260808 01:51:16.315399 13729 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:16.318032 14018 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:16.319031  9556 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:16.320135 13897 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:16.288626 14032 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260808 01:51:16.304471 13909 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:51:16.325181 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:16.265327 13811 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:16.313378 13893 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:16.329349 13811 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:16.251106 13945 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:16.331440  9556 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:16.331554 13909 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:51:16.332319 13945 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:16.334250 13897 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:51:16.337489  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2182) took 1469 ms. Trace:
I20260808 01:51:16.338042  9634 rpcz_store.cc:276] 0808 01:51:14.868435 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:14.880600 (+ 12165us) service_pool.cc:224] Handling call
0808 01:51:16.337267 (+1456667us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54000db454. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:16.337397 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2871815424}
I20260808 01:51:16.273696 13953 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:16.339223  9521 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:16.493634  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 154469us
I20260808 01:51:16.494529  9521 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:16.496815  9521 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:16.339797 13907 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:16.342461 13932 raft_consensus.cc:3055] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:16.751744 13932 raft_consensus.cc:740] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:16.344208 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:16.342197  9633 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:16.288451  9560 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:32922
I20260808 01:51:16.761752  9560 raft_consensus.cc:606] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:16.762256  9560 raft_consensus.cc:614] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:16.763581  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:16.401682  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 159282us
I20260808 01:51:16.765125  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:16.403327 14006 raft_consensus.cc:2764] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260808 01:51:16.408684 13997 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:16.767638 13997 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:16.770200 13997 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:16.409353  9475 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260808 01:51:16.771656  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:16.766173  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:16.773142  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:16.411518  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:16.412643  9473 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260808 01:51:16.774025 13935 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:16.778025 13935 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:16.778393  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:16.780385  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:16.781280 13935 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:16.781987 13935 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:16.414357  9479 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260808 01:51:16.416394 13949 raft_consensus.cc:3055] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:16.787498 13949 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:16.788684 13949 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:16.789891 13949 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:16.431531  9551 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260808 01:51:16.424013  9472 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:16.792372  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:16.793718  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:16.438699 13950 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:16.487181 13905 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:16.488426  9558 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260808 01:51:16.486188 13897 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:16.489495  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:16.491829  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:16.498099 14027 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:16.489938  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:16.520185 13967 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.435s	user 0.000s	sys 0.014s
I20260808 01:51:16.526000 13878 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:51:16.526747 14005 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:16.528165  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:16.528311 14050 consensus_queue.cc:1059] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260808 01:51:16.560235  9554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 2670) took 1994 ms (client timeout 1787 ms). Trace:
I20260808 01:51:16.563148  9550 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260808 01:51:16.604277  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:16.624867  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:16.639091  9555 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:16.641932 13907 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:16.645092  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:16.648983  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 469) took 1790 ms. Trace:
I20260808 01:51:16.500416  9521 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:16.677285  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:16.685570 10361 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:16.687151 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:16.688220 13993 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:16.688683 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:16.690635 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:16.695744  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:16.697160 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 53659us
I20260808 01:51:16.701735 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:16.706640 13886 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:16.892489 13945 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:51:16.893390 10368 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:16.752904 13932 consensus_queue.cc:260] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:16.893980 13752 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:16.894447 13932 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:16.894338 14030 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:16.894932  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:16.755162 10364 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:16.766911 13951 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:16.767904  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:16.778196 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:16.896823  9476 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:51:16.498732 14024 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:16.892990 13905 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:16.775514  9477 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:51:16.974560  9481 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:16.898684  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:16.795172 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:16.774636  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:16.775171  9480 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260808 01:51:16.978005  9554 rpcz_store.cc:269] 0808 01:51:14.565833 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:14.565957 (+   124us) service_pool.cc:224] Handling call
0808 01:51:16.559940 (+1993983us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b540008a7d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:16.560121 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2938942208}
I20260808 01:51:16.857650  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:16.980098  9552 raft_consensus.cc:2376] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7. Re-sending same reply.
I20260808 01:51:16.773821  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:16.983626  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:16.896785  9556 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:16.987949  9475 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:16.994977  9558 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 9: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:16.794595  9551 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 13: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:16.966456  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 477) took 2099 ms. Trace:
I20260808 01:51:16.997207  9556 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:16.868189 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:16.997788  9630 rpcz_store.cc:276] 0808 01:51:14.867249 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:14.883206 (+ 15957us) service_pool.cc:224] Handling call
0808 01:51:14.883579 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:14.912417 (+ 28838us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:14.912456 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:14.913105 (+   649us) log.cc:844] Serialized 26 byte log entry
0808 01:51:15.027342 (+114237us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:51:15.027418 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:15.027517 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:16.519488 (+1491971us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:16.895831 (+376343us) raft_consensus.cc:1685] finished
0808 01:51:16.895860 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:16.966351 (+ 70491us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24376,"raft.queue_time_us":29524,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":4570,"spinlock_wait_cycles":2698112,"thread_start_us":113369,"threads_started":1,"wal-append.queue_time_us":148849}
I20260808 01:51:16.998901  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:16.839541 13813 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:17.007362  9559 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 12: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:17.038848  9478 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9. Ops: []
W20260808 01:51:17.039608  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 182423us
I20260808 01:51:17.042590  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:17.038893  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:16.748597  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 57952.6us
I20260808 01:51:17.055863 10332 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:17.058995  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:17.060336  9551 raft_consensus.cc:2410] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260808 01:51:17.061370  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:17.062319  9560 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:16.896049  9637 rpcz_store.cc:276] 0808 01:51:14.858284 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:14.858418 (+   134us) service_pool.cc:224] Handling call
0808 01:51:14.858802 (+   384us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:14.868614 (+  9812us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:14.868642 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:14.869545 (+   903us) log.cc:844] Serialized 52 byte log entry
0808 01:51:14.991314 (+121769us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:16.501367 (+1510053us) spinlock_profiling.cc:243] Waited 3.44 ms on lock 0x7b18000fc0a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4453a6ac 00007f5d4186fe3b 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d41889e05 00007f5d483aada7 00007f5d4499dd87 00007f5d449ad49f
0808 01:51:16.501551 (+   184us) log.cc:844] Serialized 14 byte log entry
0808 01:51:16.506147 (+  4596us) log.cc:844] Serialized 14 byte log entry
0808 01:51:16.507016 (+   869us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:16.648449 (+141433us) spinlock_profiling.cc:243] Waited 2.59 ms on lock 0x7b5400339354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:16.648525 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:16.648597 (+    72us) raft_consensus.cc:1685] finished
0808 01:51:16.648616 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:16.648867 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11812,"prepare.queue_time_us":85906,"prepare.run_cpu_time_us":122,"prepare.run_wall_time_us":1790,"raft.queue_time_us":7386,"raft.run_cpu_time_us":510,"raft.run_wall_time_us":30647,"spinlock_wait_cycles":13864960,"thread_start_us":138254,"threads_started":4,"wal-append.queue_time_us":128059,"wal-append.run_cpu_time_us":2575,"wal-append.run_wall_time_us":1070577}
I20260808 01:51:17.069326  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:17.074724  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:17.075879  9554 raft_consensus.cc:2376] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7. Re-sending same reply.
I20260808 01:51:17.077839  9554 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260808 01:51:16.829387 13926 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:17.079876  9552 raft_consensus.cc:1217] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Deduplicated request from leader. Original: 5.3->[7.4-7.4]   Dedup: 7.4->[]
I20260808 01:51:17.088014  9552 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
W20260808 01:51:17.017174  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 516792us
I20260808 01:51:17.090476  9557 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:16.894536 13752 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:16.966992 13701 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:16.824381 14014 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260808 01:51:17.092044  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 49594.1us
I20260808 01:51:17.084048  9558 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260808 01:51:17.094236  9554 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260808 01:51:16.776989 13888 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:16.834041 13467 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:17.007273 13945 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:16.776684 13704 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:16.966305 13924 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:17.099949  9474 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:17.093853 13886 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.007s
I20260808 01:51:16.776829 13391 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:16.775336 13955 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:16.782582 13935 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.108057  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:16.983687 13951 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.069469 13909 raft_consensus.cc:3055] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:16.860878 13790 consensus_queue.cc:1059] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.241s
I20260808 01:51:17.081176 13907 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:51:17.127295  9550 raft_consensus.cc:3055] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:17.160944  9550 raft_consensus.cc:740] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:17.102072 13421 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:16.966516 13897 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.088354 14011 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:51:17.085588  9481 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: []
I20260808 01:51:17.136742  9474 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260808 01:51:17.117635  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:17.068784 13923 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.088s
I20260808 01:51:17.112219  9523 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:17.102480 13725 raft_consensus.cc:2764] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election decision vote started in defunct term 11: won
I20260808 01:51:17.015905 14005 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.102903 13926 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.078857 13813 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.155925 13909 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:17.115365 13467 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:17.016140 14027 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:17.170162 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 115923us
I20260808 01:51:17.141686 13945 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:17.199867 14063 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260808 01:51:17.183295  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:17.113427 13701 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.184232  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:17.116254 13889 consensus_queue.cc:1059] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.041s
I20260808 01:51:17.131999 13844 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:17.141104 13704 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:17.189256  9633 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 01:51:17.191080  9601 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:17.240875  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50040.1us
W20260808 01:51:17.192931  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 285587us
I20260808 01:51:17.183851  9550 consensus_queue.cc:260] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:17.008047 13950 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.195829  9560 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:17.121074 13888 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:17.197382  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 79874.6us
W20260808 01:51:17.218355 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:17.178002  9557 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:17.219327  9600 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:17.143959 13391 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.208105 14050 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:17.206317 13930 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:17.250592  9552 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260808 01:51:17.180794 13752 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.147583 13955 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.215047 14073 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:17.220695 14074 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:51:17.180299 13421 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:17.140112 13924 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.332938  9630 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260808 01:51:17.353902  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 112130us
W20260808 01:51:17.352638  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 240515us
I20260808 01:51:17.321352 13930 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:17.363440  9523 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:17.324589 14060 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:17.322243 14050 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:17.244887 13844 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:17.383433  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 140989us
I20260808 01:51:17.298086 13894 consensus_queue.cc:1059] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.116s
W20260808 01:51:17.395980  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:17.383162  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 163947us
I20260808 01:51:17.397861  9599 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:17.382510 13930 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:17.394364 13752 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:17.394160 14050 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:17.427389  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 319441us
I20260808 01:51:17.425212 13888 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:17.425657 13701 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:17.428725  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:17.429701 14065 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.279s	user 0.001s	sys 0.000s
I20260808 01:51:17.426626  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 643) took 1018 ms. Trace:
I20260808 01:51:17.430910 14018 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:17.431792 14018 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:51:17.431211  9473 rpcz_store.cc:276] 0808 01:51:16.408265 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:16.781665 (+373400us) service_pool.cc:224] Handling call
0808 01:51:16.782021 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:16.782266 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 10
0808 01:51:16.811088 (+ 28822us) log.cc:844] Serialized 14 byte log entry
0808 01:51:17.426282 (+615194us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:17.426338 (+    56us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:51:17.426385 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:17.426449 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:17.426566 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":301335,"prepare.queue_time_us":28099,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":301728,"raft.run_cpu_time_us":51,"raft.run_wall_time_us":52,"thread_start_us":313440,"threads_started":1,"wal-append.queue_time_us":368972}
W20260808 01:51:17.433643  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:17.434872 13893 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:17.435424 13893 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:17.436517 14014 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:17.437103 14014 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.437839 13893 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:17.439447 13924 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:17.430935 14062 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260808 01:51:17.415669 14076 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260808 01:51:17.396760 13905 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:17.430238  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:17.332010 13909 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.415819 14070 raft_consensus.cc:3055] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:17.477407 14070 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:17.478638 14070 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.480430 14070 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:17.463702 13955 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:17.483217 14005 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:17.484100 14005 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260808 01:51:17.364168 13467 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.486227 13467 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:17.393743 14060 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:17.489801 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:17.492602 14075 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260808 01:51:17.492942  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 95233.5us
I20260808 01:51:17.493223 14075 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:17.493698 14075 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:17.493822 14068 consensus_queue.cc:1059] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:51:17.494755 13907 consensus_queue.cc:1059] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260808 01:51:17.496793 13888 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:17.497634 13888 raft_consensus.cc:3055] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:17.498046 13888 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:17.499455 13888 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.499998 13844 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:17.501180 14014 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:17.504269  9599 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:17.505890 14071 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:17.506525 14071 raft_consensus.cc:697] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:17.507522  9599 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:17.507606 14071 consensus_queue.cc:237] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.508884 13953 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:17.509547 13953 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:17.510749 13704 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:17.510591 13953 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:17.512672 13421 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:17.513432 13897 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:17.514554 13951 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:17.515815  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 85661.5us
I20260808 01:51:17.516714  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:17.358309 14073 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:17.524420 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 468616us
W20260808 01:51:17.526662  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 165948us
W20260808 01:51:17.528369  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 223596us
I20260808 01:51:17.532114  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:17.535713  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:17.536444  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:17.537068  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:17.540158  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:17.542822  9481 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:17.543474  9472 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 01:51:17.531432 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:17.545256 14069 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260808 01:51:17.527661 14067 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.118s	user 0.003s	sys 0.001s
I20260808 01:51:17.547401 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:17.472955 13935 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:17.558709  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:17.558575  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:17.563714  9477 raft_consensus.cc:2393] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:17.570411  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:17.527076 14078 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:17.611290  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:17.547375 14018 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:17.552282 14014 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:17.618731  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:17.620436  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:17.622130  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:17.624012  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:17.624212  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:17.624867  9557 raft_consensus.cc:2393] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 6: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:17.553005 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:17.617014  9601 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:17.627192  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:17.629654  9601 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:17.625016  9551 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:17.559875 14069 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:51:17.564584 13813 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:17.619952  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:17.641404  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:17.643386  9601 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:17.613232 13924 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260808 01:51:17.613229 13725 consensus_queue.cc:1059] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260808 01:51:17.564425 13897 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.071s
I20260808 01:51:17.613015 14017 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:17.619737 14011 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:17.646986 14077 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:17.561415 13421 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.078s
I20260808 01:51:17.657974  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:17.568562 13886 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.076s
I20260808 01:51:17.609742 13997 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.075s
I20260808 01:51:17.689844  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:17.639353 13889 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.087s
I20260808 01:51:17.559756 14072 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.108s
I20260808 01:51:17.637197 14027 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:17.651937 14014 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:17.656363 13926 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:17.660130 14078 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:51:17.648080 14080 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:17.615486 14052 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.110s
I20260808 01:51:17.724661 14060 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:17.702847 13813 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:17.735185 13875 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:17.613041 13955 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.123s
I20260808 01:51:17.734185 13950 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:17.736734 14011 raft_consensus.cc:697] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:17.757179 14076 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:17.820972 14014 raft_consensus.cc:697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:17.811213 14078 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:17.744565 14082 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.193s
I20260808 01:51:17.792695 14050 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.905170 13813 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:17.908200 13875 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:17.929425 14078 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:18.053153 14073 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:18.107826 14089 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.196s	user 0.001s	sys 0.000s
W20260808 01:51:18.110945 13467 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.624s	user 0.014s	sys 0.003s
W20260808 01:51:18.109818 14024 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.096s	user 0.005s	sys 0.010s
I20260808 01:51:17.923409 14076 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.136124 13813 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:18.126356 14024 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:18.114035 14014 consensus_queue.cc:237] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.103077 14011 consensus_queue.cc:237] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:18.156836 14070 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.676s	user 0.015s	sys 0.000s
W20260808 01:51:18.160238 13752 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.718s	user 0.012s	sys 0.000s
I20260808 01:51:18.170292 13875 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:18.171876 13932 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.276s	user 0.017s	sys 0.000s
I20260808 01:51:18.173753 14050 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:18.206269 14075 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.712s	user 0.012s	sys 0.000s
I20260808 01:51:18.213084 14075 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:18.180296 13949 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.378s	user 0.007s	sys 0.011s
I20260808 01:51:18.178995 14076 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:18.219875 13953 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.705s	user 0.000s	sys 0.010s
W20260808 01:51:18.281566 13953 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.001s	sys 0.011s
W20260808 01:51:18.257346 13930 consensus_meta.cc:287] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.841s	user 0.008s	sys 0.007s
I20260808 01:51:18.282249 14075 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:18.282580 13930 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:18.300501 13930 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:18.287756 14078 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.331238 14078 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:17.802487  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.018s	sys 0.033s
W20260808 01:51:18.411630  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.170s	user 0.018s	sys 0.033s
W20260808 01:51:17.802410  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 385648us
W20260808 01:51:18.413905  9635 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.851s	user 0.025s	sys 0.049s
W20260808 01:51:18.414611  9635 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.856s	user 0.026s	sys 0.049s
W20260808 01:51:18.415063  9523 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.772s	user 0.000s	sys 0.015s
W20260808 01:51:18.415771  9523 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.773s	user 0.000s	sys 0.015s
I20260808 01:51:18.417055  9632 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260808 01:51:18.420622  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 777062us
W20260808 01:51:17.694494  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46480.8us
I20260808 01:51:18.425030 10360 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: []
W20260808 01:51:18.427785  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:18.428164 13790 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260808 01:51:18.410179  9520 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.768s	user 0.004s	sys 0.010s
W20260808 01:51:18.429826  9520 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.788s	user 0.004s	sys 0.011s
W20260808 01:51:17.906291 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 338248us
I20260808 01:51:18.435003  9556 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: []
W20260808 01:51:18.441069  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:18.443136 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:18.468559 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:18.480325  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 537) took 1983 ms. Trace:
I20260808 01:51:18.485986 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:18.492154 14076 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260808 01:51:18.499460 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:18.500993  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:18.501813 14014 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:18.503255 14014 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.506889  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 541) took 2012 ms. Trace:
I20260808 01:51:18.506889 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:18.507153 13950 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:18.510578 13932 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:18.515535 14080 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:18.523061  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2212) took 1336 ms. Trace:
I20260808 01:51:18.508297  9552 rpcz_store.cc:276] 0808 01:51:16.494270 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:17.089497 (+595227us) service_pool.cc:224] Handling call
0808 01:51:17.250093 (+160596us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:17.321922 (+ 71829us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:18.506314 (+1184392us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b54001ac550. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:18.506642 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":368318720}
I20260808 01:51:18.526513 14052 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:18.525550 14024 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:18.515719 13950 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.481274  9556 rpcz_store.cc:276] 0808 01:51:16.493167 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:17.079439 (+586272us) service_pool.cc:224] Handling call
0808 01:51:18.429365 (+1349926us) spinlock_profiling.cc:243] Waited 1.16 ms on lock 0x7b5800003c98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:51:18.430022 (+   657us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:18.458328 (+ 28306us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:18.476064 (+ 17736us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2669440}
I20260808 01:51:18.517453 14080 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:18.537988 14084 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:18.522332 14050 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:18.539137 14084 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.539384 14050 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.532450  9632 rpcz_store.cc:276] 0808 01:51:17.187159 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:17.187372 (+   213us) service_pool.cc:224] Handling call
0808 01:51:17.190389 (+  3017us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:18.418259 (+1227870us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:18.522478 (+104219us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400337f54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:18.522873 (+   395us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2817355008}
W20260808 01:51:18.544481  9559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2072) took 2054 ms (client timeout 1664 ms). Trace:
I20260808 01:51:18.535322 13932 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:18.545092  9559 rpcz_store.cc:269] 0808 01:51:16.490213 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:16.491528 (+  1315us) service_pool.cc:224] Handling call
0808 01:51:18.543852 (+2052324us) spinlock_profiling.cc:243] Waited 34.6 ms on lock 0x7b5400217fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:18.544184 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":79622912}
I20260808 01:51:18.548238 13932 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:18.540458 14080 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:18.550823 14084 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:18.554316  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:18.557107 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:18.556816 14050 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:18.568333  9582 debug-util.cc:398] Leaking SignalData structure 0x7b08001c8c00 after lost signal to thread 7834
I20260808 01:51:18.572350 14071 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:18.573412 14071 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:18.574023  9582 debug-util.cc:398] Leaking SignalData structure 0x7b08001f95e0 after lost signal to thread 9412
I20260808 01:51:18.557729 14052 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:18.571866  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.881s	user 0.013s	sys 0.033s
W20260808 01:51:18.566330  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 3b8e0e8fda6744ad877514b11658b9d9 (from 52ccf4dcca1843f7bb29e9172e2d4ab4 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:18.582445 14030 raft_consensus.cc:3055] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:18.583233 14030 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:18.583739 14071 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:18.586097 14030 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:18.583248  9582 debug-util.cc:398] Leaking SignalData structure 0x7b08002a3f60 after lost signal to thread 9506
I20260808 01:51:18.579283 10365 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:18.579692  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.964s	user 0.013s	sys 0.034s
W20260808 01:51:18.589224  9582 debug-util.cc:398] Leaking SignalData structure 0x7b08002f77c0 after lost signal to thread 9507
I20260808 01:51:18.592624 14083 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:18.593159 14024 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:18.593824 14083 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:18.592854  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:18.596487 14024 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:18.596747  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:18.592074 14014 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:18.599231 14052 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:18.601312 14078 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:18.602984 14083 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:18.602164 14078 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.602265  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:18.608744 14083 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260808 01:51:18.610539  9582 debug-util.cc:398] Leaking SignalData structure 0x7b08003bde80 after lost signal to thread 9585
I20260808 01:51:18.615362 14078 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:18.616271 13704 raft_consensus.cc:3055] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:18.616673 13704 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:18.617746 13704 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.622769 14078 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:18.626111 14078 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.623358 14011 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:18.628999 14011 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.623194 13704 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:18.619760  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:18.623268  9558 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:32922
I20260808 01:51:18.640009  9558 raft_consensus.cc:606] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:18.640506  9558 raft_consensus.cc:614] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:51:18.642297  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 5cf7e7eadc2347aa84c98413849ebfc9 (from 52ccf4dcca1843f7bb29e9172e2d4ab4 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:18.644907 14078 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:18.646796 13993 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:18.648342  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 486) took 1475 ms. Trace:
I20260808 01:51:18.648963 13993 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.652130 13993 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:18.631659 13790 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:18.644796 10360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 552) took 1204 ms. Trace:
I20260808 01:51:18.656158 10360 rpcz_store.cc:276] 0808 01:51:17.440699 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:17.440881 (+   182us) service_pool.cc:224] Handling call
0808 01:51:17.441286 (+   405us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:18.425891 (+984605us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:18.644431 (+218540us) spinlock_profiling.cc:243] Waited 983 ms on lock 0x7b54003200d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:18.644700 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2260206464}
W20260808 01:51:18.633908  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 1.02262e+06us
I20260808 01:51:18.648864  9637 rpcz_store.cc:276] 0808 01:51:17.173103 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:17.173262 (+   159us) service_pool.cc:224] Handling call
0808 01:51:17.173657 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:17.413052 (+239395us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:17.413085 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:17.413130 (+    45us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:17.413163 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:18.647874 (+1234711us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b54003faf54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:18.647952 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:18.648152 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":549969792}
I20260808 01:51:18.647467  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:18.663707  9477 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:18.674369  9558 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:32922
I20260808 01:51:18.646818 13391 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:18.756880  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:18.647791 13950 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:18.764550  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:18.771106  9558 raft_consensus.cc:606] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:18.773836  9558 raft_consensus.cc:614] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:51:18.779006 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 202005us
I20260808 01:51:18.784332  9559 raft_consensus.cc:2410] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 11 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260808 01:51:18.785612 14097 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:51:18.786866 13391 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:18.787871 13391 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:18.788877  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 111964us
I20260808 01:51:18.789103 13790 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:18.792135 14097 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:18.794334  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:18.788347 13945 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:18.795572  9559 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:18.795815 14097 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:18.800688 14097 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:18.797896 14078 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:18.806325  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:18.807210 14078 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.808252  9474 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: []
I20260808 01:51:18.797169 13945 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:18.809384 13945 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.802053 13391 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:18.803200  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:18.807210  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:18.815621 13945 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:18.815519  9481 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:51:18.781104 13889 consensus_queue.cc:1059] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260808 01:51:18.819507 13889 raft_consensus.cc:3055] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:18.819792 10365 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:18.814555 14078 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:18.818557 14030 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:18.828778 13923 raft_consensus.cc:3055] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:18.829417  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:18.831949 14104 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260808 01:51:18.832410  9557 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260808 01:51:18.850492  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2294) took 2330 ms. Trace:
I20260808 01:51:18.851053  9479 rpcz_store.cc:276] 0808 01:51:16.519732 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:16.785409 (+265677us) service_pool.cc:224] Handling call
0808 01:51:16.785922 (+   513us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:16.786391 (+   469us) raft_consensus.cc:1534] Early marking committed up to index 4
0808 01:51:16.816941 (+ 30550us) log.cc:844] Serialized 14 byte log entry
0808 01:51:17.078810 (+261869us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:18.537926 (+1459116us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b2c00134ff8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:51:18.539374 (+  1448us) log.cc:844] Serialized 52 byte log entry
0808 01:51:18.566914 (+ 27540us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:51:18.613538 (+ 46624us) log.cc:844] Serialized 14 byte log entry
0808 01:51:18.613860 (+   322us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:18.622935 (+  9075us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:18.850021 (+227086us) raft_consensus.cc:1685] finished
0808 01:51:18.850047 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:18.850280 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":387549,"prepare.queue_time_us":46515,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":105,"raft.queue_time_us":113929,"raft.run_cpu_time_us":583,"raft.run_wall_time_us":33597,"spinlock_wait_cycles":4076800,"thread_start_us":188906,"threads_started":2,"wal-append.queue_time_us":355880,"wal-append.run_cpu_time_us":3025,"wal-append.run_wall_time_us":1329264}
I20260808 01:51:18.820165 13889 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:18.854611  9478 raft_consensus.cc:1217] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Deduplicated request from leader. Original: 7.4->[9.5-9.5]   Dedup: 9.5->[]
I20260808 01:51:18.855479  9478 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260808 01:51:18.855747 13889 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:18.858632  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet c85ff498df484fc6ac0913aa57aa868b (from 52ccf4dcca1843f7bb29e9172e2d4ab4 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
W20260808 01:51:18.847591  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 234861us
W20260808 01:51:18.855020  9582 debug-util.cc:398] Leaking SignalData structure 0x7b080033cda0 after lost signal to thread 9663
I20260808 01:51:18.855327 13909 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:18.846108  9558 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:18.845218 13790 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:18.850553  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:18.866888  9582 debug-util.cc:398] Leaking SignalData structure 0x7b08001ea140 after lost signal to thread 9664
W20260808 01:51:18.865269  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 148957us
I20260808 01:51:18.874953 13701 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:18.875229 14060 raft_consensus.cc:3055] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:51:18.875722 14060 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:18.875938 13701 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.877187 14060 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.879103  9637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:18.879838  9637 raft_consensus.cc:606] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:18.880249  9637 raft_consensus.cc:614] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:18.883185 14077 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:18.884502 14018 raft_consensus.cc:3055] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Stepping down as leader of term 9
W20260808 01:51:18.884871  9582 debug-util.cc:398] Leaking SignalData structure 0x7b08001b4980 after lost signal to thread 10393
I20260808 01:51:18.885020 14018 raft_consensus.cc:740] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:18.885757  9582 debug-util.cc:398] Leaking SignalData structure 0x7b0800309040 after lost signal to thread 10394
I20260808 01:51:18.887344 14018 consensus_queue.cc:260] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.889314 14018 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:18.891095  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:18.883198 13844 raft_consensus.cc:3055] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:18.892117 13844 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:18.893613 13844 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:18.894402 13945 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:18.895211 13945 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:18.895845 13945 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:18.896209 13844 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:18.897643  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:18.898980 13701 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:18.884176 14077 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.869763  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:18.878475 14060 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:18.829540 13923 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:18.926074 13923 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:18.927716  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:18.928617  9558 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:18.929427  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2491) took 2288 ms. Trace:
I20260808 01:51:18.929870  9478 rpcz_store.cc:276] 0808 01:51:16.640915 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:17.068883 (+427968us) service_pool.cc:224] Handling call
0808 01:51:18.854268 (+1785385us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:18.855993 (+  1725us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:18.929163 (+ 73170us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b54002bb1d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:18.929298 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4104895360}
I20260808 01:51:18.930425 13704 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:18.927774 13923 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:18.933135 13704 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:18.931483  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:18.936684  9481 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:18.936926  9637 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:51:18.936954  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:18.886603  9582 debug-util.cc:398] Leaking SignalData structure 0x7b08003cf1c0 after lost signal to thread 13703
I20260808 01:51:18.938289  9557 raft_consensus.cc:2410] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20260808 01:51:18.910205  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet d0db7b84c3b040c788e0a3f66657d912 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:18.906004  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:18.902560  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:18.903015 14050 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
W20260808 01:51:18.917758  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:18.905475 10334 connection.cc:256] RPC call timeout handler was delayed by -1.5356620449747425s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2106s for a response
I20260808 01:51:18.948669 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:18.907145 13888 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:18.906785 14077 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:18.908537 13905 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:18.913082 14080 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.955545 13905 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:18.958031  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:18.959089  9472 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:18.959177  9582 debug-util.cc:398] Leaking SignalData structure 0x7b080039bb80 after lost signal to thread 13736
I20260808 01:51:18.918464  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:18.960682  9476 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:51:18.921029 14011 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:18.965782  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:18.938010  9473 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:51:18.938076  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:18.930936 13790 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:18.935829 13704 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:18.938903  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:51:18.941469 13844 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:18.941862 13889 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:18.929895  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:18.967629  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:18.962224  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:18.968076  9600 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:18.958086 14080 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:18.940282  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:18.983475  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:18.994940  9481 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:18.959350 13905 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:18.976513 13932 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:19.002081  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:18.998307  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.008569  9476 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:18.979521 13790 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:19.012355  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.019692  9481 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:19.021193  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.022886  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.024607 13790 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:19.024892  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.024281  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:18.991624 13844 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:19.030025  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.001863 13932 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:19.031988  9481 raft_consensus.cc:2410] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260808 01:51:19.012671 14080 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:19.034873  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:19.036562  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.037499  9476 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:19.027724  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:19.032537 13932 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:19.050339  9557 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:51:19.045544  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 107213us
I20260808 01:51:18.992072 14011 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260808 01:51:19.039635  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2211) took 1853 ms. Trace:
I20260808 01:51:19.039499 14081 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.009s
I20260808 01:51:19.039801  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.038667  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 488) took 1862 ms. Trace:
W20260808 01:51:19.058051 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 109602us
I20260808 01:51:19.059521 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:19.061676  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:19.065487 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:19.065487 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:19.069156 10363 raft_consensus.cc:2393] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 10: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:19.070402  9550 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:19.070602  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.071017 14011 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:19.071648  9479 raft_consensus.cc:2393] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:19.071612 14011 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:19.073045 14011 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 9.11, Last appended by leader: 11, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:19.075325 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:19.057366  9630 rpcz_store.cc:276] 0808 01:51:17.175959 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:17.176075 (+   116us) service_pool.cc:224] Handling call
0808 01:51:17.176358 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:17.359343 (+182985us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:19.038116 (+1678773us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b540008c854. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:19.038463 (+   347us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":358806400}
I20260808 01:51:19.077659  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:19.042380 13844 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:19.093199  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.094084  9633 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:51:19.094952  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:51:19.084152 14011 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:19.104041  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.099905  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.105296  9559 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:19.108784  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:19.109278  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.054077  9635 rpcz_store.cc:276] 0808 01:51:17.186448 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:17.187553 (+  1105us) service_pool.cc:224] Handling call
0808 01:51:17.207521 (+ 19968us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:17.395016 (+187495us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:17.395054 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:17.395642 (+   588us) log.cc:844] Serialized 26 byte log entry
0808 01:51:17.530349 (+134707us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:18.415450 (+885101us) log.cc:844] Serialized 14 byte log entry
0808 01:51:18.416266 (+   816us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:19.039087 (+622821us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b54001b49d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:19.039155 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:19.039240 (+    85us) raft_consensus.cc:1685] finished
0808 01:51:19.039360 (+   120us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:19.039576 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64144,"prepare.queue_time_us":594054,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":88,"raft.queue_time_us":188062,"raft.run_cpu_time_us":627,"raft.run_wall_time_us":34330,"spinlock_wait_cycles":43267840,"thread_start_us":1127631,"threads_started":3,"wal-append.queue_time_us":349404}
W20260808 01:51:19.123145  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:19.126127  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.125968 14082 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:51:19.127257  9637 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:51:19.127147  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 119941us
I20260808 01:51:19.128603  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:19.129611  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:19.130062 10360 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:19.126319  9630 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
W20260808 01:51:19.132234  9521 leader_election.cc:336] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.535s (SENT)
I20260808 01:51:19.131137  9599 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:19.130091  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:19.133582  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:19.139021  9559 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:19.139573  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:19.139565  9444 connection.cc:256] RPC call timeout handler was delayed by -1.732444333989406s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1513s for a response
W20260808 01:51:19.141157  9444 connection.cc:256] RPC call timeout handler was delayed by -1.5965443339955527s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1664s for a response
W20260808 01:51:19.141520  9444 connection.cc:256] RPC call timeout handler was delayed by -1.5965443339955527s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1664s for a response
W20260808 01:51:19.141788  9444 connection.cc:256] RPC call timeout handler was delayed by -1.5812443339964375s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16810000000000003s for a response
W20260808 01:51:19.142056  9444 connection.cc:256] RPC call timeout handler was delayed by -1.5200443339999765s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17490000000000003s for a response
W20260808 01:51:19.142323  9444 connection.cc:256] RPC call timeout handler was delayed by -1.45254433399532s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1824s for a response
I20260808 01:51:19.142928 13790 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:19.143220 13421 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:19.144747  9521 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:19.143823 13790 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:19.146582  9599 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:19.146198  9600 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:19.147596  9552 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
W20260808 01:51:19.142642  9444 connection.cc:256] RPC call timeout handler was delayed by -1.4471443340007681s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18300000000000002s for a response
W20260808 01:51:19.152309  9444 connection.cc:256] RPC call timeout handler was delayed by -1.4462443339871243s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1831s for a response
W20260808 01:51:19.154064  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1834443339903373s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21230000000000004s for a response
W20260808 01:51:19.156664 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:19.156646  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:19.159876  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.143070 13893 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:19.149272 13790 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:19.167914  9557 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:19.156025 13905 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:51:19.187433 13905 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:19.187929 13905 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:19.171056 14050 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:19.144032 14078 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:19.173578  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:19.191148 14078 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:19.192296  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:19.193305  9554 raft_consensus.cc:2410] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260808 01:51:19.194151  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:19.155474 14077 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:19.195317 14077 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:19.195701 14077 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:19.165702 13421 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:19.195894  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:19.197727 13421 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:19.198313 13421 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:51:19.181130 13893 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:19.199005 14014 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:19.200367 14014 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:19.201311 13893 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:19.186831  9560 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:19.203127 14080 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:19.203809 14080 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:19.205024 14080 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:19.192487 14018 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:19.206996  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:19.173043  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
W20260808 01:51:19.222101  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 92105.8us
I20260808 01:51:19.223488  9559 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:19.223268 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:19.231860  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:19.233007  9556 raft_consensus.cc:2376] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11. Re-sending same reply.
I20260808 01:51:19.235000 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:19.234395  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:19.239107  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 139366us
W20260808 01:51:19.240077  9521 leader_election.cc:336] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.540s (SENT)
I20260808 01:51:19.240471  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:19.243496 14078 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:19.245779  9373 catalog_manager.cc:5697] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 5 to 8, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 8 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260808 01:51:19.248769 10332 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.682s (SENT) [suppressed 14 similar messages]
I20260808 01:51:19.249641  9373 catalog_manager.cc:5697] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.251020 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:19.252873  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.254367  9631 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260808 01:51:19.254554  9556 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:19.255445  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.256454  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.273139  9637 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:19.258786  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:19.261165  9599 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:19.262970  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:19.263280  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:19.263588 13701 consensus_queue.cc:1059] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:51:19.227010  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:19.267066 13893 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:51:19.268637 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:19.228407 10360 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:19.270152 14050 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:19.274784  9638 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:19.281104  9472 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:19.290225  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.290853  9554 raft_consensus.cc:2410] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 11 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 3.
I20260808 01:51:19.295593  9550 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260808 01:51:19.296679  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.298260  9552 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:19.297642  9637 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:19.307154  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:19.308749  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.309562  9635 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:19.299410 13888 raft_consensus.cc:3055] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:19.311208 13888 raft_consensus.cc:740] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:19.312170 13888 consensus_queue.cc:260] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:19.313376 13888 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:19.315081  9629 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:19.305437  9599 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:19.316186  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.316520  9600 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:19.316514 14078 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:19.317255 14078 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:19.317632  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:19.317804  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.318106  9557 raft_consensus.cc:2410] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 11 because replica has last-logged OpId of term: 9 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260808 01:51:19.319103  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:19.319171  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:19.320427 13391 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:19.300724  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.320735  9552 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260808 01:51:19.320839 13391 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:19.321353  9631 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:19.317655  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.322458  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.323024  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.321516 13391 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:19.314532 13905 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:19.317655 14078 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:19.326954 13905 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:19.327131  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.327912  9629 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:19.320881  9557 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260808 01:51:19.330322  9557 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: [9.11-9.11]
I20260808 01:51:19.314921  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 544) took 2819 ms. Trace:
I20260808 01:51:19.331162  9550 rpcz_store.cc:276] 0808 01:51:16.495371 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:17.095580 (+600209us) service_pool.cc:224] Handling call
0808 01:51:17.127124 (+ 31544us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:19.053393 (+1926269us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b54002ce820. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44908dfd 00007f5d448ef584 00007f5d44971827 00007f5d4498a3c7 00007f5d4498bbbe 00007f5d44995f29 00007f5d449967f6 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229
0808 01:51:19.296159 (+242766us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:19.314810 (+ 18651us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":630336768}
I20260808 01:51:19.316205  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:19.334493 13704 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:19.301119 14060 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:19.335111 13704 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:19.335306  9599 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:19.336047  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:19.336369 13844 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:19.336220 13704 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.4, Last appended by leader: 4, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:19.336935  9559 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 12: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:19.335317 14060 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:19.338014  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.339515  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.339666  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.340282  9638 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:19.356300  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:19.310598  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.336858 13844 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:19.344871  9601 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:19.357323 13844 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:19.357481  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:19.358167  9555 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260808 01:51:19.358430  9601 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:19.359297  9550 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260808 01:51:19.358453  9638 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 10: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:19.347445 13894 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:19.360699 13894 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:19.304028 13421 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:19.359887  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:19.304544 13701 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:19.362888 13421 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:19.363278 13701 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:19.363818 13701 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:19.364426  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:19.364516 13421 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:19.366266 13951 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:19.366465 13421 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:19.372293 13951 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:51:19.372778  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:19.372983 13951 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:19.356405  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.375715  9637 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
W20260808 01:51:19.377898  9372 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50064 (request call id 237) took 31710 ms (client timeout 15000 ms). Trace:
W20260808 01:51:19.378801  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 62357.3us
I20260808 01:51:19.380620  9524 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:19.381502  9373 catalog_manager.cc:5697] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 4 to 6, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 6 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.383812  9373 catalog_manager.cc:5697] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.385393 14074 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:19.386037 14074 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:19.332211  9557 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 5. (term mismatch)
I20260808 01:51:19.323721  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.386875  9557 pending_rounds.cc:85] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 4
I20260808 01:51:19.387244  9557 pending_rounds.cc:107] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20260808 01:51:19.387401  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:19.387758  9557 raft_consensus.cc:2889] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:19.346835  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:19.398699  9634 raft_consensus.cc:2410] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260808 01:51:19.399478  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.381551  9524 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:19.367007  9560 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260808 01:51:19.385722  9373 catalog_manager.cc:5697] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.396389  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.403426  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:19.386479 14074 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:19.307160 13945 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:19.411471  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:19.304188 13889 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:19.363077 13950 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:51:19.418256  9523 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:19.386169 13844 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:19.429582  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.423033  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.423631  9481 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:19.432752  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.424125  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.425042  9479 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:19.425987 14098 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:19.426636  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.427320  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.439502  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:19.451551  9559 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:19.371986 13894 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:19.456614  9631 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 12: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:19.457059  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:19.458384  9476 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:19.461300  9479 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:19.479271 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 303538us
I20260808 01:51:19.486586  9477 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:19.516400  9481 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:19.439368 14098 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:19.455085 14050 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:19.451151 13909 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:19.488266  9473 raft_consensus.cc:2393] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 8: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:19.532931  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.533412  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.532981  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.489238  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:19.488189  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:19.530821  9373 catalog_manager.cc:5697] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.527236  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:19.526777 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 278039us
I20260808 01:51:19.535396  9638 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:19.538223  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.534726  9478 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 4.
I20260808 01:51:19.546342 14024 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:19.543022 10334 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:19.464347 14077 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:19.556387  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 360637us
W20260808 01:51:19.559376 14113 log.cc:927] Time spent T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260808 01:51:19.478248 14078 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:19.567534 14078 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:19.566910  9373 catalog_manager.cc:5697] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 6 to 7, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 7 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.529995 14098 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:19.568830 14060 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:19.568706  9373 catalog_manager.cc:5697] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 1 to 6, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 6 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.537590 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:51:19.571146  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 172767us
I20260808 01:51:19.574884  9638 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:19.576498  9629 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260808 01:51:19.578094  9629 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260808 01:51:19.579465  9477 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260808 01:51:19.491799 13889 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:19.528004 13391 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:19.571655 14080 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:19.585947  9631 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:19.570638 13945 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:19.592548  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:19.580587  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:19.621419 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 78503.3us
I20260808 01:51:19.625262 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:19.620930 10365 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260808 01:51:19.620927  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:19.631968  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 52883.4us
I20260808 01:51:19.632288  9480 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:19.630903  9638 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260808 01:51:19.624755  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:19.641274  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 281384us
I20260808 01:51:19.643172  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:19.631608 14082 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:19.656997  9373 catalog_manager.cc:5697] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
W20260808 01:51:19.673169 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 97763.4us
I20260808 01:51:19.630254 14118 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:51:19.608587 14080 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:19.679015  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:19.697153  9631 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9. Ops: []
W20260808 01:51:19.654417  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 206977us
W20260808 01:51:19.696484  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 53642.2us
I20260808 01:51:19.703975  9475 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:19.672104 14082 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:19.720715  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:19.732372  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 152416us
I20260808 01:51:19.734704  9601 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:19.731396  9373 catalog_manager.cc:5697] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 5 to 6, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 6 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
W20260808 01:51:19.710619 13923 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.775s	user 0.010s	sys 0.005s
I20260808 01:51:19.743471  9638 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260808 01:51:19.710491 14082 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:51:19.731474 13932 consensus_meta.cc:287] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.640s	user 0.014s	sys 0.000s
I20260808 01:51:19.708058 13950 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:19.746308  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:19.764426 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 90251.1us
W20260808 01:51:19.378376  9372 rpcz_store.cc:269] 0808 01:50:47.667527 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:47.668323 (+   796us) service_pool.cc:224] Handling call
0808 01:51:19.377600 (+31709277us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:51:10.283127 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:51:10.284017 (+   890us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:51:10.284069 (+    52us) write_op.cc:435] Acquired schema lock
0808 01:51:10.284097 (+    28us) tablet.cc:662] Decoding operations
0808 01:51:10.295161 (+ 11064us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:51:10.328358 (+ 33197us) write_op.cc:641] Partition lock acquired for write op
0808 01:51:10.334874 (+  6516us) tablet.cc:685] Acquiring locks for 17 operations
0808 01:51:10.844370 (+509496us) tablet.cc:701] Row locks acquired
0808 01:51:10.868263 (+ 23893us) write_op.cc:260] PREPARE: finished
0808 01:51:10.928090 (+ 59827us) write_op.cc:270] Start()
0808 01:51:10.956191 (+ 28101us) write_op.cc:276] Timestamp: P: 1786153870894046 usec, L: 0
0808 01:51:10.958723 (+  2532us) op_driver.cc:348] REPLICATION: starting
0808 01:51:10.959608 (+   885us) log.cc:844] Serialized 6740 byte log entry
0808 01:51:12.244853 (+1285245us) spinlock_profiling.cc:243] Waited 18.9 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4186fa5c 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb
0808 01:51:15.143669 (+2898816us) spinlock_profiling.cc:243] Waited 9.23 ms on lock 0x7b1800371260. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4453a6ac 00007f5d4186fe3b 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1
0808 01:51:15.144300 (+   631us) op_driver.cc:464] REPLICATION: finished
0808 01:51:15.170902 (+ 26602us) write_op.cc:301] APPLY: starting
0808 01:51:15.171078 (+   176us) tablet.cc:1367] starting BulkCheckPresence
0808 01:51:15.210968 (+ 39890us) tablet.cc:1370] finished BulkCheckPresence
0808 01:51:15.211013 (+    45us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:51:15.250568 (+ 39555us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:51:15.250966 (+   398us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=34,key_file_lookups=34,delta_file_lookups=34,mrs_lookups=0
0808 01:51:15.251033 (+    67us) write_op.cc:312] APPLY: finished
0808 01:51:15.251501 (+   468us) log.cc:844] Serialized 156 byte log entry
0808 01:51:15.284138 (+ 32637us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:51:15.284359 (+   221us) write_op.cc:454] Released schema lock
0808 01:51:15.284428 (+    69us) write_op.cc:341] FINISH: Updating metrics
0808 01:51:15.297103 (+ 12675us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:51:15.297147 (+    44us) write_op.cc:454] Released schema lock
0808 01:51:15.347655 (+ 50508us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540005c3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4499070a 00007f5d449162f4 00007f5d449162a3 00007f5d44916218 00007f5d449161d3 00007f5d449154bc 00007f5d4493f1f6 00007f5d4492709b 00007f5d4491177d 00007f5d44915155
Metrics: {"mutex_wait_us":544792,"reported_tablets":46,"child_traces":[["op",{"apply.queue_time_us":26188,"apply.run_cpu_time_us":49493,"apply.run_wall_time_us":126119,"cfile_cache_hit":70,"cfile_cache_hit_bytes":75289,"num_ops":17,"prepare.queue_time_us":330204,"prepare.run_cpu_time_us":111357,"prepare.run_wall_time_us":4861016,"raft.queue_time_us":206205,"raft.run_cpu_time_us":20932,"raft.run_wall_time_us":1516011,"replication_time_us":4185345,"spinlock_wait_cycles":3063412992,"thread_start_us":2574231,"threads_started":5,"wal-append.queue_time_us":2428824,"wal-append.run_cpu_time_us":26353,"wal-append.run_wall_time_us":2346763}]]}
W20260808 01:51:19.767203  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 61392.5us
W20260808 01:51:19.760146 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46352.9us
I20260808 01:51:19.774185  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:19.706107 13945 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.035s
I20260808 01:51:19.770882  9373 catalog_manager.cc:5697] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 5 to 8, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 8 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.779246  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:19.733124 14024 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:19.741649 13421 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election won for term 9
W20260808 01:51:19.788822 14121 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260808 01:51:19.789848 14140 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:19.791857 13704 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:19.792415 14132 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.096s
I20260808 01:51:19.793197 14127 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:19.778926  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:19.815363  9555 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:19.782469  9373 catalog_manager.cc:5697] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
W20260808 01:51:19.801542  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:19.746081 13844 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:19.806483  9479 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
W20260808 01:51:19.802656  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 67637.7us
W20260808 01:51:19.809049  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 427533us
W20260808 01:51:19.817893  9521 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.009s	sys 0.036s
W20260808 01:51:19.824569  9521 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.009s	sys 0.037s
W20260808 01:51:19.825122  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 554289us
I20260808 01:51:19.827950  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:19.828648  9476 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:19.836915 13391 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:19.837507  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:19.803215 14024 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:19.855919  9373 catalog_manager.cc:5697] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:19.810215 13894 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:19.814174 14127 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:19.780160 13932 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:19.866314  9559 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:19.815246 13421 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:19.870039 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:19.866621 10332 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:19.872561  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:19.875787  9599 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:19.879166  9552 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:19.879997  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:19.847931 14077 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:19.884796  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:19.893532 14005 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260808 01:51:19.883457 13951 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.510s	user 0.014s	sys 0.000s
I20260808 01:51:19.894229 14005 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:19.894735 13951 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:19.895385 14005 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:19.873755 14127 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:19.897356 13951 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:19.894240 10362 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260808 01:51:19.898131 13951 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 01:51:19.905165 10360 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:19.905668  9632 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:19.872841 13909 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:19.912899  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 73437.5us
W20260808 01:51:19.914083  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:19.891220 13889 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:19.887467 13894 raft_consensus.cc:697] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:19.994555  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:19.924896 13701 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.561s	user 0.011s	sys 0.003s
W20260808 01:51:19.974717  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:19.956256  9552 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
W20260808 01:51:19.958642 13790 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.776s	user 0.009s	sys 0.005s
W20260808 01:51:20.002460  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 267840us
W20260808 01:51:20.008929  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 121986us
I20260808 01:51:19.953557 13909 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:19.995795 14077 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:19.994467 14139 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:20.006835 14135 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.329s	user 0.002s	sys 0.000s
I20260808 01:51:20.027841  9601 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:20.010167 14132 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election won for term 9
W20260808 01:51:20.059588  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 171491us
I20260808 01:51:19.995162 14082 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:20.069214  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 115243us
I20260808 01:51:20.066623  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:20.074198  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:20.001658 13701 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:20.032675 14080 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:20.068531 14139 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:20.040349 14077 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:20.069984 13888 consensus_meta.cc:287] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.756s	user 0.016s	sys 0.000s
W20260808 01:51:20.092551 14122 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.359s	user 0.000s	sys 0.001s
W20260808 01:51:20.177402  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:20.073167 13894 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:20.007119 13889 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
I20260808 01:51:20.066272 13421 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:20.214728 10365 raft_consensus.cc:2368] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10.
I20260808 01:51:20.074863 14030 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260808 01:51:20.161847 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 295438us
I20260808 01:51:20.014379 13790 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:20.233405  9555 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:20.231266 13893 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:20.231129 14137 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:20.251186  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:20.231750 14130 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.218s	user 0.001s	sys 0.007s
I20260808 01:51:20.256136 13909 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:20.238893 14124 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.433s	user 0.000s	sys 0.009s
I20260808 01:51:20.229773  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:20.266718  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 200444us
W20260808 01:51:20.266446  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 51744.4us
W20260808 01:51:20.229460 14011 consensus_meta.cc:287] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.099s	user 0.009s	sys 0.021s
I20260808 01:51:20.274554 13893 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:20.276161 14077 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:20.276396 13790 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:20.276712 14082 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:20.284861  9635 raft_consensus.cc:2368] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10.
W20260808 01:51:20.291867 14116 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.368s	user 0.000s	sys 0.010s
I20260808 01:51:20.293392 14139 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:20.293726 13893 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:20.278874 13932 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:20.284478 13701 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:20.315907  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 101510us
I20260808 01:51:20.294544 13909 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 01:51:20.333061  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:20.343489  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:20.349069  9637 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:20.324702 14011 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260808 01:51:20.385035 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 114915us
I20260808 01:51:20.382557 13893 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260808 01:51:20.417140  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 102521us
I20260808 01:51:20.425634 13893 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:20.443584  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:20.450349  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:20.435233 14074 consensus_meta.cc:287] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.006s	user 0.015s	sys 0.004s
I20260808 01:51:20.432798 13893 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:20.438467  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2216) took 1302 ms. Trace:
I20260808 01:51:20.432086 14152 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:20.505829  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:20.520550 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 279770us
I20260808 01:51:20.507761  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:20.519469  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:20.407660 13909 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260808 01:51:20.437897 14128 log.cc:927] Time spent T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.141s	user 0.000s	sys 0.000s
I20260808 01:51:20.503721  9630 rpcz_store.cc:276] 0808 01:51:19.136066 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:19.136241 (+   175us) service_pool.cc:224] Handling call
0808 01:51:19.161125 (+ 24884us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:19.236555 (+ 75430us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:19.236594 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:19.237180 (+   586us) log.cc:844] Serialized 26 byte log entry
0808 01:51:19.359501 (+122321us) spinlock_profiling.cc:243] Waited 32 ms on lock 0x7b1800374fe0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:51:19.604489 (+244988us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:19.859939 (+255450us) log.cc:844] Serialized 14 byte log entry
0808 01:51:19.860271 (+   332us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:19.860382 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:20.426815 (+566433us) raft_consensus.cc:1685] finished
0808 01:51:20.438173 (+ 11358us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:20.438388 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":92390,"prepare.queue_time_us":103860,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":48,"raft.queue_time_us":82859,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":12648,"spinlock_wait_cycles":73663744,"thread_start_us":441045,"threads_started":2,"wal-append.queue_time_us":442629}
I20260808 01:51:20.426388 14137 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:20.443038 13421 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:20.503074 14030 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:20.509462 14011 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 01:51:20.549861  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 122118us
W20260808 01:51:20.527781 14098 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.959s	user 0.019s	sys 0.004s
I20260808 01:51:20.503391 13932 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.074s
W20260808 01:51:20.555053  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 248573us
W20260808 01:51:20.528983 14005 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.633s	user 0.010s	sys 0.009s
I20260808 01:51:20.552336  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:20.553306  9477 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:51:20.529191 13889 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:20.502799 14074 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:20.572407 14098 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:20.586004 14118 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:20.600136 14131 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:20.600607 14118 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:20.582306 14005 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:20.600855 14131 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:20.601486 14127 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.704s	user 0.014s	sys 0.000s
I20260808 01:51:20.602180 14098 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:20.590229 13889 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:20.603489 14005 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:20.600159  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:20.605285 13889 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:20.607026  9521 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:20.609367 14172 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:20.609930 14172 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:20.556808 14030 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:20.611840 14173 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:51:20.611625 14137 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 9.11, Last appended by leader: 11, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:20.612923 14030 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:20.526330 14152 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:20.613518 14173 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:20.613998 14173 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:20.614403 14173 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:20.614858 14118 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:20.608633  9521 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:20.582075 14171 consensus_queue.cc:1059] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260808 01:51:20.615778 14152 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:20.618896 14074 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:20.619706  9560 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:20.602058 14127 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:20.610351 14172 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:20.642916 13878 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:20.643643 13878 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:20.644053 13878 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:20.644284 14127 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:20.639875  9559 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 7. Current term is 8. Ops: []
I20260808 01:51:20.651337  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2218) took 1492 ms. Trace:
I20260808 01:51:20.643025  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:20.657766  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 105554us
I20260808 01:51:20.661298  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:20.658826 14024 consensus_meta.cc:287] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.785s	user 0.011s	sys 0.003s
I20260808 01:51:20.651789  9636 rpcz_store.cc:276] 0808 01:51:19.158624 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:19.158795 (+   171us) service_pool.cc:224] Handling call
0808 01:51:19.159279 (+   484us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:19.159841 (+   562us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:19.159882 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:19.164057 (+  4175us) log.cc:844] Serialized 26 byte log entry
0808 01:51:19.499890 (+335833us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:19.499951 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:19.500063 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:20.645341 (+1145278us) spinlock_profiling.cc:243] Waited 2.5 ms on lock 0x7b2400330ff8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41713594 00007f5d4171169b 00007f5d418cd6d9 00007f5d418cd365 00007f5d418cc8c0 00007f5d4455a638 00007f5d4455ad98 00007f5d44532588 00007f5d4452d95a 00007f5d4452cda6
0808 01:51:20.646843 (+  1502us) raft_consensus.cc:1685] finished
0808 01:51:20.646873 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:20.651249 (+  4376us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15120,"raft.queue_time_us":2611,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":27887,"spinlock_wait_cycles":5748480,"thread_start_us":327468,"threads_started":1,"wal-append.queue_time_us":610078}
I20260808 01:51:20.666644  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:20.558667 13421 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:20.673925 13909 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:20.666149 14152 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:20.666339 14131 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:20.678798 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:20.691721  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:20.696064 10365 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:20.698808  9523 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:20.688875 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:20.676962 13909 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:20.675873 14024 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:20.681561 14152 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:20.693997 14181 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:20.722818 14192 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:51:20.724725 14011 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:51:20.678485 14188 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260808 01:51:20.727105  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:20.727770 14188 raft_consensus.cc:3055] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:51:20.728261 14188 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:20.728541  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:20.728888 14024 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:20.729236 14188 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:20.722112 14181 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:20.730480 14181 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:20.730422 14011 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
W20260808 01:51:20.732137 14156 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.488s	user 0.003s	sys 0.000s
I20260808 01:51:20.732524 14011 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:20.732896 14196 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260808 01:51:20.733520  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 53079.3us
I20260808 01:51:20.730518 14172 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:20.740701 14172 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:20.744088  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:20.744349  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:20.744987 14184 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:20.720340 14152 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:20.721745 14171 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:20.754832 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:20.755236  9475 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:20.756255  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:20.714300 13909 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:20.757491  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:20.755314  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:20.763777  9477 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:20.762724 14192 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:20.770239  9524 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:20.721992 14131 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260808 01:51:20.775708  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 47245.2us
I20260808 01:51:20.775753  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:20.754896 14171 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:20.741569 13893 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:20.812703  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:20.809597  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:20.818837 14197 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.080s	user 0.002s	sys 0.000s
W20260808 01:51:20.838963 14114 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.841s	user 0.007s	sys 0.000s
I20260808 01:51:20.840179 13894 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:51:20.844709 13878 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:20.852301 13893 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:20.853581  9479 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:20.758905 14184 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:20.815863 14171 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:20.813890 14172 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:20.871371 14172 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:20.871953 14172 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:20.873900 13421 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:20.880635  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:20.791659 14192 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:20.887742  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:20.888715  9474 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:51:20.858322 14184 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:20.895895  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 78335.5us
I20260808 01:51:20.894519 14192 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:20.899214 14184 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:20.906373 14082 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:20.926301  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:51:20.926239  9477 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:20.907915 13894 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:20.926637 14192 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:20.931439  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:20.931813 14192 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:20.934181 14192 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:20.936215 14181 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:20.945052 14200 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
I20260808 01:51:20.925132 14184 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:20.942951 13894 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:20.940413 14082 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:20.958093  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2234) took 1649 ms. Trace:
I20260808 01:51:20.977579  9637 rpcz_store.cc:276] 0808 01:51:19.308482 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:19.376636 (+ 68154us) service_pool.cc:224] Handling call
0808 01:51:19.403173 (+ 26537us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b5800231698. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:51:20.957763 (+1554590us) spinlock_profiling.cc:243] Waited 844 ms on lock 0x7b5400420ed4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:20.957951 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1968349440}
I20260808 01:51:20.963732 13878 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:20.964051  9524 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:20.965134  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 65603.3us
I20260808 01:51:20.968564 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:20.970568  9555 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:20.971203 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:20.980109 10365 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:20.982590 14131 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:20.983255 14131 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:20.983676 14131 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:20.983709  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:20.984659 14074 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:20.985257 14074 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:20.985102 14175 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
W20260808 01:51:20.986091 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:20.987500  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:20.950412 14181 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:20.987926  9445 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:20.989486 14192 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:20.984335 14166 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:20.999925  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:21.081607  9555 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
W20260808 01:51:21.081969  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 1.07285e+06us
W20260808 01:51:21.016498  9635 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2231) took 1709 ms (client timeout 2189 ms). Trace:
W20260808 01:51:21.082518  9635 rpcz_store.cc:269] 0808 01:51:19.307486 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:19.365516 (+ 58030us) service_pool.cc:224] Handling call
0808 01:51:21.016226 (+1650710us) spinlock_profiling.cc:243] Waited 891 ms on lock 0x7b54003fa054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:21.016417 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2048782592}
W20260808 01:51:21.034425  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 1.0031e+06us
I20260808 01:51:21.025512 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:20.993023 10362 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:21.034909  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2737) took 1237 ms. Trace:
I20260808 01:51:21.026896 14163 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:21.041321 13889 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:20.993678  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:21.087153 13889 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:21.088069  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 94523.6us
I20260808 01:51:21.089257  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:21.089830 13889 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:20.998298 14082 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:21.001138 13894 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:21.044675 14136 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:21.090725 14201 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:21.091684 14201 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:21.091310 14136 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:21.020920 14184 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:21.000399 14074 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.024628 14011 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:21.023465 14173 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:20.991804 14118 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.049283 14132 raft_consensus.cc:697] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:21.050132  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 814793us
I20260808 01:51:21.001421 14188 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:21.053491 14180 raft_consensus.cc:3055] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:21.053218 13888 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:51:21.057817 13421 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:21.020135 14172 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.059892 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:21.058869 13893 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:21.074141  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:21.131726 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 71990.1us
W20260808 01:51:21.075943  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 94073.2us
I20260808 01:51:21.084064 10365 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:21.084903  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:20.992117 14183 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260808 01:51:21.094259  9599 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:21.096182  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2230) took 1789 ms. Trace:
I20260808 01:51:21.084486  9560 rpcz_store.cc:276] 0808 01:51:19.797228 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:19.805189 (+  7961us) service_pool.cc:224] Handling call
0808 01:51:19.840089 (+ 34900us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:20.642051 (+801962us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:21.034583 (+392532us) spinlock_profiling.cc:243] Waited 779 ms on lock 0x7b54003264d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:21.034805 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1792368000}
I20260808 01:51:21.099021  9601 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:21.081374 14196 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:21.099519  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:21.136127  9599 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:21.100395 14171 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:21.100893 14153 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:21.093331 14163 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:21.108834 14166 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:21.124779  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:21.127528  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:21.134514  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:21.145959  9472 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:21.146458  9476 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:21.096469 13909 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.094307 14152 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:21.140872  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 51738.4us
I20260808 01:51:21.137733  9473 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:51:21.149355 13951 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:21.149091 13893 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.137589 14196 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:21.150249 14196 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:21.150364 14163 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:21.136756 14137 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:21.151887 14163 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:21.126446 14180 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:21.151429 13893 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:21.140121 14153 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:21.168485 14153 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:21.168746 13894 consensus_queue.cc:1059] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 01:51:21.170130 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:21.137745 13421 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:21.171173  9475 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:21.172310 13421 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:21.137744 14080 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:21.172646  9523 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:21.158738 14146 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:21.160477 14172 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:21.173686 14060 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:21.174116 14060 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:21.174482 14060 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:21.176403 14184 raft_consensus.cc:3055] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:21.176884 14184 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:21.178844 13909 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:21.180652 14208 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260808 01:51:21.162515 14118 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:21.162734 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:21.163127 14183 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 01:51:21.182821 14183 raft_consensus.cc:3055] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:21.183251 14183 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:21.183517 14152 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:21.184159 14183 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.185514 14183 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:21.186507 14160 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:21.187105 14160 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:21.187495 14160 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:21.162319 14203 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.276s	user 0.006s	sys 0.000s
I20260808 01:51:21.167238 14132 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.196063 14011 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
W20260808 01:51:21.167724  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 68837.1us
I20260808 01:51:21.177937 14184 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.181898  9478 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:21.200374 14136 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:21.195940 13704 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:21.201822 13704 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:21.191447 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:21.202378 14184 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:21.203151 10365 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 10: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:21.204247 13704 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:21.204531 13909 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:21.196223  9480 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:21.206565 14171 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:21.207026 13888 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:21.210189 13888 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:21.205327 13909 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:21.212404 13888 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:21.213085  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 44237.1us
I20260808 01:51:21.213181 14080 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:51:21.135655  9629 rpcz_store.cc:276] 0808 01:51:19.306985 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:19.584793 (+277808us) service_pool.cc:224] Handling call
0808 01:51:19.585122 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:19.593291 (+  8169us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:51:19.852549 (+259258us) log.cc:844] Serialized 14 byte log entry
0808 01:51:20.003207 (+150658us) log.cc:844] Serialized 14 byte log entry
0808 01:51:20.731626 (+728419us) spinlock_profiling.cc:243] Waited 6.79 ms on lock 0x7b2400330ff8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41713594 00007f5d4171169b 00007f5d418cd6d9 00007f5d418cd365 00007f5d418cc9ad 00007f5d445162fe 00007f5d4452da7c 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411
0808 01:51:21.031544 (+299918us) spinlock_profiling.cc:243] Waited 3.04 ms on lock 0x7b180035bb40. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667 00007f5d449ae5e8
0808 01:51:21.085791 (+ 54247us) log.cc:844] Serialized 14 byte log entry
0808 01:51:21.095892 (+ 10101us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:21.095953 (+    61us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:21.095987 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:21.096038 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:21.096135 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":131583,"prepare.queue_time_us":193037,"prepare.run_cpu_time_us":146,"prepare.run_wall_time_us":4219,"raft.queue_time_us":2144936,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":9310,"spinlock_wait_cycles":26105984,"thread_start_us":235765,"threads_started":3,"wal-append.queue_time_us":202642}
I20260808 01:51:21.202112  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:21.192263  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 44401us
I20260808 01:51:21.223254 14180 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:21.224948 14180 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:21.225791  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:21.227995 13421 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:21.196574  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:21.227752 13909 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:21.228729  9472 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:21.207743  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:21.214746 14146 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:21.214313 14080 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:21.230854 14080 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:21.231092 14146 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.215975  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:21.229588  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:21.233013  9554 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:21.215653  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:21.166538 14131 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.216310  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:21.234217  9558 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 10
I20260808 01:51:21.219194 10334 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:21.227797 14011 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:51:21.215653 13951 raft_consensus.cc:697] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:21.196568  9475 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:21.231864 14080 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.242076  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:21.247869 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:21.249205 10365 raft_consensus.cc:2410] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 14 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260808 01:51:21.245492 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:21.250432 10360 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:21.252558  9554 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:21.252292  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:21.254832  9560 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 11
I20260808 01:51:21.254916  9600 leader_election.cc:400] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10.
I20260808 01:51:21.255291  9600 leader_election.cc:403] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260808 01:51:21.256057  9476 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260808 01:51:21.258991  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:21.259678  9557 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:21.261313 14131 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:21.261471  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 53827.7us
I20260808 01:51:21.262024 13888 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:21.262887 13888 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:21.263051  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:21.264513  9601 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:21.264907 14127 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:21.277048  9601 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:21.275164  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 79554us
I20260808 01:51:21.284436 10334 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:21.261816 14080 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:21.291054  9559 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 10 index: 4. Preceding OpId from leader: term: 12 index: 5. (index mismatch)
I20260808 01:51:21.258244 14137 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260808 01:51:21.292636  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 63265.4us
I20260808 01:51:21.297174  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:21.304802  9475 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 10
I20260808 01:51:21.272414 14127 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260808 01:51:21.273175 14166 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:21.313637  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:21.316434  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:21.317425 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:21.283789 14136 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:21.311692  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:21.323264  9551 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:21.283351 13704 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:21.325917  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:21.326049  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:21.327919  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:21.293008 14011 raft_consensus.cc:2764] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election decision vote started in defunct term 10: won
I20260808 01:51:21.323107 10363 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:21.338135  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:21.339650  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:21.340123  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:21.340801  9636 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 4.
I20260808 01:51:21.340937  9557 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:21.341389  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:21.341866  9559 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:21.263432 13888 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:21.309263 14137 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:21.329138  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:21.345031 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:21.292624 13889 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:21.296587 13945 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:21.343421  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:21.351125  9476 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:21.352109  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:21.343613  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:21.344039  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:21.335299  9629 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:21.354502  9635 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:21.345534  9472 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:21.345551  9632 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:21.356769  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:21.345994  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:21.290132 14131 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
W20260808 01:51:21.331674  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:21.352856  9638 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:21.357864  9524 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:21.359606  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:21.360013  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:21.360314  9557 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 10: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:21.358320  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:21.353760 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:21.361896  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:21.365111  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:21.343159 13704 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.367792 13945 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:51:21.369570 13704 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:21.371737  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet b335bc0170854e26a77dbec13dbe9eac: Illegal state: Replica 6e7856bed1be43acaa821290d506ee97 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:21.376120 13888 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:21.321291 13893 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.042s
I20260808 01:51:21.379082  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:21.380826  9556 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260808 01:51:21.382257 14196 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:21.384704  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:21.382491 14160 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:21.386227  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:21.387298  9632 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 10: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:21.385798 14196 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:21.387729 14173 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:21.388057  9560 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:21.388226  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:21.391368 14060 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:21.392081 14131 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:21.392586 14131 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:21.392696  9554 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:21.393918 14060 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:21.394531  9629 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:21.364990  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:21.397586  9637 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:51:21.397908  9475 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
W20260808 01:51:21.404186  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46415.6us
I20260808 01:51:21.404521  9560 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: [8.6-10.7]
I20260808 01:51:21.352046 14137 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:21.393591 14131 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.408720  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:21.333536 13909 consensus_queue.cc:1059] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260808 01:51:21.408721  9558 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:21.353930 14212 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:21.414785  9550 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:21.415369  9631 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:21.354844 14146 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260808 01:51:21.418052  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 61342.3us
I20260808 01:51:21.415155 14171 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:21.338124 13894 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260808 01:51:21.427234  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:21.428332 14160 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:21.430182 14140 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:21.420414 14212 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.323719 14152 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.086s
I20260808 01:51:21.413285 14074 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:21.438544 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:21.439205  9474 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 9 index: 11. Preceding OpId from leader: term: 10 index: 12. (index mismatch)
W20260808 01:51:21.416148 14187 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260808 01:51:21.350088 13951 consensus_queue.cc:237] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.443872 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:21.366900 13889 raft_consensus.cc:697] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:21.451263  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:21.553536  9631 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:21.455410 13909 raft_consensus.cc:3055] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Stepping down as leader of term 9
W20260808 01:51:21.459097  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46609.7us
I20260808 01:51:21.460944 13421 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:21.461826 14192 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:21.465852 13893 consensus_queue.cc:1059] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:51:21.469913 14131 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:21.474843 13888 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:21.360754 14153 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:21.480074  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:21.486527 14213 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.170s	user 0.011s	sys 0.000s
W20260808 01:51:21.488487 14211 log.cc:927] Time spent T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Append to log took a long time: real 0.152s	user 0.002s	sys 0.000s
W20260808 01:51:21.493559  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 407cb6b113ce432587849bc840ba6872: Illegal state: Replica 52ccf4dcca1843f7bb29e9172e2d4ab4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 11 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:21.527091 14196 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
W20260808 01:51:21.587646  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 107635us
I20260808 01:51:21.588441  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:21.588835  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:21.559927 14074 raft_consensus.cc:697] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:21.512657  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:21.514294  9560 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:21.592649  9554 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:21.516254  9638 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:21.520920  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:21.593469  9629 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:21.521665  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:21.501698 14011 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260808 01:51:21.544806  9559 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:21.546648 10360 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:21.414026 14080 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.106s
I20260808 01:51:21.553617  9552 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260808 01:51:21.555999  9551 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
W20260808 01:51:21.565594  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:21.519330 14171 raft_consensus.cc:697] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:21.581606  9635 raft_consensus.cc:606] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:21.580487 13888 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:21.500790  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:21.510787  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:21.627033  9550 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:51:21.627583 13889 consensus_queue.cc:237] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:21.629084  9632 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260808 01:51:21.634344 14171 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:21.558296 13909 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:21.576905 13932 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.027s
I20260808 01:51:21.637049 13909 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.590906  9599 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:21.638896 14150 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:21.639516 14150 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:21.640647  9636 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260808 01:51:21.640413 14150 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.577261 13421 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.642117  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:21.597059 13945 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:21.643095 13945 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:21.643510 13945 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:51:21.638484  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 47944.1us
I20260808 01:51:21.527439 14181 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
W20260808 01:51:21.673822  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 173102us
W20260808 01:51:21.601637  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:21.600451  9476 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:21.623420  9635 raft_consensus.cc:614] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:51:21.676528  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:21.645344 14074 consensus_queue.cc:237] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.680367  9550 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260808 01:51:21.681604  9599 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:21.679623  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:21.684480 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:21.685809  9524 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:21.678589  9600 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:21.688953 14136 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10.
I20260808 01:51:21.690131  9479 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260808 01:51:21.693751  9637 raft_consensus.cc:2410] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 10 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260808 01:51:21.694700  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:21.695246  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:21.695621  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:21.704919 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:21.596318 13391 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:21.717468  9631 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:21.685576 14151 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:21.700445 14158 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:21.582150 14192 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.700080 14127 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:21.699826 14138 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:21.700841 13894 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:21.744019 13421 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:21.746479 14212 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:21.713078 14136 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:51:21.716005 14133 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:21.700282 13704 consensus_queue.cc:1059] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20260808 01:51:21.734634 14151 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:21.737823 13391 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:21.716092 14184 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.037s
I20260808 01:51:21.759608 14137 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:21.779577  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:21.808667  9555 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260808 01:51:21.758559 14136 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:21.761308 14133 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:21.761113 13894 raft_consensus.cc:697] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:21.819804 13894 consensus_queue.cc:237] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.769029 14151 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:21.764602 13421 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:21.734429 13888 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.818634 14133 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:21.828064 10369 raft_consensus.cc:2368] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12.
W20260808 01:51:21.816015 14220 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.118s	user 0.000s	sys 0.001s
I20260808 01:51:21.803848 13893 consensus_queue.cc:1243] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260808 01:51:21.815730 14212 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:51:21.806730 14127 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:51:21.835539  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 140947us
I20260808 01:51:21.817994 14136 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:51:21.839184 14192 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:21.840631 13945 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:21.829962 13421 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:21.852095 13421 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.853257 13421 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:21.852963 14127 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.855513 14127 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:21.847194 14212 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:21.857075 14212 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:51:21.857522  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:21.863125 14161 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:21.864283 14221 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260808 01:51:21.864033 14161 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.867996 14152 raft_consensus.cc:3055] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:21.868543 14152 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:21.869511 14152 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.873668 13701 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:51:21.862936 14133 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:21.880439  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:21.888234  9638 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:21.889921  9601 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:21.883352  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:21.892017  9559 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:51:21.884214  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 50fd024764ea49c7a1f16e992e7599f3 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:21.904213  9601 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:21.859625 13945 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:21.881592 14161 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:21.880573 14127 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:21.910346 14152 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:21.924042 14080 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260808 01:51:21.959573  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:21.961793  9600 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:21.969128  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:21.972025  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:21.978029 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:21.982393 10365 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 14
I20260808 01:51:21.979215  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:21.863214 13391 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.942451 13701 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:21.988704  9635 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:21.996634  9633 raft_consensus.cc:2368] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 13. Current term is 14.
I20260808 01:51:22.000464  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:22.001466  9551 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:22.002529  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.003360  9637 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:22.004621  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:22.012862  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.016001  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.017112  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.021003  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:21.995785 14137 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:22.026568  9632 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:22.034116 13421 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:22.034931 13421 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:21.966161 14127 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:22.045040  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:22.052028  9480 raft_consensus.cc:2393] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:22.054419 13421 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:22.050143  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:22.056418  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 414365us
I20260808 01:51:22.057601  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:22.056355  9632 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 12
I20260808 01:51:22.025377 14161 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:21.978937 13945 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260808 01:51:22.058418 13421 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:22.059849  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:22.061407  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:22.062256  9478 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:22.054797  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:22.072891  9551 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:22.088424  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:22.091832  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:22.092512  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:22.092974  9473 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:22.065769  9479 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:22.110741  9481 raft_consensus.cc:2368] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10.
W20260808 01:51:22.111574  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:22.110819  9557 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:22.115733  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:22.117381  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 57611.9us
W20260808 01:51:22.115140  9629 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.521s	user 0.010s	sys 0.005s
I20260808 01:51:22.128793  9629 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:22.129415  9635 raft_consensus.cc:606] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:22.129823  9635 raft_consensus.cc:614] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:22.130228 10365 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:22.135673 14229 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:22.137583  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:22.110407 14133 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:22.124686  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:22.137106 14212 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:22.145186  9481 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:22.116776 13421 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:22.146260 13421 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:22.146790 13421 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:22.151664  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:22.121511  9559 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:22.156045  9479 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:22.156898  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:22.159379  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:22.163388  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:22.166579  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:22.169807  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:22.150159 14196 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:22.099530 14161 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:22.192945  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:22.194662  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:22.195963  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:22.196942  9475 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 10
I20260808 01:51:22.149938 14181 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.042s
I20260808 01:51:22.199056 14181 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:22.199487 14181 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:22.201052 14181 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:22.201647 14212 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:22.202526 14181 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:22.203238 14151 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:22.205653 14151 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:22.165750 14230 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:51:22.209532 14201 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:22.210127 14201 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:22.210275 14153 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:22.210595 14201 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:22.213045 14226 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:51:22.213631 14226 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:22.214076 14226 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:22.160749 14158 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:22.186802 14196 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:22.240129 14196 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:22.242134  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:22.243099 14230 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:22.243815 14230 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:22.243564 14158 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:22.244889 14230 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:22.243139 14127 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:22.249280 13701 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:22.243878  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:22.174160 14224 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:22.251133 14224 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:22.251595 14224 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:22.252384  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 0ea307d9adc442539683547d0176a05b: Illegal state: Replica 6e7856bed1be43acaa821290d506ee97 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:22.252169 14118 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:51:22.257094  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:22.255056  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:22.258698  9559 raft_consensus.cc:2368] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10.
I20260808 01:51:22.253041 14227 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260808 01:51:22.274036 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 255088us
I20260808 01:51:22.236481 13421 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:22.277318  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:22.291972 10332 leader_election.cc:336] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.682s (SENT)
I20260808 01:51:22.279567  9630 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:22.292007  9475 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:22.293021 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:22.293956  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:22.278090  9632 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:22.295370  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:22.305773  9550 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260808 01:51:22.269505 14118 consensus_queue.cc:1243] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20260808 01:51:22.290824  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 315009us
I20260808 01:51:22.319335  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:22.320382  9554 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:22.322006  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.323886  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.323982  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:22.325404  9637 raft_consensus.cc:2410] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260808 01:51:22.325956  9557 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 10: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:22.330534  9560 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: [8.6-10.7]
I20260808 01:51:22.311538  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:22.270798 14231 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:22.340778  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.340952  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:22.342115  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.342804  9632 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:22.341195  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.343067  9633 raft_consensus.cc:2376] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11. Re-sending same reply.
I20260808 01:51:22.346746  9600 leader_election.cc:400] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12.
I20260808 01:51:22.346349  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.351198  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.351559 13421 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:22.347275  9600 leader_election.cc:403] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:51:22.347141 14236 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:22.352499  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:22.352216  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 312101us
I20260808 01:51:22.356277  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:22.347078  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:22.309418 14163 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:22.358429  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.363637  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.362361 13421 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:51:22.357178  9638 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:22.364812 13421 consensus_queue.cc:1243] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 11.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260808 01:51:22.363755  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:22.362368  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.366526  9637 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:22.367651  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.368928  9629 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 12: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:22.364488  9633 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:22.371882  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:22.372920  9629 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:22.369124  9524 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:22.382409  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:22.340150 13932 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:22.410758  9478 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:22.410758  9634 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
W20260808 01:51:22.420847 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 128901us
I20260808 01:51:22.385615  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.422726 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:22.423991  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:22.425666  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:22.436470  9633 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:22.436807 14137 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:22.440603  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:22.407138 14050 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:22.450296 14050 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:22.450906 14050 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:22.412627 14150 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12.
I20260808 01:51:22.456822  9481 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12. Ops: []
W20260808 01:51:22.416320  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 46402.2us
I20260808 01:51:22.400571 13893 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:22.401983 13888 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:22.469002 13893 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:22.469259  9481 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 10 index: 4. Preceding OpId from leader: term: 12 index: 5. (index mismatch)
I20260808 01:51:22.470252 13893 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:22.471515 13893 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:22.471984  9474 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:51:22.472036 14011 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:22.402228 14161 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:22.403369 14241 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
I20260808 01:51:22.350337 14237 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:22.404934  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2233) took 1065 ms. Trace:
I20260808 01:51:22.489547  9630 rpcz_store.cc:276] 0808 01:51:21.339973 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:21.340108 (+   135us) service_pool.cc:224] Handling call
0808 01:51:21.354380 (+ 14272us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:22.309028 (+954648us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:22.404658 (+ 95630us) spinlock_profiling.cc:243] Waited 925 ms on lock 0x7b540034fb54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:22.404863 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2127185280}
I20260808 01:51:22.491261  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:22.376482 14163 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:22.406152 14146 raft_consensus.cc:3055] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:22.420935  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:22.369980 14236 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:22.451331  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:22.485117 14242 log.cc:927] Time spent T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.054s	user 0.002s	sys 0.000s
I20260808 01:51:22.463321 10334 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:22.464372  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:22.442389  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:22.465973  9473 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:22.473776 14011 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:22.508792 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 45591.4us
I20260808 01:51:22.494184  9478 raft_consensus.cc:2368] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12.
W20260808 01:51:22.477320 14136 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.574s	user 0.011s	sys 0.002s
I20260808 01:51:22.494315  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:22.511276  9630 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:22.461264 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:22.498299  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.524250  9473 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260808 01:51:22.505291  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:22.492708  9600 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:22.528630  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:22.531689 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 70424.6us
I20260808 01:51:22.539213 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:22.385831  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:22.546980  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:22.549981  9475 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:22.558647  9637 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:22.567286  9638 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
W20260808 01:51:22.570475  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:22.496686 14146 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:22.578105  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:22.586898  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2105) took 1073 ms. Trace:
I20260808 01:51:22.516769  9629 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:22.577710  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:22.590708  9637 raft_consensus.cc:2376] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11. Re-sending same reply.
I20260808 01:51:22.587642  9556 rpcz_store.cc:276] 0808 01:51:21.513291 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:21.513448 (+   157us) service_pool.cc:224] Handling call
0808 01:51:21.515463 (+  2015us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:21.537763 (+ 22300us) raft_consensus.cc:1534] Early marking committed up to index 4
0808 01:51:21.770655 (+232892us) log.cc:844] Serialized 14 byte log entry
0808 01:51:22.146209 (+375554us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:22.154803 (+  8594us) log.cc:844] Serialized 26 byte log entry
0808 01:51:22.154992 (+   189us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:51:22.155027 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:22.155097 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:22.586491 (+431394us) raft_consensus.cc:1685] finished
0808 01:51:22.586524 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:22.586794 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9877,"prepare.queue_time_us":51660,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":41888,"raft.run_cpu_time_us":383,"raft.run_wall_time_us":34468,"thread_start_us":492772,"threads_started":2,"wal-append.queue_time_us":461855}
I20260808 01:51:22.588449  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.594431  9638 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:22.596503  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:22.604202  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 111602us
I20260808 01:51:22.607348 13909 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:22.609979 14150 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:22.610785 14150 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:22.577625 14133 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.037s
I20260808 01:51:22.613171 14150 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:22.494211 14237 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:22.618805 14237 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:22.621520  9632 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:22.622840  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.623773  9629 raft_consensus.cc:2376] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10. Re-sending same reply.
I20260808 01:51:22.550149 14160 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:22.628816  9601 leader_election.cc:400] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10.
I20260808 01:51:22.626196 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:22.629853  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:22.578296 14176 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:22.551081 14145 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:22.627944 14240 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:22.625851  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:22.644593  9479 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:22.644644  9638 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260808 01:51:22.650956  9474 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:22.651055  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.651887  9631 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:22.652688  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:22.647684 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:22.652889  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.648509  9475 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
W20260808 01:51:22.645416  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 298859us
I20260808 01:51:22.654366  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:22.550437 14136 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:22.686890  9601 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:22.693176  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 924b85a2ee7e46ecbcfd57257902abef (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:22.702095 10369 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: []
W20260808 01:51:22.703140  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:22.703629  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:22.704604  9638 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:22.699072  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:22.707532  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 52838.4us
W20260808 01:51:22.710750  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:22.647337 14160 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:22.715190  9636 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:51:22.716040 10360 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
W20260808 01:51:22.693980 14243 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.251s	user 0.001s	sys 0.001s
I20260808 01:51:22.628202 14201 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:22.729833  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:22.730065  9520 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.041s	sys 0.073s
W20260808 01:51:22.730526  9520 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.041s	sys 0.073s
I20260808 01:51:22.724040  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:22.721469  9521 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:22.726964  9601 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:22.707512 14140 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:51:22.701732 13888 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:22.711150 14240 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:22.631343 14011 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:22.656670 14188 consensus_queue.cc:1059] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.082s
I20260808 01:51:22.749325  9601 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:22.833921 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:22.738109 14227 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:22.714573 14118 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.037s
I20260808 01:51:22.841470  9629 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:22.842996 14236 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:22.843466 14258 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260808 01:51:22.844475 14011 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:22.845396 14201 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:22.846593  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:22.746968 14152 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.779s	user 0.012s	sys 0.001s
I20260808 01:51:22.728273 13704 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:22.714393 14161 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260808 01:51:22.795639 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:22.647284 14224 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:22.859797 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 284422us
I20260808 01:51:22.739653 14230 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.155s
W20260808 01:51:22.732779 14226 consensus_meta.cc:287] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.518s	user 0.009s	sys 0.003s
W20260808 01:51:22.757025  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:22.919679  9480 raft_consensus.cc:2410] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260808 01:51:22.839442  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:51:22.885052  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 135848us
I20260808 01:51:22.920994  9629 raft_consensus.cc:606] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:22.921422  9629 raft_consensus.cc:614] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:22.921350  9601 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:22.875344 14138 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260808 01:51:22.847276 14246 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.216s	user 0.000s	sys 0.001s
I20260808 01:51:22.709961 14146 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:22.903965  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:22.794934 14080 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.094s
I20260808 01:51:22.871410  9630 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:22.913743 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:22.931675  9633 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:22.931854  9474 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260808 01:51:22.849990 14257 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:22.885828 13790 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:51:22.926168  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:22.847366 14148 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.328s
I20260808 01:51:22.941336  9478 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 11. Current term is 12. Ops: []
W20260808 01:51:22.946132  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 195138us
I20260808 01:51:22.946516  9601 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:22.946997  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:22.902282 14227 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:22.955022  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:22.959097  9444 leader_election.cc:400] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10.
W20260808 01:51:22.926802 14196 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.686s	user 0.015s	sys 0.000s
I20260808 01:51:22.959666  9444 leader_election.cc:403] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:51:22.918308 14163 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:22.954022 14158 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:22.942926 14166 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election won for term 10
W20260808 01:51:22.977205  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 255812us
I20260808 01:51:22.937538 14257 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:51:22.978117  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 74049.7us
I20260808 01:51:22.988274  9521 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:22.946583 14171 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:22.995669  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:22.962816 14230 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:22.896564 14160 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:23.020994  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:22.970430 13893 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:22.930048 14226 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:22.974460 14080 raft_consensus.cc:3055] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:22.975383 14201 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10.
I20260808 01:51:23.030483  9601 leader_election.cc:400] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12.
I20260808 01:51:22.992333 13790 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:23.022429 14171 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:23.052062 14250 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.342s	user 0.000s	sys 0.002s
I20260808 01:51:23.052405 14171 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:23.054380 14265 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260808 01:51:22.968040 14196 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.075621  9481 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 6. Current term is 7. Ops: []
I20260808 01:51:22.996848 14257 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:23.054944 14236 raft_consensus.cc:3055] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:23.090595  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2765) took 1207 ms. Trace:
W20260808 01:51:23.065636 14264 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
W20260808 01:51:23.069311 14263 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
I20260808 01:51:23.037984 14230 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:23.072916 14227 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.073822  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 677) took 1398 ms. Trace:
I20260808 01:51:23.016119 14152 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:23.096694  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:23.035095 14050 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.584s	user 0.014s	sys 0.000s
I20260808 01:51:23.019368 14166 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:23.081899  9474 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 11 index: 5. (index mismatch)
I20260808 01:51:23.082803  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:23.100346  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:23.100711  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:23.101033  9475 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:23.028888 14080 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:23.091140  9555 rpcz_store.cc:276] 0808 01:51:21.883585 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:21.883766 (+   181us) service_pool.cc:224] Handling call
0808 01:51:21.884122 (+   356us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:21.884547 (+   425us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:21.884584 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:21.885190 (+   606us) log.cc:844] Serialized 26 byte log entry
0808 01:51:22.331984 (+446794us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:22.332025 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:22.332095 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:23.051416 (+719321us) spinlock_profiling.cc:243] Waited 3.77 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:23.056320 (+  4904us) raft_consensus.cc:1685] finished
0808 01:51:23.056351 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:23.090466 (+ 34115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2544,"raft.queue_time_us":1226,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":21342,"spinlock_wait_cycles":8670464,"thread_start_us":446391,"threads_started":1,"wal-append.queue_time_us":711188}
I20260808 01:51:23.096772  9472 rpcz_store.cc:276] 0808 01:51:21.675578 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:21.675764 (+   186us) service_pool.cc:224] Handling call
0808 01:51:21.710981 (+ 35217us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:22.054791 (+343810us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:22.054894 (+   103us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:22.112034 (+ 57140us) log.cc:844] Serialized 26 byte log entry
0808 01:51:22.495333 (+383299us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:51:22.495399 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:22.495510 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:23.073576 (+578066us) raft_consensus.cc:1685] finished
0808 01:51:23.073601 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:23.073746 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":139690,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":102,"thread_start_us":687393,"threads_started":2,"wal-append.queue_time_us":784594}
I20260808 01:51:23.106815  9637 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:23.108421  9479 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:23.102814  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:23.102027  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:23.021041 14158 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.060s
I20260808 01:51:22.956377 14240 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:23.044530 13893 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:23.020748 14151 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260808 01:51:23.111488  9474 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260808 01:51:23.111244  9475 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 5: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:23.056727 14140 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:23.115634  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:23.116434  9635 raft_consensus.cc:2376] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10. Re-sending same reply.
W20260808 01:51:23.126029  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 109794us
I20260808 01:51:23.130013 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:23.130966 10363 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:23.143090  9638 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260808 01:51:23.090046 14136 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:23.089587 14236 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:23.153641 14267 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20260808 01:51:23.154458 14262 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:23.153927 14080 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:23.165634 14080 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:23.165287 14166 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:23.167594 14247 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:23.171099  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2393) took 1203 ms. Trace:
I20260808 01:51:23.120798 14158 consensus_queue.cc:1243] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 10.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260808 01:51:23.111956 14226 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:23.120344 13391 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:23.114610 14224 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:23.153815 14230 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.180274  9635 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:23.171622  9477 rpcz_store.cc:276] 0808 01:51:21.967824 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:21.969941 (+  2117us) service_pool.cc:224] Handling call
0808 01:51:21.992610 (+ 22669us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:21.993139 (+   529us) raft_consensus.cc:1534] Early marking committed up to index 5
0808 01:51:22.301973 (+308834us) log.cc:844] Serialized 14 byte log entry
0808 01:51:22.651190 (+349217us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:22.728680 (+ 77490us) log.cc:844] Serialized 26 byte log entry
0808 01:51:22.728851 (+   171us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:51:22.728903 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:22.728999 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:23.170824 (+441825us) raft_consensus.cc:1685] finished
0808 01:51:23.170847 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:23.171024 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17102,"prepare.queue_time_us":356811,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":431,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":32017,"thread_start_us":527429,"threads_started":2,"wal-append.queue_time_us":465336}
I20260808 01:51:23.105801 14050 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.111480 14152 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:23.206046  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 170215us
I20260808 01:51:23.179378 14274 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:23.208020 13909 consensus_meta.cc:287] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.598s	user 0.013s	sys 0.002s
I20260808 01:51:23.111222 13893 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.179733 14284 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:23.203212 14262 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:23.276516 14262 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:23.266436 14284 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:23.196825 13391 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:23.269563  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 168932us
I20260808 01:51:23.230072 14288 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260808 01:51:23.303773  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 344880us
I20260808 01:51:23.308414  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:23.306077  9601 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:23.314132  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:23.266197 13909 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:23.391276  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 85295.4us
I20260808 01:51:23.254166 14171 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:23.283023 14267 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:23.389400  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:23.386118 14262 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:23.272320 14236 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:23.424381  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:23.393944 14163 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:23.429143  9556 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 10 index: 12. (index mismatch)
I20260808 01:51:23.432557  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:23.395433 13790 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:23.406270 14136 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:23.251952 14274 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:23.425550 14262 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:23.421936 14196 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:23.385908 13704 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260808 01:51:23.404453 14282 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:23.400436 14118 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:23.413662 14201 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:23.442829  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 218342us
I20260808 01:51:23.434618  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:23.394037 13909 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.447755 14161 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:23.406364 14247 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260808 01:51:23.410980 13888 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260808 01:51:23.433296 14227 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:23.408558 14171 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.442814 14136 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:23.482406 14136 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:23.483744 14284 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:23.421562 14267 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:23.460183 14283 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:23.489884 14283 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:23.490293 14283 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:23.490633  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2400) took 1108 ms. Trace:
I20260808 01:51:23.491114  9481 rpcz_store.cc:276] 0808 01:51:22.382195 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:22.493650 (+111455us) service_pool.cc:224] Handling call
0808 01:51:22.493988 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:23.078260 (+584272us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:23.490345 (+412085us) spinlock_profiling.cc:243] Waited 581 ms on lock 0x7b5400399c54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:23.490566 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1336967424}
I20260808 01:51:23.436087 14226 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:23.441716 14236 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:23.448897 14262 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:51:23.497936 14266 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.609s	user 0.000s	sys 0.012s
I20260808 01:51:23.470630 14161 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:23.500077 14269 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.233s	user 0.001s	sys 0.000s
I20260808 01:51:23.500849 14161 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:23.503291 14278 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:23.503867 14278 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:23.459687 14282 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:23.504882 14278 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.506417 14282 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:23.481359 13391 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:23.478058 14050 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:23.481104 14227 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:23.470618 14118 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.468264 14201 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.436069 14163 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:23.511186 14152 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:23.511427 14299 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:23.471390 14196 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:51:23.512275  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 687) took 1055 ms. Trace:
W20260808 01:51:23.515388 14289 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260808 01:51:23.515576  9550 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
W20260808 01:51:23.516451 14280 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
W20260808 01:51:23.517302  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 9949664c54584858ae8fc74c9dc07a4a (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:23.523468 14284 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:23.518564  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:23.522490  9473 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260808 01:51:23.506363 14278 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:23.443612 13790 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:23.531201 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 132149us
I20260808 01:51:23.535491  9478 rpcz_store.cc:276] 0808 01:51:22.457172 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:22.613679 (+156507us) service_pool.cc:224] Handling call
0808 01:51:22.614045 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:22.942098 (+328053us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:23.511966 (+569868us) spinlock_profiling.cc:243] Waited 327 ms on lock 0x7b54002c0954. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:23.512203 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":752070400}
I20260808 01:51:23.557773  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:23.558872  9551 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 11: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:51:23.527472 14011 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.617s	user 0.013s	sys 0.001s
I20260808 01:51:23.517691  9630 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:23.563100  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 128528us
I20260808 01:51:23.525602 13888 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:23.554128  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 93366.6us
I20260808 01:51:23.564535  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:23.565390  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:23.543998 14131 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:23.571434  9476 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260808 01:51:23.572839  9476 pending_rounds.cc:85] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 5
I20260808 01:51:23.573133  9476 pending_rounds.cc:107] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260808 01:51:23.573371  9476 raft_consensus.cc:2889] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:23.547592 14138 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:23.544601 13894 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:23.576498  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:23.577098  9555 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260808 01:51:23.578411  9551 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 10: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:23.550752 14050 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:23.584743  9552 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260808 01:51:23.531457 14226 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:23.591051  9630 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260808 01:51:23.591464  9637 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:51:23.591744  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:23.544457 14299 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.593398 14301 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.046s
W20260808 01:51:23.621915  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:23.632936  9559 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:23.577586 14011 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:23.634410  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:23.559557 14284 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.572407 14267 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:23.647325  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:23.647229  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:23.646562  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 58656.5us
I20260808 01:51:23.632095  9560 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:23.574018 14248 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:23.633512  9637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:23.657128  9637 raft_consensus.cc:606] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:23.657128  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:23.657568  9637 raft_consensus.cc:614] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:23.658172  9554 raft_consensus.cc:2393] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 13: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:23.661535  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:23.578768 14160 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:23.662766  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:23.578030 14138 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:23.677032  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:23.678424  9560 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:23.680665  9560 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 11 index: 5. (index mismatch)
I20260808 01:51:23.682760  9560 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260808 01:51:23.685387  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:23.686309  9551 raft_consensus.cc:2393] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 9: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:23.687999  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:23.690788 14160 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:23.691488 14136 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.693353 14160 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:23.693838 14136 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:23.672560  9552 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:23.674003 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:23.669847  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:23.705703  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet affdd24b03ae4ee795a90703613305f7 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:23.706084  9552 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260808 01:51:23.697747  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:23.704221  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:23.634183 13909 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:23.631892 14201 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:23.626816 14292 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:23.696049 14248 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:23.700448 10369 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:51:23.627978 14118 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:23.624027 14252 consensus_queue.cc:1059] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.155s
I20260808 01:51:23.654934 14011 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:23.582867 13888 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:23.582659 14131 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.654840 14301 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20260808 01:51:23.704939  9554 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
W20260808 01:51:23.719758  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:23.720577  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:23.743219  9560 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:23.743945  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:23.641598 13704 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.068s
I20260808 01:51:23.589727 14148 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.118s
I20260808 01:51:23.712627  9599 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:23.589491 14050 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.582221 13894 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.758455 13894 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:23.654788 14224 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
W20260808 01:51:23.709117 14280 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260808 01:51:23.674256 14267 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.065s
I20260808 01:51:23.719028 14305 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:23.748648  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:23.715926 14227 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:23.717128 14226 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:23.715209 14163 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:23.750420  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:23.735183 14011 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:23.720273 14299 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:23.769366  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:23.747172 14257 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.645s	user 0.003s	sys 0.009s
I20260808 01:51:23.705866 14262 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:23.779526 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 138962us
I20260808 01:51:23.764560 13704 consensus_queue.cc:1243] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 10.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260808 01:51:23.786881 10332 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:23.763686 14136 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:23.795550 14299 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:23.796221  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47717.2us
I20260808 01:51:23.796437 14050 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:23.797745  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:23.739264 14292 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:23.773654 14307 raft_consensus.cc:3055] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:51:23.806491  9521 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:23.794872  9560 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:23.815956  9472 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260808 01:51:23.778825 13894 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:23.812119  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:23.807308  9599 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:23.813311 13893 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:23.812979 14299 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:23.844449  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:23.798509 14131 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:23.813452 14237 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260808 01:51:23.819444  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:23.819959 14283 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.784577 14227 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:23.793115 13391 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:23.831442  9481 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:51:23.833541  9521 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:23.777801 14080 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260808 01:51:23.835557  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:23.813513 14307 raft_consensus.cc:740] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:23.821609 14292 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:23.864560 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:23.784752 14293 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
I20260808 01:51:23.780583 14150 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260808 01:51:23.855643  9560 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:32922
I20260808 01:51:23.814183 14201 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:23.859334  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:23.822171 14050 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:23.844257 14267 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260808 01:51:23.849396  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 2f740cd3515641a8b2c4add8f809dd82: Illegal state: Replica 52ccf4dcca1843f7bb29e9172e2d4ab4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:23.844282 13701 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:23.865684  9473 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:23.799191 14257 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.841593  9599 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:23.855583 14196 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:23.872462  9560 raft_consensus.cc:606] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:23.776682 14305 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.856451 14227 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:23.851164 13893 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.887972 10334 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:23.875569 14201 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:23.893296 14309 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260808 01:51:23.883776 14080 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:23.894510 13893 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:23.886816 14196 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.895622 13888 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:23.898015 14196 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:23.899477 14262 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:23.874578 14267 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:23.882985  9560 raft_consensus.cc:614] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:23.862777 13790 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:23.866818 14278 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:23.924220  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:23.924551  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 108358us
I20260808 01:51:23.925073  9560 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:23.911927  9556 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:23.926770  9599 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:23.889797 14227 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:23.915160  9559 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: []
W20260808 01:51:23.952710  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet e57f5706d264426eb123cb26c6f3b89c (from 52ccf4dcca1843f7bb29e9172e2d4ab4 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:23.965489  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:51:23.966912  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:23.967911  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:51:23.914000  9473 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:23.969064  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:51:23.916569 14267 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:23.984455 14305 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:23.970032  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:23.976262 13932 raft_consensus.cc:3055] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:23.977272 13704 raft_consensus.cc:3055] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:23.979360 14163 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:23.979108 14321 consensus_queue.cc:1059] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.007s
I20260808 01:51:23.979640 14262 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:23.980582 14178 raft_consensus.cc:3055] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Stepping down as leader of term 4
W20260808 01:51:23.982085 14313 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
W20260808 01:51:23.983731 14310 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.150s	user 0.002s	sys 0.000s
I20260808 01:51:24.034363 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:51:23.987542  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 60894.5us
W20260808 01:51:23.992637  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 227942us
I20260808 01:51:23.926987 14080 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:23.986183 14131 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:23.902980 14307 consensus_queue.cc:260] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:23.950709 14201 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:24.005043 13932 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:51:24.027472  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:24.010658 13704 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:24.008383 14163 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:24.008018 13888 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:24.021008  9521 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:24.028757  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:24.051409  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:24.022384 14305 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:24.055997  9478 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:24.057317 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:24.060503 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:24.024930 14161 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:24.022725 14178 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:24.061232 10369 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:24.032516 14248 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:24.028452 14181 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:24.067171 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:24.073894  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:24.074957  9478 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 12: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:24.074994 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:24.076236  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:24.078022  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:24.079378  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:24.080077  9478 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:24.081035  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:24.057466 14163 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:51:24.084913  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 66300.5us
I20260808 01:51:24.057883 13888 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:24.060182 14293 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:24.054631 14151 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:24.064366 14074 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:24.115396 14178 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.117497 14178 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:51:24.119016 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:24.119683 14283 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:24.066246 14305 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.120512 14293 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.123654  9637 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:24.032847 14262 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.068540 14321 raft_consensus.cc:3055] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:24.065815 14307 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:24.135249 13704 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.137287 13704 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:24.084081 14181 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.095311 14302 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:24.144878 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:24.097415  9521 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:24.099043 14257 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:24.099650 14317 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
W20260808 01:51:24.100760 14311 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260808 01:51:24.102377  9557 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260808 01:51:24.110661 14312 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:24.090848 14303 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.081s
I20260808 01:51:24.132382 14321 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:24.123932 14305 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:24.085064 14299 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:24.154225  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:24.163709 14293 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:24.164613 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:24.142518 14181 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:24.168728 14296 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.655s	user 0.002s	sys 0.006s
I20260808 01:51:24.172379 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:24.173303 14192 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:24.181491 14267 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.183279 14227 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:24.185084  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:24.191076 14158 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:24.191282 14196 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:24.191162 14201 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:24.192551  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:24.305514  9478 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:24.194425 14262 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:24.195227 13704 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:24.145424 13932 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.092667 13888 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:24.063630 14080 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.113978 14151 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.327239 14267 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:24.328711 14151 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:24.148407 14283 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:24.329746 14283 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.234239 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:24.254159 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 179440us
I20260808 01:51:24.254513  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:24.333684  9474 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:51:24.146807 14188 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.086s
I20260808 01:51:24.214634 14230 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:24.221728 14150 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:24.232291 14312 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:24.231817 14226 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:24.255609 14118 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:24.312235  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:24.316247  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2280) took 1218 ms. Trace:
I20260808 01:51:24.320611 10369 raft_consensus.cc:2368] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11.
I20260808 01:51:24.269050 14236 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:24.322218  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:24.323424 14321 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.339884  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:24.344118  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:24.347875  9630 rpcz_store.cc:276] 0808 01:51:23.098404 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:23.098750 (+   346us) service_pool.cc:224] Handling call
0808 01:51:23.517421 (+418671us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:23.591604 (+ 74183us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:24.315926 (+724322us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x7b540032cb50. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:24.316156 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":961770880}
I20260808 01:51:24.346560  9481 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:24.385982  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:24.218668 14011 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.421828  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:24.386991  9552 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
W20260808 01:51:24.347419 14296 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260808 01:51:24.343922 14330 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:24.314890 14158 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.310443 14196 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.338959 13704 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.342178 14312 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:24.285921 14192 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.425616  9556 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:24.418927  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 321662us
I20260808 01:51:24.463148  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:24.427903  9599 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:24.466009 14136 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:24.466554 14136 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:24.466996 14136 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:24.468014 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:24.468600 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:24.388301  9557 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:24.439471  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:24.469766 13391 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:24.470494  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:24.427078 14150 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.470441 13391 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.472366  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:24.473043  9478 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:51:24.473140 14150 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:24.474092  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:24.434877  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:24.440908 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:24.420136 13701 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:24.475276 13701 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.475751  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:24.477620 13701 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:24.481338 13888 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.416193 14237 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.033s
I20260808 01:51:24.488122 10362 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:51:24.428341 14163 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.421458 14236 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.490005  9478 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:51:24.490550 14236 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:24.491192 14236 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:24.451674 14224 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:24.492403 14224 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:24.492648 10363 raft_consensus.cc:2368] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9.
I20260808 01:51:24.492834  9556 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:51:24.493417 14224 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.495398 14192 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:24.495810  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:24.354466 14267 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:24.497437 14267 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.498190 14136 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.502411 14267 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:24.503592 14237 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:51:24.504222 14237 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:24.504654 14237 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:24.504972  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:24.505342 14227 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:24.491686 14236 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.511054  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:24.336082 14201 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.514232 14158 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:24.474251 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:24.515233 14163 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:24.513759  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:24.516067  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:24.440937 14337 raft_consensus.cc:3055] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:51:24.517410 14337 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:24.517017 14196 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:24.516171 14201 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:24.473158 13391 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:24.518931 14236 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:24.388837  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:24.518620 14337 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.520227 14337 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:24.523871 14293 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:51:24.524161  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:24.524241 13701 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:24.524839 10332 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:24.524887 13701 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.526286  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:24.526612  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 63570.7us
I20260808 01:51:24.527464  9521 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:24.527753 14325 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:24.528688  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:24.521054  9472 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 4. Current term is 5. Ops: []
I20260808 01:51:24.529333  9552 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:24.531242  9631 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:24.530089  9637 pending_rounds.cc:85] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 3
I20260808 01:51:24.548141  9637 pending_rounds.cc:107] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260808 01:51:24.548460  9637 raft_consensus.cc:2889] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:24.527369 13701 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:24.422379 14118 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.552223  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:24.555562  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:24.556915  9634 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 10: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:24.443800 14330 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.528323 14325 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:24.558319 14325 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:24.559657  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:24.560618  9632 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:24.557643 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:24.563999  9631 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:24.549933 14136 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:24.566099 13909 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:24.566725 14330 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:24.566844 13909 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:24.568275 13909 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.568946 14138 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.571969 13888 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:24.559242 14074 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:24.575048  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:24.524979 13704 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:24.576385  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2559) took 1177 ms. Trace:
I20260808 01:51:24.576910  9476 rpcz_store.cc:276] 0808 01:51:23.398612 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:23.399223 (+   611us) service_pool.cc:224] Handling call
0808 01:51:23.418331 (+ 19108us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:23.573800 (+155469us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:24.576089 (+1002289us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b5400403dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:24.576301 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":338242304}
I20260808 01:51:24.547187 14293 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.579527  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2255) took 1928 ms. Trace:
I20260808 01:51:24.580102  9633 rpcz_store.cc:276] 0808 01:51:22.650912 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:22.685599 (+ 34687us) service_pool.cc:224] Handling call
0808 01:51:22.931403 (+245804us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:22.932294 (+   891us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:24.579208 (+1646914us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b540008c350. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:24.579426 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":564411776}
I20260808 01:51:24.581355 14293 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:24.341300 14230 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.583067  9557 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:24.583847 14230 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:24.462087 14283 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:24.355242 14226 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.527491 14268 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:24.585798  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:24.589692  9552 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:51:24.596669  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:24.599931  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:24.603456  9633 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:24.602617  9523 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:24.608436  9523 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:24.574388 14118 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:24.570596 14138 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:24.622000  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:24.622983  9634 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:24.628317  9472 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260808 01:51:24.630474 10332 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:24.631903  9476 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260808 01:51:24.585600 14336 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:51:24.634270 14267 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:24.635037 14267 raft_consensus.cc:697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:24.636088 14267 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.637220 14118 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:24.637776 14118 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.640239 14118 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:24.612738 14173 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:24.618002 14303 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:24.653141  9524 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:24.656289  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:24.658361  9560 raft_consensus.cc:2393] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:24.617871 14252 raft_consensus.cc:2764] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260808 01:51:24.617892 14307 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:24.618817 14268 raft_consensus.cc:697] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:24.608481 14293 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260808 01:51:24.661531  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:24.662708  9556 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 13: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:24.657344  9634 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
W20260808 01:51:24.659653 14306 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.010s	sys 0.013s
I20260808 01:51:24.671835  9632 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:24.678941  9635 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:24.666124  9445 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:24.681169  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:24.689592  9634 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:24.694124  9633 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:51:24.690037  9445 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:24.699062  9600 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:24.663229 14307 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:24.703531 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:24.706869  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:24.654373 14118 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:24.723479  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:24.723225 14118 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.725108  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:24.663875 14074 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:24.713181  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:24.682240 14306 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.010s	sys 0.013s
I20260808 01:51:24.662961 14283 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:24.709854 14212 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:24.699447 14230 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:24.760599  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:24.761629  9560 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:24.763821  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 236439us
I20260808 01:51:24.711508 14011 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:24.716382 14307 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:24.771822  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:24.754168  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47456.3us
I20260808 01:51:24.775084  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:24.733568 14080 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:24.749542  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:24.732106 13701 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:51:24.764849  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:24.741364 14118 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:24.737107 14074 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:24.778239  9550 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:51:24.780766  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:24.785701  9521 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:24.787653  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:24.793068  9476 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 9
I20260808 01:51:24.758447 14230 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:24.800832  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:24.797629  9560 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:24.768848 14341 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:24.806087  9599 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:24.807555  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:24.809888 13945 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:24.798374 14074 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:24.790416 13701 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:24.811897 13701 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.812813 14286 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:24.815513  9521 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:24.817422 14348 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:51:24.817974  9521 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:24.818624  9475 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 6. (term mismatch)
I20260808 01:51:24.819109  9475 pending_rounds.cc:85] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 5
I20260808 01:51:24.819453  9475 pending_rounds.cc:107] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 11.6
I20260808 01:51:24.819870  9475 raft_consensus.cc:2889] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:24.757618 14283 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.823972  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:24.825341  9521 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:24.826347  9480 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2533) took 2734 ms (client timeout 1720 ms). Trace:
W20260808 01:51:24.826958  9480 rpcz_store.cc:269] 0808 01:51:22.087703 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:22.087955 (+   252us) service_pool.cc:224] Handling call
0808 01:51:24.803326 (+2715371us) spinlock_profiling.cc:243] Waited 830 ms on lock 0x7b5400392bd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:24.821622 (+ 18296us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1908522752}
I20260808 01:51:24.829281  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 520) took 1238 ms. Trace:
I20260808 01:51:24.829670  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:24.829883  9629 rpcz_store.cc:276] 0808 01:51:23.591270 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:23.591508 (+   238us) service_pool.cc:224] Handling call
0808 01:51:23.591784 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:23.768455 (+176671us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:23.768525 (+    70us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:23.768594 (+    69us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:51:23.768643 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:24.829016 (+1060373us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b540025bcd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:24.829070 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:24.829218 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":405640064}
I20260808 01:51:24.798590 14306 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:24.833849  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:24.834107  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:24.834548  9560 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:24.804806 14230 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:24.804407 14212 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:24.819005 14011 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:51:24.789947 14224 consensus_queue.cc:1059] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260808 01:51:24.823593 13945 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:24.783818 14080 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.778149 14312 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:24.824828 14236 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:24.845525  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:24.847748  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:24.828384 14286 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:24.850601  9550 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:24.853266  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:24.840039 14348 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260808 01:51:24.855995 14291 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.405s	user 0.000s	sys 0.003s
I20260808 01:51:24.861769 14283 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:24.863530  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2560) took 1464 ms. Trace:
I20260808 01:51:24.866045  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 695) took 1045 ms. Trace:
I20260808 01:51:24.903093  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:24.906307  9629 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:24.906605  9481 rpcz_store.cc:276] 0808 01:51:23.820445 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:23.859060 (+ 38615us) service_pool.cc:224] Handling call
0808 01:51:24.865806 (+1006746us) spinlock_profiling.cc:243] Waited 476 ms on lock 0x7b54002c1354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:24.865908 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1093901184}
I20260808 01:51:24.908315  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:24.909040  9630 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:24.865801 14080 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:24.867059 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:24.914412  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:24.915382  9629 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:24.914235 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:24.918089  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:24.868608 14312 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:24.869755 14344 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260808 01:51:24.834009 14226 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:24.870023 10365 raft_consensus.cc:2393] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 11: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:24.875540  9524 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:24.877656 13945 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:24.879117 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:24.879254 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:24.932459  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 57014.8us
I20260808 01:51:24.880620 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:24.881711  9521 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:24.882720 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:24.883061 14183 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:51:24.884647 14340 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:24.885402 13932 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:24.887938 14351 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.059s	user 0.002s	sys 0.000s
I20260808 01:51:24.943612  9560 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:24.893734  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2772) took 1265 ms. Trace:
I20260808 01:51:24.838414 14201 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:51:24.838449 14118 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:24.898090  9636 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260808 01:51:24.899314  9476 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:51:24.974224  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:24.914263  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:24.919704  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:24.976733  9633 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:24.977003  9630 raft_consensus.cc:2410] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260808 01:51:24.920297 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:24.953176  9558 rpcz_store.cc:276] 0808 01:51:23.628384 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:23.639062 (+ 10678us) service_pool.cc:224] Handling call
0808 01:51:23.666989 (+ 27927us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:23.757136 (+ 90147us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:23.757192 (+    56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:23.757266 (+    74us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:51:23.757318 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:24.893476 (+1136158us) spinlock_profiling.cc:243] Waited 89.7 ms on lock 0x7b540015bb54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:24.893528 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:24.893666 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":206265216}
I20260808 01:51:24.918831  9629 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:24.921319  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:24.905489  9477 rpcz_store.cc:276] 0808 01:51:23.399028 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:23.399362 (+   334us) service_pool.cc:224] Handling call
0808 01:51:23.406912 (+  7550us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:23.417119 (+ 10207us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:23.417163 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:23.433045 (+ 15882us) log.cc:844] Serialized 26 byte log entry
0808 01:51:23.552157 (+119112us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:51:23.566960 (+ 14803us) log.cc:844] Serialized 14 byte log entry
0808 01:51:23.573535 (+  6575us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:23.573632 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:24.859080 (+1285448us) spinlock_profiling.cc:243] Waited 54.5 ms on lock 0x7b18003e1400. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:51:24.863190 (+  4110us) raft_consensus.cc:1685] finished
0808 01:51:24.863307 (+   117us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:24.863461 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10540,"prepare.queue_time_us":14562,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":3960,"raft.queue_time_us":12950,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":17825,"spinlock_wait_cycles":125426432,"thread_start_us":84090,"threads_started":1,"wal-append.queue_time_us":126679}
I20260808 01:51:24.934458 14304 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:24.927480  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:24.852414 14236 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:24.778802 14268 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:24.930081 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:24.896154 14173 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:24.932052 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:24.879421 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:24.935165  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 53543.6us
I20260808 01:51:24.935092  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:24.940052  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:24.891590 14284 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:24.992676  9472 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:24.946341 14340 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:24.949327 14171 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:24.958206  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:24.958369  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:25.011292  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:24.960361  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:24.962100 14350 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260808 01:51:24.938738 14226 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:24.942435 14286 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260808 01:51:24.969022 14346 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.170s	user 0.000s	sys 0.000s
I20260808 01:51:24.920446  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:25.023123  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:25.024276  9635 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:25.024410  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:25.025806  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:24.972723  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:24.981930  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:24.926539 14349 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260808 01:51:24.942054 14252 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:24.993158  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:24.996155  9629 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260808 01:51:25.031077  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:24.997745  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:24.999243  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:25.003988 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 72051.3us
I20260808 01:51:24.987751 14201 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:25.036283  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:25.037048  9521 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:25.037292  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:25.008697 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:24.980798 14118 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:25.004071 14301 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:24.989363 14236 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:25.028000  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:25.041129  9477 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:25.034659 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:25.042474  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:25.028581  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:25.029592  9636 raft_consensus.cc:2393] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 12: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:24.953125 14080 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:25.032367  9633 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:51:25.064581  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2562) took 1562 ms. Trace:
I20260808 01:51:25.034196  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:25.065905  9634 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:25.066247  9472 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 12: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:25.066953  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:25.068358  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:25.069072  9634 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:25.070150  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:25.035329  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:25.071916  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:25.020061 14161 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.052s
I20260808 01:51:25.073048  9629 raft_consensus.cc:2410] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260808 01:51:25.076815  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:25.078292  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:25.065166  9475 rpcz_store.cc:276] 0808 01:51:23.502479 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:23.502783 (+   304us) service_pool.cc:224] Handling call
0808 01:51:24.803564 (+1300781us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:24.827054 (+ 23490us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:25.064178 (+237124us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b5400392bd0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:25.064458 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2990343040}
W20260808 01:51:25.078783  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:25.079447  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:25.044826 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:25.021560 14286 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:25.045804 14118 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:25.046545  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:25.087910  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2405) took 2096 ms. Trace:
W20260808 01:51:25.088762  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 51807.3us
I20260808 01:51:25.089614  9521 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:25.051424 14224 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:25.052376 14305 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:51:25.055222  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:25.055475  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2127) took 1423 ms. Trace:
I20260808 01:51:25.055835 14331 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.085s
I20260808 01:51:25.057579 14340 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:25.005806 14347 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.270s
I20260808 01:51:24.980374 14299 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.043s
I20260808 01:51:25.059644 14011 raft_consensus.cc:3055] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:25.007270 14304 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:25.080786  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:25.082357  9634 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 11: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:25.083611 14312 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260808 01:51:25.086115 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:25.076588  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:25.050814 14196 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:25.146051 14196 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:25.146631 14196 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:25.088485  9479 rpcz_store.cc:276] 0808 01:51:22.992038 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:22.992252 (+   214us) service_pool.cc:224] Handling call
0808 01:51:22.992596 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:23.109054 (+116458us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:25.087187 (+1978133us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54002dc3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:25.087795 (+   608us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":265743616}
I20260808 01:51:25.074412 14201 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:25.151329  9480 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260808 01:51:25.152653 14146 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:25.092810 14252 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:51:25.100323 14237 consensus_meta.cc:287] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.595s	user 0.014s	sys 0.002s
I20260808 01:51:25.101962  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:25.031625 14173 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:25.106892  9559 rpcz_store.cc:276] 0808 01:51:23.632383 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:23.690174 (+ 57791us) service_pool.cc:224] Handling call
0808 01:51:23.690493 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:23.964765 (+274272us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:25.055194 (+1090429us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b540016a3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:25.055404 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":515923712}
I20260808 01:51:25.106549  9560 raft_consensus.cc:2393] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 10: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:25.092856  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:25.029354 14226 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:25.118489  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:25.173254  9633 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:25.174031  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:25.174881  9634 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
W20260808 01:51:25.122658  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 84484.3us
W20260808 01:51:25.084880  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 126667us
I20260808 01:51:25.175904  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:25.176168  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:25.180691 14290 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 1.670s	user 0.000s	sys 0.012s
I20260808 01:51:25.140620  9636 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:25.103716 14131 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:25.141981 14161 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:25.145200  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:25.117523 14074 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:25.158191  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:25.159400  9635 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:25.161612  9477 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:25.112942 14224 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:25.115420 14171 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:25.065351 14192 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:25.036875 14325 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:25.169184 13421 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:25.165477 10361 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:25.120456  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:25.211414 13421 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:25.212000  9472 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:25.089076 14286 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:25.083215 14080 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:25.128046 14011 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:25.186308  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:25.184243  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:25.218664  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:25.157269 14312 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:25.189352  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:25.197134  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:25.200999 14274 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:25.203547  9476 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:25.204469  9635 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260808 01:51:25.191468 14161 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:25.221877  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:25.169291 14237 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:25.224524 13421 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:25.188656 14358 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:25.207613 14224 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:25.216413 14192 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:25.186622 14173 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:25.250753  9475 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:25.190586 14285 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260808 01:51:25.254799  9629 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 10: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:25.256512  9477 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260808 01:51:25.186614 14257 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260808 01:51:25.231630 14312 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:25.238268 14305 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:51:25.276962  9630 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: []
I20260808 01:51:25.265090 13421 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:25.231779 14347 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:25.289263 14347 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:25.289903 14347 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:25.295820 14173 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:25.250000 14161 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:25.203862 14230 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:25.198267 14252 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:25.296916 14074 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:25.308449 10363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 565) took 1212 ms. Trace:
I20260808 01:51:25.263108 14192 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:25.262650 14358 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:25.244014 14304 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:25.313863  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:25.302805 14011 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:25.267755  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:25.297798  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:25.355046  9559 raft_consensus.cc:2368] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11.
I20260808 01:51:25.340162 10363 rpcz_store.cc:276] 0808 01:51:24.096160 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:24.096593 (+   433us) service_pool.cc:224] Handling call
0808 01:51:25.307881 (+1211288us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b5400681d54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:25.308173 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":856704384}
I20260808 01:51:25.354734  9555 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:25.364451  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:25.367821  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:25.371418  9636 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: [11.6-11.6]
W20260808 01:51:25.381088  9473 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2567) took 1661 ms (client timeout 1561 ms). Trace:
I20260808 01:51:25.381286  9479 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
W20260808 01:51:25.381711  9473 rpcz_store.cc:269] 0808 01:51:23.697293 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:23.697478 (+   185us) service_pool.cc:224] Handling call
0808 01:51:25.358188 (+1660710us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b54002c0454. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:25.358347 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":364193152}
W20260808 01:51:25.351676 14138 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.723s	user 0.018s	sys 0.000s
I20260808 01:51:25.399336  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 694) took 1583 ms. Trace:
I20260808 01:51:25.399823  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:25.399850  9478 rpcz_store.cc:276] 0808 01:51:23.816076 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:24.491995 (+675919us) service_pool.cc:224] Handling call
0808 01:51:24.492314 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:24.529378 (+ 37064us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:24.529411 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:24.529455 (+    44us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:24.529487 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:25.398841 (+869354us) spinlock_profiling.cc:243] Waited 36.9 ms on lock 0x7b540005fd54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:25.398898 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:25.399258 (+   360us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":84779904}
I20260808 01:51:25.416326  9634 raft_consensus.cc:2410] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260808 01:51:25.277565 14283 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.110s
I20260808 01:51:25.419780  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:25.420502  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:25.420827 14011 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:25.430357 14307 consensus_meta.cc:287] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.650s	user 0.008s	sys 0.013s
I20260808 01:51:25.431751  9473 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:25.351599 14358 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:25.464148 14307 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:25.366998 14080 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:25.472340  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:25.473630 14362 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260808 01:51:25.477447  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:25.478411  9555 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:25.479558  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:25.480849  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:25.481601  9555 raft_consensus.cc:2368] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11.
I20260808 01:51:25.488438 14226 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:25.482457  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:25.495434  9555 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260808 01:51:25.534789 14325 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:25.535872  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 446286us
I20260808 01:51:25.536769  9472 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:51:25.541005 13421 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:25.542083 13421 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:25.547894  9555 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: []
W20260808 01:51:25.551875 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:25.552722  9521 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:25.561690  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 340586us
I20260808 01:51:25.569339  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:25.569644 13421 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:25.577311  9521 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:25.566169  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:25.582136 13421 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:25.582823 13421 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:25.583308 13421 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:25.561900 14074 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:25.563867 14324 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:25.597306 14173 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:25.608278 14312 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:25.597005  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:25.603389  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:25.589022 14285 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:25.607807 14347 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:25.608855 14324 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:25.626025  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:25.644114  9479 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:25.628647  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:25.645591  9480 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:25.658362  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:25.660897  9476 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260808 01:51:25.662599  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:25.613215 14173 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260808 01:51:25.665805 14236 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.576s	user 0.011s	sys 0.009s
I20260808 01:51:25.666332 14236 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:25.666949 14286 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:25.668905 14236 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:25.669559 14347 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:25.672638 14196 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.525s	user 0.022s	sys 0.000s
I20260808 01:51:25.685210  9480 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:25.691675 13421 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:25.691236 14080 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:25.696092  9472 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:25.713479 13421 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:25.615384 14230 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:25.703339  9474 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.500s	user 0.020s	sys 0.062s
W20260808 01:51:25.694445 14118 consensus_meta.cc:287] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.578s	user 0.041s	sys 0.042s
I20260808 01:51:25.711900  9479 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:25.673528 14196 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:25.669577 14286 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:25.695779 14074 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:25.613718 14285 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:25.691555 14138 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:25.648177 14361 consensus_queue.cc:1059] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.094s
W20260808 01:51:25.706192  9523 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.015s	sys 0.014s
W20260808 01:51:25.750062  9523 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.016s	sys 0.014s
I20260808 01:51:25.717998 14307 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:25.763612 14171 raft_consensus.cc:3055] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:25.764318 14171 raft_consensus.cc:740] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:51:25.772233  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:25.749426 14285 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:25.746578 14312 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:25.717195 14080 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:25.718667 14118 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:25.780679  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 583) took 1994 ms. Trace:
I20260808 01:51:25.784629  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:25.785812  9479 raft_consensus.cc:2393] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:25.781960  9554 rpcz_store.cc:276] 0808 01:51:23.786512 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:23.855586 (+ 69074us) service_pool.cc:224] Handling call
0808 01:51:23.858555 (+  2969us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:23.862655 (+  4100us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:23.862711 (+    56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:23.862754 (+    43us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:23.862787 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:25.780299 (+1917512us) spinlock_profiling.cc:243] Waited 3.82 ms on lock 0x7b54000ff854. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:25.780365 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:25.780592 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8785792}
I20260808 01:51:25.792941  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:25.793627 14324 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:25.793927  9560 raft_consensus.cc:2368] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
I20260808 01:51:25.794140 14321 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:25.795460  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2778) took 2165 ms. Trace:
I20260808 01:51:25.795996  9551 rpcz_store.cc:276] 0808 01:51:23.630243 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:23.687143 (+ 56900us) service_pool.cc:224] Handling call
0808 01:51:23.687581 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:23.707610 (+ 20029us) raft_consensus.cc:1534] Early marking committed up to index 5
0808 01:51:23.876131 (+168521us) log.cc:844] Serialized 14 byte log entry
0808 01:51:23.912858 (+ 36727us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:23.912919 (+    61us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:51:23.912967 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:25.795115 (+1882148us) spinlock_profiling.cc:243] Waited 19.7 ms on lock 0x7b5400325ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:25.795193 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:25.795369 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":645172,"prepare.queue_time_us":99272,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":10115,"raft.queue_time_us":112,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":581761,"spinlock_wait_cycles":45402368,"thread_start_us":192971,"threads_started":2,"wal-append.queue_time_us":184459,"wal-append.run_cpu_time_us":1804,"wal-append.run_wall_time_us":1046673}
I20260808 01:51:25.758065 14138 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:25.801806  9551 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260808 01:51:25.803124 14138 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:25.811635  9479 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260808 01:51:25.815457 14224 consensus_meta.cc:287] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.546s	user 0.028s	sys 0.010s
I20260808 01:51:25.811883 14118 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:25.814373 14080 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:25.795377 14371 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260808 01:51:25.811909 14196 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:25.826145  9554 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260808 01:51:25.765535 14171 consensus_queue.cc:260] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:25.840493 14171 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:25.857590 14356 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.518s	user 0.001s	sys 0.001s
I20260808 01:51:25.868218  9632 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:25.869302 14224 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:25.871929 14224 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:25.872581 14160 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:25.873409 14358 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:25.874804 10367 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:25.873198 14160 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:25.880880 14160 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:25.888803 14161 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.582s	user 0.009s	sys 0.013s
W20260808 01:51:25.894673 14146 consensus_meta.cc:287] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.741s	user 0.011s	sys 0.017s
I20260808 01:51:25.891248 14160 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:25.891268 14137 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:25.890991 14166 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.382s	user 0.031s	sys 0.034s
W20260808 01:51:25.912088 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:25.914383  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2808) took 1083 ms. Trace:
I20260808 01:51:25.922044  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:25.922852  9552 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:25.913180 14166 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.402s	user 0.032s	sys 0.034s
I20260808 01:51:25.930624  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2807) took 1118 ms. Trace:
I20260808 01:51:25.910826 14248 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:25.940248  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:25.941146  9559 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:25.109146  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:51:25.951663  9560 raft_consensus.cc:2393] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:25.948331  9555 rpcz_store.cc:276] 0808 01:51:24.811888 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:25.563844 (+751956us) service_pool.cc:224] Handling call
0808 01:51:25.589579 (+ 25735us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:25.589888 (+   309us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:51:25.749381 (+159493us) log.cc:844] Serialized 14 byte log entry
0808 01:51:25.930163 (+180782us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:25.930251 (+    88us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:25.930305 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:25.930393 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:25.930546 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":130,"prepare.queue_time_us":316138,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":11084,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"spinlock_wait_cycles":1263872,"thread_start_us":307437,"threads_started":2}
W20260808 01:51:25.938299 14379 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.140s	user 0.000s	sys 0.002s
I20260808 01:51:25.903475 14161 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:25.962117  9552 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:25.911733 14137 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:25.963338 14312 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:25.963764 14377 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260808 01:51:25.964056 14312 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:25.954664 14248 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:25.970268 14248 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:25.970903  9631 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:51:25.973207 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:25.977635  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2262) took 1448 ms. Trace:
I20260808 01:51:25.978293  9637 rpcz_store.cc:276] 0808 01:51:24.529299 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:24.529416 (+   117us) service_pool.cc:224] Handling call
0808 01:51:24.529806 (+   390us) raft_consensus.cc:1471] Updating replica for 4 ops
0808 01:51:24.644728 (+114922us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:24.644781 (+    53us) raft_consensus.cc:1539] Triggering prepare for 4 ops
0808 01:51:24.658626 (+ 13845us) log.cc:844] Serialized 104 byte log entry
0808 01:51:24.824084 (+165458us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:24.837340 (+ 13256us) log.cc:844] Serialized 14 byte log entry
0808 01:51:24.841045 (+  3705us) log.cc:844] Serialized 14 byte log entry
0808 01:51:24.843307 (+  2262us) log.cc:844] Serialized 14 byte log entry
0808 01:51:24.849633 (+  6326us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:24.849709 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:25.494095 (+644386us) spinlock_profiling.cc:243] Waited 44.1 ms on lock 0x7b18003f9460. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188ce62 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:51:25.860171 (+366076us) spinlock_profiling.cc:243] Waited 3.2 ms on lock 0x7b48002379d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:51:25.966955 (+106784us) spinlock_profiling.cc:243] Waited 4.95 ms on lock 0x7b18003f9460. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:51:25.977263 (+ 10308us) raft_consensus.cc:1685] finished
0808 01:51:25.977295 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:25.977540 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4934,"prepare.queue_time_us":94821,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":103,"raft.queue_time_us":14136,"raft.run_cpu_time_us":3837,"raft.run_wall_time_us":1040559,"spinlock_wait_cycles":230732544,"thread_start_us":112933,"threads_started":1,"wal-append.queue_time_us":245163}
I20260808 01:51:25.970386 14312 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:25.994105  9554 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: []
W20260808 01:51:25.982053 14192 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.614s	user 0.006s	sys 0.008s
I20260808 01:51:25.982107 14306 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:26.000523 14306 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:26.002159 14161 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:26.005815 14306 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:26.006584 14192 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:25.994218 14324 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:26.011740 14324 raft_consensus.cc:697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:26.012151  9475 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:26.015786  9477 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260808 01:51:25.992664 14295 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:26.017251  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 413941us
I20260808 01:51:26.016973 14295 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:26.018396  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:26.019248  9445 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.196:33455 timed out after 1.762s (SENT) [suppressed 14 similar messages]
I20260808 01:51:26.020752 14341 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:51:26.021721 14341 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:25.996219  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:26.025431  9560 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:51:26.030884 14295 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:26.031122 14299 consensus_queue.cc:1059] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:26.032223 14295 raft_consensus.cc:3055] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:26.032541 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:26.032728 14295 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:26.033819 14295 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:26.035079 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:26.035686  9560 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260808 01:51:26.034142 14321 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:26.037207  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:26.037415 14321 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:26.035380 14295 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:26.038026 14321 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:26.038018 14312 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:26.036175  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 698) took 1378 ms. Trace:
I20260808 01:51:26.039366 14299 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:26.047371 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:26.038378 14306 consensus_queue.cc:1059] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20260808 01:51:26.045661  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:26.051116 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:26.051925 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:26.046093 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:26.053236 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:26.038995  9524 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:26.053691  9444 leader_election.cc:400] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11.
I20260808 01:51:26.054170  9444 leader_election.cc:403] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260808 01:51:26.044721  9474 rpcz_store.cc:276] 0808 01:51:24.658024 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:24.867066 (+209042us) service_pool.cc:224] Handling call
0808 01:51:24.867380 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:24.867599 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 10
0808 01:51:25.013489 (+145890us) log.cc:844] Serialized 14 byte log entry
0808 01:51:25.715791 (+702302us) log.cc:844] Serialized 14 byte log entry
0808 01:51:26.035747 (+319956us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:26.035817 (+    70us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:51:26.035868 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:26.035952 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:26.036106 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":389935,"prepare.queue_time_us":102070,"prepare.run_cpu_time_us":129,"prepare.run_wall_time_us":133,"raft.queue_time_us":337086,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":10750,"spinlock_wait_cycles":18406272,"thread_start_us":759935,"threads_started":3,"wal-append.queue_time_us":203468}
I20260808 01:51:26.052999 10368 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 5.
W20260808 01:51:25.993719 14240 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.481s	user 0.009s	sys 0.018s
I20260808 01:51:26.058764  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:26.059255 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:26.061192  9523 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:26.066129 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:26.069697  9475 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260808 01:51:26.071630 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:26.067188 10361 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:26.070297  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:26.075093 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:26.075347  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:26.077183  9523 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:26.062572 14240 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.550s	user 0.009s	sys 0.018s
I20260808 01:51:26.060853 14192 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:26.055055 14267 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:26.057976 14146 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:26.081139  9444 leader_election.cc:400] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11.
I20260808 01:51:26.058737 14161 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:26.059640 13893 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:26.085116  9444 leader_election.cc:403] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260808 01:51:26.012923 14324 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:26.061004 14299 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:26.087953  9444 leader_election.cc:400] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
I20260808 01:51:26.089176  9444 leader_election.cc:403] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260808 01:51:26.055249 14201 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.027s
I20260808 01:51:26.071815 14367 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:26.071365 14368 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11.
I20260808 01:51:26.076820 13909 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:26.076654 14341 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:26.076644 14347 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:26.079679 14348 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:26.084144 14284 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:26.081573 14361 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:26.114240 14284 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:51:26.107051  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:26.114936 14284 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:26.058645 14312 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:26.090372 14267 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:26.093617 14146 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:26.094527 14192 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
I20260808 01:51:26.089427 14340 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11.
W20260808 01:51:26.107637  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 88609.8us
I20260808 01:51:26.117828  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:26.118752  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:26.119675  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:26.122781 10368 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:26.147527  9477 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:26.122941  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:26.095022 14161 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:26.084025 14285 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.131657 14312 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:26.186362 10369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 564) took 2116 ms (client timeout 1817 ms). Trace:
I20260808 01:51:26.115417 14341 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:26.186913 10369 rpcz_store.cc:269] 0808 01:51:24.066652 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:24.066879 (+   227us) service_pool.cc:224] Handling call
0808 01:51:26.182250 (+2115371us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b540064aad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:26.182453 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":575300992}
I20260808 01:51:26.188129 14285 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:26.132262  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2289) took 1146 ms. Trace:
I20260808 01:51:26.189038  9555 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:26.189244  9634 rpcz_store.cc:276] 0808 01:51:24.986181 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:25.175642 (+189461us) service_pool.cc:224] Handling call
0808 01:51:26.131975 (+956333us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b54003fb454. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:26.132186 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":550408576}
I20260808 01:51:26.132632 14299 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:26.190121 14161 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.191363 14146 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:25.915241  9556 rpcz_store.cc:276] 0808 01:51:24.831031 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:25.565021 (+733990us) service_pool.cc:224] Handling call
0808 01:51:25.565391 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:25.565676 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 5
0808 01:51:25.768833 (+203157us) log.cc:844] Serialized 14 byte log entry
0808 01:51:25.914044 (+145211us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:25.914102 (+    58us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:51:25.914135 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:25.914213 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:25.914328 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":333338,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":1299,"spinlock_wait_cycles":1770368,"thread_start_us":322831,"threads_started":2}
I20260808 01:51:26.122884 14145 consensus_queue.cc:1059] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
I20260808 01:51:26.101718 14224 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
W20260808 01:51:26.133770 10367 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 770) took 1667 ms (client timeout 1728 ms). Trace:
I20260808 01:51:26.137739 14267 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:26.138027 14137 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:26.180366  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62624.2us
I20260808 01:51:26.123840 14348 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.213438 14348 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:26.195933 10367 rpcz_store.cc:269] 0808 01:51:24.466738 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:24.547491 (+ 80753us) service_pool.cc:224] Handling call
0808 01:51:26.133560 (+1586069us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400646c54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:26.133700 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3019639552}
I20260808 01:51:26.221282 14348 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:26.221995 14348 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:26.227519 14348 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:26.193077 13909 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:26.126670 14274 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20260808 01:51:26.240520 14181 raft_consensus.cc:3055] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:26.241067 14181 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:26.241112  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:26.242081 14181 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.242713  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:26.243920 14368 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:26.245261 14237 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:26.243534 14181 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:26.249001  9481 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:26.255007  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:26.258307  9479 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 12: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:26.258133 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:26.197381 14145 raft_consensus.cc:3055] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:26.257169  9480 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:26.259768  9474 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:26.255404  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:26.268393  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:26.261217 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:26.268489  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:26.270089  9556 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:51:26.271071 13421 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:26.271705 13421 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:26.272197 13421 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:26.263127  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:26.283806 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:26.288903 14388 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260808 01:51:26.288903 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:26.291107  9556 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260808 01:51:26.279165 14367 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:26.297005 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:26.298851  9474 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:26.297005 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:26.264226 14145 raft_consensus.cc:740] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:26.303013 10368 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:26.310325  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:26.296118  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:26.315796  9637 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260808 01:51:26.293987 14237 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:26.318872  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:26.320582  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:26.321753  9479 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:26.321995  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:26.322513  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:26.300428 13701 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:26.293684 14285 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:26.323567  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2143) took 1356 ms. Trace:
I20260808 01:51:26.323345  9636 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:26.337970  9554 rpcz_store.cc:276] 0808 01:51:24.967281 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:25.834960 (+867679us) service_pool.cc:224] Handling call
0808 01:51:25.835344 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:25.994709 (+159365us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:26.323242 (+328533us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b540008acd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:26.323482 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":361908096}
W20260808 01:51:26.324957 10365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 566) took 2181 ms (client timeout 2227 ms). Trace:
W20260808 01:51:26.339480 10365 rpcz_store.cc:269] 0808 01:51:24.143926 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:24.152801 (+  8875us) service_pool.cc:224] Handling call
0808 01:51:26.324709 (+2171908us) spinlock_profiling.cc:243] Waited 599 ms on lock 0x7b54006108d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:26.324890 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1378554240}
I20260808 01:51:26.324851  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:26.341266  9637 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:26.328352  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 528) took 1400 ms. Trace:
I20260808 01:51:26.342429  9630 rpcz_store.cc:276] 0808 01:51:24.928435 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:24.989671 (+ 61236us) service_pool.cc:224] Handling call
0808 01:51:25.018105 (+ 28434us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:25.277544 (+259439us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:26.327997 (+1050453us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b54004204d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:26.328249 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":594720000}
I20260808 01:51:26.302503 14248 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:26.333096  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:26.298542 14146 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:26.334256  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:26.335237 10366 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:26.302719 14367 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:26.345435  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:26.347450  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:26.350838  9600 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:26.370837  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:26.372176  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:26.372421  9523 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:26.351768  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 55760.4us
I20260808 01:51:26.353173  9633 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:26.380419  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:26.372761 13421 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.353976  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:26.382421  9632 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:26.356406  9445 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:26.356743  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:26.378039  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 131749us
I20260808 01:51:26.383538  9554 raft_consensus.cc:2410] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 15 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260808 01:51:26.358242  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:26.358578  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:26.385001  9630 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 12
I20260808 01:51:26.358139 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:26.375838  9600 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:26.400441 10363 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260808 01:51:26.400931  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:26.401968  9637 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:26.348961 14305 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:26.404047  9557 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: []
W20260808 01:51:26.405529  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 49258.9us
I20260808 01:51:26.407770 13421 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:26.411931  9445 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:26.412521  9556 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:26.413196  9556 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:26.416864  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:26.417900  9445 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:26.356207 14336 raft_consensus.cc:3055] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:26.420737  9445 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:26.423367  9637 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 01:51:26.418111  9560 raft_consensus.cc:2393] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 9: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:26.415042  9556 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.352823 14224 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:26.457703 14224 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.461199 14224 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:26.488417  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 67910.9us
I20260808 01:51:26.489748 13421 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260808 01:51:26.467444  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:26.336829 14237 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.503584  9445 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:26.505338  9445 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:26.509760  9630 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:26.348691 14285 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.340318 13701 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:26.511833  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 139488us
I20260808 01:51:26.402163 13888 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260808 01:51:26.453709 13945 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.376s	user 0.035s	sys 0.058s
I20260808 01:51:26.453070 14262 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260808 01:51:26.523329  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:26.529311  9633 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:26.355227 14248 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.416477 14347 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:51:26.373457 14367 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:26.514715  9556 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:26.570744  9631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 527) took 1995 ms (client timeout 2278 ms). Trace:
W20260808 01:51:26.571228  9631 rpcz_store.cc:269] 0808 01:51:24.575738 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:24.575910 (+   172us) service_pool.cc:224] Handling call
0808 01:51:25.297598 (+721688us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b5800231698. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:51:26.570505 (+1272907us) spinlock_profiling.cc:243] Waited 615 ms on lock 0x7b540008d754. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:26.570636 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1438248576}
I20260808 01:51:26.453219 14192 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:26.531157 14224 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260808 01:51:26.535156  9632 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
W20260808 01:51:26.535567 13945 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.479s	user 0.035s	sys 0.058s
I20260808 01:51:26.577042 14224 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:26.577585 14224 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 5 FOLLOWER]: Advancing to term 6
I20260808 01:51:26.578109  9635 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:26.544425 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:26.546612 10366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 638) took 1637 ms. Trace:
I20260808 01:51:26.580374 10366 rpcz_store.cc:276] 0808 01:51:24.908990 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:24.988741 (+ 79751us) service_pool.cc:224] Handling call
0808 01:51:26.335002 (+1346261us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:26.367506 (+ 32504us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:26.546371 (+178865us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54006108d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:26.546533 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3033327872}
I20260808 01:51:26.530171 14274 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:26.583066 14274 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:26.416224 14340 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.121s
I20260808 01:51:26.583474 14274 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:51:26.549517 14321 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.508s	user 0.012s	sys 0.001s
I20260808 01:51:26.552466  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:26.552925 14248 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:26.359135 14146 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:26.661360 14283 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260808 01:51:26.661516  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 109198us
I20260808 01:51:26.662792 14146 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:26.535982 13888 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:26.558290 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:26.558951 14240 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:26.561054 13701 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:26.562992  9559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2137) took 1648 ms (client timeout 2178 ms). Trace:
I20260808 01:51:26.563643 14285 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:26.564227 14358 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:26.566272 14237 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:26.567821 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:26.569053 14196 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 01:51:26.569785 14341 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:26.587867  9475 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 5. (term mismatch)
I20260808 01:51:26.589542  9445 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:26.702198  9475 pending_rounds.cc:85] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 4
I20260808 01:51:26.702657  9475 pending_rounds.cc:107] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 10.5
I20260808 01:51:26.703189  9475 raft_consensus.cc:2889] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:51:26.531049 14304 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.856s	user 0.008s	sys 0.011s
W20260808 01:51:26.594388 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 453068us
I20260808 01:51:26.467519 14305 raft_consensus.cc:697] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:26.637564  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 261838us
I20260808 01:51:26.645676  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:26.455751 14336 raft_consensus.cc:740] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:26.648665  9636 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:26.675457 13888 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:26.677065 14240 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:26.651341  9632 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:26.718362 14237 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20260808 01:51:26.653060  9633 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260808 01:51:26.726771  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:26.726979 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:26.727162  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:26.698145 14161 consensus_queue.cc:1059] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.030s
I20260808 01:51:26.727625 10366 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:26.728135  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:26.728339  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:26.729265  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:26.729588  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:26.513559 14201 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:26.664678 10364 raft_consensus.cc:2393] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 11: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:26.657784  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:26.730399 14201 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:26.730947 14201 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:26.685263 14306 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:26.644982 14382 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
W20260808 01:51:26.656193 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 375003us
I20260808 01:51:26.726385 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:26.626876 14348 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:26.676263  9556 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260808 01:51:26.677968  9599 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:26.629639 14368 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260808 01:51:26.677508  9559 rpcz_store.cc:269] 0808 01:51:24.914915 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:25.470654 (+555739us) service_pool.cc:224] Handling call
0808 01:51:26.562739 (+1092085us) spinlock_profiling.cc:243] Waited 468 ms on lock 0x7b540029f4d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:26.562902 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1075612032}
I20260808 01:51:26.662968 14283 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 01:51:26.685602  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:26.676108 14284 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.528s	user 0.012s	sys 0.002s
I20260808 01:51:26.696920  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:26.678864 14248 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:26.722817 14341 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:26.675258 14173 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:26.705354 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:26.710604  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 121164us
I20260808 01:51:26.756932  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:26.757854  9633 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:51:26.650507  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2146) took 1681 ms. Trace:
I20260808 01:51:26.713277  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:26.713873  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:26.714699  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:26.694000 14295 consensus_meta.cc:287] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.653s	user 0.013s	sys 0.001s
I20260808 01:51:26.718518  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:26.686306 14298 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:26.742854 14361 raft_consensus.cc:3055] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:26.746039 14240 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:26.729839  9632 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:26.652433  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:26.659942  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:26.776629  9631 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:26.777936  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:26.732141 10361 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:26.468524 14145 consensus_queue.cc:260] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:26.733281  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:26.783362  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:26.739535  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:26.784152  9633 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:26.784868  9635 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 8: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:26.675189 14321 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:26.741657 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:26.787222 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45661.6us
W20260808 01:51:26.590299  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:26.748715  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 70844.4us
I20260808 01:51:26.789238  9599 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:26.737689  9600 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:26.758636  9555 rpcz_store.cc:276] 0808 01:51:24.968822 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:25.952824 (+984002us) service_pool.cc:224] Handling call
0808 01:51:25.975500 (+ 22676us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:26.189533 (+214033us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:26.650248 (+460715us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b54000b3bd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:26.650443 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":490589440}
I20260808 01:51:26.760435  9554 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 12: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:51:26.760959  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 46418.4us
I20260808 01:51:26.761077  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:26.763235  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:26.752486 14131 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
W20260808 01:51:26.709602 14304 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.062s	user 0.008s	sys 0.012s
I20260808 01:51:26.718580 14183 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:26.768141 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 62899.3us
I20260808 01:51:26.781893  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:26.761157 14403 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:26.792160  9630 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:26.760860 14237 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:26.794569 10365 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:26.811239  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:26.791812  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:26.765074 14368 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:26.744267 14348 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:26.806730 10334 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:26.766844 14196 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:26.820835  9636 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:26.822358  9636 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260808 01:51:26.823097  9632 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:26.744050 14306 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.795553  9635 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:26.773558 14361 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:26.813324  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:26.803637  9631 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:26.808101 14399 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:26.779987 14324 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:26.818367  9633 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: [9.5-11.6]
I20260808 01:51:26.758622 14284 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:26.818426  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 80854.3us
W20260808 01:51:26.809576  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 161415us
I20260808 01:51:26.760190 14248 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.824705  9632 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:26.819110 14341 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:51:26.833768  9635 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260808 01:51:26.835878  9445 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:26.780826 14298 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.839180  9630 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:26.840890  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:26.811210 14237 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:26.876963  9638 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260808 01:51:26.762902 14173 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.853720 14295 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:26.854893 14299 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:26.848732 14283 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:26.814631 14368 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:26.866977  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 77874.1us
I20260808 01:51:26.867565  9636 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 11 index: 5. (index mismatch)
W20260808 01:51:26.814921  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 81809.9us
I20260808 01:51:26.902837  9601 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:26.868204  9600 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:26.868791 13421 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:26.870175  9475 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260808 01:51:26.871526  9478 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:26.842139 14146 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:26.824519 14196 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:26.937628 14196 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:26.812750 14267 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260808 01:51:26.879355  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:26.941272  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62066.7us
I20260808 01:51:26.941156 14201 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:26.942891 13421 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:26.943445 13421 raft_consensus.cc:697] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:26.944269 13421 consensus_queue.cc:237] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.880062  9474 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260808 01:51:26.823650 14406 raft_consensus.cc:2764] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260808 01:51:26.766605 14336 consensus_queue.cc:260] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.812788 14183 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.805271 14305 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.816872 14403 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.892019 14295 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.865674 14386 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.098s
I20260808 01:51:26.892956 14321 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:26.894575 14400 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260808 01:51:26.981961 14262 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:26.994463 14262 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.996999 14262 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:26.998334 14368 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:26.896131 14398 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260808 01:51:26.897089 14393 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
I20260808 01:51:26.842317 14399 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:26.867046 14397 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:27.010571 14399 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:26.888378 14285 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.099s
I20260808 01:51:27.016772 14285 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:51:26.886111 14237 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:27.019233 14403 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:26.997185 14183 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:26.875695 14391 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.124s
I20260808 01:51:26.890431 14340 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election won for term 12
W20260808 01:51:26.986152 14338 consensus_queue.cc:1175] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6e7856bed1be43acaa821290d506ee97" responder_term: 12 status { last_received { term: 11 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 01:51:27.000267 14368 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:27.022993 14340 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:27.023964 14224 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:27.024148 14340 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.908643 14361 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.028061 14262 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:27.028726 14262 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.952314 14283 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:26.919781 14181 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.656s	user 0.003s	sys 0.010s
I20260808 01:51:26.920143 14299 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:26.922125 14284 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:27.031971 14299 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:27.032934 14299 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.033820 14295 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:27.034258 14299 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:26.922691 14408 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:51:27.036131 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:26.924356 14348 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:26.924098 14410 consensus_queue.cc:1059] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:51:26.926714 14306 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:27.038275 14306 raft_consensus.cc:3055] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Stepping down as leader of term 8
W20260808 01:51:27.038570 13909 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.804s	user 0.014s	sys 0.000s
I20260808 01:51:27.038842 14306 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:27.039093 13909 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:26.982802 14341 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:27.040364 14341 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:26.928157  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:27.042905  9478 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:27.043879 14367 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.045192  9477 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:26.929737 14248 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:26.931749 14298 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:26.932196 14240 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:27.061678  9629 raft_consensus.cc:2368] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13.
I20260808 01:51:27.062835 14240 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:27.063094 14348 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:26.935406  9556 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260808 01:51:26.936364 14173 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:26.937036 14146 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:51:26.947297  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 44617us
I20260808 01:51:27.067282  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:27.069185  9601 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:26.948156  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 80041.9us
I20260808 01:51:26.900682  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:26.942415  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:27.071038  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:27.071854  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 129521us
I20260808 01:51:27.063663 14348 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.072753  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:26.951411 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:27.073752 14347 raft_consensus.cc:3055] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:27.074259 14348 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:27.026511 14224 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:26.951716 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 104098us
I20260808 01:51:26.957424  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:27.079891  9475 raft_consensus.cc:2410] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 12 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260808 01:51:26.958469  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:27.082554 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:26.961272  9550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2136) took 2051 ms (client timeout 1771 ms). Trace:
W20260808 01:51:27.091499  9550 rpcz_store.cc:269] 0808 01:51:24.909459 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:24.910975 (+  1516us) service_pool.cc:224] Handling call
0808 01:51:26.960957 (+2049982us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b54000b4ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:26.961188 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4205697280}
W20260808 01:51:26.928192 14404 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260808 01:51:27.081319  9474 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:51:26.848318 14358 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.098s
I20260808 01:51:26.894948 14342 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:27.030725 14283 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:26.959124 13945 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:26.990978 14405 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:27.050717 13909 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:26.984627 14336 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:27.127897  9560 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260808 01:51:26.855702 14131 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:51:26.950660 14201 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:27.078744 14240 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260808 01:51:27.076632  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 176070us
I20260808 01:51:27.100433  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:27.142056  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:27.142977  9557 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:27.108907  9601 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:27.103973 14406 raft_consensus.cc:2764] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260808 01:51:27.116433  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:27.044643 14306 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.079383 14284 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:27.127031  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:27.144786  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:27.075686 14146 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:27.104377 14181 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:27.074442 14347 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:27.127934 14405 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:27.177086  9554 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 01:51:27.153957 13421 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:27.110561 14403 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:27.145605  9558 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:27.110198 13945 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:27.167351 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:27.161012  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:27.105510 14324 raft_consensus.cc:697] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:27.102164 14386 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
W20260808 01:51:27.189244 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 229192us
W20260808 01:51:27.190146  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 81341.9us
I20260808 01:51:27.144073 14397 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:27.192334 14342 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:27.194778  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 49330.1us
I20260808 01:51:27.122786 14293 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260808 01:51:27.199877  9555 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:51:27.179865 14201 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260808 01:51:27.220234 10364 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:27.178436 14284 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:27.178720 14131 consensus_queue.cc:1243] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 10.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260808 01:51:27.195384 14340 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:27.199963 14367 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:27.182696 14405 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:27.273698  9557 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:27.283041  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:27.224545 14306 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:27.204666 13945 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:27.150091 14226 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
I20260808 01:51:27.136897 13909 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:27.276561  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:27.279250  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:27.280429 13421 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:27.283876  9557 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:27.304965  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:27.179584 14181 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:27.284886  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 144037us
W20260808 01:51:27.294332  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:27.230656 14426 raft_consensus.cc:2764] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260808 01:51:27.272533 14403 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
W20260808 01:51:27.323895  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:27.295789 14274 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.712s	user 0.011s	sys 0.003s
I20260808 01:51:27.330574  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:27.334758  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 120215us
I20260808 01:51:27.335062  9557 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:27.330101 13421 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.339493  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:27.340341  9557 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:27.286959 14429 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.047s
I20260808 01:51:27.351186  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:27.289915 14340 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:27.355486  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:27.357894 14358 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260808 01:51:27.357892 14399 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:27.277872 14347 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.405237 14196 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:27.423556 14348 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:27.425968 14262 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:27.436450 14402 log.cc:927] Time spent T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.591s	user 0.004s	sys 0.000s
I20260808 01:51:27.438025 14341 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:27.349964 14432 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.181s
I20260808 01:51:27.440227 14295 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:27.451720 14367 raft_consensus.cc:3055] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:27.463299  9445 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:27.463760 14410 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Stepping down as leader of term 11
W20260808 01:51:27.549404  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 86223.6us
I20260808 01:51:27.549754 14410 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:27.546573 14367 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:27.551575 14367 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.551415 14410 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 11.5, Last appended by leader: 5, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.553158 14367 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:27.553587 14410 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:27.556048 14417 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.315s	user 0.001s	sys 0.000s
I20260808 01:51:27.556797 14262 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:27.558017 14262 raft_consensus.cc:697] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:27.472831 14304 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:27.361800 14284 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.563167 14304 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.565580 14304 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:27.478998 14401 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.674s	user 0.003s	sys 0.000s
I20260808 01:51:27.339782 14342 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.568305  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2442) took 1306 ms. Trace:
W20260808 01:51:27.485374  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:27.338959 14224 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:27.340147 14201 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:27.339481 14424 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:27.345821 13888 consensus_meta.cc:287] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.597s	user 0.014s	sys 0.000s
I20260808 01:51:27.573191  9479 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260808 01:51:27.568707  9472 rpcz_store.cc:276] 0808 01:51:26.262499 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:26.263908 (+  1409us) service_pool.cc:224] Handling call
0808 01:51:26.264521 (+   613us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:26.264915 (+   394us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:26.264943 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:26.284286 (+ 19343us) log.cc:844] Serialized 52 byte log entry
0808 01:51:26.630278 (+345992us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:51:26.805040 (+174762us) log.cc:844] Serialized 14 byte log entry
0808 01:51:26.813968 (+  8928us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:26.814061 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:27.478192 (+664131us) spinlock_profiling.cc:243] Waited 6.34 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:27.567970 (+ 89778us) raft_consensus.cc:1685] finished
0808 01:51:27.568073 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:27.568235 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":259446,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":74,"raft.queue_time_us":8942,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":4601,"spinlock_wait_cycles":14575488,"thread_start_us":432147,"threads_started":2,"wal-append.queue_time_us":446239}
I20260808 01:51:27.334244 14324 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:27.521613 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 298339us
W20260808 01:51:27.521549  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 170634us
I20260808 01:51:27.545233 14347 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:27.347828 14431 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:27.541018  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:27.602095  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:27.541193 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:27.513589 14237 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.362437 14426 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:51:27.337437 14274 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:27.558177 14422 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.214s	user 0.000s	sys 0.000s
I20260808 01:51:27.623370  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:27.624776 14399 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:27.532215 14430 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.072s
I20260808 01:51:27.625623  9551 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:27.625064 14341 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:51:27.626204 13909 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:27.628077 14196 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:27.628170 13945 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.628979 14196 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:27.629462 14196 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:27.629871 14196 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:27.569767 14295 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.632889 14295 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:27.636754 14237 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:27.531613 14358 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:51:27.337762 14428 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:27.603101 10365 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 13
W20260808 01:51:27.542642 14420 log.cc:927] Time spent T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.205s	user 0.000s	sys 0.000s
I20260808 01:51:27.575831 14424 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.697049  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:27.697275  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:27.605739 14312 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 01:51:27.656433 13945 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:27.538612 14348 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.606400 14431 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.608311  9559 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260808 01:51:27.579767 13888 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:27.575620 14201 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.615294 14361 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:27.616305 14340 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:27.617164  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2296) took 1313 ms. Trace:
I20260808 01:51:27.742280  9634 rpcz_store.cc:276] 0808 01:51:26.303936 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:26.304156 (+   220us) service_pool.cc:224] Handling call
0808 01:51:26.304471 (+   315us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:26.304878 (+   407us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:26.304908 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:26.317814 (+ 12906us) log.cc:844] Serialized 26 byte log entry
0808 01:51:26.658991 (+341177us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:26.659060 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:26.659133 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:27.435568 (+776435us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:27.616822 (+181254us) raft_consensus.cc:1685] finished
0808 01:51:27.616846 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:27.617084 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1554,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":3069,"spinlock_wait_cycles":33507072,"thread_start_us":300357,"threads_started":1,"wal-append.queue_time_us":443237}
I20260808 01:51:27.622974 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:27.639801 10361 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:27.466991 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:27.624842  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:27.649991 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:27.748519 13421 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:51:27.362588 14181 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.661006 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:27.519330  9524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.011s	sys 0.000s
W20260808 01:51:27.752506  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 795498us
W20260808 01:51:27.755049  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 113654us
I20260808 01:51:27.755492  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:27.756564  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:27.577319 14423 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.227s	user 0.000s	sys 0.001s
W20260808 01:51:27.530258  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 220093us
W20260808 01:51:27.617672 14299 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.579s	user 0.010s	sys 0.001s
I20260808 01:51:27.759958 14403 consensus_queue.cc:1059] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.004s
I20260808 01:51:27.764061 10365 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
W20260808 01:51:27.764176  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:27.764196 14340 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:27.760728  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:27.765276 14340 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:27.767693 14340 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:27.679102 14428 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:27.768169  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:27.768806 14428 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:27.769088  9481 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:27.583750 14414 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:27.772581 14414 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:27.773783 10366 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:27.776358 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:27.695724 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 72948.6us
I20260808 01:51:27.605321 14262 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:27.625137 14274 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:27.698897  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:27.699925  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:27.700908  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:27.782187 14428 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:27.782998  9629 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:27.784730 14274 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:27.785722 14274 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:27.625641 14430 consensus_queue.cc:1243] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260808 01:51:27.701823 10367 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260808 01:51:27.576021 14224 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:27.729687 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:27.795400  9479 raft_consensus.cc:2410] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260808 01:51:27.650331 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:27.748956 13421 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:51:27.736107 14368 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.713s	user 0.010s	sys 0.003s
I20260808 01:51:27.796562 13421 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:27.799294  9636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:27.800299  9636 raft_consensus.cc:606] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:27.800740  9636 raft_consensus.cc:614] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:51:27.808960  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:27.809688  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 03b17f21031f4ff4828c85b7332fcc2d (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:27.760779 14295 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:27.810732 14295 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:27.814544  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:27.824791 13421 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.825951  9476 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:27.826315 14224 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:27.827615 13421 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:27.828653 14295 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:27.771890  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 147169us
W20260808 01:51:27.687268  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 382398us
I20260808 01:51:27.816768 14397 consensus_queue.cc:1059] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.012s
I20260808 01:51:27.829989 14295 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:27.830354 14267 raft_consensus.cc:3055] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:27.830838 14267 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:27.830567 14295 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:27.831843 14267 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:27.832679  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:27.830639  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:27.833205 14267 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:27.833339  9478 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:27.833878  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:27.834437  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:27.835155  9478 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:27.835347  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:27.836616  9478 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 5. Current term is 6. Ops: []
I20260808 01:51:27.838055  9636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "244e080726394210b5467bf688813cf0"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:27.845502  9636 raft_consensus.cc:606] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:27.846223  9636 raft_consensus.cc:614] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:51:27.842010  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:27.843569  9521 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:27.839270  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 75756.2us
I20260808 01:51:27.851766  9524 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:27.914273  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 64359.5us
I20260808 01:51:27.915437  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:27.917300  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:27.920706  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 69071.2us
I20260808 01:51:27.748908 14444 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:27.930998  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:27.931615  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 244e080726394210b5467bf688813cf0 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:27.931921  9630 raft_consensus.cc:2376] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12. Re-sending same reply.
W20260808 01:51:27.854296 14439 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.116s	user 0.001s	sys 0.000s
I20260808 01:51:27.925050  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:27.926051  9524 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:27.918851  9371 catalog_manager.cc:5697] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 3 to 11. New cstate: current_term: 11 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:27.912235 14414 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:27.922801  9634 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260808 01:51:27.741736 14429 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.180s
I20260808 01:51:27.847571 14299 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:27.927474  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:27.950239  9632 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 13: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:27.853102 14173 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:27.949658  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:27.950243  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:27.934515  9521 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:27.852849 14368 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:27.916934 14338 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:51:27.945492  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:27.945598 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 112192us
W20260808 01:51:27.947116 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 111629us
I20260808 01:51:27.947892  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:27.942270  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:27.850373 14224 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:27.951072  9473 raft_consensus.cc:2376] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11. Re-sending same reply.
I20260808 01:51:27.985641  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:27.934294  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:27.945339  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:27.917215 14430 raft_consensus.cc:3055] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Stepping down as leader of term 10
W20260808 01:51:27.996059 14454 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260808 01:51:27.932140 14446 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260808 01:51:28.005141  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 70708.1us
I20260808 01:51:27.958241  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:28.006179  9521 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:28.007383  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:28.008956  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:28.009663  9632 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:27.957963 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:28.011690 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 53818.5us
I20260808 01:51:28.012674 14312 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:28.013110 14312 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:28.013443 14312 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:51:27.985962  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:27.945644  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:27.988165  9633 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:28.015890  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:27.991261 14338 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:51:28.017118 14338 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:28.017532 14338 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:27.992864 14348 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:27.956744 14299 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:27.937238 14428 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:27.943360 14441 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:27.915139 13909 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:27.939684 14161 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:27.970708  9600 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:28.004302  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 58900.7us
I20260808 01:51:27.923830 14304 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:28.026433 14304 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.024562 14441 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.028332 14338 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.017401  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:28.029588  9633 raft_consensus.cc:2393] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:28.024586 13909 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.030570  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:27.999402  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:28.031939  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:28.031952 13421 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:28.033167  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:28.033416 14441 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.034379  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:28.035802  9633 raft_consensus.cc:2410] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260808 01:51:28.037236  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:28.042989  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 43650.7us
I20260808 01:51:28.044369  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:28.047425 14441 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:28.050657 14441 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:28.055583 14441 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:28.056519  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:28.062484 13421 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260808 01:51:28.066094  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 60016.5us
I20260808 01:51:28.067128  9521 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:28.068229 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 60902.7us
W20260808 01:51:28.070531 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:28.072989  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:28.074101  9632 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:28.075256  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:27.945142 14425 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:28.083756  9629 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:27.958933 14358 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:28.087554  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:28.091197  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 120645us
I20260808 01:51:28.091810  9473 raft_consensus.cc:2410] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
W20260808 01:51:28.088762  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 143220us
I20260808 01:51:28.093390  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:28.093663  9632 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260808 01:51:28.093449  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:28.081758 14298 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:51:27.983021 14463 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:27.995030 14295 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:28.118309 13421 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:28.184398  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 92340.1us
I20260808 01:51:28.195611 13421 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:28.138847  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:27.954394 14368 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:28.000012 14274 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:27.996603 14430 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:28.005533 14446 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:28.188129 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:28.199795 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:51:28.084812 14415 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260808 01:51:28.077121 14224 raft_consensus.cc:697] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:28.210636  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 117338us
I20260808 01:51:27.941177 14444 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.216179  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:28.213711  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 289519us
I20260808 01:51:28.207510  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:28.092527 14342 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:27.959169 14173 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.002606 14406 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.205s
I20260808 01:51:28.201764 13701 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:28.228905 14173 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.226164 14201 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.238060  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:28.233150  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:28.225363  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:28.227416 14224 consensus_queue.cc:237] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.098870 14428 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260808 01:51:28.237322 14146 consensus_meta.cc:287] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.062s	user 0.012s	sys 0.000s
I20260808 01:51:28.195122 14463 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:28.259428 13421 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:28.261080  9629 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
W20260808 01:51:28.270413  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 45242.2us
W20260808 01:51:28.215380 14448 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.536s	user 0.002s	sys 0.000s
I20260808 01:51:28.263710  9481 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
W20260808 01:51:28.235766 14347 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.568s	user 0.011s	sys 0.001s
I20260808 01:51:28.202016 14368 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:28.271147  9629 raft_consensus.cc:606] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:28.275710  9629 raft_consensus.cc:614] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:28.275916  9630 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 8: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:51:28.277938  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:28.280857  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:28.223523 14295 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:28.289975  9481 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260808 01:51:28.216459 14446 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:28.256613 14403 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:28.186154 14358 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.331033  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:28.265750 14166 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:28.278155 13701 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:28.315120 14340 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:28.317175 14196 consensus_meta.cc:287] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.686s	user 0.006s	sys 0.005s
I20260808 01:51:28.319063 14431 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.262310 14427 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:28.251435 14342 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:28.273471 14173 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:28.250276 14298 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.072s
I20260808 01:51:28.277143 14463 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:28.320602 14424 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.271793 14146 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:28.329398 14453 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.396s	user 0.001s	sys 0.000s
I20260808 01:51:28.286099 14462 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:28.211679 14445 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.194s
W20260808 01:51:28.333127  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 7b5ea129398a42fb9981fca652027cd6: Illegal state: Replica 52ccf4dcca1843f7bb29e9172e2d4ab4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:51:28.382575  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 166493us
W20260808 01:51:28.380920  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 163744us
I20260808 01:51:28.338217  9473 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:28.308766 13421 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:28.386970 13421 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:28.387527 13421 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:28.292717 14295 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:28.355038  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 716) took 1269 ms. Trace:
I20260808 01:51:28.367153 14342 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:28.393260  9480 rpcz_store.cc:276] 0808 01:51:27.085379 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:27.088291 (+  2912us) service_pool.cc:224] Handling call
0808 01:51:27.088737 (+   446us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:27.107437 (+ 18700us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:27.107473 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:27.406365 (+298892us) spinlock_profiling.cc:243] Waited 3.29 ms on lock 0x7b2c00134ff8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:51:27.406993 (+   628us) log.cc:844] Serialized 26 byte log entry
0808 01:51:27.747372 (+340379us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:27.747427 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:27.747523 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:28.332874 (+585351us) raft_consensus.cc:1685] finished
0808 01:51:28.332896 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:28.354920 (+ 22024us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18154,"raft.queue_time_us":18724,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":15750,"spinlock_wait_cycles":7570816,"thread_start_us":339860,"threads_started":1,"wal-append.queue_time_us":776568}
W20260808 01:51:28.395457  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:28.437222  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:28.438396  9559 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:28.439862  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:28.441058  9554 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:28.444193  9550 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 6. Current term is 7. Ops: []
I20260808 01:51:28.371834 14298 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:28.362257 14340 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.363446 14196 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.455343  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:28.464450  9554 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 13: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:28.458299  9559 raft_consensus.cc:1217] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[4.3-12.6]   Dedup: 4.3->[7.4-12.6]
I20260808 01:51:28.333216 14430 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.383045 14462 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:28.404531 14295 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:28.465106  9559 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: [4.3-12.6]
I20260808 01:51:28.460342  9599 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:28.484937 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 296940us
I20260808 01:51:28.474534 13421 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:28.466061  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:28.466390  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:28.369655 14173 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:28.460134  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:28.477355  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 410284us
I20260808 01:51:28.428050 14358 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.480779  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:28.492823  9474 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 13: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:28.431818 14161 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:28.502280  9550 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 13: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:28.496680 10332 leader_election.cc:400] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13.
I20260808 01:51:28.427127 14427 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.052s
I20260808 01:51:28.505502 10332 leader_election.cc:403] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
W20260808 01:51:28.490183 14410 consensus_meta.cc:287] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.936s	user 0.016s	sys 0.002s
I20260808 01:51:28.496433  9521 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:28.504607  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 297196us
I20260808 01:51:28.436028 14248 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.052s
I20260808 01:51:28.510849  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:28.511991 13421 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:28.481413 14462 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:28.532979  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:28.557154  9481 raft_consensus.cc:2393] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:28.496125 14201 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:28.532701 14444 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:28.541733 14299 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.544736  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:28.548341 13888 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:28.549590 13945 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:28.551155 14459 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13.
I20260808 01:51:28.457712 14298 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.535079  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:28.555253  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:28.537256 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:28.604666  9480 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
W20260808 01:51:28.605465 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 68326.6us
I20260808 01:51:28.497745 14080 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:28.567251  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:28.571125 14145 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:28.570894 14201 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:28.575430 14306 consensus_meta.cc:287] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.284s	user 0.015s	sys 0.000s
W20260808 01:51:28.517765 14405 consensus_meta.cc:287] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.216s	user 0.014s	sys 0.002s
I20260808 01:51:28.459018 13701 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:28.508183 14161 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.617928  9551 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:28.595322 14274 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:28.599607 13945 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:28.577982 14312 consensus_meta.cc:287] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.562s	user 0.007s	sys 0.003s
I20260808 01:51:28.580243 14470 raft_consensus.cc:3055] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:28.538363 14166 raft_consensus.cc:3055] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:51:28.508435 14403 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260808 01:51:28.582747 14461 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.190s	user 0.000s	sys 0.001s
W20260808 01:51:28.584617 14465 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
W20260808 01:51:28.581997 14267 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.748s	user 0.011s	sys 0.000s
I20260808 01:51:28.612854 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:28.557960 14472 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:28.607712 13888 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:28.683391 14472 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.685927 14472 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.613621 14137 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:28.607482 14299 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260808 01:51:28.688133 14161 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.630745 14306 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:28.689098 14306 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:28.689245 14173 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:28.689865 14201 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.690138 14306 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.692327 14298 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.635736 13421 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:28.657122 14312 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:28.683507 13888 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.693845 13421 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:28.636655 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 169632us
I20260808 01:51:28.695964 13421 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:28.606916 14468 consensus_queue.cc:1059] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.097s
I20260808 01:51:28.696591 14312 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.630436 14459 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20260808 01:51:28.697139 14427 raft_consensus.cc:3055] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:51:28.697357 14312 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:28.697646 14427 raft_consensus.cc:740] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:28.614792 14080 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.695994 13888 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:28.698854 14427 consensus_queue.cc:260] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.700387 14340 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.701009 10369 raft_consensus.cc:2410] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260808 01:51:28.678422 14146 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:28.702355 14196 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.700573 14427 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:28.675028 14403 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:28.703760 14403 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:28.646399 14470 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:28.617347 14405 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:28.709141 14405 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:28.669818 14336 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.532s	user 0.016s	sys 0.000s
I20260808 01:51:28.710578 14348 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:28.711161 14348 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:28.712246 14441 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.652s	user 0.010s	sys 0.003s
I20260808 01:51:28.712860 14441 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.713943 14368 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.723160 14386 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:28.731062 14386 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:28.731949 14441 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.735474 14368 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.723712 13945 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:28.737648 13945 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:28.738242 13909 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.707887 14470 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.671409 14477 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:28.740164 14477 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:28.740247 14470 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:28.741247 14430 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:28.701074 14080 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:28.666497 14166 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:28.743108 14338 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.743397 14166 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:28.680370 14361 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.931s	user 0.014s	sys 0.000s
I20260808 01:51:28.747720 14477 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:28.745105 14166 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:28.751897 14463 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.756664 14342 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.757655 14428 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:28.758119 14428 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:28.759145 14428 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:28.760815 14428 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:28.764744 14446 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.765249 14295 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.767781 14295 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.769001 14342 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:28.771126 14304 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.772464 14403 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.754524 14463 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.780687 14403 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:28.788079 14446 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.737689 10366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 586) took 1117 ms. Trace:
I20260808 01:51:28.792793 10366 rpcz_store.cc:276] 0808 01:51:27.620779 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:27.620915 (+   136us) service_pool.cc:224] Handling call
0808 01:51:28.737434 (+1116519us) spinlock_profiling.cc:243] Waited 28.7 ms on lock 0x7b5400646c54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:28.737595 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":65932032}
I20260808 01:51:28.782236 14386 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.840437 14462 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:28.883183  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 718) took 1551 ms. Trace:
I20260808 01:51:28.993922  9477 rpcz_store.cc:276] 0808 01:51:27.330737 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:27.332212 (+  1475us) service_pool.cc:224] Handling call
0808 01:51:27.346567 (+ 14355us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:27.741479 (+394912us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:27.741534 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:28.332506 (+590972us) spinlock_profiling.cc:243] Waited 75.1 ms on lock 0x7b2c00134ff8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:51:28.354713 (+ 22207us) log.cc:844] Serialized 26 byte log entry
0808 01:51:28.607809 (+253096us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:28.607848 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:28.607922 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:28.881289 (+273367us) raft_consensus.cc:1685] finished
0808 01:51:28.881397 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:28.882472 (+  1075us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7822,"raft.queue_time_us":186746,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"spinlock_wait_cycles":172647296,"thread_start_us":616455,"threads_started":2,"wal-append.queue_time_us":337916}
W20260808 01:51:28.883970  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 372985us
I20260808 01:51:28.998088 14462 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.894032  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2458) took 1138 ms. Trace:
I20260808 01:51:28.897414  9445 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:28.899125  9476 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260808 01:51:28.605835  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:28.612854  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:29.023629  9481 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 13: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:29.005533  9474 rpcz_store.cc:276] 0808 01:51:27.755573 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:27.756321 (+   748us) service_pool.cc:224] Handling call
0808 01:51:28.893327 (+1137006us) spinlock_profiling.cc:243] Waited 735 ms on lock 0x7b540009a1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:28.893736 (+   409us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1691161984}
I20260808 01:51:28.900018  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2456) took 1145 ms. Trace:
I20260808 01:51:29.030265  9479 rpcz_store.cc:276] 0808 01:51:27.754844 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:27.755060 (+   216us) service_pool.cc:224] Handling call
0808 01:51:28.899397 (+1144337us) spinlock_profiling.cc:243] Waited 38.4 ms on lock 0x7b540019d7d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:28.899750 (+   353us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":88423680}
I20260808 01:51:29.031895  9474 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
W20260808 01:51:29.032493  9523 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.753s	user 0.016s	sys 0.008s
W20260808 01:51:29.033119  9523 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.754s	user 0.016s	sys 0.008s
I20260808 01:51:28.900509  9473 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
W20260808 01:51:28.610591  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 55488.3us
I20260808 01:51:29.035900  9555 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
W20260808 01:51:28.904138  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:29.038643 14148 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260808 01:51:28.638437  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 153636us
W20260808 01:51:28.907629  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:28.907665  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 3b8e0e8fda6744ad877514b11658b9d9 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
W20260808 01:51:29.048995  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:28.917562  9633 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260808 01:51:28.920239  9637 raft_consensus.cc:2368] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13.
I20260808 01:51:28.920455  9629 raft_consensus.cc:2410] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260808 01:51:28.925928 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:29.052874  9550 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260808 01:51:28.930779 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:28.932027 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:28.916436  9371 catalog_manager.cc:5697] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 6 to 10. New cstate: current_term: 10 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:28.937215 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:28.940109 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:28.945245 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:28.951282 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:28.955453  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2331) took 1003 ms. Trace:
I20260808 01:51:28.956037 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:28.957774 14137 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:28.967357 14181 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:28.968591 14224 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:29.062485 10368 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 13: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:29.063192 14224 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:29.066315  9550 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:32922
I20260808 01:51:28.971982 14240 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:28.979205 14284 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:28.980538 14324 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:28.981628 14341 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:28.982072 14348 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:28.985812 14368 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:28.891727  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2457) took 1136 ms. Trace:
W20260808 01:51:29.015753  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:29.054975 13945 consensus_queue.cc:1059] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260808 01:51:29.020603 14441 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:29.083313  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2461) took 1242 ms. Trace:
I20260808 01:51:29.029781  9477 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: [12.6-12.6]
I20260808 01:51:29.092338  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:29.092950 13945 consensus_queue.cc:1243] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260808 01:51:29.093307  9477 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:29.095388  9554 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:29.096329 13945 raft_consensus.cc:3055] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:29.049680  9555 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260808 01:51:29.096241 14490 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:51:29.097002 13945 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:29.096995 14462 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:51:29.097939  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:29.058979 10360 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 13: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:29.059514 14181 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:29.098915 13945 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:29.099581 14181 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:29.100481  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2314) took 1165 ms. Trace:
I20260808 01:51:29.099128 14470 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:29.027292  9480 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:29.100967 13945 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:29.060536 14137 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:29.102445  9637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2318) took 1407 ms (client timeout 1865 ms). Trace:
I20260808 01:51:29.101709 14470 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:51:29.102980  9637 rpcz_store.cc:269] 0808 01:51:27.695203 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:27.696962 (+  1759us) service_pool.cc:224] Handling call
0808 01:51:29.102023 (+1405061us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54003faf54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:29.102346 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2802845440}
I20260808 01:51:29.104387 14470 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:29.021392  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 124493us
I20260808 01:51:29.076349  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:29.106875  9560 raft_consensus.cc:2410] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 12 because replica has last-logged OpId of term: 11 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260808 01:51:29.107842 14470 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:29.108459 14428 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:29.109295 14428 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:29.109604  9550 raft_consensus.cc:606] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:29.110213  9550 raft_consensus.cc:614] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:29.106915  9445 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:29.099130  9559 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 9: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:29.089816 14284 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:29.067638  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:29.111047 10366 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:29.095192  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:29.087669  9476 rpcz_store.cc:276] 0808 01:51:27.840754 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:28.243793 (+403039us) service_pool.cc:224] Handling call
0808 01:51:28.244363 (+   570us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:29.016379 (+772016us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:29.082724 (+ 66345us) spinlock_profiling.cc:243] Waited 654 ms on lock 0x7b5400404cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:29.083189 (+   465us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1503830400}
I20260808 01:51:29.100976  9633 rpcz_store.cc:276] 0808 01:51:27.934867 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:28.038500 (+103633us) service_pool.cc:224] Handling call
0808 01:51:28.038942 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:29.050283 (+1011341us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:29.100106 (+ 49823us) spinlock_profiling.cc:243] Waited 877 ms on lock 0x7b54004218d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:29.100382 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2016087808}
I20260808 01:51:29.057643  9630 rpcz_store.cc:276] 0808 01:51:27.951514 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:27.953256 (+  1742us) service_pool.cc:224] Handling call
0808 01:51:28.954480 (+1001224us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x7b54000dfa54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:28.954919 (+   439us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":665254656}
I20260808 01:51:29.126051 14463 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:29.127171 14463 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:29.130550 14499 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.118s	user 0.000s	sys 0.002s
I20260808 01:51:29.131052 14441 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:51:29.126109  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:29.135923 14368 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:29.136451  9371 catalog_manager.cc:5697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:29.139421 14324 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:29.141944 10369 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:29.141584 14341 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:29.142993  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:29.144197 14341 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:29.144940  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:29.145375 14284 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:29.146970 14137 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:29.149278  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:29.149778  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:29.150619  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 460e1059f5504707b891c4b0e0ac6a09 (from 52ccf4dcca1843f7bb29e9172e2d4ab4 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:29.151247  9555 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:29.151134  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:29.150447  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:29.152100  9550 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:29.153394 14348 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:29.066566 14224 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:29.156487  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:29.159791  9474 raft_consensus.cc:2368] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13.
W20260808 01:51:29.162017  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:29.177649  9477 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:29.178977  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:29.179394 14368 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:29.180303  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:29.180076  9633 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:29.086267  9472 rpcz_store.cc:276] 0808 01:51:27.755255 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:27.755741 (+   486us) service_pool.cc:224] Handling call
0808 01:51:27.756137 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:27.853756 (+ 97619us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:51:27.917238 (+ 63482us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:27.917712 (+   474us) log.cc:844] Serialized 26 byte log entry
0808 01:51:28.256651 (+338939us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:51:28.256720 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:28.543354 (+286634us) spinlock_profiling.cc:243] Waited 91.9 ms on lock 0x7b540022b5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:28.543435 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:28.727654 (+184219us) spinlock_profiling.cc:243] Waited 7.92 ms on lock 0x7b4800211158. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:51:28.891024 (+163370us) raft_consensus.cc:1685] finished
0808 01:51:28.891092 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:28.891505 (+   413us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5566,"raft.run_cpu_time_us":10236,"raft.run_wall_time_us":813479,"spinlock_wait_cycles":229683840,"thread_start_us":271586,"threads_started":1,"wal-append.queue_time_us":428216}
I20260808 01:51:29.157866 14324 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:29.188026  9480 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:29.190613  9630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cacb9622c47c4737beff104125e98b77"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:29.197896  9550 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260808 01:51:29.196501  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:29.199106  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:29.197981  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:29.199193  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:29.203070  9478 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:29.203934  9474 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 12: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:51:29.203722  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 54805.3us
I20260808 01:51:29.201910  9630 raft_consensus.cc:606] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:29.208091  9630 raft_consensus.cc:614] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:29.188927 13701 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:29.198300  9445 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:29.217730  9473 raft_consensus.cc:2410] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260808 01:51:29.222510  9555 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:29.225525  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:29.229763  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:29.230363  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:29.211598 14463 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:29.224383  9560 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:29.234472  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:29.236091  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:29.236799  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:29.237143  9558 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:29.221743 14486 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:51:29.238205  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet fa45ab4daad84ee78514f98cb3738f23: Illegal state: Replica 259bcb3f0d254e8b9effac3277e654e0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:29.236202  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:29.240044  9632 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
W20260808 01:51:29.238705 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 48089.8us
I20260808 01:51:29.198712 14240 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:29.233807  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:29.236984  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:29.245424  9629 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:29.253391 14201 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:29.257513  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:29.255242  9638 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:29.257326  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:29.260588  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet cacb9622c47c4737beff104125e98b77 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:29.262180  9476 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 13: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:29.264750  9472 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:29.276573  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:29.278477  9477 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:29.278857  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:29.279098 14295 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:29.280375  9634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "df6982ec010444a8854140f7b64e985d"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:29.281194  9556 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:29.281318  9634 raft_consensus.cc:606] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:29.281773 14368 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:29.282667 14368 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:29.280123 14295 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:29.286108  9473 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 7. (term mismatch)
I20260808 01:51:29.243690  9371 catalog_manager.cc:5697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 6 to 7, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 7 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:29.259334 14428 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:29.286895  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:29.238432 14486 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:29.286798  9633 raft_consensus.cc:2376] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12. Re-sending same reply.
I20260808 01:51:29.286784  9473 pending_rounds.cc:85] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 6
W20260808 01:51:29.296795 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:29.292622  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260808 01:51:29.281517 14348 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:29.300158  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:29.257337 13701 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:29.305267  9636 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: []
I20260808 01:51:29.295631  9473 pending_rounds.cc:107] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260808 01:51:29.308728  9473 raft_consensus.cc:2889] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:29.310604 14459 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:29.311523 14459 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:29.313213 10369 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:29.291342 14224 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:29.315424  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:29.312810  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:29.319273  9552 raft_consensus.cc:2368] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9.
W20260808 01:51:29.323441  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 163863us
I20260808 01:51:29.290350 14181 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:29.323737  9630 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:29.328510  9556 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260808 01:51:29.332537  9560 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:29.333529  9634 raft_consensus.cc:2393] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 11: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:51:29.333513  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 116150us
I20260808 01:51:29.328101  9371 catalog_manager.cc:5697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 7 to 10, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 10 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:51:29.337950  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:29.343483 10366 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
W20260808 01:51:29.357734  9373 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50084 (request call id 232) took 38102 ms (client timeout 15000 ms). Trace:
I20260808 01:51:29.372943  9636 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260808 01:51:29.383740 14181 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:29.377596  9371 catalog_manager.cc:5697] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 8 to 11, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 11 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:29.404896  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:29.401995  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:29.342393 14224 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:29.410250  9599 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:29.415877 14501 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.151s	user 0.000s	sys 0.001s
I20260808 01:51:29.420145  9371 catalog_manager.cc:5697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 6 to 10. New cstate: current_term: 10 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:29.423153  9632 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260808 01:51:29.353534 14284 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:51:29.427003 14502 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.158s	user 0.000s	sys 0.002s
W20260808 01:51:29.427858  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 53655us
I20260808 01:51:29.433213  9479 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:29.400990 14348 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:29.435279  9552 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:29.438632  9556 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:29.386253 14240 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:29.437713  9630 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:29.448380  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:29.450778 14368 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:29.460124  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 170104us
I20260808 01:51:29.461270  9552 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:29.459213 14137 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:29.465193 14137 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:29.472915 14497 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.268s	user 0.002s	sys 0.002s
I20260808 01:51:29.474255 14137 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:29.470808  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:29.474766 14201 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:51:29.482107  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:29.476195  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:29.452723  9371 catalog_manager.cc:5697] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 6 to 8, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 8 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:29.478346 14284 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:29.490934 14284 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:29.492017  9479 raft_consensus.cc:2410] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260808 01:51:29.495527  9636 raft_consensus.cc:2393] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 13: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:29.496127  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:29.496908  9638 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:29.499934  9521 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:29.500443  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:29.463248 14240 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:29.507531  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:29.502300  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2459) took 1736 ms. Trace:
I20260808 01:51:29.488039  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:29.520875  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:29.523021 14405 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:29.523825 14405 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:29.526718  9635 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260808 01:51:29.514451  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:51:29.451814 14348 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:29.517776  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:29.537874  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 201226us
I20260808 01:51:29.500048 14295 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:29.518805  9634 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:51:29.535926  9600 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:29.544020 14405 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:29.549921  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:29.529471 14459 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:29.552309  9630 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 12: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:29.510458 14428 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:29.553514  9444 leader_election.cc:400] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13.
I20260808 01:51:29.513126  9475 rpcz_store.cc:276] 0808 01:51:27.766197 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:27.766757 (+   560us) service_pool.cc:224] Handling call
0808 01:51:27.767256 (+   499us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:27.767590 (+   334us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:27.767611 (+    21us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:28.895826 (+1128215us) spinlock_profiling.cc:243] Waited 70.8 ms on lock 0x7b2c00134ff8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:51:28.897301 (+  1475us) log.cc:844] Serialized 26 byte log entry
0808 01:51:29.121284 (+223983us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:51:29.125229 (+  3945us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:29.157625 (+ 32396us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:29.476085 (+318460us) raft_consensus.cc:1685] finished
0808 01:51:29.476117 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:29.501991 (+ 25874us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7288,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":154,"spinlock_wait_cycles":162827520,"thread_start_us":188850,"threads_started":1,"wal-append.queue_time_us":293522}
I20260808 01:51:29.563452  9600 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:29.563707  9636 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260808 01:51:29.523039 14450 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:29.528911 14490 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:29.580966  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:29.581269 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:29.581900 10360 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:51:29.583184  9635 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260808 01:51:29.583568 14342 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:29.587641  9600 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:29.581008  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:29.594197 14348 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:29.602738 14500 log.cc:927] Time spent T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.234s	user 0.001s	sys 0.001s
W20260808 01:51:29.604519  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 45977.9us
I20260808 01:51:29.607033  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:29.555030 14428 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:29.611891  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2608) took 1078 ms. Trace:
I20260808 01:51:29.627943  9634 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260808 01:51:29.560125 14486 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:29.640844  9555 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:29.645579  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:29.571696 13701 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:29.579051 13945 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:29.659198  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:29.661931  9638 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:29.666868  9551 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260808 01:51:29.622970  9481 rpcz_store.cc:276] 0808 01:51:28.533463 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:29.025307 (+491844us) service_pool.cc:224] Handling call
0808 01:51:29.051532 (+ 26225us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:29.052404 (+   872us) raft_consensus.cc:1534] Early marking committed up to index 4
0808 01:51:29.052443 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:29.053170 (+   727us) log.cc:844] Serialized 26 byte log entry
0808 01:51:29.122217 (+ 69047us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:51:29.122246 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:29.122305 (+    59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:29.611600 (+489295us) raft_consensus.cc:1685] finished
0808 01:51:29.611633 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:29.611814 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":718,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":275,"thread_start_us":68529,"threads_started":1,"wal-append.queue_time_us":156730}
I20260808 01:51:29.669538 14427 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:29.629834 14428 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:29.567116 14240 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:29.675346  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:29.677290  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 154703us
I20260808 01:51:29.677399  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:29.677634  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:29.681303  9554 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260808 01:51:29.686465  9480 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:51:29.681551 14430 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.126s
I20260808 01:51:29.681377  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:29.689360  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:29.673815 13945 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:29.694473 14470 consensus_meta.cc:287] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.572s	user 0.013s	sys 0.008s
W20260808 01:51:29.695041  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 73285.3us
I20260808 01:51:29.657685 14342 raft_consensus.cc:697] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:29.687497  9473 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:29.695364 13945 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:29.689392  9371 catalog_manager.cc:5697] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 5 to 9, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 9 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:29.707688 10334 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:29.711262  9371 catalog_manager.cc:5697] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 6 to 9, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 9 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:51:29.731667  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:29.734244 14166 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:29.739778  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:29.740321 14166 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:29.740803 14166 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:29.741437  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:29.742957 14284 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:29.743448  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:29.743957 14284 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:29.744500  9551 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 7: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:29.748142  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:29.750118  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 289744us
I20260808 01:51:29.752640  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:29.358346  9373 rpcz_store.cc:269] 0808 01:50:51.255282 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:51.262845 (+  7563us) service_pool.cc:224] Handling call
0808 01:51:29.357337 (+38094492us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:51:20.709135 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:51:20.710125 (+   990us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:51:20.710200 (+    75us) write_op.cc:435] Acquired schema lock
0808 01:51:20.710229 (+    29us) tablet.cc:662] Decoding operations
0808 01:51:20.712046 (+  1817us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:51:20.712235 (+   189us) write_op.cc:641] Partition lock acquired for write op
0808 01:51:20.712281 (+    46us) tablet.cc:685] Acquiring locks for 13 operations
0808 01:51:20.949870 (+237589us) tablet.cc:701] Row locks acquired
0808 01:51:20.954995 (+  5125us) write_op.cc:260] PREPARE: finished
0808 01:51:20.987550 (+ 32555us) write_op.cc:270] Start()
0808 01:51:21.039729 (+ 52179us) write_op.cc:276] Timestamp: P: 1786153880977677 usec, L: 0
0808 01:51:21.046284 (+  6555us) op_driver.cc:348] REPLICATION: starting
0808 01:51:21.047253 (+   969us) log.cc:844] Serialized 5188 byte log entry
0808 01:51:21.486904 (+439651us) spinlock_profiling.cc:243] Waited 4.66 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:22.134752 (+647848us) op_driver.cc:464] REPLICATION: finished
0808 01:51:22.732293 (+597541us) write_op.cc:301] APPLY: starting
0808 01:51:22.751863 (+ 19570us) tablet.cc:1367] starting BulkCheckPresence
0808 01:51:23.627236 (+875373us) tablet.cc:1370] finished BulkCheckPresence
0808 01:51:23.627288 (+    52us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:51:24.045690 (+418402us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:51:24.173771 (+128081us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=26,key_file_lookups=26,delta_file_lookups=26,mrs_lookups=0
0808 01:51:24.173835 (+    64us) write_op.cc:312] APPLY: finished
0808 01:51:24.174280 (+   445us) log.cc:844] Serialized 121 byte log entry
0808 01:51:24.462074 (+287794us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:51:24.487399 (+ 25325us) write_op.cc:454] Released schema lock
0808 01:51:24.487492 (+    93us) write_op.cc:341] FINISH: Updating metrics
0808 01:51:24.488091 (+   599us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:51:24.488122 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":109831,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":521397,"apply.run_cpu_time_us":144941,"apply.run_wall_time_us":1770965,"cfile_cache_hit":54,"cfile_cache_hit_bytes":58545,"num_ops":13,"prepare.queue_time_us":202936,"prepare.run_cpu_time_us":43903,"prepare.run_wall_time_us":436197,"raft.queue_time_us":481000,"raft.run_cpu_time_us":18403,"raft.run_wall_time_us":397678,"replication_time_us":1069115,"spinlock_wait_cycles":10718080,"thread_start_us":1119270,"threads_started":5,"wal-append.queue_time_us":529272,"wal-append.run_cpu_time_us":28502,"wal-append.run_wall_time_us":2709619}]]}
I20260808 01:51:29.670540 14486 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:51:29.758301  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:29.758960  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 62754.1us
I20260808 01:51:29.763500  9552 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260808 01:51:29.709909 14342 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:29.753599  9371 catalog_manager.cc:5697] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 6 to 8, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 8 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:51:29.641993 14450 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:29.744458 14284 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:29.774722  9599 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:29.772661  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 272880us
I20260808 01:51:29.735291 14224 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:29.717686  9630 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.5, Last appended by leader: 4, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:51:29.760215 14486 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:29.783661  9472 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:29.778627  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:29.782338  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:29.790825  9478 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:29.793392  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:29.777709 14284 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:29.792179  9551 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:29.797034  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:29.788494 14486 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:51:29.812892 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:29.794461 14405 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:29.810544  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:51:29.781759 14430 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260808 01:51:29.821401  9552 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:29.822947  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:29.812269  9371 catalog_manager.cc:5697] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 6 to 9, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 9 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:51:29.829763  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:29.825455  9599 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:29.839116  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:29.839607  9557 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260808 01:51:29.840752  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:29.841526  9556 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 9. (term mismatch)
I20260808 01:51:29.841792  9554 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 10: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:29.841965  9556 pending_rounds.cc:85] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 8
I20260808 01:51:29.842348  9556 pending_rounds.cc:107] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260808 01:51:29.842613  9556 raft_consensus.cc:2889] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:29.819726 14224 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:51:29.819787 14405 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:29.871461 14405 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:29.875675  9473 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:29.876387  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:29.884023  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 87079.6us
I20260808 01:51:29.884383  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:29.853612 14240 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:29.894886  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:29.865751 14181 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:29.895011  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:29.899377  9481 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:51:29.901225 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 425713us
W20260808 01:51:29.902640  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 52082.1us
I20260808 01:51:29.876276 13701 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:29.909992  9478 raft_consensus.cc:2410] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260808 01:51:29.903997  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:29.890347 14430 raft_consensus.cc:3055] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Stepping down as leader of term 5
I20260808 01:51:29.916379  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:29.916337  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:29.928918  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:29.931825  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 343995us
I20260808 01:51:29.932602  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:29.943799  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:29.945264  9521 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:29.913187 13701 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:29.947960  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:29.951591  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:29.931962 14240 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:29.928530 14430 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:29.953862  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:29.963753  9555 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:29.974982 14470 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:29.975741 14470 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:29.976253 14470 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:29.980854  9478 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260808 01:51:29.982172  9478 pending_rounds.cc:85] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 5
W20260808 01:51:29.982172  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 49895.7us
I20260808 01:51:29.983433  9478 pending_rounds.cc:107] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260808 01:51:29.984140  9478 raft_consensus.cc:2889] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:29.963479 14240 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:29.989801  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:30.003670  9475 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:29.992688  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:29.932475 14441 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.037s
I20260808 01:51:29.941051 14181 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:30.011871  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 209825us
I20260808 01:51:29.944633 14224 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.043s
W20260808 01:51:30.038165  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 103367us
I20260808 01:51:29.991396 14450 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:30.003378 14240 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:51:30.041425  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 57030.9us
I20260808 01:51:30.046936 14341 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:51:30.047200 14430 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:30.047888 14506 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.057s	user 0.002s	sys 0.000s
I20260808 01:51:30.048390 14262 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:30.050875 14181 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:30.057999  9629 consensus_meta.cc:287] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.772s	user 0.014s	sys 0.005s
I20260808 01:51:30.070989  9629 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:51:30.065167  9475 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:30.042411 13945 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:30.073575 13945 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.055224  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:30.054095 14341 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:30.085350  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:30.068414 14262 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:30.091907  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 730) took 1027 ms. Trace:
I20260808 01:51:30.093220  9477 rpcz_store.cc:276] 0808 01:51:29.064896 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:29.279760 (+214864us) service_pool.cc:224] Handling call
0808 01:51:29.283747 (+  3987us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:29.306100 (+ 22353us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:29.306133 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:29.338458 (+ 32325us) log.cc:844] Serialized 26 byte log entry
0808 01:51:29.793071 (+454613us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:29.793116 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:29.793203 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:30.046975 (+253772us) spinlock_profiling.cc:243] Waited 22.9 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:30.091487 (+ 44512us) raft_consensus.cc:1685] finished
0808 01:51:30.091524 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:30.091813 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17017,"raft.queue_time_us":27144,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":100,"spinlock_wait_cycles":52612992,"thread_start_us":388933,"threads_started":1,"wal-append.queue_time_us":559427}
I20260808 01:51:30.081156 13701 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.127121 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:30.128006 13701 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:30.128799 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:30.077400 14450 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260808 01:51:30.089876 14341 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:30.099175 14262 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:30.121196  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:30.138262  9551 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 10: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:51:30.139618  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 344022us
I20260808 01:51:30.142269 13945 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:30.166338  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 111231us
I20260808 01:51:30.098419 14490 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:51:30.204622  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:30.209004  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:30.211490 14430 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 5 FOLLOWER]: Advancing to term 6
W20260808 01:51:30.227650 14515 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260808 01:51:30.230314 14342 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:30.137888 14368 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.073s
I20260808 01:51:30.214752 14510 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260808 01:51:30.313781  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 508275us
I20260808 01:51:30.314080  9481 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:30.337097 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:30.331887  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 169268us
I20260808 01:51:30.343675 10360 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:51:30.337759 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:30.350549 10366 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: []
W20260808 01:51:30.331180 14516 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.128s	user 0.000s	sys 0.006s
W20260808 01:51:30.335537 14166 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.592s	user 0.012s	sys 0.008s
I20260808 01:51:30.359762 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:30.342554 10332 leader_election.cc:336] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.602s (SENT)
I20260808 01:51:30.360633  9524 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:30.359694 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:30.365075 10368 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 10: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:30.333840 14224 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:30.374224 14510 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:30.375077 14510 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.377292 14224 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.377460 14262 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.379469 14403 consensus_queue.cc:1059] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:51:30.379858 14224 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:30.381461 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47988.9us
I20260808 01:51:30.382290 14527 raft_consensus.cc:3055] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:30.382774 14527 raft_consensus.cc:740] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:30.384038  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:30.349406  9445 connection.cc:256] RPC call timeout handler was delayed by -1.0176228460040875s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
W20260808 01:51:30.401757  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 75174.7us
I20260808 01:51:30.352054 14405 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.356419 14441 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
W20260808 01:51:30.402395 14428 consensus_meta.cc:287] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.642s	user 0.014s	sys 0.001s
I20260808 01:51:30.412544  9445 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:30.350014 14342 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.377493 14510 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:30.405709 14224 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election won for term 11
W20260808 01:51:30.417717 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66870.2us
I20260808 01:51:30.371353 14201 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260808 01:51:30.365957 14166 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:30.432658 14342 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:30.449571 14224 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:30.454053  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1934302919835318s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1514s for a response
I20260808 01:51:30.383819 14527 consensus_queue.cc:260] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.415277 14368 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:30.459533  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:30.462095 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:30.441395 14450 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:30.473172  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:30.473769  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:30.472028  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 109487us
I20260808 01:51:30.477856  9638 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: []
W20260808 01:51:30.456256 14284 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.638s	user 0.012s	sys 0.006s
W20260808 01:51:30.478559 14520 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
I20260808 01:51:30.478881 14284 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.479609 14450 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:30.480203 14450 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:30.481403 14284 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:30.482236  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:30.483207 10366 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260808 01:51:30.483508 14368 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.484658  9633 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:30.486006 14368 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:30.440971 14428 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.486961  9629 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:30.487910  9630 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 13 index: 6. (term mismatch)
I20260808 01:51:30.488375  9630 pending_rounds.cc:85] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 5
I20260808 01:51:30.488649  9630 pending_rounds.cc:107] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 12.6
I20260808 01:51:30.488755 14428 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:30.488900  9630 raft_consensus.cc:2889] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:30.425758 14470 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.490060 14262 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:30.491223 14470 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:30.470371 14181 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:30.491952 14470 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:30.492376 14181 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.494369 14166 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:30.486999  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:30.495524  9632 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:30.498526 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:30.490752 14224 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.492784 14470 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:30.503836  9632 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
W20260808 01:51:30.461990  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 114566us
I20260808 01:51:30.450464 14441 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:30.520993 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 77540.6us
I20260808 01:51:30.503726 14324 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.027s
I20260808 01:51:30.529489  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:30.524778 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:30.531392 14449 consensus_queue.cc:1059] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.016s
I20260808 01:51:30.532496 14449 raft_consensus.cc:3055] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:30.532984 14449 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:30.533917 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:30.534157  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 112606us
W20260808 01:51:30.537962 14533 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260808 01:51:30.537422 14449 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:30.538982 14449 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:30.522302 14527 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:30.545033  9601 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:30.547091  9523 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:30.547247  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:30.549717  9523 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:30.559662 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:30.525820 14181 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:30.533761 14537 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:30.561816 14486 consensus_meta.cc:287] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.732s	user 0.014s	sys 0.000s
I20260808 01:51:30.581049 14486 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:30.583846 14486 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:30.554935 14441 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:30.523218 14470 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.591259 14543 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:30.592166 14543 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.592490 14441 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:30.593287 14470 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:30.593972 14470 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:30.579077 14405 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:30.522040 14368 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260808 01:51:30.522233 14262 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260808 01:51:30.525025 14341 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.579973 14537 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:30.589756 14240 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.515s	user 0.010s	sys 0.000s
I20260808 01:51:30.576157  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:30.607440  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 70944.2us
I20260808 01:51:30.563207 14284 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:30.617395  9599 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:30.617775 14284 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.621585 14284 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:30.627012 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:30.616184  9632 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:30.582580 14544 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260808 01:51:30.630841  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 62826.4us
W20260808 01:51:30.664206  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 119281us
I20260808 01:51:30.666435  9599 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:30.636685 14470 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:51:30.676118  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:30.650374 14490 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:30.662330 14428 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:30.662302 14544 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:30.630035 14240 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:30.694414 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 167948us
W20260808 01:51:30.702080  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 149340us
I20260808 01:51:30.707589  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:30.708693  9555 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:30.710163  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:30.717566  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:30.714083 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 214366us
I20260808 01:51:30.724663  9551 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 8: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:30.716940  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:30.691346 14428 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:30.733966  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:30.735169  9477 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:30.727198  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:30.661036 14542 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.080s
I20260808 01:51:30.693207 14543 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:30.699359 14405 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:30.739408  9560 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 12: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:30.741600  9629 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:30.743479  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:30.745959  9550 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:30.736600  9559 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:30.697505 14544 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:30.707113 14342 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260808 01:51:30.763463 14534 raft_consensus.cc:993] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:51:30.764127 14537 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:30.765242  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:30.766175  9560 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 13: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:30.725121 14470 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:30.739614 14428 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:30.777814  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:30.774430 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45585.7us
I20260808 01:51:30.717134 14284 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260808 01:51:30.780920 10332 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:30.783329  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 107369us
W20260808 01:51:30.792863  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:30.778769  9631 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:30.815140 10366 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260808 01:51:30.815225 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:30.816219 10369 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
W20260808 01:51:30.816504  9600 consensus_peers.cc:597] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c617cb141a34007b6c1b8af0d911923. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:30.819401 14522 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:51:30.822129  9559 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:30.776652 14470 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.835731  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:30.777755 14430 raft_consensus.cc:1081] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Signalling peer 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:51:30.769618 14536 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:51:30.752245 14545 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.095s
I20260808 01:51:30.860210  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:30.880466  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 140315us
I20260808 01:51:30.883780  9559 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:30.887166  9599 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:30.890111 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 135443us
I20260808 01:51:30.848031 14405 raft_consensus.cc:697] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:30.850868 14459 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:30.892462  9477 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:51:30.846907 14284 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:30.884847  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47698.5us
I20260808 01:51:30.912405  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:30.910559 10334 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:30.901206  9551 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "df6982ec010444a8854140f7b64e985d"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:53208
I20260808 01:51:30.916620  9551 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:51:30.917055  9551 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:30.896301 14341 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:30.881716 14450 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:30.924964 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:30.922883 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 141897us
I20260808 01:51:30.925024  9633 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
W20260808 01:51:30.930536  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 380960us
I20260808 01:51:30.932721 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:30.933560  9524 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:30.936494 10369 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:30.936998 14405 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.923563 14137 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:30.921068 14441 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:30.928277 14486 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:51:30.952267 14450 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:30.913384 14284 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:30.955282 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:30.950264  9444 leader_election.cc:400] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9.
I20260808 01:51:30.972179  9444 leader_election.cc:403] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260808 01:51:30.958573 14560 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:30.964260 14224 raft_consensus.cc:3055] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:51:30.967571 14470 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:30.968959 14553 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20260808 01:51:30.969970 14546 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.120s	user 0.000s	sys 0.001s
I20260808 01:51:30.972206 10369 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:30.951360  9523 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:30.974597 14224 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:30.992246 14224 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.993872 14224 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:30.953308 14137 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:51:31.004379 10332 leader_election.cc:336] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.635s (SENT)
I20260808 01:51:31.005205  9559 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
W20260808 01:51:31.005347  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 55192.2us
I20260808 01:51:31.011850  9551 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:31.014495  9601 leader_election.cc:400] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13.
I20260808 01:51:31.015157  9601 leader_election.cc:403] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260808 01:51:31.016215 14137 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:31.017580  9551 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:31.020951 14262 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:51:31.022845 14284 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:31.023980 14490 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:31.024497 14490 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:31.025528 14490 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:30.982014 14450 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9.
I20260808 01:51:31.033281  9554 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
W20260808 01:51:31.033807  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:31.034171 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:31.036228 14556 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.086s	user 0.002s	sys 0.000s
I20260808 01:51:31.021150  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:31.043105 10369 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:51:31.045055  9445 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:31.079219 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 103154us
I20260808 01:51:31.058306 14284 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:51:31.051880 14449 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:31.086908 14324 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:31.087788 14324 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:31.089700 14560 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:31.091842 14324 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:31.082928 14564 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:31.093700 14564 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:31.096272 14564 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:31.099112  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 78061.6us
I20260808 01:51:31.099141  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:31.100311  9480 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:51:31.101606  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:31.101987 10334 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:31.102896 14450 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:31.103293  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:31.103485 14324 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:31.104013 14324 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:31.104298 14262 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:51:31.104419 14324 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:51:31.104873  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 175943us
I20260808 01:51:31.105902  9524 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:31.107944 14562 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:31.108441 14562 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:31.108903 14562 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:31.109627 14568 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:31.082752 14490 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:31.110996  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 163652us
I20260808 01:51:31.111886  9523 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:31.114931 14470 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260808 01:51:31.116596  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 229543us
W20260808 01:51:31.119658  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:31.108191 14537 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:31.151098 14537 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:31.152500 10332 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.591s (SENT) [suppressed 13 similar messages]
W20260808 01:51:31.119028  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 209034us
I20260808 01:51:31.118971 13701 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:31.157397 10369 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260808 01:51:31.149115 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: true dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:31.147352  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:31.090372 14560 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:31.151698 14537 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:31.164628  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 44617.4us
I20260808 01:51:31.219563  9599 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:31.158833 13701 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:31.224687 13701 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:31.229641 14562 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:31.170603  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 64864.2us
I20260808 01:51:31.232270  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:31.162904  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:31.235387 14490 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:31.181473 14405 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260808 01:51:31.173519 14524 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:31.237473  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:31.183423  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 169068us
I20260808 01:51:31.215118 10368 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:31.226063  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:31.225317 14486 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:31.239490 14486 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:31.240012 14486 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:31.241065 14430 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:31.241580 14430 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:31.242028 14430 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:31.244165  9601 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:31.245307 13945 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:51:31.246941 14535 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:31.171829  9600 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:31.251163  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 79464.1us
I20260808 01:51:31.251649 14522 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:31.252200 14522 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:31.219498 14428 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:31.253252  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:31.254453  9480 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 6: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:31.234344 14560 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:31.259202  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 154473us
I20260808 01:51:31.154635 14284 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:31.262195  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:31.260629  9599 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:31.264117  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:31.170717 14510 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13.
I20260808 01:51:31.164523 14240 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:31.265404  9558 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 13 index: 6. (index mismatch)
I20260808 01:51:31.265484  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:31.263186 14449 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:31.267155  9559 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:31.267987  9550 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:31.257846 14536 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:51:31.263270  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:31.270004  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:31.273866  9632 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260808 01:51:31.276572  9560 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:51:31.276270  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:31.258642 14490 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:31.281455  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:31.280298  9524 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:31.283617  9599 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:31.164350 14544 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:31.288791  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:31.274231 14449 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:31.290643  9554 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260808 01:51:31.282307 14403 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:31.237764 14524 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:31.291915  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:31.298434  9560 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:31.298653  9550 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:31.265486 14522 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:51:31.276217 14536 raft_consensus.cc:697] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:31.267907 14542 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260808 01:51:31.274446 14224 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260808 01:51:31.308358 14536 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:31.310344 14522 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:31.279505 14545 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:31.304255  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:31.314852  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:31.315158  9555 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:31.315487  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:31.318590 14403 raft_consensus.cc:697] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:31.300421 14449 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:31.320210 14403 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:31.322391 14544 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:31.342392  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:31.318948 14527 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:31.344158 14527 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:31.344367  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:31.323604 14428 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:31.345361 14527 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:31.350838  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:31.285149 14324 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:31.352890 14224 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:31.353104  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:31.354099 14545 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:31.355310 14324 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:31.352170  9479 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:51:31.355551 14545 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:31.334764  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:31.313894 13701 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:31.357030  9480 raft_consensus.cc:2410] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260808 01:51:31.336567 14524 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:31.337858 14430 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:31.340097  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:31.349548  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:31.366909 10368 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:31.351341  9478 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:31.350396  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:31.368153 14428 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:31.363835  9599 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:31.356899  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:31.376991  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:31.377790  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 77618.4us
I20260808 01:51:31.378664  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:31.379079  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:31.379416  9472 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:51:31.377163  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 73639.5us
I20260808 01:51:31.377951  9557 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:31.381776 14552 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:31.394199  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:31.401871  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:31.397157  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:31.400143  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:31.374377 14428 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:31.374907 13701 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:31.374799 14542 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:31.389812  9524 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:31.387647 14530 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:31.404482  9560 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:31.387427 14552 raft_consensus.cc:697] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:31.395349  9550 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:51:31.433653  9472 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:51:31.437588 10334 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:31.393910 14224 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.041s
I20260808 01:51:31.366335 14284 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:31.443971  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46974.7us
W20260808 01:51:31.451330  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:31.451431 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 299017us
W20260808 01:51:31.503540  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 103508us
I20260808 01:51:31.505019  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:31.427868 14543 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:51:31.404488 14534 consensus_queue.cc:1059] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 11.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
W20260808 01:51:31.508688  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 117043us
I20260808 01:51:31.438773 14240 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:31.446363 14544 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:31.438414 14490 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 11.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.077s
I20260808 01:51:31.516700 14428 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:31.405143 14405 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.096s
I20260808 01:51:31.513355  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:31.524272  9472 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:31.438926 14368 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:31.441035 14403 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260808 01:51:31.526657  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:31.441721 14530 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:31.510923 14543 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:31.552820  9478 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
W20260808 01:51:31.557691  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 52906us
I20260808 01:51:31.531327 14341 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:51:31.535465 14368 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:31.567114  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:31.542734 14403 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:31.535673 14544 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:31.575937 14537 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:31.576870  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:31.548056 14543 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:31.577762  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:31.578543 14530 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:31.580308  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:31.562845 14441 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260808 01:51:31.552796 14552 consensus_queue.cc:237] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:31.583382  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:31.590411  9473 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:51:31.595260  9481 raft_consensus.cc:2393] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:31.549918 14342 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:31.599267  9476 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 12: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:31.586941 14542 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:31.626150  9472 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:31.588738 14368 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:31.594034 14403 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:31.592487 14510 raft_consensus.cc:3055] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:31.636193 14342 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:31.593135 14450 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:31.646178 10332 leader_election.cc:336] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.729s (SENT)
I20260808 01:51:31.625594 14535 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:31.622605 14341 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:31.630779 14542 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:31.574659 14449 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:31.593505 14486 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:31.643378 14510 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:31.652067 14450 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:31.676848  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:31.677451  9550 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
W20260808 01:51:31.670740 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:31.671360  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:31.674118 14544 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:31.662806 14341 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:31.678263  9555 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:31.679366  9524 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:31.646603 14240 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:31.694052  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:31.697762  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:31.700083 14450 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:31.701305  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:31.701944  9557 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260808 01:51:31.703053  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:31.704268  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:31.704851  9557 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:31.705209  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:31.705840  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:31.706617  9550 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:31.707994 14240 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:31.710875 14524 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:31.711632 14524 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:31.714147 14524 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:31.715207  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:31.716068  9554 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:31.706621 14430 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:31.733948  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:31.729276  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:31.770210  9559 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260808 01:51:31.775590  9558 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:31.785573  9560 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:31.786775  9552 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260808 01:51:31.784219  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:31.846257 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 200071us
I20260808 01:51:31.868652  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:32.124150  9477 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:31.922714  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:31.936578 14583 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
W20260808 01:51:31.936635 14584 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260808 01:51:31.873903 14510 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.5, Last appended by leader: 5, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:31.782836 14450 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:31.936647 14578 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:32.102006 14537 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:32.111280  9475 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260808 01:51:31.728864 14470 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:31.793503  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:32.114331  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:32.115255  9554 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
W20260808 01:51:32.159907  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 366464us
I20260808 01:51:32.161131  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:32.163223  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.114490  9555 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:32.117344  9480 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:32.117803  9632 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260808 01:51:32.118382 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:32.120049  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:31.727803 14542 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.5, Last appended by leader: 5, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:32.128656  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:32.138602  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:32.141003  9638 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260808 01:51:32.141773  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:32.167273  9634 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:32.120540 13945 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:32.150017  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.155871  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:32.155915  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.149202  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:32.099126 14522 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:32.121472 14224 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:32.168227  9637 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260808 01:51:32.114867 14591 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:32.168895 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:32.146870 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:32.177879  9476 pending_rounds.cc:85] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 6
I20260808 01:51:32.195225  9476 pending_rounds.cc:107] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260808 01:51:32.195590  9476 raft_consensus.cc:2889] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:32.136202 14545 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:32.150801 14534 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:32.201610 14534 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:32.168766 14450 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:32.203908 14590 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.354s	user 0.010s	sys 0.000s
I20260808 01:51:32.204269 14534 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:32.205005 14486 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:32.205173 14588 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:32.206548  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:32.194154  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:32.199236  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:32.208592  9629 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:32.200647 14545 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:32.210075 10369 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:32.205914  9636 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:32.157209 14536 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:32.121102 14284 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20260808 01:51:32.225157  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:32.180087 14441 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:32.236334  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:32.237334  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.236238  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:32.238485  9475 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:32.239120  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:32.242230  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:32.242931 14537 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:32.243647  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:32.193415 14470 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:32.244465  9479 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 13: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:32.245549 14450 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:32.246238 14450 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:32.246512  9477 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:32.188155 13945 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:32.195081 14224 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:32.248426 14224 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:32.249692 13945 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:32.216738 14560 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:32.242362 14341 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:32.245250 14470 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:32.242233  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:32.257289  9635 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:32.257622 10360 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:32.245709  9479 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:32.243494 14537 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:32.238184  9638 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:32.254921  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:32.260907  9478 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 12: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:32.262591  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:32.176941 14578 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:32.266237 14578 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:32.244490 14545 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:32.273357  9475 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:32.331061  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:32.333252  9599 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:32.334787 14240 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:32.335345 14240 raft_consensus.cc:697] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:32.336301 14240 consensus_queue.cc:237] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:32.340104  9554 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:32.238444 14542 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:32.342315  9629 raft_consensus.cc:2410] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260808 01:51:32.238703 14441 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:32.347941  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:32.193483 14522 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:32.356000  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.365938  9638 raft_consensus.cc:2410] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260808 01:51:32.276306 14560 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:32.215026 14591 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:32.323216 14403 consensus_meta.cc:287] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.683s	user 0.013s	sys 0.001s
I20260808 01:51:32.393004  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:32.394757  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:32.397686  9629 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:51:32.399384  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 66264.1us
I20260808 01:51:32.412487  9599 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:32.416965  9635 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:32.419917  9634 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:32.225050 14592 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.152s
I20260808 01:51:32.354410 14449 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:32.444142  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.437129  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.445274  9638 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:32.362157 14441 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:32.446825  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.240425 14536 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:32.445329  9633 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:32.448762  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:32.449872  9631 raft_consensus.cc:2410] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260808 01:51:32.451370 14536 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:32.455163  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.457531  9635 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:32.464293  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.466423  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.217079 14588 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:32.480368  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 319309us
I20260808 01:51:32.440344  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.436138 14524 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:32.481838  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:32.514421  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:32.418644 14403 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:32.511181 14560 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:32.515921 14449 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:32.535029  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:32.539561  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:32.529963 14524 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:32.533402 14543 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.904s	user 0.011s	sys 0.007s
I20260808 01:51:32.530722 14262 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:32.537940 14240 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:32.549957 14240 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:51:32.553465 14240 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:32.555982 14595 log.cc:927] Time spent T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.160s	user 0.000s	sys 0.000s
I20260808 01:51:32.556133 14522 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:32.556517 14403 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:32.558280 14588 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:32.432740  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:32.562570  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 447738us
W20260808 01:51:32.567266  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 134650us
I20260808 01:51:32.566937  9630 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:32.541751 14449 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:32.573655  9629 raft_consensus.cc:2410] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
W20260808 01:51:32.565585 14593 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.359s	user 0.000s	sys 0.002s
I20260808 01:51:32.565145 14284 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:32.563591 14262 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:32.576200 14403 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election won for term 12
W20260808 01:51:32.588560  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 49107.3us
I20260808 01:51:32.576881 14450 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:32.580229 14449 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:32.563028 14224 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:32.596076 14591 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:32.595794 14524 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:32.565837 14543 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:32.649752  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:32.669895 14586 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.001s	user 0.002s	sys 0.000s
W20260808 01:51:32.718750  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 119323us
I20260808 01:51:32.726899 14562 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:32.735494 14341 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:51:32.600952  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.736824  9635 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 13: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:32.737987  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:32.738281 14615 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.155s	user 0.001s	sys 0.001s
I20260808 01:51:32.739066  9635 raft_consensus.cc:2410] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260808 01:51:32.740999 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:32.741793 10368 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 5.
I20260808 01:51:32.742798  9629 raft_consensus.cc:2410] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260808 01:51:32.098253  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:51:32.749505 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:32.750048 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:32.751394 10367 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260808 01:51:32.752911 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:32.753566 10369 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:32.756819  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.757634  9631 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:51:32.758556  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:32.759310  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2221) took 1059 ms. Trace:
I20260808 01:51:32.759761  9552 rpcz_store.cc:276] 0808 01:51:31.700160 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:31.704274 (+  4114us) service_pool.cc:224] Handling call
0808 01:51:31.786486 (+ 82212us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:31.787295 (+   809us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:32.759086 (+971791us) spinlock_profiling.cc:243] Waited 81.8 ms on lock 0x7b5400257e50. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:32.759224 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":188084096}
I20260808 01:51:32.760322  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.761029  9631 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:32.762116  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:32.762830  9631 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:32.766812 14543 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:32.768046  9556 raft_consensus.cc:2368] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8.
I20260808 01:51:32.769469 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:32.769948 14582 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.198s	user 0.007s	sys 0.007s
I20260808 01:51:32.741474 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:32.729031  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2499) took 1370 ms. Trace:
I20260808 01:51:32.774828 10360 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:32.770340  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2223) took 1069 ms. Trace:
I20260808 01:51:32.730973 14342 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:32.778795  9473 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:32.600704 14284 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:32.776655  9556 rpcz_store.cc:276] 0808 01:51:31.700790 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:31.701032 (+   242us) service_pool.cc:224] Handling call
0808 01:51:32.769959 (+1068927us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540015bb54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:32.770176 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2451123072}
I20260808 01:51:32.786783  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:32.775262  9480 rpcz_store.cc:276] 0808 01:51:31.358314 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:31.358780 (+   466us) service_pool.cc:224] Handling call
0808 01:51:31.359112 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:32.164882 (+805770us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:32.728651 (+563769us) spinlock_profiling.cc:243] Waited 511 ms on lock 0x7b540009a6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:32.728936 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1175700352}
I20260808 01:51:32.600539 14262 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:32.790589 14470 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:32.790966  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 191212us
I20260808 01:51:32.791350 14470 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:32.793295  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:32.793797 14470 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:32.602614 14537 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:32.794615  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:32.723407 14611 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.158s	user 0.001s	sys 0.000s
I20260808 01:51:32.803040 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:32.804962 10369 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:32.800089  9629 raft_consensus.cc:2410] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260808 01:51:32.724532 14527 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:32.743068 14403 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:32.795127 14284 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:32.801025 14605 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:32.815122 14605 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:32.816411 14537 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:32.817095 14537 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:32.825341  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2229) took 1042 ms. Trace:
I20260808 01:51:32.829977  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.831020  9634 raft_consensus.cc:2410] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 14 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260808 01:51:32.831544  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:32.832055 14368 consensus_meta.cc:287] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.192s	user 0.015s	sys 0.000s
I20260808 01:51:32.788184 14224 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:32.833058 14224 consensus_queue.cc:1059] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 01:51:32.832782 14368 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:32.834019 14224 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:51:32.834839 14224 consensus_queue.cc:1243] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 10.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260808 01:51:32.807174  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.810158 14604 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:32.839447 14604 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:32.808894 14527 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:32.811794 14403 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:32.813213 14441 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:32.846851 13701 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:32.848227  9474 raft_consensus.cc:2368] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10.
I20260808 01:51:32.795262 14564 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:32.831987  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:32.847431 14527 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:32.827600  9551 rpcz_store.cc:276] 0808 01:51:31.783138 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:31.788921 (+  5783us) service_pool.cc:224] Handling call
0808 01:51:32.111844 (+322923us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:32.298868 (+187024us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:32.298899 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:32.353493 (+ 54594us) log.cc:844] Serialized 26 byte log entry
0808 01:51:32.570444 (+216951us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:51:32.570485 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:32.570564 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:32.825005 (+254441us) raft_consensus.cc:1685] finished
0808 01:51:32.825031 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:32.825238 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":23152,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":17321,"thread_start_us":362194,"threads_started":2,"wal-append.queue_time_us":425115}
W20260808 01:51:32.852973  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:32.844498 14604 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:32.832207  9630 raft_consensus.cc:3055] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:32.849936  9634 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:32.857661  9630 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:32.809391  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.745549  9638 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:32.859102  9629 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:51:32.860349  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.860670  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.854048  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.862612  9635 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
W20260808 01:51:32.863101 14530 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.002s	sys 0.006s
W20260808 01:51:32.863543 14530 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.002s	sys 0.006s
W20260808 01:51:32.798153 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 242986us
I20260808 01:51:32.854902 14564 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:32.853469  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:32.862336  9629 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:32.870977  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:32.872018  9600 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:32.874913  9600 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:32.877944  9600 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:32.883118  9631 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
W20260808 01:51:32.851902 14428 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.671s	user 0.004s	sys 0.009s
I20260808 01:51:32.892891  9635 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
W20260808 01:51:32.875934  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 141969us
I20260808 01:51:32.847599 13701 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:32.896347  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:32.899504  9633 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: [6.4-11.5]
I20260808 01:51:32.900274  9638 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260808 01:51:32.861707 14342 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:32.872839 14450 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:32.902516  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:32.915705  9637 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
W20260808 01:51:32.892505 14428 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.716s	user 0.005s	sys 0.009s
I20260808 01:51:32.885145 14591 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:51:32.891870 14240 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:32.876251 14405 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
I20260808 01:51:32.939815  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:32.936769  9521 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:32.905642 14441 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:32.938526 14631 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:32.905283 14368 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:32.941844  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:32.955425  9481 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:32.915076 14627 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:32.905236 14536 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:32.943250  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:32.934434 14342 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:32.944347  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:32.944744 14633 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:32.945734 14543 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:32.933800  9634 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:51:32.904110 13701 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:32.878417 14527 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.055s
I20260808 01:51:32.992938 14612 raft_consensus.cc:3055] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:32.993501 14612 raft_consensus.cc:740] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:32.989735 14441 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 01:51:32.951632  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:32.952946  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:32.885373 14522 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20260808 01:51:32.940490 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:32.937456  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:32.960507  9445 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:32.934514 14591 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:32.998373  9480 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 12: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:32.941366 14341 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Stepping down as leader of term 9
W20260808 01:51:33.002967 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 62766.7us
I20260808 01:51:32.980489 14536 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:32.983325 14627 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:33.007496 14522 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:33.007975 14166 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:33.008494 14166 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:33.008167 14522 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.008921 14166 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:33.009260 14627 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:33.002588 14341 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:33.011464 14636 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260808 01:51:33.012467 14341 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:32.974412  9481 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:51:32.983420  9476 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260808 01:51:32.985549  9521 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:32.986891  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:32.987233 14552 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:32.950670 14631 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:32.994994 14629 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:32.957646 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:32.938144 14240 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:32.936638 14604 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
W20260808 01:51:33.034310  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 94574.7us
I20260808 01:51:33.011737 14405 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:32.992977 14527 consensus_queue.cc:1243] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 7201605bb2f54c949a4ebe6e9b3b4b2a log is divergent from this leader: its last log entry 14.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
W20260808 01:51:33.038403  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 52933.6us
I20260808 01:51:33.000406 14633 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:32.985105 14470 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.044s
W20260808 01:51:33.037067 14542 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.684s	user 0.005s	sys 0.009s
I20260808 01:51:33.027150 14522 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:33.041622  9521 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:33.002732 14591 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:33.041088  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:33.014925 14536 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:33.056958  9521 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:33.058075  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:32.950256 14450 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.016495 14341 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:33.034296 14368 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:33.048902 14405 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:33.053455 14470 consensus_queue.cc:1243] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 15.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260808 01:51:33.056908  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:33.040184 14240 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:33.091897  9631 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:33.093701  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:33.073431  9521 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:33.079560  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:33.083827  9444 connection.cc:256] RPC call timeout handler was delayed by -1.2860766550293192s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1552s for a response
I20260808 01:51:33.051039 14543 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260808 01:51:33.098065  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1348766550072469s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17200000000000001s for a response
I20260808 01:51:33.098184  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:33.098572  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1033766550244763s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1755s for a response
W20260808 01:51:33.099125  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0124766550143249s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18560000000000001s for a response
I20260808 01:51:33.059904 14544 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:33.097348  9523 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:33.063061 14562 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:33.067582 14368 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:33.096325 14470 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.011s
I20260808 01:51:33.103915 14641 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:33.068511 14578 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:33.102715  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:33.096171 14631 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.108839  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:33.077464 14542 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:33.096890 14262 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.093405 14441 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:33.121235  9636 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:33.112735  9601 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:33.097523 14530 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:33.148444 14634 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260808 01:51:33.148994 14631 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:33.150125 14634 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:33.120529 14641 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:33.150908 14645 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:33.151054 14641 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:33.124815 14542 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:33.152154 14542 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:33.135192 14441 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:33.152868 14441 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:33.151691 14645 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.157614  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:33.043768 14629 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:33.121207 14543 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:33.160928  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:33.118124 14368 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:33.117237 14564 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:33.126948 14510 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:33.136732 14562 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election won for term 10
W20260808 01:51:33.187578  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 90360.4us
I20260808 01:51:33.194397  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:33.162312 14637 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260808 01:51:33.195407  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:33.155489 14470 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:33.197481 14648 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260808 01:51:33.198570  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:33.156561 14634 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.198840 14562 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:33.159278 14522 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:33.155135 14530 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.130726 14284 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.117660 14544 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:33.151424 14449 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:33.280826 10332 leader_election.cc:336] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.591s (SENT)
I20260808 01:51:33.302897 10332 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:33.293174  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 190582us
W20260808 01:51:33.294456  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:33.265467 14342 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:33.296110  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:33.306118  9444 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:33.328281  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:33.331593 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:33.331866 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 55653.2us
I20260808 01:51:33.337862 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:33.344053 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:33.345069 10369 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:33.281699 14470 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.183259 14633 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 11.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.347908 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:33.354462  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 242428us
I20260808 01:51:33.335568 14648 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:33.360430 10366 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:33.361135  9632 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:33.366331  9632 raft_consensus.cc:606] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:33.366916  9632 raft_consensus.cc:614] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:33.290454 14632 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.243s
I20260808 01:51:33.340873 14602 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:33.340468 14450 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:51:33.388607  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:33.419866  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 3b8e0e8fda6744ad877514b11658b9d9 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:33.422746  9481 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260808 01:51:33.430725  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:33.372011 14649 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:33.376492 14632 consensus_queue.cc:1243] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260808 01:51:33.435726 14664 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:33.436125  9524 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:33.439296 14605 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:33.439683  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:33.431945  9478 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 6.
I20260808 01:51:33.471505  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:33.435799  9481 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260808 01:51:33.439087  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:33.374598 14652 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260808 01:51:33.451409  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 338693us
I20260808 01:51:33.376111 14604 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:33.476801  9476 raft_consensus.cc:2368] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11.
I20260808 01:51:33.476632  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:33.423560 14262 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:33.477844  9601 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:33.484102  9478 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:33.426342 14543 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:33.486353  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:33.491137  9481 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:33.497603  9477 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 8: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:33.492223  9480 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:33.503628  9551 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260808 01:51:33.370823 14648 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.503731 14664 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:33.506031 14663 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:33.507009  9550 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: [8.6-12.8]
I20260808 01:51:33.391182 14450 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.517493  9523 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:33.520074  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:33.381367 14562 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:33.472858 14649 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.503757 14605 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:51:33.524544  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:33.525478  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:33.530738  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:33.616815  9633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:33.642849  9633 raft_consensus.cc:606] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:33.493863 14604 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:33.540328 14530 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:33.644809 14166 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.635s	user 0.011s	sys 0.004s
I20260808 01:51:33.526499 14543 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.522145 14664 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.518914 14645 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:33.690891  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 237129us
I20260808 01:51:33.663691 14449 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:33.698431  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:33.709631  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:33.711640  9478 raft_consensus.cc:1217] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Deduplicated request from leader. Original: 12.10->[14.11-14.11]   Dedup: 14.11->[]
I20260808 01:51:33.726783  9478 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260808 01:51:33.712539  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:33.728875  9480 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:33.730206  9477 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
W20260808 01:51:33.701193 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 47526.5us
I20260808 01:51:33.519999 14663 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.724113  9551 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
W20260808 01:51:33.676608 14654 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.244s	user 0.001s	sys 0.000s
W20260808 01:51:33.771000  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:33.697892 14649 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:33.772706  9559 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:33.693430 14650 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:33.668615 14166 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:33.694769 14634 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:33.697144 14664 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:33.777650 14552 consensus_meta.cc:287] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.738s	user 0.012s	sys 0.003s
I20260808 01:51:33.752290 14605 raft_consensus.cc:697] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:33.675161 14662 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.106s
I20260808 01:51:33.775961 14470 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:33.779685 14450 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:33.809592  9559 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:33.810889  9554 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260808 01:51:33.785698 14649 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:33.807709 14552 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:33.810109 14544 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:33.785132 14650 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:33.820482 14341 consensus_meta.cc:287] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.750s	user 0.014s	sys 0.001s
I20260808 01:51:33.897626 14650 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:33.852108 14470 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:33.898959 14470 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:33.901760 14470 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:33.902233 14629 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:33.870136 14612 consensus_queue.cc:260] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.827385 14166 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:33.904222 14612 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:33.904620  9555 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:33.864571 14648 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:33.865274  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:33.905448 10360 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260808 01:51:33.905578 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:33.850069 14668 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:33.906736 14668 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.906765 10369 raft_consensus.cc:2410] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260808 01:51:33.868065 14604 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:33.910254  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:33.913143 14648 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:33.828413  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:33.847190 14654 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260808 01:51:33.914852  9550 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:33.915058 14605 consensus_queue.cc:237] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.835201  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:33.840865 14543 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:33.917073 14668 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:33.813059  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:33.903167  9475 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 5. (term mismatch)
I20260808 01:51:33.907910  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2504) took 1759 ms. Trace:
I20260808 01:51:33.860764 14663 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:33.879262  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 124615us
I20260808 01:51:33.923513  9552 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:33.839649  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:33.924969  9557 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:33.913828 14648 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:33.923861  9473 rpcz_store.cc:276] 0808 01:51:32.148215 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:32.148382 (+   167us) service_pool.cc:224] Handling call
0808 01:51:32.182929 (+ 34547us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:32.786553 (+603624us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:33.907646 (+1121093us) spinlock_profiling.cc:243] Waited 590 ms on lock 0x7b54002dbed4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:33.907840 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1357139712}
W20260808 01:51:33.866302 14542 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.714s	user 0.010s	sys 0.001s
I20260808 01:51:33.933075  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:33.923455  9475 pending_rounds.cc:85] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 4
I20260808 01:51:33.929571  9480 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:33.934089  9552 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:33.931666  9478 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260808 01:51:33.931964 10360 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
W20260808 01:51:33.891994 14441 consensus_meta.cc:287] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.739s	user 0.015s	sys 0.000s
I20260808 01:51:33.934084  9475 pending_rounds.cc:107] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20260808 01:51:33.938297  9475 raft_consensus.cc:2889] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:33.936147  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:33.944835  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:33.947845  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:33.946596  9557 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:33.950662 10369 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:33.950878  9551 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:33.927898 14612 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:33.955498  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:33.955999  9551 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12. Ops: []
W20260808 01:51:33.956490  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.007s	sys 0.022s
W20260808 01:51:33.963688  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.634s	user 0.007s	sys 0.022s
I20260808 01:51:33.965121  9477 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260808 01:51:33.964282  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 636109us
I20260808 01:51:33.941100 14647 raft_consensus.cc:476] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Not starting pre-election -- already a leader
I20260808 01:51:33.976037  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:33.975140  9554 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260808 01:51:33.976749  9474 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:51:33.988487  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:33.989809  9552 raft_consensus.cc:2410] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260808 01:51:33.989284  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:33.943917 14441 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:33.940452 14542 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:33.995734  9637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2377) took 1849 ms (client timeout 1536 ms). Trace:
I20260808 01:51:33.996515  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:33.991247 14631 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.842s	user 0.011s	sys 0.005s
I20260808 01:51:33.998145  9550 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:33.979233 14450 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.013s
I20260808 01:51:33.999814  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:51:33.998154  9637 rpcz_store.cc:269] 0808 01:51:32.138848 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:32.193850 (+ 55002us) service_pool.cc:224] Handling call
0808 01:51:33.988127 (+1794277us) spinlock_profiling.cc:243] Waited 311 ms on lock 0x7b540027ef54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:33.988235 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":715312512}
I20260808 01:51:33.999922  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:34.000769  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 483418us
I20260808 01:51:34.002398  9552 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:33.959955 14612 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:34.008836  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:33.979262 14341 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.024s
I20260808 01:51:34.010174  9479 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:33.998631 14664 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:34.014156  9556 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:34.006338  9523 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:34.003849 14592 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.007s	sys 0.018s
W20260808 01:51:34.016630  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:34.012352  9554 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
W20260808 01:51:34.013496  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:34.008725 14641 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.857s	user 0.005s	sys 0.012s
I20260808 01:51:34.022416  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:34.022784 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:34.020326  9444 leader_election.cc:400] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8.
I20260808 01:51:34.003639 14449 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:34.012200 14341 consensus_queue.cc:1243] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 11.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260808 01:51:34.020819 14605 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:34.046833 14605 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:34.046662 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:34.047959 14605 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:34.048403 14648 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:34.044392 14663 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:34.051369 14663 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:34.053642 14663 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:34.056154 14591 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.978s	user 0.015s	sys 0.000s
I20260808 01:51:34.056798 14591 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:34.043994 14641 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:34.060570 14441 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:34.060971 14641 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:34.061610 14536 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.972s	user 0.008s	sys 0.006s
W20260808 01:51:34.063271 14240 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.963s	user 0.015s	sys 0.001s
I20260808 01:51:34.063023 14605 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:34.063746 14240 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:34.065119 14542 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:33.997722 14634 consensus_queue.cc:1059] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20260808 01:51:34.024513  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:34.062131 14510 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.875s	user 0.016s	sys 0.000s
W20260808 01:51:34.019786 14592 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.007s	sys 0.018s
I20260808 01:51:34.013978 14600 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.056s
I20260808 01:51:34.070472 14240 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:34.070744  9557 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:34.011312 14594 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.060s
I20260808 01:51:34.062059 14536 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:34.066478  9444 leader_election.cc:403] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
W20260808 01:51:34.095536  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 75354.8us
I20260808 01:51:34.096449 14536 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:34.096554 14224 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8.
I20260808 01:51:34.097908  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:34.072145  9552 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
W20260808 01:51:34.063079 14368 consensus_meta.cc:287] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.872s	user 0.013s	sys 0.004s
I20260808 01:51:34.075095  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:34.075424  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:34.027876  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:34.110275  9555 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 11
I20260808 01:51:34.076246  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:34.111510  9476 raft_consensus.cc:2393] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:34.076731  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:34.014164 14664 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:34.012092 14655 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:34.076695 14614 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:34.087343 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:34.162781 10369 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:34.088927 14591 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:34.090794  9560 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:34.093047 14612 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:34.098394  9477 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:34.100538  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:34.176149  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:34.108022 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:34.114837 14624 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:34.178666  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:34.115257 14594 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:34.117154  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:34.181890  9473 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:34.117813 14663 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260808 01:51:34.183573  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 83143us
I20260808 01:51:34.120165 14632 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:34.119740  9630 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:34.133195 14368 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:34.141260 14284 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:34.166257 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:34.180817  9472 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 12: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:51:34.180606 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 162015us
W20260808 01:51:34.182485 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 170450us
I20260808 01:51:34.079640 14542 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.11, Next index: 11, Last known committed idx: 10, Time since last communication: 0.088s
I20260808 01:51:34.231184 14542 consensus_queue.cc:1243] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 15.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260808 01:51:34.232380 14592 raft_consensus.cc:3055] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:34.233134 14592 raft_consensus.cc:740] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:34.238147  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 86590.2us
I20260808 01:51:34.184316  9523 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:34.224773  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:34.198238  9630 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:34.183780 14663 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:51:34.239355  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:34.239593 10366 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:34.181635 14594 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:34.240206 14428 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
W20260808 01:51:34.244357  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 60129.5us
I20260808 01:51:34.199527 14612 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:34.183457 14655 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:34.198061 14632 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:34.243956  9555 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:51:34.266624  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:34.268029  9554 raft_consensus.cc:2393] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:34.269244  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:34.270069  9554 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:34.252676 14428 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:34.275127  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:34.276046  9557 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:34.275434 14428 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:34.265257  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:34.289243  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:34.240518 14676 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.056s
I20260808 01:51:34.234468 14592 consensus_queue.cc:260] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:34.160378 14664 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:34.294409 14592 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:34.295451  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:34.296823  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:34.297544  9560 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:34.261869 14602 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.037s
I20260808 01:51:34.299016 14602 consensus_queue.cc:1243] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 10.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260808 01:51:34.301542 14605 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:34.165630 14614 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:34.305400  9629 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:34.306191  9629 raft_consensus.cc:606] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:34.298332  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:34.281368 14594 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 10.12, Last appended by leader: 12, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:34.309289 14594 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:34.262758 14544 consensus_queue.cc:1059] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.048s
I20260808 01:51:34.258949 10334 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:34.311474 14544 raft_consensus.cc:3055] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:34.312027 14544 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:51:34.312054 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 53218.8us
I20260808 01:51:34.313215 10334 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:34.313406 14544 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:34.271260  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:34.260191  9550 raft_consensus.cc:1217] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[4.3-13.6]   Dedup: 4.3->[7.4-13.6]
I20260808 01:51:34.247203 14668 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:34.246631 14663 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:34.345341  9630 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:34.263792  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:34.357579  9558 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:34.301446 14405 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:34.199378 14562 consensus_queue.cc:1059] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260808 01:51:34.360257 14562 raft_consensus.cc:3055] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:34.360678 14562 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:34.308306  9560 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:34.260761  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:34.333285  9550 pending_rounds.cc:85] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 3
I20260808 01:51:34.337481 14668 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:34.341912 14665 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.553s	user 0.002s	sys 0.000s
I20260808 01:51:34.361186 14537 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:34.368223  9556 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:34.368842  9550 pending_rounds.cc:107] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20260808 01:51:34.369199  9550 raft_consensus.cc:2889] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:34.364192 14682 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:34.370478 14682 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:34.248078 14542 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:51:34.365222 14490 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:51:34.371255 14668 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:34.371515 14490 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:34.372576 14490 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:34.374190 14490 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:34.348639  9634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:34.349246 14605 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:34.353520 14663 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:34.338220 14166 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:51:34.277496 14428 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:34.427671  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2388) took 1316 ms. Trace:
W20260808 01:51:34.366416 14667 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.548s	user 0.001s	sys 0.000s
I20260808 01:51:34.363637 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:34.429414  9634 raft_consensus.cc:606] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:34.429795  9634 raft_consensus.cc:614] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:34.376765 14544 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:34.365330 14655 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:34.378645 14614 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:34.370951 14682 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:51:34.361208 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46515.1us
I20260808 01:51:34.361642 14562 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:34.381549  9557 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:34.441939 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:51:34.439726 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 178878us
I20260808 01:51:34.442670 10362 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 5 FOLLOWER]: Advancing to term 7
I20260808 01:51:34.442759  9555 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:34.261402 14612 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:34.384861  9560 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
W20260808 01:51:34.385521  9630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2351) took 1811 ms (client timeout 1923 ms). Trace:
W20260808 01:51:34.447217  9630 rpcz_store.cc:269] 0808 01:51:32.574300 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:32.579588 (+  5288us) service_pool.cc:224] Handling call
0808 01:51:32.831380 (+251792us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b5800231698. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:51:34.119321 (+1287941us) spinlock_profiling.cc:243] Waited 99.3 ms on lock 0x7b540016ea20. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4491144f 00007f5d4498a340 00007f5d4498bbbe 00007f5d4499765c 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922
0808 01:51:34.385440 (+266119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":276905600}
I20260808 01:51:34.411435 14657 raft_consensus.cc:2764] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260808 01:51:34.385371  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:34.381855  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2386) took 1284 ms. Trace:
I20260808 01:51:34.384136  9551 pending_rounds.cc:85] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 4
I20260808 01:51:34.449316  9551 pending_rounds.cc:107] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 11.5
I20260808 01:51:34.449839  9551 raft_consensus.cc:2889] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:51:34.446571  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 522007us
W20260808 01:51:34.392390  9521 leader_election.cc:336] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.527s (SENT)
I20260808 01:51:34.450912  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:34.409662 14624 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:51:34.418015  9481 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
W20260808 01:51:34.418907  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:34.423763 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:34.425390 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:34.459661 10361 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:34.426149 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:34.460502 10363 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 11: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:34.432879  9556 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:34.383296 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:34.461026 10360 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:34.384285  9552 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260808 01:51:34.461844  9552 pending_rounds.cc:85] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 5
I20260808 01:51:34.462100  9552 pending_rounds.cc:107] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260808 01:51:34.462328  9552 raft_consensus.cc:2889] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:51:34.444468  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 247043us
W20260808 01:51:34.385426  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 907625us
I20260808 01:51:34.428220  9636 rpcz_store.cc:276] 0808 01:51:33.111278 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:33.191823 (+ 80545us) service_pool.cc:224] Handling call
0808 01:51:33.192299 (+   476us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:33.192620 (+   321us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:51:33.192991 (+   371us) log.cc:844] Serialized 14 byte log entry
0808 01:51:34.365622 (+1172631us) spinlock_profiling.cc:243] Waited 29 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:34.427271 (+ 61649us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:34.427344 (+    73us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:34.427404 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:34.427482 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:34.427608 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":737946,"prepare.queue_time_us":2049,"prepare.run_cpu_time_us":6257,"prepare.run_wall_time_us":25093,"raft.queue_time_us":738124,"spinlock_wait_cycles":66669824,"thread_start_us":495862,"threads_started":1,"wal-append.queue_time_us":587415}
I20260808 01:51:34.408701  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:34.455006  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:34.465410  9559 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:34.420488 14224 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:34.448779  9631 rpcz_store.cc:276] 0808 01:51:33.101129 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:33.110532 (+  9403us) service_pool.cc:224] Handling call
0808 01:51:33.110893 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:33.330853 (+219960us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:51:33.336580 (+  5727us) log.cc:844] Serialized 14 byte log entry
0808 01:51:33.447799 (+111219us) log.cc:844] Serialized 14 byte log entry
0808 01:51:34.122856 (+675057us) spinlock_profiling.cc:243] Waited 6.83 ms on lock 0x7b1800430f00. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:51:34.379738 (+256882us) spinlock_profiling.cc:243] Waited 7.48 ms on lock 0x7b1800430f00. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667 00007f5d449ae5e8
0808 01:51:34.379932 (+   194us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:34.379985 (+    53us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:34.380036 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:34.381521 (+  1485us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b540032cb54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:34.381585 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:34.381770 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":899895,"prepare.queue_time_us":579,"prepare.run_cpu_time_us":134,"prepare.run_wall_time_us":5054,"raft.queue_time_us":959621,"raft.run_cpu_time_us":626,"raft.run_wall_time_us":38876,"spinlock_wait_cycles":539185536,"thread_start_us":109409,"threads_started":1,"wal-append.queue_time_us":176845}
I20260808 01:51:34.467317 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:34.467754  9601 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:34.468919  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 76632us
I20260808 01:51:34.469280  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:34.470371 14537 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:34.471025  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:34.471547  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:34.471705  9557 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:34.472178  9554 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:34.472590 14224 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:34.475276  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66687.4us
I20260808 01:51:34.477147 14664 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:34.477881 14676 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:34.478374 14676 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:34.490093 14676 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:34.491683 14664 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:34.478910 14543 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 01:51:34.495404 14543 raft_consensus.cc:3055] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:34.495857 14543 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:34.496835 14543 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:34.498293 14543 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:34.480618 14405 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:34.506673  9601 leader_election.cc:400] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10.
I20260808 01:51:34.461498  9472 pending_rounds.cc:85] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 9
I20260808 01:51:34.508607  9472 pending_rounds.cc:107] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260808 01:51:34.509056  9472 raft_consensus.cc:2889] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:51:34.482769  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet b335bc0170854e26a77dbec13dbe9eac (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:34.440773 14632 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:34.484510 14430 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260808 01:51:34.485496  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:34.457056 14612 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:34.487460  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:34.488288 14663 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:34.489382 14542 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:51:34.479826 14403 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260808 01:51:34.498911 10360 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:34.498471  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:34.503911 10362 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:51:34.506369 10361 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:51:34.516038  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:34.526988  9636 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:34.520341  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:34.529378  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:34.529595  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:34.530536  9637 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:34.520486  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:34.521600  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:34.534482  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:34.528398  9523 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:34.522336 10367 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260808 01:51:34.531688  9445 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:34.536579  9629 raft_consensus.cc:2410] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 14 because replica has last-logged OpId of term: 12 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 5.
I20260808 01:51:34.420631 14605 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:34.536799  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:34.537760  9632 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:34.540045  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:34.538262 14624 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:34.545816  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:34.546000  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:34.547068  9638 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:34.547691  9638 raft_consensus.cc:606] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:34.516547 14562 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:34.553167  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:34.541956  9629 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:34.542168 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:34.554001  9637 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:51:34.549108  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:34.555382  9559 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 11: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:34.513322 14224 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:34.540649  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:34.562175  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260808 01:51:34.556512  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:34.564888  9601 leader_election.cc:400] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11.
I20260808 01:51:34.565488  9601 leader_election.cc:403] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
W20260808 01:51:34.558774  9521 leader_election.cc:336] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.720s (SENT)
I20260808 01:51:34.566740  9521 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:34.568943  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:34.541801 14664 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:34.562476  9630 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:34.576764  9633 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:34.575956  9636 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
W20260808 01:51:34.589977  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:34.604070  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:34.605208  9557 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:34.541157 14592 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260808 01:51:34.615880  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 57141.2us
I20260808 01:51:34.608377  9601 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:34.523983 14645 consensus_queue.cc:1059] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260808 01:51:34.625770  9475 raft_consensus.cc:1217] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Deduplicated request from leader. Original: 12.10->[14.11-14.11]   Dedup: 14.11->[]
I20260808 01:51:34.625636 14668 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:34.549054 14534 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.090s
I20260808 01:51:34.676664 14224 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:34.677196 14668 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:34.677325 14224 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:34.679791 14224 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:34.675051  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 134472us
W20260808 01:51:34.681272  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:34.606751 14405 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11.
I20260808 01:51:34.571805 13701 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.072s
I20260808 01:51:34.644420  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:34.610045 14524 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260808 01:51:34.649765  9371 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55058 (request call id 128) took 20483 ms (client timeout 15000 ms). Trace:
I20260808 01:51:34.651026 14368 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:34.709141  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 100864us
I20260808 01:51:34.651293  9557 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:34.652024  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:51:34.626040 14605 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:34.606087 14663 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:34.649065 14682 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:34.584613 14664 raft_consensus.cc:697] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:34.666128  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 56437.9us
I20260808 01:51:34.666491  9475 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260808 01:51:34.622368 14657 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20260808 01:51:34.665195 14656 raft_consensus.cc:993] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 6e7856bed1be43acaa821290d506ee97 to start an election
I20260808 01:51:34.675577 14624 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260808 01:51:34.682627  9521 leader_election.cc:336] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.774s (SENT)
I20260808 01:51:34.699110  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:34.721248  9521 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:34.708559 14490 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:34.712479  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:51:34.710893 14524 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:51:34.712592  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 235498us
I20260808 01:51:34.723119  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:34.700570  9632 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:34.704476 14647 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:34.728924  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:34.737100 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 274025us
I20260808 01:51:34.654599 14676 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:34.735670  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:34.743052  9478 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:51:34.743070  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 60492.5us
W20260808 01:51:34.743968  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:34.718639 14534 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:34.742372  9600 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:34.744359  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:34.744781  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:34.738405  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:34.745543  9521 leader_election.cc:336] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.854s (SENT)
I20260808 01:51:34.745765  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:34.746999 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:34.747607  9521 leader_election.cc:336] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.881s (SENT)
I20260808 01:51:34.748364  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:34.748818  9475 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 9. (term mismatch)
I20260808 01:51:34.749328  9521 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:34.749788  9475 pending_rounds.cc:85] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 8
I20260808 01:51:34.750304  9475 pending_rounds.cc:107] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260808 01:51:34.750658  9475 raft_consensus.cc:2889] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:34.735823 14524 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:51:34.704532  9371 rpcz_store.cc:269] 0808 01:51:14.191937 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:14.198063 (+  6126us) service_pool.cc:224] Handling call
0808 01:51:15.865715 (+1667652us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540005c3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fd470 00007f5d4499c31a 00007f5d4a78c38c 00007f5d4a888bfe 00007f5d48d424a4 00007f5d48d42432 00007f5d48d42361 00007f5d48d422dc 00007f5d48d41592 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:34.649633 (+18783918us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:51:30.634387 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:51:30.635420 (+  1033us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:51:30.635484 (+    64us) write_op.cc:435] Acquired schema lock
0808 01:51:30.635503 (+    19us) tablet.cc:662] Decoding operations
0808 01:51:30.781075 (+145572us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:51:30.861293 (+ 80218us) write_op.cc:641] Partition lock acquired for write op
0808 01:51:30.874178 (+ 12885us) tablet.cc:685] Acquiring locks for 12 operations
0808 01:51:30.952405 (+ 78227us) tablet.cc:701] Row locks acquired
0808 01:51:30.952433 (+    28us) write_op.cc:260] PREPARE: finished
0808 01:51:30.952654 (+   221us) write_op.cc:270] Start()
0808 01:51:30.952866 (+   212us) write_op.cc:276] Timestamp: P: 1786153890952568 usec, L: 0
0808 01:51:30.952917 (+    51us) op_driver.cc:348] REPLICATION: starting
0808 01:51:30.963690 (+ 10773us) log.cc:844] Serialized 4750 byte log entry
0808 01:51:31.021830 (+ 58140us) spinlock_profiling.cc:243] Waited 9.04 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4186fa5c 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb
0808 01:51:31.068086 (+ 46256us) op_driver.cc:464] REPLICATION: finished
0808 01:51:31.148937 (+ 80851us) write_op.cc:301] APPLY: starting
0808 01:51:31.149069 (+   132us) tablet.cc:1367] starting BulkCheckPresence
0808 01:51:31.225020 (+ 75951us) tablet.cc:1370] finished BulkCheckPresence
0808 01:51:31.225066 (+    46us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:51:31.341888 (+116822us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:51:31.342332 (+   444us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=24,key_file_lookups=24,delta_file_lookups=24,mrs_lookups=0
0808 01:51:31.342406 (+    74us) write_op.cc:312] APPLY: finished
0808 01:51:31.387640 (+ 45234us) log.cc:844] Serialized 113 byte log entry
0808 01:51:31.420333 (+ 32693us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:51:31.428152 (+  7819us) write_op.cc:454] Released schema lock
0808 01:51:31.443022 (+ 14870us) write_op.cc:341] FINISH: Updating metrics
0808 01:51:31.560146 (+117124us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:51:31.562954 (+  2808us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1881976,"reported_tablets":21,"spinlock_wait_cycles":2639642240,"child_traces":[["op",{"apply.queue_time_us":80447,"apply.run_cpu_time_us":22764,"apply.run_wall_time_us":382299,"cfile_cache_hit":50,"cfile_cache_hit_bytes":54359,"num_ops":12,"prepare.queue_time_us":410868,"prepare.run_cpu_time_us":27686,"prepare.run_wall_time_us":427465,"raft.queue_time_us":2038,"raft.run_cpu_time_us":28036,"raft.run_wall_time_us":88835,"replication_time_us":114857,"spinlock_wait_cycles":20802688,"thread_start_us":251528,"threads_started":4,"wal-append.queue_time_us":97581,"wal-append.run_cpu_time_us":6269,"wal-append.run_wall_time_us":1409295}]]}
I20260808 01:51:34.752933  9479 raft_consensus.cc:2410] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260808 01:51:34.729592 14368 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election won for term 10
W20260808 01:51:34.755156  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 226838us
I20260808 01:51:34.739367 14645 raft_consensus.cc:3055] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:34.771703  9473 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:34.723649 14663 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:51:34.792017  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46310.3us
I20260808 01:51:34.795058  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:34.793128 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 344409us
I20260808 01:51:34.807257  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:34.809413  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:34.811275  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:34.812211  9480 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:34.820827  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:34.796517 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:34.745654 14534 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:34.827047  9476 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260808 01:51:34.840351  9476 pending_rounds.cc:85] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 7
I20260808 01:51:34.840857  9476 pending_rounds.cc:107] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260808 01:51:34.841296  9476 raft_consensus.cc:2889] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:51:34.828927  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 86631.8us
I20260808 01:51:34.829290 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:34.841619  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:34.831257  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:34.842264  9636 raft_consensus.cc:2376] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13. Re-sending same reply.
I20260808 01:51:34.843658  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:34.843861  9637 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260808 01:51:34.844920  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:34.845235  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:34.845722  9478 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:34.832823  9475 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260808 01:51:34.846007  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:34.845611  9629 raft_consensus.cc:2376] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11. Re-sending same reply.
I20260808 01:51:34.736222 14490 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:34.752882 14633 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:34.854709  9479 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260808 01:51:34.855507  9477 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
I20260808 01:51:34.739591 14430 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.083s
I20260808 01:51:34.865914  9477 pending_rounds.cc:85] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 6
I20260808 01:51:34.866415  9477 pending_rounds.cc:107] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260808 01:51:34.866734  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:34.866797  9477 raft_consensus.cc:2889] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:34.871006  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:34.768184 14605 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:51:34.900134  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:34.793669 14645 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:34.829414 14656 raft_consensus.cc:1081] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Signalling peer 6e7856bed1be43acaa821290d506ee97 to start an election
I20260808 01:51:34.916872  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:34.917868  9637 raft_consensus.cc:2376] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14. Re-sending same reply.
I20260808 01:51:34.921191 14594 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election won for term 11
W20260808 01:51:34.923413  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 128455us
I20260808 01:51:34.924702  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:34.850514 14534 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:34.887110  9629 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 11: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:34.900246  9634 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:34.901327  9632 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:34.887324  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:34.829248 14624 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:34.910491 14490 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:34.910741  9636 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 11. (term mismatch)
I20260808 01:51:34.941025  9636 pending_rounds.cc:85] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 10
I20260808 01:51:34.922041 14594 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:34.931712 10367 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:34.856741 14535 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Leader election won for term 7
I20260808 01:51:34.935806  9630 raft_consensus.cc:2376] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11. Re-sending same reply.
I20260808 01:51:34.941414  9636 pending_rounds.cc:107] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260808 01:51:34.960774  9636 raft_consensus.cc:2889] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:34.937330 13701 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:34.933092  9599 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:34.947903  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:34.975503  9599 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:34.895059  9480 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:51:34.912384 14543 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260808 01:51:34.997367  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 49609.9us
I20260808 01:51:35.006008  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:34.946358 14624 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:34.994717  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:34.939188 14631 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:35.016713  9635 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260808 01:51:35.012091  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.018893  9632 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:35.013813  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:34.944687 14668 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:35.023198  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:35.024164  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.031235  9630 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:51:34.971349 14490 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:34.997334 14614 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:35.033800  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:35.042517  9636 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:34.943871 14428 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.082s
I20260808 01:51:35.045063  9377 catalog_manager.cc:5697] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 4 to 7, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195), no longer has a pending config: VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193), VOTER 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194), VOTER 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195), NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 7 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:34.996623 13701 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.048316 14490 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:35.048012  9377 catalog_manager.cc:5697] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 6 to 11, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 11 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:35.050082 13701 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:35.002583 14530 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:34.865389 14664 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:35.040786  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.062655  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.064111  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:35.071167 14594 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:35.050505 14490 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:35.002343 14663 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:35.084362  9636 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:51:35.044812  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.093312 14676 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:35.009107 14535 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:35.094506  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.094786 14535 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:35.095361  9629 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:35.096101 14530 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:35.096408  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.097266  9629 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:35.097568 14614 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:35.092501 14682 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:35.098138 14614 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:35.029542 14624 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:35.098668  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.100538  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.097326 14676 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:35.101831  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:35.102474  9629 raft_consensus.cc:2376] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12. Re-sending same reply.
I20260808 01:51:35.039181 14668 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.103637  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.105021 14536 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:35.105310  9633 raft_consensus.cc:2368] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8.
I20260808 01:51:35.042693 14486 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:35.036931 14656 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.011s
I20260808 01:51:35.104775  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.107856  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.110071  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:35.028885 14645 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.111016 14403 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:51:35.111729  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:35.111908 14403 consensus_queue.cc:1243] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 12.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260808 01:51:35.105023 14668 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:35.114253 14608 raft_consensus.cc:3055] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:35.114751 14608 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:35.115888 14612 consensus_meta.cc:287] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.571s	user 0.010s	sys 0.001s
I20260808 01:51:35.115785 14608 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.116413 14612 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.117477 14608 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:35.105472 14536 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:35.119824 14536 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.121126 14536 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:35.119732  9377 catalog_manager.cc:5697] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
W20260808 01:51:35.104679 14562 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.533s	user 0.011s	sys 0.000s
W20260808 01:51:35.136860 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 305056us
I20260808 01:51:35.113003  9636 raft_consensus.cc:2376] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14. Re-sending same reply.
I20260808 01:51:35.141538  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:35.142614  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:35.143503  9480 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:51:35.144757  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52041.6us
I20260808 01:51:35.148831  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:35.149678  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:35.138653 14709 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:35.137562 14631 raft_consensus.cc:697] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:35.155313 14486 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:35.155432 14555 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:51:35.140997 10332 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:35.155432 14405 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260808 01:51:35.160960 10332 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:35.160741 14614 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:35.161381 14405 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:35.163074 14688 raft_consensus.cc:2764] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election decision vote started in defunct term 13: won
I20260808 01:51:35.158774  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.155577 14656 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:35.165618 14534 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.165994  9629 raft_consensus.cc:2410] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260808 01:51:35.167052  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.162963  9377 catalog_manager.cc:5697] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 5 to 8, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 8 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:35.167553  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:35.179459  9377 catalog_manager.cc:5697] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 5 to 8. New cstate: current_term: 8 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:35.184340 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:35.098060  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 55360.4us
I20260808 01:51:35.189730  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:35.190415  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:35.205066  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:35.194007  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:35.195047  9377 catalog_manager.cc:5697] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 4 to 9, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 9 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:35.196756  9638 raft_consensus.cc:2368] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12.
I20260808 01:51:35.216332  9480 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
W20260808 01:51:35.192682  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:35.229628 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:35.230309  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:35.192651 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:35.167933 14224 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:35.168129 14634 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:35.236357  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:35.253319  9476 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:35.253458  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:35.237309  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.238116  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:35.240243  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:35.160621 14524 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:51:35.170162 14405 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:35.256016  9636 raft_consensus.cc:2376] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10. Re-sending same reply.
I20260808 01:51:35.261772 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:35.268131  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.138994 14655 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.110s
I20260808 01:51:35.195711 14682 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:35.272691  9476 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:35.190486 14709 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:35.195629 14612 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:35.278833  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.009s	sys 0.016s
I20260808 01:51:35.253401 14691 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:35.260701 14562 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:35.284212  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:35.271147 14702 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
W20260808 01:51:35.293434  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.009s	sys 0.016s
I20260808 01:51:35.293077  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:35.294296  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:35.311970  9555 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:35.299918  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.300580  9557 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260808 01:51:35.274725 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:35.307977  9554 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:35.294090  9377 catalog_manager.cc:5697] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:35.316071  9377 catalog_manager.cc:5697] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:35.318966  9476 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 11: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:35.250530 14530 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.046s
I20260808 01:51:35.291946  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:35.303472  9552 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260808 01:51:35.306243  9551 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260808 01:51:35.310114  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:35.324000 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 57304.1us
I20260808 01:51:35.197467 14688 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.081s
I20260808 01:51:35.327744  9377 catalog_manager.cc:5697] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:35.263334 14624 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.332775  9524 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:35.332700  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 585081us
W20260808 01:51:35.333918  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:35.334839  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:35.332015 14708 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:35.320677 14709 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.387879 14708 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.336048 14542 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:35.337317 14624 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:35.338197  9473 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 13: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:35.320509 14631 consensus_queue.cc:237] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.403321  9523 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:35.339243  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.407887  9633 raft_consensus.cc:2368] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 10. Current term is 11.
I20260808 01:51:35.340044 14524 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:35.343305 14600 raft_consensus.cc:3055] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:35.346500  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2240) took 1087 ms. Trace:
I20260808 01:51:35.347514  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:35.349252 14632 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:35.351141  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:35.423794  9629 raft_consensus.cc:2376] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9. Re-sending same reply.
I20260808 01:51:35.351940  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:35.424726  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 90007.9us
I20260808 01:51:35.378113 14534 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:35.353590 14712 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:35.339243  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:35.286273 14707 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:35.280521 14368 raft_consensus.cc:697] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:35.267966 14634 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.267833 14405 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.375600  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.453941  9636 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:35.375885 10366 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:35.380955  9377 catalog_manager.cc:5697] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 6 to 9. New cstate: current_term: 9 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:35.290452 14537 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
I20260808 01:51:35.397668 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:35.304316 14430 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.075s
I20260808 01:51:35.466476 14711 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:35.467607 14711 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:35.322976 14691 raft_consensus.cc:697] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:35.423197 14647 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:35.334419 14614 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:35.377661 14676 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:35.423843 14542 raft_consensus.cc:697] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:35.484653  9521 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:35.352682  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.491618  9630 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:35.418888  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:35.438446  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:35.318704 14562 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:35.440193  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:35.417299  9550 rpcz_store.cc:276] 0808 01:51:34.258726 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:34.259475 (+   749us) service_pool.cc:224] Handling call
0808 01:51:34.259927 (+   452us) raft_consensus.cc:1471] Updating replica for 4 ops
0808 01:51:34.370261 (+110334us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:51:34.578310 (+208049us) log.cc:844] Serialized 14 byte log entry
0808 01:51:34.929224 (+350914us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0808 01:51:34.935349 (+  6125us) log.cc:844] Serialized 78 byte log entry
0808 01:51:34.937068 (+  1719us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:51:34.991986 (+ 54918us) log.cc:844] Serialized 14 byte log entry
0808 01:51:35.015378 (+ 23392us) log.cc:844] Serialized 14 byte log entry
0808 01:51:35.095442 (+ 80064us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:35.095521 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:35.113079 (+ 17558us) spinlock_profiling.cc:243] Waited 5.33 ms on lock 0x7b48000c03d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:51:35.346142 (+233063us) raft_consensus.cc:1685] finished
0808 01:51:35.346179 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:35.346414 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":79251,"prepare.queue_time_us":411002,"prepare.run_cpu_time_us":177,"prepare.run_wall_time_us":3389,"raft.queue_time_us":82209,"raft.run_cpu_time_us":3244,"raft.run_wall_time_us":115633,"spinlock_wait_cycles":12261760,"thread_start_us":509856,"threads_started":2,"wal-append.queue_time_us":568687}
I20260808 01:51:35.443336  9478 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:35.445346  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.500003  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:35.449091  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.459542  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:35.461724  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:35.509773  9629 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:35.461879  9555 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:35.446691 14712 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:35.425796 14284 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:35.531793  9476 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:35.426177 14655 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:35.409610  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:35.417016 14600 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:35.497274 14707 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:35.434626 14543 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:35.419831  9551 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
W20260808 01:51:35.420805  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 426236us
I20260808 01:51:35.523954  9552 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:35.525694  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:35.547300  9630 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:35.486418 14676 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:35.527101  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.529489  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:35.525457 14712 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:35.528650  9377 catalog_manager.cc:5697] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 4 to 10. New cstate: current_term: 10 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:35.536868  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:35.390380 14692 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.072s
I20260808 01:51:35.515880 14430 consensus_queue.cc:1243] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 14.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260808 01:51:35.526718 13701 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:35.574434  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:35.576192  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:35.579501  9473 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 12: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:51:35.544723  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 60153.8us
I20260808 01:51:35.587746  9521 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:35.538563 14708 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:35.577334  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:35.604568  9630 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:35.541215 14537 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:35.580389  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 774) took 1119 ms. Trace:
I20260808 01:51:35.554925 14711 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:35.647606  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:35.648489  9473 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:35.594357  9478 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
W20260808 01:51:35.617832  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:35.559309 14676 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:35.545540 14647 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:35.623206  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 433536us
I20260808 01:51:35.627230  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.731247  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:35.627513  9377 catalog_manager.cc:5697] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 6 to 10. New cstate: current_term: 10 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:35.585572 14224 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:35.600684  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:35.638104  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 50333.3us
I20260808 01:51:35.540480 14655 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.739475 14655 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:35.739880 14722 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260808 01:51:35.707293 14534 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.042s
I20260808 01:51:35.621773 14542 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:35.751699  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 151233us
I20260808 01:51:35.587805 14715 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:35.630932  9472 rpcz_store.cc:276] 0808 01:51:34.460654 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:34.460834 (+   180us) service_pool.cc:224] Handling call
0808 01:51:34.461217 (+   383us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:34.527255 (+ 66038us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:34.527287 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:34.527906 (+   619us) log.cc:844] Serialized 52 byte log entry
0808 01:51:34.828503 (+300597us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:51:35.137325 (+308822us) log.cc:844] Serialized 14 byte log entry
0808 01:51:35.579830 (+442505us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:35.579956 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:35.580034 (+    78us) raft_consensus.cc:1685] finished
0808 01:51:35.580056 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:35.580272 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":124015,"prepare.queue_time_us":301375,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":257434,"raft.run_cpu_time_us":438,"raft.run_wall_time_us":29407,"thread_start_us":877399,"threads_started":3,"wal-append.queue_time_us":542214}
I20260808 01:51:35.587301 14718 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:35.573680 14530 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20260808 01:51:35.760571  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:35.628983 14537 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:35.691159  9633 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:51:35.642282 14708 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260808 01:51:35.691965 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 274951us
I20260808 01:51:35.639282 14706 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:35.585371 13701 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.701855  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.634372 14486 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.069s
I20260808 01:51:35.719108  9634 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:51:35.721962  9476 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:50140
I20260808 01:51:35.792963 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:35.793258  9476 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:51:35.793829  9634 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260808 01:51:35.793898  9476 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:35.726644  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:35.795218  9478 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:35.735380  9521 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:35.797286  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62573.1us
I20260808 01:51:35.599663 14368 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.655809  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:35.799610  9630 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 11: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:35.627943 14614 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:35.798533  9521 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:35.772684 10332 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:35.764931 14669 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:35.631872 14691 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:35.768252 14405 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:35.727537 14707 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.5, Last appended by leader: 5, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.778393  9377 catalog_manager.cc:5697] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 5 to 8. New cstate: current_term: 8 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:35.766076 14715 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.747578 14647 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.780355 14524 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:35.785109  9633 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
W20260808 01:51:35.785728  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:35.843881  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:35.790791  9632 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260808 01:51:35.791023  9638 raft_consensus.cc:2410] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 15 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260808 01:51:35.795913  9599 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:35.759953 14166 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:35.806588  9521 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:35.806803 14668 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260808 01:51:35.851742  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 45218.1us
I20260808 01:51:35.853319  9521 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:35.774032 14634 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:35.790190 14719 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
I20260808 01:51:35.812716 10368 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260808 01:51:35.685842 14600 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:35.785190 14645 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:35.824076 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:35.877164  9601 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:35.880363  9521 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:35.795713 14706 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:35.847644  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 87247.2us
I20260808 01:51:35.853572  9481 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260808 01:51:35.768849 14537 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:35.836576 14524 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:35.852257 14614 raft_consensus.cc:697] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:35.898325 14614 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:35.901719  9633 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260808 01:51:35.903201  9633 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:35.897390 14524 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:35.809871 14708 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:35.907325 14555 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:35.907960 14555 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:35.805871 14403 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260808 01:51:35.910894 14403 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:35.911342 14403 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:35.831808 14562 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:35.867640 14652 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:35.884737 14428 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:35.902513  9377 catalog_manager.cc:5697] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 2 to 9, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 9 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:35.916533  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:35.855041 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 82519.2us
W20260808 01:51:35.868803  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 72989.8us
I20260808 01:51:35.845490  9473 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:51:35.870395 14663 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:35.875501 14695 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:35.939392  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:35.907630 14591 raft_consensus.cc:3055] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:35.943706 10334 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:51:35.931919 14608 consensus_meta.cc:287] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.814s	user 0.015s	sys 0.004s
I20260808 01:51:35.942926  9479 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260808 01:51:35.954550  9377 catalog_manager.cc:5697] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:35.955546  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:35.893422 14657 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:35.887511 14740 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260808 01:51:35.878604 14543 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260808 01:51:35.964422  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:35.925078 14600 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:35.979677  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:35.988104 10367 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:36.023092  9473 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
W20260808 01:51:35.981546 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 43503.1us
W20260808 01:51:36.013077  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 96634.9us
W20260808 01:51:36.024161  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:35.836539 14224 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.026221  9377 catalog_manager.cc:5697] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 1 to 7. New cstate: current_term: 7 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:51:36.029867  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 74600.2us
I20260808 01:51:36.028265  9377 catalog_manager.cc:5697] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 4 to 6, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 6 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:51:36.011611  9635 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:36.011528  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:36.032573  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:51:35.933497 14428 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:36.034291  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:36.035944  9599 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:35.958204 14668 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:35.933383 14562 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:51:36.012277 14536 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.891s	user 0.015s	sys 0.002s
W20260808 01:51:36.073462  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 100296us
I20260808 01:51:35.960108 14591 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:35.912288 14403 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:35.988768 14530 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260808 01:51:36.070708  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 190421us
I20260808 01:51:36.003729  9476 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.093246  9521 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:36.094135  9631 raft_consensus.cc:2410] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
W20260808 01:51:36.094599  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 60631.1us
I20260808 01:51:35.954118 14534 consensus_queue.cc:1059] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.120s
I20260808 01:51:36.098719  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:36.035633 14709 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:36.072135 14428 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:36.067807 14663 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:36.070080 14544 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:36.081228 10368 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:51:36.069901 14659 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:36.106370  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:36.092530 14530 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:36.068806 14744 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:36.081931  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46096.5us
I20260808 01:51:36.149973  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:36.114338  9476 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:35.969436 14695 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.075661 14535 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.043s
I20260808 01:51:36.078890 14727 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:36.094161  9473 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:36.154657  9444 leader_election.cc:400] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 7. Current term is 8.
I20260808 01:51:36.103461 14449 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:36.109155 14486 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:36.092197 14562 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:36.164477  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 58198.9us
I20260808 01:51:36.172562 14668 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:36.191062 14403 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:36.208609 14732 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.274s	user 0.000s	sys 0.000s
I20260808 01:51:36.201179 13701 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:36.209497  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:36.210147  9554 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:36.211222 14647 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:36.211591 14631 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:36.155390 14544 raft_consensus.cc:697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:36.212446 14631 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:36.213047 14544 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:36.160346 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:36.157579 14744 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.216414 10334 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:36.171116 14681 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.099s
I20260808 01:51:36.143123 14659 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:36.172770 14731 raft_consensus.cc:2764] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election decision vote started in defunct term 11: won
I20260808 01:51:36.175000 14726 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election won for term 12
W20260808 01:51:36.176031 14735 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260808 01:51:36.118819 14543 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:36.243131  9558 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260808 01:51:36.187573  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:36.191752  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:36.245793  9557 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:36.191826  9552 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260808 01:51:36.191992  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:36.107187 14709 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:36.154755 14530 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:36.197211  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:36.199765  9521 leader_election.cc:400] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 10. Current term is 11.
I20260808 01:51:36.257323  9521 leader_election.cc:403] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:51:36.079519 14739 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.093s
I20260808 01:51:36.208257 14695 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:36.092275 14284 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.064s
I20260808 01:51:36.264380 14428 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.156945  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:36.265563 14669 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:36.266110 14669 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:36.266310  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 109505us
I20260808 01:51:36.266932 14428 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:36.267314  9599 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:36.267750 14428 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:36.242910 14659 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:36.154587 14656 raft_consensus.cc:3055] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:36.269028 14656 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:36.269201 14428 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:36.269757 14428 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:36.270169 14656 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.271840 14656 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:36.271864 14631 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:36.273337 14631 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:36.273506  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2681) took 1017 ms. Trace:
I20260808 01:51:36.273542 14535 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:51:36.166541  9632 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:36.275164 14695 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 10. Current term is 11.
W20260808 01:51:36.201042 14632 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.767s	user 0.010s	sys 0.006s
I20260808 01:51:36.276369 14632 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:36.276963 14632 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:36.207424 14737 log.cc:927] Time spent T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.239s	user 0.000s	sys 0.001s
I20260808 01:51:36.194695 14633 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:36.280373 14633 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:36.208377 14745 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.273s	user 0.001s	sys 0.000s
I20260808 01:51:36.192636 14486 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:36.283116 14706 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:36.283427 14486 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:36.285421  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2679) took 1029 ms. Trace:
I20260808 01:51:36.216219 14750 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:36.195416 14449 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:36.231896 14744 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:36.148018 14711 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.054s
I20260808 01:51:36.232600 14524 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.235030 14562 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:36.236418 14740 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:36.145236 14663 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:36.163494 14612 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:36.178918 14608 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:36.250842  9560 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:51:36.274452  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 78215.3us
W20260808 01:51:36.307476  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:36.308251  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:36.275265  9632 raft_consensus.cc:606] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:36.207131  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:36.274132  9479 rpcz_store.cc:276] 0808 01:51:35.256406 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:35.900217 (+643811us) service_pool.cc:224] Handling call
0808 01:51:35.942526 (+ 42309us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:35.985749 (+ 43223us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:36.273291 (+287542us) spinlock_profiling.cc:243] Waited 41.9 ms on lock 0x7b54002bacd0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:36.273431 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":96312192}
W20260808 01:51:36.275501 14758 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260808 01:51:36.285948  9477 rpcz_store.cc:276] 0808 01:51:35.255847 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:35.749411 (+493564us) service_pool.cc:224] Handling call
0808 01:51:35.749798 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:35.847264 (+ 97466us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:35.847296 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:35.847827 (+   531us) log.cc:844] Serialized 26 byte log entry
0808 01:51:35.991639 (+143812us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:35.992035 (+   396us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:35.992497 (+   462us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:36.285177 (+292680us) raft_consensus.cc:1685] finished
0808 01:51:36.285202 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:36.285350 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":96808,"raft.queue_time_us":102328,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":15408,"thread_start_us":77582,"threads_started":1,"wal-append.queue_time_us":420749}
I20260808 01:51:36.325408  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:36.296576  9550 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260808 01:51:36.299777  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:36.254236 14709 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:36.351099  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:36.352099  9557 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:36.353449  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:36.270169 14428 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:36.286296 14655 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:36.379215  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:36.295464 14726 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:36.373381  9550 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260808 01:51:36.375041  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:36.382571  7831 auto_leader_rebalancer.cc:456] leader transfer for tablet 924b85a2ee7e46ecbcfd57257902abef from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97 failed: Remote error: Service unavailable: leadership transfer for 924b85a2ee7e46ecbcfd57257902abef already in progress
I20260808 01:51:36.384276 14592 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:36.384523 14449 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:36.384935 14592 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:36.297736 14633 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:36.387063 14524 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:36.375672 14534 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260808 01:51:36.377534  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:36.302425 14632 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:36.303193 14740 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:51:36.389428  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 81918.2us
I20260808 01:51:36.312625 14588 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:36.390810  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:36.392040  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:36.393334  9558 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:36.376679  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:36.400954  9521 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:36.337069 14728 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.444s	user 0.002s	sys 0.000s
I20260808 01:51:36.407717  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:36.426585  9521 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:36.430051  9550 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:36.425583  9632 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:36.431229  9632 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:36.302443 14631 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:36.432773  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2382) took 1374 ms. Trace:
I20260808 01:51:36.442520  9631 rpcz_store.cc:276] 0808 01:51:35.058257 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:35.158436 (+100179us) service_pool.cc:224] Handling call
0808 01:51:36.432542 (+1274106us) spinlock_profiling.cc:243] Waited 930 ms on lock 0x7b540003bbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:36.432711 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2138267392}
I20260808 01:51:36.444416  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:36.446064  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:36.447377  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:36.430015  9557 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:36.454169  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:36.454995  9558 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:36.313359 14612 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:36.485006 14612 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:36.405488 14740 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:36.494199  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2672) took 1689 ms. Trace:
I20260808 01:51:36.495844 14534 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:36.494702  9475 rpcz_store.cc:276] 0808 01:51:34.804813 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:34.847403 (+ 42590us) service_pool.cc:224] Handling call
0808 01:51:34.855135 (+  7732us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:35.187733 (+332598us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:35.187785 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:35.188520 (+   735us) log.cc:844] Serialized 26 byte log entry
0808 01:51:35.586006 (+397486us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:51:35.650003 (+ 63997us) log.cc:844] Serialized 14 byte log entry
0808 01:51:35.650352 (+   349us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:35.650482 (+   130us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:35.909917 (+259435us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b1800435c40. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188ce62 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:51:36.493877 (+583960us) raft_consensus.cc:1685] finished
0808 01:51:36.493987 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:36.494130 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":152431,"prepare.queue_time_us":64010,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":4366,"raft.queue_time_us":170971,"raft.run_cpu_time_us":479,"raft.run_wall_time_us":43868,"spinlock_wait_cycles":30445440,"thread_start_us":659502,"threads_started":2,"wal-append.queue_time_us":569499}
I20260808 01:51:36.496486 14534 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.500591 14534 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:36.500881 14741 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.526s	user 0.000s	sys 0.003s
I20260808 01:51:36.439163  9632 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.431416 14543 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:36.375880 13701 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.059s
I20260808 01:51:36.396442 14655 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.463376  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:36.463874 14712 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.875s	user 0.012s	sys 0.000s
W20260808 01:51:36.450425 14758 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
W20260808 01:51:36.464983 14752 log.cc:927] Time spent T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.163s	user 0.000s	sys 0.000s
I20260808 01:51:36.465468 14605 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260808 01:51:36.467815 14753 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.211s	user 0.001s	sys 0.000s
I20260808 01:51:36.563670 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:36.468583  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:36.473205 14631 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:36.531798 14691 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.009s	sys 0.011s
I20260808 01:51:36.480379 14663 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:36.301110 14750 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.437950  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:36.505756 14652 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.556s	user 0.009s	sys 0.003s
I20260808 01:51:36.514668  9638 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:36.525255 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:36.525494  9550 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:36.527302  9632 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:36.527828  9560 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:36.543118  9554 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
W20260808 01:51:36.545128  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:36.584295  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:36.548797  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:36.566009  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 97521.1us
I20260808 01:51:36.589371  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:36.477425 14486 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.571635 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:36.548102 14543 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:51:36.554469 14676 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.799s	user 0.014s	sys 0.000s
I20260808 01:51:36.586169  9478 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:36.612247  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:36.437078 14588 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.589075  9558 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:36.520802 14592 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:36.592013  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 374122us
I20260808 01:51:36.589507  9638 raft_consensus.cc:606] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:36.594295  9554 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:36.622718  9638 raft_consensus.cc:614] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:36.510589 14715 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:36.594350  9474 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:36.552635 14608 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:36.612846 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:36.582746 14657 raft_consensus.cc:697] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:36.572451 14691 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.010s	sys 0.011s
I20260808 01:51:36.620463  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:36.627434  9630 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:36.573299 14706 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:36.648977  9521 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:36.648483  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:36.652768 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:36.592386 14652 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:36.559960 14712 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.609498 14543 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:36.682976  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 780) took 1107 ms. Trace:
I20260808 01:51:36.679818  9632 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
W20260808 01:51:36.628533 14594 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.010s	sys 0.022s
W20260808 01:51:36.681313 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:36.619129 14663 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:36.618934 14605 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260808 01:51:36.685405 14562 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260808 01:51:36.708371 14753 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260808 01:51:36.687606  9479 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:36.688222  9477 rpcz_store.cc:276] 0808 01:51:35.580240 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:36.681979 (+1101739us) service_pool.cc:224] Handling call
0808 01:51:36.682362 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:36.682546 (+   184us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:36.682569 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:36.682622 (+    53us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:36.682653 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:36.682744 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:36.682899 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:36.733469  9477 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:36.703707 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:36.649032 14668 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:36.705235  9638 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260808 01:51:36.704079  9557 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
W20260808 01:51:36.682215 14752 log.cc:927] Time spent T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260808 01:51:36.748971  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2680) took 1492 ms. Trace:
I20260808 01:51:36.629305 14726 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.749421  9472 rpcz_store.cc:276] 0808 01:51:35.256135 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:35.754575 (+498440us) service_pool.cc:224] Handling call
0808 01:51:35.755111 (+   536us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:35.789307 (+ 34196us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:35.789340 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:35.789858 (+   518us) log.cc:844] Serialized 26 byte log entry
0808 01:51:35.935844 (+145986us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:51:35.970796 (+ 34952us) log.cc:844] Serialized 14 byte log entry
0808 01:51:36.073413 (+102617us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:36.073554 (+   141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:36.748713 (+675159us) raft_consensus.cc:1685] finished
0808 01:51:36.748740 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:36.748900 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":176527,"prepare.queue_time_us":31876,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":142977,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":17378,"spinlock_wait_cycles":22977152,"thread_start_us":145626,"threads_started":1,"wal-append.queue_time_us":511985}
I20260808 01:51:36.752589  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 782) took 1169 ms. Trace:
I20260808 01:51:36.753263  9472 rpcz_store.cc:276] 0808 01:51:35.582836 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:36.751539 (+1168703us) service_pool.cc:224] Handling call
0808 01:51:36.751907 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:36.752100 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:36.752124 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:36.752167 (+    43us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:51:36.752200 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:36.752261 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:36.752506 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:51:36.687497 14537 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.790s	user 0.013s	sys 0.001s
I20260808 01:51:36.619575 14676 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.724866  9630 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:36.754338 14537 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.629189 14695 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.074s
I20260808 01:51:36.755398  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 783) took 1115 ms. Trace:
I20260808 01:51:36.755856  9472 rpcz_store.cc:276] 0808 01:51:35.640317 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:36.754504 (+1114187us) service_pool.cc:224] Handling call
0808 01:51:36.754874 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:36.755053 (+   179us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:36.755080 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:36.755122 (+    42us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:51:36.755165 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:36.755220 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:36.755328 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:36.756635 14537 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:36.756713 14555 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:36.657382 14647 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.068s
I20260808 01:51:36.757321  9472 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:36.758023 14676 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:36.758227  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 784) took 1114 ms. Trace:
I20260808 01:51:36.758832  9472 rpcz_store.cc:276] 0808 01:51:35.643693 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:36.756811 (+1113118us) service_pool.cc:224] Handling call
0808 01:51:36.757140 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:36.757807 (+   667us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:36.758162 (+   355us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:36.759593 14486 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:36.760686  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 785) took 1109 ms. Trace:
I20260808 01:51:36.761094  9472 rpcz_store.cc:276] 0808 01:51:35.651874 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:36.759752 (+1107878us) service_pool.cc:224] Handling call
0808 01:51:36.760109 (+   357us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:36.760314 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:36.760341 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:36.760384 (+    43us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:51:36.760423 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:36.760508 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:36.760624 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:36.651367 14608 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.762667  9472 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:36.763391  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2543) took 1030 ms. Trace:
I20260808 01:51:36.768267  9472 rpcz_store.cc:276] 0808 01:51:35.732617 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:36.762083 (+1029466us) service_pool.cc:224] Handling call
0808 01:51:36.762466 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:36.763116 (+   650us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:36.763334 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:51:36.725217 14645 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.830s	user 0.015s	sys 0.000s
I20260808 01:51:36.739578 14712 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:36.741456 13701 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:36.823182  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 200336us
I20260808 01:51:36.824275  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2684) took 1421 ms. Trace:
I20260808 01:51:36.742238 14729 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:36.742833 14755 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.118s
I20260808 01:51:36.745239 14588 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:36.710256  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:36.754634 14743 raft_consensus.cc:3055] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:36.764645 14608 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:36.683677  9559 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:36.734405  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 781) took 1152 ms. Trace:
W20260808 01:51:36.704409 14594 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.011s	sys 0.023s
I20260808 01:51:36.770830 14657 consensus_queue.cc:237] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.770982 14535 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:36.841681 14535 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.800523  9557 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260808 01:51:36.852378  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:36.853329  9557 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:36.772050  9551 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:51:36.739663 14773 log.cc:927] Time spent T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
W20260808 01:51:36.861745 14770 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.146s	user 0.001s	sys 0.000s
I20260808 01:51:36.831878 14729 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.831626 14428 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.800427  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:36.770604 14530 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.801126  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:36.815063  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 166150us
I20260808 01:51:36.820473  9637 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 6. (term mismatch)
I20260808 01:51:36.881585  9637 pending_rounds.cc:85] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 5
I20260808 01:51:36.841566 14769 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260808 01:51:36.882040  9637 pending_rounds.cc:107] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 11.6
I20260808 01:51:36.882242  9560 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:51:36.830699  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:36.881724 14750 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:36.833951  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:36.771819  9558 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:36.847776  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2376) took 1791 ms. Trace:
I20260808 01:51:36.840250  9477 rpcz_store.cc:276] 0808 01:51:35.582476 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:36.732724 (+1150248us) service_pool.cc:224] Handling call
0808 01:51:36.733203 (+   479us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:36.734062 (+   859us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:36.734325 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:36.854666  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:36.871387  9554 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:36.889585  9557 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:36.888391  9636 rpcz_store.cc:276] 0808 01:51:35.056690 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:35.782973 (+726283us) service_pool.cc:224] Handling call
0808 01:51:35.783430 (+   457us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:35.791860 (+  8430us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:35.791892 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:35.792401 (+   509us) log.cc:844] Serialized 26 byte log entry
0808 01:51:35.900828 (+108427us) spinlock_profiling.cc:243] Waited 4.04 ms on lock 0x7b48002379d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:51:35.948216 (+ 47388us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:35.982540 (+ 34324us) log.cc:844] Serialized 14 byte log entry
0808 01:51:35.982821 (+   281us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:35.982942 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:36.847432 (+864490us) raft_consensus.cc:1685] finished
0808 01:51:36.847545 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:36.847704 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13023,"prepare.queue_time_us":34739,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":6149,"raft.queue_time_us":35452,"raft.run_cpu_time_us":2864,"raft.run_wall_time_us":129477,"spinlock_wait_cycles":9293056,"thread_start_us":151061,"threads_started":1,"wal-append.queue_time_us":360864}
I20260808 01:51:36.893774  9633 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:36.824747  9473 rpcz_store.cc:276] 0808 01:51:35.402812 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:36.025098 (+622286us) service_pool.cc:224] Handling call
0808 01:51:36.079357 (+ 54259us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:36.169295 (+ 89938us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:36.823996 (+654701us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b54002bb6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:36.824205 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":33483520}
I20260808 01:51:36.882423  9637 raft_consensus.cc:2889] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:36.843654 14763 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:36.837231 14743 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:36.888175  9638 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260808 01:51:36.897120  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:36.887926  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:36.887365  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:36.903155  9631 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 13: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:36.903367  9550 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:36.904565  9550 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: []
W20260808 01:51:36.904973  9521 connection.cc:256] RPC call timeout handler was delayed by -1.084792499023024s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1936s for a response
I20260808 01:51:36.907059  9551 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:36.826326 13701 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.906978  9560 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:36.907244  9557 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
W20260808 01:51:36.910142  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 76295us
I20260808 01:51:36.912999  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:36.913548  9636 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:36.915422 14743 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:36.903786 14656 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.627s	user 0.011s	sys 0.003s
I20260808 01:51:36.915900 14729 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:36.917465  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:36.923897 14428 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:36.935165  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:36.926005 13701 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:36.926276 14659 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.658s	user 0.011s	sys 0.000s
W20260808 01:51:36.927781 14709 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.548s	user 0.010s	sys 0.003s
I20260808 01:51:36.929072 14530 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:36.971204  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:36.932749  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:36.933583  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:36.897872 14534 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:36.971992  9630 raft_consensus.cc:2410] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260808 01:51:36.969800  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:36.916666 14769 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:36.976225  9638 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:36.970489  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2540) took 1498 ms. Trace:
I20260808 01:51:36.974102  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:36.976053  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:36.977131  9560 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:36.976608  9473 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260808 01:51:36.979897  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2688) took 1019 ms. Trace:
I20260808 01:51:36.980351  9473 rpcz_store.cc:276] 0808 01:51:35.960532 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:36.974402 (+1013870us) service_pool.cc:224] Handling call
0808 01:51:36.976078 (+  1676us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:36.979483 (+  3405us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:36.979816 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:36.976670  9480 rpcz_store.cc:276] 0808 01:51:35.472621 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:36.176184 (+703563us) service_pool.cc:224] Handling call
0808 01:51:36.176655 (+   471us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:36.177084 (+   429us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:36.177116 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:36.177684 (+   568us) log.cc:844] Serialized 26 byte log entry
0808 01:51:36.400849 (+223165us) raft_consensus.cc:1645] Marking committed up to 4
0808 01:51:36.400890 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:36.402231 (+  1341us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:36.970137 (+567906us) raft_consensus.cc:1685] finished
0808 01:51:36.970205 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:36.970411 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":703,"raft.queue_time_us":2931,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":4682,"thread_start_us":222750,"threads_started":1,"wal-append.queue_time_us":386340}
W20260808 01:51:36.987841  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 54385.3us
I20260808 01:51:36.908078 14763 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.995398  9560 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:51:36.997877  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:36.997682  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:36.998736  9552 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 7.
I20260808 01:51:36.998896  9638 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
W20260808 01:51:36.986660 14783 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260808 01:51:36.975481 14534 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.936188 14712 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.062s
I20260808 01:51:36.975133 14729 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:37.001417  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:37.001132  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:36.975274 14769 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:37.003281  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:37.010325  9638 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:37.003894  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:37.005149  9477 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:37.010834  9473 raft_consensus.cc:1217] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Deduplicated request from leader. Original: 11.6->[12.7-13.8]   Dedup: 12.7->[13.8-13.8]
I20260808 01:51:36.994666 14631 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:37.005690  9552 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:37.012110  9560 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:51:37.008266  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:37.013300  9557 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:37.014276  9473 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: [12.7-13.8]
I20260808 01:51:37.015631  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:36.975510 14709 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.973462 14659 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:36.995303 14743 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:37.115950  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2687) took 1156 ms. Trace:
I20260808 01:51:37.107131  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:37.035549  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:37.113584  9635 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2388) took 2035 ms (client timeout 1936 ms). Trace:
I20260808 01:51:37.130996  9638 raft_consensus.cc:2368] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13.
I20260808 01:51:37.034456  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2686) took 1523 ms. Trace:
I20260808 01:51:37.028376  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:37.043350  9554 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
W20260808 01:51:37.118621  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 9949664c54584858ae8fc74c9dc07a4a (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:37.116864  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2677) took 1862 ms. Trace:
I20260808 01:51:37.042593  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:37.133317  9481 rpcz_store.cc:276] 0808 01:51:35.255033 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:35.712036 (+457003us) service_pool.cc:224] Handling call
0808 01:51:35.712591 (+   555us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:35.895774 (+183183us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:37.048824 (+1153050us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b54000938d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:37.116801 (+ 67977us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":323260160}
I20260808 01:51:37.117297  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2685) took 1610 ms. Trace:
I20260808 01:51:37.028839  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:37.133865  9633 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:37.108007  9521 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.936s (SENT) [suppressed 33 similar messages]
I20260808 01:51:37.119911  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:37.116515 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:37.134433  9479 rpcz_store.cc:276] 0808 01:51:35.507248 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:36.337193 (+829945us) service_pool.cc:224] Handling call
0808 01:51:36.337625 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:36.773969 (+436344us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:37.117038 (+343069us) spinlock_profiling.cc:243] Waited 350 ms on lock 0x7b5400403dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:37.117235 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":804255488}
I20260808 01:51:37.016074  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2689) took 1037 ms. Trace:
I20260808 01:51:37.136788  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:37.136744  9599 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:37.136850 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:37.136629  9473 rpcz_store.cc:276] 0808 01:51:35.978296 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:36.981667 (+1003371us) service_pool.cc:224] Handling call
0808 01:51:37.010583 (+ 28916us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:37.014844 (+  4261us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:37.015090 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:37.043767  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2369) took 2007 ms. Trace:
I20260808 01:51:37.138440  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:37.138978  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:37.114295  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:37.129946  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 787) took 1223 ms. Trace:
I20260808 01:51:37.141201  9473 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:37.113214  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:37.143086  9551 raft_consensus.cc:2410] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260808 01:51:37.141688  9472 rpcz_store.cc:276] 0808 01:51:35.907079 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:36.770568 (+863489us) service_pool.cc:224] Handling call
0808 01:51:36.770938 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:36.771173 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:51:36.771545 (+   372us) log.cc:844] Serialized 14 byte log entry
0808 01:51:37.129515 (+357970us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:37.129582 (+    67us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:51:37.129637 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:37.129723 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:37.129874 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":760,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":3689,"thread_start_us":357445,"threads_started":1}
I20260808 01:51:37.143996  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:37.144275  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:37.036453  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:37.129304  9478 rpcz_store.cc:276] 0808 01:51:35.960110 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:36.011097 (+ 50987us) service_pool.cc:224] Handling call
0808 01:51:37.115777 (+1104680us) spinlock_profiling.cc:243] Waited 554 ms on lock 0x7b54002bb1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:37.115886 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1273075456}
W20260808 01:51:37.131044  9635 rpcz_store.cc:269] 0808 01:51:35.078344 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:35.195473 (+117129us) service_pool.cc:224] Handling call
0808 01:51:37.048540 (+1853067us) spinlock_profiling.cc:243] Waited 743 ms on lock 0x7b540032d554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:37.113493 (+ 64953us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1707863936}
W20260808 01:51:37.147487  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 111145us
I20260808 01:51:37.131785  9475 rpcz_store.cc:276] 0808 01:51:35.511296 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:36.496752 (+985456us) service_pool.cc:224] Handling call
0808 01:51:36.497143 (+   391us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:36.497569 (+   426us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:51:36.531055 (+ 33486us) log.cc:844] Serialized 14 byte log entry
0808 01:51:36.752392 (+221337us) spinlock_profiling.cc:243] Waited 6.13 ms on lock 0x7b1800435be0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4889671f 00007f5d4188ceb1 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d
0808 01:51:36.898349 (+145957us) log.cc:844] Serialized 14 byte log entry
0808 01:51:36.901119 (+  2770us) log.cc:844] Serialized 14 byte log entry
0808 01:51:36.901218 (+    99us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:36.908946 (+  7728us) log.cc:844] Serialized 26 byte log entry
0808 01:51:36.909113 (+   167us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:36.909156 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:36.918563 (+  9407us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:37.034217 (+115654us) raft_consensus.cc:1685] finished
0808 01:51:37.034239 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:37.034388 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":195970,"prepare.queue_time_us":42754,"prepare.run_cpu_time_us":160,"prepare.run_wall_time_us":3288,"raft.queue_time_us":169568,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":27523,"spinlock_wait_cycles":15035648,"thread_start_us":137981,"threads_started":2,"wal-append.queue_time_us":258823}
I20260808 01:51:37.138491  9637 rpcz_store.cc:276] 0808 01:51:35.036584 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:35.686950 (+650366us) service_pool.cc:224] Handling call
0808 01:51:35.687426 (+   476us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:36.896649 (+1209223us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:37.043508 (+146859us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540003b1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:37.043705 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2604788480}
I20260808 01:51:37.149554  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:37.134721  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:37.150225  9472 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:37.152453  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:37.107650  9636 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260808 01:51:37.154086  9555 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:37.154836  9476 raft_consensus.cc:2393] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:37.161129  9473 raft_consensus.cc:2368] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13.
I20260808 01:51:37.169319  9599 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:37.011744 14769 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:37.117009 14405 raft_consensus.cc:3055] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:37.188303 14405 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:37.189334 14405 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:37.190805 14645 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260808 01:51:37.191990 14781 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:37.192548 14781 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:37.192936 14781 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:37.194933 14787 consensus_queue.cc:1059] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:51:37.195744  9636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:37.196511  9636 raft_consensus.cc:606] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:37.196930  9636 raft_consensus.cc:614] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:37.179813  9599 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:37.197337 14769 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:37.108098 14740 consensus_meta.cc:287] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.622s	user 0.011s	sys 0.004s
I20260808 01:51:37.198985 14740 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:37.200240 10366 raft_consensus.cc:2368] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12.
I20260808 01:51:37.201371 14740 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:37.032101 14730 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.944s	user 0.001s	sys 0.001s
I20260808 01:51:37.201880 14769 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:37.021202 14756 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.550s	user 0.000s	sys 0.001s
I20260808 01:51:37.202546  9479 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:37.116744 14655 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:37.184649 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:37.204533 14645 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:37.205266 14645 raft_consensus.cc:697] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:37.205591  9523 leader_election.cc:400] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12.
I20260808 01:51:37.205966  9523 leader_election.cc:403] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:51:37.207096  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 588) took 1393 ms. Trace:
I20260808 01:51:37.208103 10364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 673) took 1979 ms. Trace:
I20260808 01:51:37.206149 14645 consensus_queue.cc:237] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:37.198583  9599 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:37.209486 14659 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:37.208531 10364 rpcz_store.cc:276] 0808 01:51:35.229155 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:35.229299 (+   144us) service_pool.cc:224] Handling call
0808 01:51:35.260576 (+ 31277us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:35.261009 (+   433us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:35.261062 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:35.272711 (+ 11649us) log.cc:844] Serialized 26 byte log entry
0808 01:51:35.749043 (+476332us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:35.749082 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:35.749156 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:37.031445 (+1282289us) spinlock_profiling.cc:243] Waited 45.6 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:37.207770 (+176325us) raft_consensus.cc:1685] finished
0808 01:51:37.207792 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:37.208015 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2702,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":16215,"spinlock_wait_cycles":104877312,"thread_start_us":471004,"threads_started":1,"wal-append.queue_time_us":673958}
I20260808 01:51:37.210278 14659 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:37.195940 14787 raft_consensus.cc:3055] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:37.210882 14537 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 12
I20260808 01:51:37.211398  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2662) took 2472 ms. Trace:
I20260808 01:51:37.212127 14709 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:37.015794 14631 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:37.234654  9599 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:37.213517 14534 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:37.042418 14729 raft_consensus.cc:697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:37.042212 14543 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:37.170032 13701 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:37.028815 14763 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:37.167249 14428 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:37.252341 14428 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:37.253021 14428 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:37.037859 14669 consensus_meta.cc:287] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.771s	user 0.013s	sys 0.000s
I20260808 01:51:37.255225 14669 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:37.257138 14543 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:37.119644 14403 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:37.258881 14784 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20260808 01:51:37.259579 14403 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:37.260442 14784 raft_consensus.cc:3055] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:37.260881 14784 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:37.219713 14779 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.311s	user 0.002s	sys 0.000s
I20260808 01:51:37.262946 10366 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:37.263679 14718 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:37.264148 14718 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:37.264560 14718 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:37.219568  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet affdd24b03ae4ee795a90703613305f7 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:37.181802 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:37.169853 14708 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:37.181077 14656 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260808 01:51:37.221395  9475 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
W20260808 01:51:37.019989 14736 log.cc:927] Time spent T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.087s	user 0.004s	sys 0.000s
I20260808 01:51:37.207499  9634 rpcz_store.cc:276] 0808 01:51:35.814204 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:35.830894 (+ 16690us) service_pool.cc:224] Handling call
0808 01:51:35.867517 (+ 36623us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:35.875024 (+  7507us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:35.875074 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:35.875711 (+   637us) log.cc:844] Serialized 26 byte log entry
0808 01:51:36.027610 (+151899us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:51:36.027661 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:36.027742 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:37.020561 (+992819us) spinlock_profiling.cc:243] Waited 26.1 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:37.206760 (+186199us) raft_consensus.cc:1685] finished
0808 01:51:37.206785 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:37.207024 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4237,"raft.queue_time_us":5325,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":12913,"spinlock_wait_cycles":60116224,"thread_start_us":129581,"threads_started":1,"wal-append.queue_time_us":503911}
I20260808 01:51:37.222798  9524 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:37.275480  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52791.2us
I20260808 01:51:37.230108  9633 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:37.227221 14659 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:51:37.201406  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:37.297307  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:37.299667  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:37.299741  9524 leader_election.cc:400] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13.
I20260808 01:51:37.232899  9474 rpcz_store.cc:276] 0808 01:51:34.738647 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:34.738995 (+   348us) service_pool.cc:224] Handling call
0808 01:51:34.739314 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:36.629044 (+1889730us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:37.211095 (+582051us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b54002c0e54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:37.211301 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4260403584}
I20260808 01:51:37.300490  9632 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:51:37.301551  9474 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260808 01:51:37.301774  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:37.302281  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:37.300514  9524 leader_election.cc:403] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260808 01:51:37.235116  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2690) took 1245 ms. Trace:
I20260808 01:51:37.190518 14405 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:37.233197 14787 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:37.245721 14631 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:37.037774 14591 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:37.036111 14750 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:37.258224  9557 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: [12.6-12.6]
I20260808 01:51:37.310105  9556 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:37.268757 10367 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:37.261770 14784 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:37.258132 14669 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:37.230890  9473 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:37.318051  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:37.282454  9472 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:37.302907  9633 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:37.303273  9480 rpcz_store.cc:276] 0808 01:51:35.989701 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:36.983651 (+993950us) service_pool.cc:224] Handling call
0808 01:51:36.984217 (+   566us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:36.984584 (+   367us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:51:37.001923 (+ 17339us) log.cc:844] Serialized 14 byte log entry
0808 01:51:37.234369 (+232446us) log.cc:844] Serialized 14 byte log entry
0808 01:51:37.234594 (+   225us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:37.234658 (+    64us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:37.234746 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:37.234857 (+   111us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:37.235028 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":19797,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":95,"raft.queue_time_us":876,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":5443,"spinlock_wait_cycles":9248384,"thread_start_us":230386,"threads_started":1}
I20260808 01:51:37.244510  9599 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:37.314081  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:37.320747  9474 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:37.321708  9636 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260808 01:51:37.246012 14608 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.064s
I20260808 01:51:37.271955 14768 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:37.270373 14708 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:37.322817  9524 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:37.323400  9634 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260808 01:51:37.412967  9634 pending_rounds.cc:85] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 7
I20260808 01:51:37.413342  9634 pending_rounds.cc:107] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260808 01:51:37.413599  9634 raft_consensus.cc:2889] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:37.414984  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:37.316903 14750 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:37.309645 14755 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260808 01:51:37.326412  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 82050.2us
I20260808 01:51:37.314644  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:37.301277 14659 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:37.324115  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:37.306931 14403 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:37.424232  9632 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
W20260808 01:51:37.423621  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 315598us
I20260808 01:51:37.426028  9638 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:37.416481  9634 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:37.424134  9636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:37.423350  9599 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:37.427901  9636 raft_consensus.cc:606] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:37.429375  9558 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:37.329304 14591 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:37.316175 14542 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13.
I20260808 01:51:37.260366 14729 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:37.414863 14708 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:37.331763 13701 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:37.321506 14647 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.109s
I20260808 01:51:37.440357  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:37.425992 14784 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:37.453845 13701 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:37.424975 14755 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:37.468703 14536 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:37.470072  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2897) took 1127 ms. Trace:
I20260808 01:51:37.499701  9559 rpcz_store.cc:276] 0808 01:51:36.342402 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:36.342598 (+   196us) service_pool.cc:224] Handling call
0808 01:51:36.382519 (+ 39921us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:36.838436 (+455917us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:37.469718 (+631282us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b54001acf54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:37.469984 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":634606464}
I20260808 01:51:37.494596  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:37.413702 14768 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:37.452320 14707 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:37.497735  9634 raft_consensus.cc:2410] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260808 01:51:37.498292 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:37.505726  9599 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:37.476643 13701 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:37.502418 14755 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:37.489319 14787 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:37.525484 14787 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:37.506299 14782 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.025s
I20260808 01:51:37.531906 14782 raft_consensus.cc:3055] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:51:37.532361 14782 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:37.532428 14768 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:37.532632 14535 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:37.533520 14782 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:37.534268 14795 log.cc:927] Time spent T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260808 01:51:37.501896 14647 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:51:37.534950 14647 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:37.487342 14682 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:37.499470 14536 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:37.542198 14536 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:37.514254 14588 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:37.537119 14608 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
W20260808 01:51:37.558189 14798 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260808 01:51:37.538597 14707 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:51:37.568445 10332 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.478s	user 0.008s	sys 0.037s
W20260808 01:51:37.568893 10332 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.479s	user 0.008s	sys 0.037s
W20260808 01:51:37.569195 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 1.47917e+06us
W20260808 01:51:37.589694  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:37.592935  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 270191us
I20260808 01:51:37.539187 14781 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:37.544672 14682 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:37.567304 14588 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:37.535917 14647 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:37.620796 14801 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260808 01:51:37.622073  9521 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:37.654764  9521 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:37.652261 14774 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.916s	user 0.005s	sys 0.000s
I20260808 01:51:37.682888  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2411) took 2665 ms. Trace:
W20260808 01:51:37.688784  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:37.683420  9629 rpcz_store.cc:276] 0808 01:51:35.017932 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:35.681450 (+663518us) service_pool.cc:224] Handling call
0808 01:51:35.706223 (+ 24773us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:35.816391 (+110168us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:51:35.816813 (+   422us) log.cc:844] Serialized 14 byte log entry
0808 01:51:35.932283 (+115470us) log.cc:844] Serialized 14 byte log entry
0808 01:51:35.935213 (+  2930us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:35.935287 (+    74us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:35.935345 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:36.428207 (+492862us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b1800404140. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4185ac56 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229
0808 01:51:36.505338 (+ 77131us) spinlock_profiling.cc:243] Waited 19.5 ms on lock 0x7b1800404140. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:51:37.682584 (+1177246us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b54004213d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:37.682643 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:37.682824 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28263,"prepare.queue_time_us":1702,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":93,"raft.queue_time_us":7347,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":10003,"spinlock_wait_cycles":437342208,"thread_start_us":112273,"threads_started":1,"wal-append.queue_time_us":329767,"wal-append.run_cpu_time_us":1907,"wal-append.run_wall_time_us":1059443}
W20260808 01:51:37.682451 14797 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
W20260808 01:51:37.702885  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 186779us
I20260808 01:51:37.708236 14805 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.010s
I20260808 01:51:37.717262 14755 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:37.704612  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:37.700730 14542 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260808 01:51:37.740891  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:37.742297  9599 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:37.744374 10368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 847) took 1468 ms. Trace:
W20260808 01:51:37.744535  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:37.745155  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:37.746178  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:37.710469 14800 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260808 01:51:37.744885 10368 rpcz_store.cc:276] 0808 01:51:36.276266 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:36.276514 (+   248us) service_pool.cc:224] Handling call
0808 01:51:36.276910 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:36.283205 (+  6295us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:36.283238 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:36.283794 (+   556us) log.cc:844] Serialized 26 byte log entry
0808 01:51:36.545124 (+261330us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:36.545173 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:36.545255 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:37.517639 (+972384us) spinlock_profiling.cc:243] Waited 4.27 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:37.736049 (+218410us) raft_consensus.cc:1685] finished
0808 01:51:37.736087 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:37.744282 (+  8195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5819,"raft.queue_time_us":7170,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":102,"spinlock_wait_cycles":9831808,"thread_start_us":260880,"threads_started":1,"wal-append.queue_time_us":374621}
W20260808 01:51:37.752171 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:37.666705 14707 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:37.721855 14608 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:37.710693 14806 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:37.734577 14805 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260808 01:51:37.744978 14676 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.009s	sys 0.017s
W20260808 01:51:37.753971 14743 consensus_meta.cc:287] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.601s	user 0.013s	sys 0.000s
I20260808 01:51:37.700157 14588 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:37.809067  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 154430us
I20260808 01:51:37.810045  9632 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 8. (term mismatch)
I20260808 01:51:37.810463  9632 pending_rounds.cc:85] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 7
I20260808 01:51:37.810721  9632 pending_rounds.cc:107] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 9.8
I20260808 01:51:37.810987  9632 raft_consensus.cc:2889] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:51:37.804133 14676 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.009s	sys 0.018s
I20260808 01:51:37.813010  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:37.815968 14608 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:37.802620 14806 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:37.807056 14534 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:37.807607 14763 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:51:37.827654  9521 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:37.828207  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 86047.7us
I20260808 01:51:37.832397  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:37.833079  9560 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:37.834084  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:37.835099  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:37.835264  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:37.835871  9550 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:37.836886  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:37.837445  9550 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:37.839653  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:37.816038 14682 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:37.845717  9638 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:37.843778  9521 leader_election.cc:400] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13.
I20260808 01:51:37.848603  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:37.838253 14534 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:37.825163 14781 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:37.829221 14818 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260808 01:51:37.850837  9551 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
W20260808 01:51:37.851943 14631 consensus_meta.cc:287] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.529s	user 0.004s	sys 0.013s
I20260808 01:51:37.855953 14817 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:37.856321 14537 consensus_meta.cc:287] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.608s	user 0.012s	sys 0.000s
I20260808 01:51:37.845620 14542 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:37.860576 14536 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:37.868021 14755 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:37.873497 14750 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:37.830727 14659 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:37.918159  9560 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
W20260808 01:51:37.887681 14718 consensus_meta.cc:287] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.623s	user 0.012s	sys 0.004s
W20260808 01:51:37.874975 14771 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 1.111s	user 0.001s	sys 0.002s
I20260808 01:51:37.922228  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:37.827188 14805 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:37.831210 14405 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:37.891142  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:37.891422  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:37.904218  9550 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:37.905655  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:37.930150  9637 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:37.921271 14428 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.660s	user 0.015s	sys 0.000s
I20260808 01:51:37.910769  9559 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260808 01:51:37.911777  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:37.915300  9634 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260808 01:51:37.877575 14647 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:37.923573  9560 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
W20260808 01:51:37.920274 14807 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
W20260808 01:51:37.908937  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 106886us
W20260808 01:51:37.918277 14794 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.495s	user 0.000s	sys 0.004s
I20260808 01:51:37.913996 14537 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12.
I20260808 01:51:37.925985 14805 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:37.932072 14405 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:37.917671 13701 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:37.916443 14631 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:37.917232 14817 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:37.929458 14750 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
W20260808 01:51:37.973584  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:37.926796 14718 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:37.962143 14805 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:51:37.973536  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 131692us
I20260808 01:51:37.967883 14405 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:37.981853  9481 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:37.982126  9633 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:37.944698 14428 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:37.991194 14428 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:37.991601 14659 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:37.984031  9637 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:51:38.000070  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:37.992566 14822 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.022s
I20260808 01:51:38.002974 14718 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:37.981056 14821 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:38.006824 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:38.003917 14811 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
I20260808 01:51:38.032631 14806 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:38.035827 14828 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260808 01:51:38.037240  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:38.039855  9599 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:38.040903 14428 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:38.041337 14428 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:38.041726 14428 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:38.039719 14824 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:38.043253 14824 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:38.043753  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:38.044595  9555 raft_consensus.cc:2368] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12.
I20260808 01:51:38.045573  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:38.046178  9555 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:38.024009 14821 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:38.049381  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:38.051082  9635 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:38.041347  9599 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:38.051116  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:38.053366  9559 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:38.049680 14821 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:38.004212 13701 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:38.058022  9635 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 6. (term mismatch)
I20260808 01:51:38.012053 14631 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:38.051488 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:38.012743 14536 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:38.069278  9635 pending_rounds.cc:85] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 5
I20260808 01:51:38.075441 14821 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:51:38.087703  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:38.089745  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 205512us
I20260808 01:51:38.092358  9521 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:38.193403  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 101148us
I20260808 01:51:38.194509  9521 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:38.196084  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:38.198166  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:38.216779  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 129157us
I20260808 01:51:38.063992 14824 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:38.024470 14682 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:38.217691 14695 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:38.216780 14809 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:51:38.242012 14707 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:38.219949 14806 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.027s
I20260808 01:51:38.243089 14707 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.217178 14831 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.155s
I20260808 01:51:38.217029 14542 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:38.229956 14659 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:38.247159 14542 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:38.251425  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 789) took 2031 ms. Trace:
I20260808 01:51:38.087004 14827 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:38.063871 14830 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:38.217876  9599 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:38.087338 14787 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.561s	user 0.016s	sys 0.000s
I20260808 01:51:38.077862  9635 pending_rounds.cc:107] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 12.6
I20260808 01:51:38.267104  9635 raft_consensus.cc:2917] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Aborting config change with OpId 12.6: Aborted: Op aborted by new leader
I20260808 01:51:38.267693  9635 raft_consensus.cc:2889] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:51:38.226197 14708 consensus_meta.cc:287] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.753s	user 0.015s	sys 0.000s
W20260808 01:51:38.277658  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 59878.8us
I20260808 01:51:38.249650 14755 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:38.277944  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:45650 (request call id 16) took 8746 ms. Trace:
I20260808 01:51:38.252307  9477 rpcz_store.cc:276] 0808 01:51:36.219637 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:37.011633 (+791996us) service_pool.cc:224] Handling call
0808 01:51:37.029986 (+ 18353us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:37.030202 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 12
0808 01:51:37.030583 (+   381us) log.cc:844] Serialized 14 byte log entry
0808 01:51:37.271301 (+240718us) log.cc:844] Serialized 14 byte log entry
0808 01:51:37.659063 (+387762us) spinlock_profiling.cc:243] Waited 23.8 ms on lock 0x7b18003a3d80. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:51:38.250594 (+591531us) spinlock_profiling.cc:243] Waited 2.76 ms on lock 0x7b18003a3d80. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667 00007f5d449ae5e8
0808 01:51:38.250822 (+   228us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:38.250895 (+    73us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:51:38.251072 (+   177us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:38.251175 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:38.251347 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1081,"prepare.queue_time_us":2532,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":96,"raft.queue_time_us":137,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":26185,"spinlock_wait_cycles":61046400,"thread_start_us":239446,"threads_started":1,"wal-append.queue_time_us":411614}
I20260808 01:51:38.288545  9559 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:38.251485 14818 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:38.083220  9555 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:38.295256  9599 leader_election.cc:400] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12.
I20260808 01:51:38.295624  9599 leader_election.cc:403] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260808 01:51:38.295759  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:38.288096  9635 rpcz_store.cc:276] 0808 01:51:29.521621 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:29.559520 (+ 37899us) service_pool.cc:224] Handling call
0808 01:51:29.852220 (+292700us) log.cc:844] Serialized 506 byte log entry
0808 01:51:30.435830 (+583610us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x7b1800219de0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449d00e1 00007f5d4490ca71 00007f5d448ed28a 00007f5d448f4133 00007f5d448f40b9 00007f5d448f4049 00007f5d448f4011 00007f5d448f31dd 00000000003aa554 00000000003aa399
0808 01:51:30.584688 (+148858us) spinlock_profiling.cc:243] Waited 2.45 ms on lock 0x7b1800219de0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449d00e1 00007f5d4490ca71 00007f5d448ed28a 00007f5d448f4133 00007f5d448f40b9 00007f5d448f4049 00007f5d448f4011 00007f5d448f31dd 00000000003aa554 00000000003aa399
0808 01:51:38.268234 (+7683546us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37724,"raft.queue_time_us":458470,"raft.run_cpu_time_us":53703,"raft.run_wall_time_us":700024,"spinlock_wait_cycles":53238016,"thread_start_us":465782,"threads_started":4,"wal-append.queue_time_us":259768,"wal-append.run_cpu_time_us":2332,"wal-append.run_wall_time_us":1012945}
I20260808 01:51:38.302517  9555 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:38.299185  9599 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:38.267107 14682 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:38.318259  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:38.280339 14659 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:38.330438  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2554) took 1014 ms. Trace:
I20260808 01:51:38.292501 14824 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:38.331139  9475 rpcz_store.cc:276] 0808 01:51:37.316284 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:37.319308 (+  3024us) service_pool.cc:224] Handling call
0808 01:51:37.412375 (+ 93067us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:37.412610 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:51:37.509137 (+ 96527us) log.cc:844] Serialized 14 byte log entry
0808 01:51:38.329917 (+820780us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:38.330005 (+    88us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:38.330072 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:38.330174 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:38.330351 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":520972,"prepare.queue_time_us":201258,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":30249,"thread_start_us":381189,"threads_started":2,"wal-append.queue_time_us":340252}
I20260808 01:51:38.287855 14755 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:38.334581  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 2c617cb141a34007b6c1b8af0d911923: Aborted: Op aborted by new leader
I20260808 01:51:38.313006 14827 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:38.347306  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:38.349314  9559 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:51:38.332720 14826 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
I20260808 01:51:38.276804 14707 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:38.286187 14708 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:38.352267  9555 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:51:38.363731  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:38.323557 14817 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:38.332619 14534 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:38.345350 14755 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:38.371516 14631 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.010s
I20260808 01:51:38.380195 14631 consensus_queue.cc:1059] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 01:51:38.282357 14830 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.383137 14830 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:38.313637 14818 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:38.384250 14659 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.350370 14827 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.386806 14818 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:38.388139 14827 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:38.389477 14799 log.cc:927] Time spent T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Append to log took a long time: real 0.897s	user 0.001s	sys 0.002s
W20260808 01:51:38.390210 14832 log.cc:927] Time spent T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260808 01:51:38.402961 14682 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:38.410041 14708 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:38.410880  9550 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:38.411836  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 658) took 3042 ms. Trace:
I20260808 01:51:38.412264  9556 rpcz_store.cc:276] 0808 01:51:35.369262 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:35.369440 (+   178us) service_pool.cc:224] Handling call
0808 01:51:35.377247 (+  7807us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:37.310544 (+1933297us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:38.411561 (+1101017us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b54001ac554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:38.411750 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4444656000}
I20260808 01:51:38.413151 14405 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.372799 14781 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:38.383219 14715 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260808 01:51:38.362746 14729 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:38.419337 14729 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.419898 10332 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:38.351245 13701 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:38.410984  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2280) took 1381 ms. Trace:
I20260808 01:51:38.423123  9476 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
W20260808 01:51:38.427491  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 206817us
I20260808 01:51:38.428810  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2701) took 1541 ms. Trace:
I20260808 01:51:38.429180  9476 rpcz_store.cc:276] 0808 01:51:36.887406 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:37.207503 (+320097us) service_pool.cc:224] Handling call
0808 01:51:37.207874 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:38.421788 (+1213914us) spinlock_profiling.cc:243] Waited 18.9 ms on lock 0x7b5400194c20. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490b1a0 00007f5d449955a0 00007f5d44996611 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c
0808 01:51:38.428519 (+  6731us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:38.428746 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":43478784}
I20260808 01:51:38.429912  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2696) took 1907 ms. Trace:
I20260808 01:51:38.430305  9481 rpcz_store.cc:276] 0808 01:51:36.522204 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:37.150822 (+628618us) service_pool.cc:224] Handling call
0808 01:51:37.151145 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:37.992652 (+841507us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:38.429658 (+437006us) spinlock_profiling.cc:243] Waited 830 ms on lock 0x7b54002bbbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:38.429833 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1909993216}
I20260808 01:51:38.426316  9479 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260808 01:51:38.431038  9559 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:38.434075  9635 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260808 01:51:38.427632  9557 rpcz_store.cc:276] 0808 01:51:37.029539 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:37.128884 (+ 99345us) service_pool.cc:224] Handling call
0808 01:51:37.129283 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:37.310315 (+181032us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:38.410668 (+1100353us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b54003255d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:38.410916 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":295797504}
I20260808 01:51:38.439474 14405 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:38.448442  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:38.449122  9477 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:38.449641  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:38.459990  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:38.455348  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2435) took 1039 ms. Trace:
I20260808 01:51:38.469576  9632 rpcz_store.cc:276] 0808 01:51:37.416113 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:37.426616 (+ 10503us) service_pool.cc:224] Handling call
0808 01:51:37.426975 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:37.811417 (+384442us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:38.454959 (+643542us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b5400421dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:38.455252 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":880122112}
I20260808 01:51:38.473619  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:38.474983  9632 raft_consensus.cc:2393] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 12: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:38.476258  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:38.421769 14542 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:38.431950 14284 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:38.480698  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:38.500663  9473 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:38.484572 14652 raft_consensus.cc:3055] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:38.432649 14715 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:38.436303 14688 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.049s
I20260808 01:51:38.486032 14602 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:38.421626 14729 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:38.464118 14647 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:38.440779 14681 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260808 01:51:38.487339  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:38.532977  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:38.533653  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:38.534030  9560 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:51:38.460498 14781 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:38.402561 14805 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.491333 14708 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:38.544131  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 245023us
I20260808 01:51:38.506103 14755 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.510846 14428 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.514881 14542 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:38.493652 14707 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260808 01:51:38.505460 14715 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:38.545886  9599 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:38.671389  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 125605us
I20260808 01:51:38.672490  9599 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:38.674149  9599 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:38.470234  9477 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:38.537644 14284 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:38.648689 13701 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12.
I20260808 01:51:38.678996  9557 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260808 01:51:38.679705  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:38.680855  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:38.681720  9558 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:38.499877 14676 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.141s
I20260808 01:51:38.537153 14681 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:38.551080  9523 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:38.502890 14652 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:38.687898  9560 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:38.692368  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:38.679257 13701 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:38.729964 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:38.690318  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:38.742367 14656 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:38.639664 14831 consensus_queue.cc:1059] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.103s
I20260808 01:51:38.537392 14806 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:38.743075  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:38.763957  9556 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:38.764473  9554 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:38.768397  9558 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260808 01:51:38.742905 14827 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:51:38.779389  9599 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:38.781019 13701 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:38.544701 14647 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:38.793268 14787 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:51:38.792128  9556 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:38.756978 14681 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:38.787626  9635 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:38.534830 14602 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:38.795233 13701 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.640093 14708 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.783164 14537 raft_consensus.cc:3055] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:38.672241 14542 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.842358 14781 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.883555  9557 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:38.885089 14428 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:38.802757 14657 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:38.811187 14676 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:38.810220 14827 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:38.875073 14825 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260808 01:51:38.897008  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:38.785581 14806 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:38.851643 14652 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.813443 14537 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:38.810819 14831 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.084s
I20260808 01:51:38.878978 14695 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:38.814666 14715 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.907513 14827 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.909960  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:38.903991 14676 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.841445 14284 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.940467 14695 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:38.952216 14695 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:38.942981 14535 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:38.953768 14535 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:38.954185 14535 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:38.954902 14647 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:38.902930 14657 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.956293 14805 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:38.956786  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34802 (request call id 59) took 1627 ms. Trace:
I20260808 01:51:38.957170 14841 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:38.957414 14657 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:38.957742 14841 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:38.960215 14841 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:38.913312 14428 raft_consensus.cc:3055] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:38.957206  9636 rpcz_store.cc:276] 0808 01:51:37.329033 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:37.329953 (+   920us) service_pool.cc:224] Handling call
0808 01:51:38.408911 (+1078958us) spinlock_profiling.cc:243] Waited 37.9 ms on lock 0x7b540016f920. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44910900 00007f5d4498da32 00007f5d4998d41a 00007f5d42ee1934 00007f5d42ee18c2 00007f5d42ee17f1 00007f5d42ee176c 00007f5d42ee0a22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:38.956446 (+547535us) spinlock_profiling.cc:243] Waited 41.9 ms on lock 0x7b540016f920. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490e9c7 00007f5d4498e048 00007f5d4498dc2a 00007f5d4998d41a 00007f5d42ee1934 00007f5d42ee18c2 00007f5d42ee17f1 00007f5d42ee176c 00007f5d42ee0a22 00007f5d42ccd07c
0808 01:51:38.956685 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":183483008}
I20260808 01:51:38.970042  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:38.951361 14816 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:38.936213 14755 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:38.947979 14602 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:38.961349 14715 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:51:38.980679  9600 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.020s	sys 0.058s
I20260808 01:51:38.982096  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:38.983253  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.045s	sys 0.062s
I20260808 01:51:38.966540 14428 raft_consensus.cc:740] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:38.973174 14588 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:38.973130 14816 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:39.001431  9445 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.658s	user 0.015s	sys 0.030s
W20260808 01:51:39.003058  9445 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.016s	sys 0.030s
W20260808 01:51:38.985522  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.046s	sys 0.062s
W20260808 01:51:39.004892  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 634368us
W20260808 01:51:38.984078  9600 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.020s	sys 0.059s
W20260808 01:51:38.985097  9629 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.017s	sys 0.027s
W20260808 01:51:39.009877  9629 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.017s	sys 0.028s
W20260808 01:51:38.986644  9552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.006s	sys 0.048s
W20260808 01:51:39.012272  9552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.007s	sys 0.048s
W20260808 01:51:39.009183 14824 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.032s	sys 0.024s
W20260808 01:51:39.050072  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 43693.2us
I20260808 01:51:39.052229  9636 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:39.055660 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:39.056685 10366 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:39.054141  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:39.059860  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:39.061240  9600 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:39.007799 14708 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:39.068025  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:39.068782  9636 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:51:39.053215 14824 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.032s	sys 0.024s
I20260808 01:51:38.977979 14537 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:38.996512 14588 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.086486  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:39.100914  9446 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.004s	sys 0.017s
W20260808 01:51:39.101447  9446 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.004s	sys 0.017s
I20260808 01:51:39.108052  9636 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
W20260808 01:51:39.087756 14682 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.681s	user 0.004s	sys 0.015s
I20260808 01:51:39.076014 14845 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:39.054057 14729 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
W20260808 01:51:39.142349  9638 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.653s	user 0.003s	sys 0.014s
W20260808 01:51:39.216893  9638 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.004s	sys 0.014s
I20260808 01:51:39.218839  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 591) took 1395 ms. Trace:
I20260808 01:51:39.089802 14830 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:39.095939 14645 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.269s	user 0.005s	sys 0.012s
I20260808 01:51:39.222360  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:39.085122 14843 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:39.137813 14632 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260808 01:51:39.148466 14682 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.742s	user 0.005s	sys 0.015s
I20260808 01:51:39.219475  9631 rpcz_store.cc:276] 0808 01:51:37.823381 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:37.823523 (+   142us) service_pool.cc:224] Handling call
0808 01:51:37.823933 (+   410us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:37.922871 (+ 98938us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:37.923000 (+   129us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:37.931796 (+  8796us) log.cc:844] Serialized 52 byte log entry
0808 01:51:38.087852 (+156056us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:51:38.223553 (+135701us) log.cc:844] Serialized 14 byte log entry
0808 01:51:38.224015 (+   462us) log.cc:844] Serialized 14 byte log entry
0808 01:51:38.263166 (+ 39151us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:38.263292 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:38.517921 (+254629us) spinlock_profiling.cc:243] Waited 3.74 ms on lock 0x7b18003c7e40. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4185ac56 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121
0808 01:51:38.532710 (+ 14789us) spinlock_profiling.cc:243] Waited 5.88 ms on lock 0x7b18003c7e40. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:51:38.731810 (+199100us) spinlock_profiling.cc:243] Waited 5.77 ms on lock 0x7b18003c7e40. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188ce62 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:51:38.953570 (+221760us) spinlock_profiling.cc:243] Waited 15.8 ms on lock 0x7b18003c7e40. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:51:38.954229 (+   659us) spinlock_profiling.cc:243] Waited 6.12 ms on lock 0x7b18003c7e40. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:51:39.218504 (+264275us) raft_consensus.cc:1685] finished
0808 01:51:39.218539 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:39.218747 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":242215,"prepare.queue_time_us":197140,"prepare.run_cpu_time_us":122,"prepare.run_wall_time_us":908,"raft.queue_time_us":227490,"raft.run_cpu_time_us":414,"raft.run_wall_time_us":16460,"spinlock_wait_cycles":85904512,"thread_start_us":159577,"threads_started":2,"wal-append.queue_time_us":337514}
I20260808 01:51:39.231858  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:39.232661  9631 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
W20260808 01:51:39.216460 14669 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.466s	user 0.000s	sys 0.039s
I20260808 01:51:39.233729  9631 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "df6982ec010444a8854140f7b64e985d"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:39.235011  9631 raft_consensus.cc:606] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:39.235575  9631 raft_consensus.cc:614] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:39.237517  9631 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260808 01:51:39.234020 14632 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:39.087221 14824 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:39.067673 14428 consensus_queue.cc:260] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:39.227196 14645 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.404s	user 0.006s	sys 0.012s
I20260808 01:51:39.222177 14806 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:39.242017 14669 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.492s	user 0.000s	sys 0.040s
I20260808 01:51:39.239338 14632 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:39.249097 14847 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:39.259740 14799 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.799s	user 0.010s	sys 0.027s
I20260808 01:51:39.265085 14676 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:39.087200 14816 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.261948 14729 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:39.286628 14842 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.183s	user 0.001s	sys 0.000s
I20260808 01:51:39.239207 14849 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:39.230232 14830 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.225355 14845 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:39.302017 14799 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.841s	user 0.010s	sys 0.027s
I20260808 01:51:39.300855 14848 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:39.296710 14847 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.234632 14843 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.241904 14824 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.329618 14849 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:39.331921 14848 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.365244 14830 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:39.354174 14852 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:39.376770 14852 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:39.423130  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet df6982ec010444a8854140f7b64e985d (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
W20260808 01:51:39.424705  9523 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.660s	user 0.009s	sys 0.008s
W20260808 01:51:39.425407  9523 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.009s	sys 0.008s
I20260808 01:51:39.425459  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 1
W20260808 01:51:39.428270  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 877177us
W20260808 01:51:39.433014  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:39.435608  9555 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.004s	sys 0.007s
W20260808 01:51:39.436337  9555 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.004s	sys 0.007s
W20260808 01:51:39.437387  9632 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.945s	user 0.000s	sys 0.035s
W20260808 01:51:39.438019  9632 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.946s	user 0.000s	sys 0.035s
I20260808 01:51:39.439409  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2938) took 1139 ms. Trace:
I20260808 01:51:39.439755  9630 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14. Ops: []
W20260808 01:51:39.436342  9599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.016s	sys 0.045s
I20260808 01:51:39.441184  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2420) took 1011 ms. Trace:
W20260808 01:51:39.441884  9599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.017s	sys 0.045s
W20260808 01:51:39.442577  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 663244us
I20260808 01:51:39.246457  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:39.445948  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:39.448633 10332 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.997s	user 0.008s	sys 0.032s
W20260808 01:51:39.449493 10333 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.028s	user 0.002s	sys 0.033s
W20260808 01:51:39.450026 10333 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.028s	user 0.002s	sys 0.033s
I20260808 01:51:39.451136  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2410) took 1453 ms. Trace:
W20260808 01:51:39.453037 10334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.006s	sys 0.005s
W20260808 01:51:39.453569 10334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.512s	user 0.006s	sys 0.006s
I20260808 01:51:39.453748  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2421) took 1023 ms. Trace:
I20260808 01:51:39.238622 14591 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:39.508996 14866 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:39.449152  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:39.449226 10332 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.997s	user 0.008s	sys 0.032s
W20260808 01:51:39.454038 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 512707us
W20260808 01:51:39.435598  9559 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.006s	sys 0.012s
W20260808 01:51:39.304766 14763 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.376s	user 0.018s	sys 0.028s
I20260808 01:51:39.289033 14682 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.013s
W20260808 01:51:39.374272 14846 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260808 01:51:39.527045  9559 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.754s	user 0.006s	sys 0.013s
I20260808 01:51:39.329006 14816 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:39.441962  9632 rpcz_store.cc:276] 0808 01:51:38.429809 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:38.483679 (+ 53870us) service_pool.cc:224] Handling call
0808 01:51:38.489928 (+  6249us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:38.491221 (+  1293us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:38.491267 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:38.491889 (+   622us) log.cc:844] Serialized 26 byte log entry
0808 01:51:39.439163 (+947274us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:39.439968 (+   805us) log.cc:844] Serialized 14 byte log entry
0808 01:51:39.440454 (+   486us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:39.440634 (+   180us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:39.440743 (+   109us) raft_consensus.cc:1685] finished
0808 01:51:39.440760 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:39.441024 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":573,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"thread_start_us":946318,"threads_started":1,"wal-append.queue_time_us":880289}
W20260808 01:51:39.533432 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 1.08176e+06us
I20260808 01:51:39.356072 14849 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:39.454224  9638 rpcz_store.cc:276] 0808 01:51:38.430096 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:38.487413 (+ 57317us) service_pool.cc:224] Handling call
0808 01:51:38.487863 (+   450us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:38.488312 (+   449us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:38.488341 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:38.489004 (+   663us) log.cc:844] Serialized 26 byte log entry
0808 01:51:39.217501 (+728497us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:39.217562 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:39.217664 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:39.453077 (+235413us) raft_consensus.cc:1685] finished
0808 01:51:39.453121 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:39.453481 (+   360us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":96,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"thread_start_us":727962,"threads_started":1,"wal-append.queue_time_us":869201}
I20260808 01:51:39.546130  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2937) took 1494 ms. Trace:
I20260808 01:51:39.440155  9552 rpcz_store.cc:276] 0808 01:51:38.300070 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:38.392294 (+ 92224us) service_pool.cc:224] Handling call
0808 01:51:38.392793 (+   499us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:38.393354 (+   561us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:38.393395 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:38.394055 (+   660us) log.cc:844] Serialized 26 byte log entry
0808 01:51:39.012882 (+618827us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:39.013056 (+   174us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:39.013143 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:39.438447 (+425304us) raft_consensus.cc:1685] finished
0808 01:51:39.438536 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:39.439031 (+   495us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":52039,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":23688,"thread_start_us":618243,"threads_started":1,"wal-append.queue_time_us":844957}
I20260808 01:51:39.283099 14656 raft_consensus.cc:697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:39.559324 14591 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:39.561307 14853 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:39.561930 14591 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:39.562559 14853 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:39.564972  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:51:39.565157 10368 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260808 01:51:39.246248 14542 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:39.576424  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:39.576785  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 592) took 1753 ms. Trace:
I20260808 01:51:39.576697 14708 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:39.552644 14695 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:39.567638 14647 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:39.566829 14750 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:39.572676 14805 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:39.575130  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:39.571429  9634 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
W20260808 01:51:39.573436 14543 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.023s	sys 0.013s
I20260808 01:51:39.451913  9637 rpcz_store.cc:276] 0808 01:51:37.998731 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:38.018489 (+ 19758us) service_pool.cc:224] Handling call
0808 01:51:38.021523 (+  3034us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:38.088999 (+ 67476us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:38.089036 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:38.089618 (+   582us) log.cc:844] Serialized 52 byte log entry
0808 01:51:38.587472 (+497854us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:38.744771 (+157299us) log.cc:844] Serialized 14 byte log entry
0808 01:51:38.780448 (+ 35677us) log.cc:844] Serialized 14 byte log entry
0808 01:51:38.780734 (+   286us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:38.780848 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:38.960050 (+179202us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b1800366ac0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:51:39.450278 (+490228us) raft_consensus.cc:1685] finished
0808 01:51:39.450343 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:39.450868 (+   525us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":149266,"prepare.queue_time_us":202686,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":107,"raft.queue_time_us":166490,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":24362,"spinlock_wait_cycles":84412416,"thread_start_us":321608,"threads_started":1,"wal-append.queue_time_us":818796}
I20260808 01:51:39.565425 14645 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:39.616091  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 169414us
I20260808 01:51:39.628832 14428 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:39.629201 14695 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.630818 14707 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:39.313205 14729 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.632226 14707 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.634390 14817 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:51:39.634756 14763 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.621s	user 0.019s	sys 0.029s
I20260808 01:51:39.634863 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:39.635565 14763 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:39.636771 14750 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:39.638854  9550 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.001s	sys 0.028s
W20260808 01:51:39.639632  9550 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.002s	sys 0.028s
I20260808 01:51:39.640050 14851 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:39.630978  9638 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:39.578226 14708 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.639482 14676 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:39.638773 14849 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:39.637657 14763 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.595777 14706 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:39.577523  9629 rpcz_store.cc:276] 0808 01:51:37.823712 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:37.823877 (+   165us) service_pool.cc:224] Handling call
0808 01:51:37.824158 (+   281us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:37.937524 (+113366us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:37.937566 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:37.946740 (+  9174us) log.cc:844] Serialized 26 byte log entry
0808 01:51:38.365841 (+419101us) spinlock_profiling.cc:243] Waited 1.86 ms on lock 0x7b1800400d80. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:51:39.036478 (+670637us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:51:39.036551 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:39.036663 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:39.576084 (+539421us) raft_consensus.cc:1685] finished
0808 01:51:39.576142 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:39.576569 (+   427us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29369,"raft.queue_time_us":137625,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":2479,"spinlock_wait_cycles":4275072,"thread_start_us":739292,"threads_started":2,"wal-append.queue_time_us":1324895}
W20260808 01:51:39.644387 14543 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.812s	user 0.024s	sys 0.013s
I20260808 01:51:39.664974 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:39.665380 14656 consensus_queue.cc:237] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.666136 10362 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
W20260808 01:51:39.666457 14864 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260808 01:51:39.304869 14681 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:39.666846 14859 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:39.621196 14866 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:39.621351 14647 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:39.666941 14542 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:39.686976  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 534203us
I20260808 01:51:39.665390 14706 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.698489 14859 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:39.700546 14886 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:39.701265 14706 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:39.701478 14886 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:39.687973 14867 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260808 01:51:39.648653 14851 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:39.678274 14876 raft_consensus.cc:3055] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:51:39.669744 14806 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:39.702610 14859 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:39.716804 14876 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:39.720396 14542 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.684237 14729 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:39.720259 14602 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.018s
I20260808 01:51:39.689651 14841 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:39.689206 14645 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:39.682060 14805 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.687774 14853 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:39.685279 14695 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:39.734759 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:39.687196 14781 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:39.727747 10361 raft_consensus.cc:3055] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Stepping down as leader of term 12
W20260808 01:51:39.770578  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet e3814b146ccd472f9304a0be09ad3738: Illegal state: Replica 259bcb3f0d254e8b9effac3277e654e0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 12 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:39.770133 14876 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.771486 14681 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:39.722934 14877 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:39.774160 14848 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:39.774195 14886 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:39.744443 14817 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.737672 14859 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:39.784221 14862 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.073s
I20260808 01:51:39.732095  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:39.766921  9599 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:39.786152 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:39.788342 14542 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:39.793732 10361 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:39.736038 14647 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:39.843376 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:39.805454 14876 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:39.846982 10362 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:39.712931 14676 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:39.847033 14871 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.196s	user 0.000s	sys 0.001s
I20260808 01:51:39.848310  9551 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:39.849788 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:39.851774 14588 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:39.853986  9523 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:39.781839 14847 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:39.819794 14681 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.036s
I20260808 01:51:39.731492  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:39.806998 14750 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:39.807586 14645 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:39.808813 14805 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:39.813062 14880 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260808 01:51:39.732784 14851 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:39.806449 14857 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:39.816895 14877 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:39.817924 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:39.818295  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2940) took 1140 ms. Trace:
I20260808 01:51:39.819077 14841 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:39.840755 14708 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:39.842978 14695 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:39.856068 14647 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:51:39.838670 14729 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.018s
I20260808 01:51:39.859728  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:39.805738 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:39.796159 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 129163us
I20260808 01:51:39.850337 14676 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:39.875792 14681 consensus_queue.cc:1243] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 13.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260808 01:51:39.876441 14781 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:39.877605  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 183942us
I20260808 01:51:39.877889 14885 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:39.736644 14806 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:39.887850  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:39.817950 14602 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:39.878511 14535 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.866s	user 0.015s	sys 0.008s
I20260808 01:51:39.901845 14866 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:39.902906 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:39.906152 14845 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:39.906888 14843 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:39.907594 14841 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:39.863560 14707 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:39.910959  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:39.925642  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 118788us
I20260808 01:51:39.922053 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:39.932066  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 72637.5us
I20260808 01:51:39.927198  9480 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260808 01:51:39.904160  9559 rpcz_store.cc:276] 0808 01:51:38.677756 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:38.677889 (+   133us) service_pool.cc:224] Handling call
0808 01:51:38.770290 (+ 92401us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:38.773050 (+  2760us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:51:39.562083 (+789033us) log.cc:844] Serialized 14 byte log entry
0808 01:51:39.755614 (+193531us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:39.790149 (+ 34535us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:39.817919 (+ 27770us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:39.818058 (+   139us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:39.818237 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2226,"prepare.queue_time_us":473279,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":102648,"raft.run_cpu_time_us":383,"raft.run_wall_time_us":11247,"thread_start_us":933348,"threads_started":3}
I20260808 01:51:39.916821 14647 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:39.943064  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:39.878981 14886 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.052s
I20260808 01:51:39.970068 14632 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.981600 10334 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:39.981832 14632 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:39.548290  9555 rpcz_store.cc:276] 0808 01:51:38.043332 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:38.370423 (+327091us) service_pool.cc:224] Handling call
0808 01:51:38.370886 (+   463us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:38.432779 (+ 61893us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:51:38.471090 (+ 38311us) log.cc:844] Serialized 14 byte log entry
0808 01:51:38.823750 (+352660us) spinlock_profiling.cc:243] Waited 2.01 ms on lock 0x7b18004a0500. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d4453242a 00007f5d4499de0a
0808 01:51:39.438204 (+614454us) log.cc:844] Serialized 14 byte log entry
0808 01:51:39.438669 (+   465us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:39.463433 (+ 24764us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:39.515686 (+ 52253us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:39.524492 (+  8806us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:39.537914 (+ 13422us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":104083,"prepare.run_cpu_time_us":174,"prepare.run_wall_time_us":5465,"spinlock_wait_cycles":4631424,"thread_start_us":612909,"threads_started":1,"wal-append.queue_time_us":896962}
I20260808 01:51:39.956331  9478 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 5. Preceding OpId from leader: term: 14 index: 6. (index mismatch)
I20260808 01:51:39.957253 14822 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:39.951574 14535 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:40.008905 14822 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:40.009547 14822 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:40.010224 14535 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:40.010869  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:39.960085 14806 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:39.960397  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:39.960763 14857 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:39.961683  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:39.962113 14885 raft_consensus.cc:697] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:40.035743  9638 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:40.036540 14885 consensus_queue.cc:237] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:40.037621  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:39.963613  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:40.040181  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:39.967496  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.043732  9476 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:40.045084  9638 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:40.045509  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.046270  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:40.046919  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:40.046286 14857 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:40.048748 14878 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.321s	user 0.003s	sys 0.000s
W20260808 01:51:40.052531  9635 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2418) took 1623 ms (client timeout 2001 ms). Trace:
W20260808 01:51:40.053110  9635 rpcz_store.cc:269] 0808 01:51:38.429292 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:38.449273 (+ 19981us) service_pool.cc:224] Handling call
0808 01:51:40.052064 (+1602791us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b54000db1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:40.052296 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":547524992}
I20260808 01:51:40.037721 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:39.968562 14781 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:40.063014  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:40.064157  9635 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:40.065383  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:40.047923  9632 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:40.068183 14806 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:40.069278  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.069073 14806 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:40.072817 14715 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.089s	user 0.017s	sys 0.004s
I20260808 01:51:40.073769 14806 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:39.971236 14707 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:39.971741 14695 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:39.973428 14602 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:39.974236 14681 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:39.974535 14647 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:39.975102  9599 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:39.932536 14859 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:39.976131 14875 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:39.983024 14895 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:39.983796 10334 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:39.984575  9554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 669) took 2101 ms (client timeout 1560 ms). Trace:
I20260808 01:51:39.985853 14817 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:39.987469  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:39.953924  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 222912us
I20260808 01:51:39.994894  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2432) took 2677 ms. Trace:
W20260808 01:51:39.995177 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 177385us
I20260808 01:51:39.947741 10363 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:40.083885  9630 rpcz_store.cc:276] 0808 01:51:37.317721 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:37.320232 (+  2511us) service_pool.cc:224] Handling call
0808 01:51:37.320559 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:39.440752 (+2120193us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:39.994427 (+553675us) spinlock_profiling.cc:243] Waited 2.12 s on lock 0x7b540034fb54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:39.994768 (+   341us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4871431552}
W20260808 01:51:39.999822  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 48547.8us
I20260808 01:51:40.007683  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.005705  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2426) took 1123 ms. Trace:
I20260808 01:51:40.014627  9474 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 6.
I20260808 01:51:40.086974 10332 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:40.016319  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2431) took 2701 ms. Trace:
I20260808 01:51:40.088088  9633 rpcz_store.cc:276] 0808 01:51:37.314805 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:37.319101 (+  4296us) service_pool.cc:224] Handling call
0808 01:51:37.413128 (+ 94027us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:37.985069 (+571941us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:40.015706 (+2030637us) spinlock_profiling.cc:243] Waited 569 ms on lock 0x7b540003bbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:40.016131 (+   425us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1308099328}
I20260808 01:51:40.089172 14868 consensus_queue.cc:1059] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:51:40.089680 14821 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:40.090127 14821 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:40.090485 14821 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:40.017856 14877 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:40.018659  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.093050  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.021517  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:40.030474  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:40.031661 14892 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:40.096889 14892 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:40.015920 14852 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:40.102469 14821 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:40.105975 14821 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:40.107728  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:40.109915 14892 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:40.047154  9638 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:39.968598 14849 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:40.086388  9637 rpcz_store.cc:276] 0808 01:51:38.876686 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:39.655024 (+778338us) service_pool.cc:224] Handling call
0808 01:51:39.744363 (+ 89339us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:39.874850 (+130487us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:39.874892 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:39.875638 (+   746us) log.cc:844] Serialized 26 byte log entry
0808 01:51:39.937679 (+ 62041us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:51:39.937730 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:39.937820 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:39.999587 (+ 61767us) raft_consensus.cc:1685] finished
0808 01:51:39.999612 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:39.999794 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8582,"raft.queue_time_us":22477,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":149,"thread_start_us":61505,"threads_started":1,"wal-append.queue_time_us":99314}
I20260808 01:51:40.112718  9481 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 14: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:40.094527  9480 raft_consensus.cc:2410] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 13 because replica has last-logged OpId of term: 12 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260808 01:51:40.115592  9637 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260808 01:51:40.116317  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.117697  9480 raft_consensus.cc:2393] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:40.119450  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:39.969520 14591 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:40.128198 14591 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:40.131322  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:40.122330 14849 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:40.083493  9554 rpcz_store.cc:269] 0808 01:51:37.883092 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:37.888173 (+  5081us) service_pool.cc:224] Handling call
0808 01:51:39.984150 (+2095977us) spinlock_profiling.cc:243] Waited 792 ms on lock 0x7b540029efd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:39.984352 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1821372288}
I20260808 01:51:40.133172  9523 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:40.135435  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:40.079397 14681 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:40.144043  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:40.144871 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 161286us
I20260808 01:51:40.081430 14875 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:40.177857  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 92626.5us
I20260808 01:51:40.085549 14895 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:40.180202  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 233100us
W20260808 01:51:40.193120  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 43594.7us
W20260808 01:51:40.198181  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:40.200927  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.149191 14681 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:51:40.205415  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:40.231770  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:40.184742 14892 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:40.182646 14875 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:40.135165 14695 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:40.235375  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:40.242031  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.190614 14895 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:40.239353  9474 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:40.243670  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:40.244607 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 114767us
I20260808 01:51:40.241536  9638 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:51:40.181617 14707 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:40.253904  9472 raft_consensus.cc:2410] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
W20260808 01:51:40.255860 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 76974.2us
I20260808 01:51:40.237773 14892 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:40.271169 14892 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:40.250499  9478 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 7.
I20260808 01:51:40.257937 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:40.258764 14898 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260808 01:51:40.239398 14806 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:40.263053 14849 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:40.263998 14899 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:40.267693 14602 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:40.268695 14695 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:40.269815  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 77443.1us
I20260808 01:51:40.255048  9480 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:51:40.276298  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:40.277029  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.279670 14676 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:40.291424 14676 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:40.291781 14824 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:40.293943  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.294263 14824 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:40.294855  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.296411  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.296690 14715 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:40.297248  9472 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:40.298504 14852 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:40.298630 14632 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:51:40.299228 14631 consensus_queue.cc:1059] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:51:40.299754 14886 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:40.300356 14631 consensus_queue.cc:1243] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 14.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260808 01:51:40.300738 14886 consensus_queue.cc:1243] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 13.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260808 01:51:40.301071 14715 raft_consensus.cc:3055] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:40.301285 14876 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:51:40.301544 14715 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:40.301749 14876 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:40.302992 14876 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:40.299193 14852 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:40.306036 14852 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:40.307808 14852 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:40.311168 14824 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:40.302906 14715 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:40.316001 14715 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:40.317689 10369 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:40.319576 14708 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:51:40.320313 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:40.320261 14890 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:51:40.323366  9480 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:40.322620  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:40.324587  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.325502 14805 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:40.319608 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:40.326229 14805 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:40.327353 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:40.327831 10366 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
I20260808 01:51:40.327632 14805 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:40.328034 14806 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:40.318261  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:40.329284  9479 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:40.324645  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:40.328208 10360 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:40.304587 14876 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:40.325515  9480 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:40.322872 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:40.294261  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.277117 14849 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:40.353134 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:51:40.354918  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 70468.4us
I20260808 01:51:40.289219 14809 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:40.319687  9524 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:40.368737  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:40.330518 14806 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:40.373350  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48906.4us
W20260808 01:51:40.376165 14901 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.130s	user 0.001s	sys 0.001s
I20260808 01:51:40.383136  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:40.384212  9552 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:40.384950 14652 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:40.383143  9474 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
W20260808 01:51:40.387346  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:40.386889 14809 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:40.387567  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:40.389756  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.389448  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.397312  9480 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 15: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:40.361332 14849 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:40.306473 14822 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:40.396013  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:40.404639  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.407814  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:40.411067  9472 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 12: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:40.412014  9474 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:40.413450  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.417912  9629 raft_consensus.cc:2368] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12.
I20260808 01:51:40.412168 14750 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:40.419914  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:40.421204 14908 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:40.422117 14908 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:40.423485 14908 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 10.12, Last appended by leader: 12, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:40.426895 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 145812us
I20260808 01:51:40.427105  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:40.427884  9551 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:40.429373  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:40.417039 14806 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election won for term 13
W20260808 01:51:40.431138  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 111634us
I20260808 01:51:40.431605 14806 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:40.432982  9558 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260808 01:51:40.433069 14806 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:40.437932 14284 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:51:40.438552 14859 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:40.438827  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:40.439268 14284 raft_consensus.cc:3055] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:40.439800 14284 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:40.440526  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:40.441679  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:40.442912 14781 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:40.443799 14781 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:40.440384  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:40.445794  9556 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:40.446940  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:40.447724  9556 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:40.443238  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:40.434448  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:40.457033 14833 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:40.450702  9478 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:40.433856 14818 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:40.453179  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:40.444737 14892 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:40.455121  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:40.458452  9632 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:40.458318 14818 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:40.456094  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:40.453991  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:40.457808 14833 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:40.464777  9558 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:40.457057 14695 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:40.466226 14695 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:40.466634  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:40.467708  9551 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:40.469378  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:40.471292  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:40.469570 14822 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:40.466704 14695 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:40.475499  9551 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:40.478191  9480 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
W20260808 01:51:40.489884  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 115592us
I20260808 01:51:40.497782  9556 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
W20260808 01:51:40.502537  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 84108.1us
W20260808 01:51:40.504230  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 71189us
I20260808 01:51:40.360666 14899 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:40.446488 14781 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:40.513584  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:40.530496  9479 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:40.507723  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:40.541393  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 54717.9us
I20260808 01:51:40.511637  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:40.519573 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:40.549021 10367 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:40.555619  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:40.561635  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50134.7us
I20260808 01:51:40.563755  9551 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 13: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:40.562801  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:40.564283  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:40.573184  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.573520  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.570541  9599 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:40.579350  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:40.578853  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:40.567256 14906 log.cc:927] Time spent T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.092s	user 0.002s	sys 0.000s
I20260808 01:51:40.579995  9479 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:40.514266 14715 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.058s
I20260808 01:51:40.586009  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:40.561650 14892 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:40.589226  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.565342 14805 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:40.571408 14750 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:51:40.536475 14824 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:40.602196  9632 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:51:40.607527  9524 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:40.624675  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:40.627703  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:40.596457 14809 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:40.634117  9634 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:40.603116 14750 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:40.643677  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:40.702178  9631 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:51:40.663084  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:40.703567  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:40.599668 14805 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:40.720564 10332 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:40.730419  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:40.732676  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 89133.4us
I20260808 01:51:40.733848  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:40.734462  9551 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260808 01:51:40.743194  9480 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:40.594954 14781 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.066s
I20260808 01:51:40.747467  9472 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:51:40.699044 14892 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:40.752617  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:40.724148 14750 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:40.638484 14818 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:40.721824 14805 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:40.774305  9599 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:40.778959  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.746269 14866 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:40.787216  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:40.653914 14602 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.129s
I20260808 01:51:40.797055  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:40.802737  9630 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:40.808081  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.763568 14892 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:40.836252 14659 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:40.837118 14659 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:40.837647 14659 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:40.801415 14817 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:40.839743 14817 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:40.840306 14817 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:40.843015 14844 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:40.843622  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:40.773778 14763 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:40.846891 14763 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:40.793970 14813 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:40.847409 14763 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:40.853641 14915 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260808 01:51:40.855134 14918 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:51:40.855739 14918 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:40.855829 14899 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:40.856225 14918 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:40.845175  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:40.862646 10365 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
W20260808 01:51:40.863518 14912 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.272s	user 0.001s	sys 0.000s
I20260808 01:51:40.856918 14892 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:40.867674  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:40.869560  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:40.875319 10367 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:40.887512 14899 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:40.888564 14899 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:40.891053 14833 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:40.892181 14899 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:40.895668  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2703) took 1227 ms. Trace:
W20260808 01:51:40.897397  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 341503us
I20260808 01:51:40.902081  9634 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:40.787621 14809 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:40.789615 14866 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:40.896288  9481 rpcz_store.cc:276] 0808 01:51:39.818802 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:40.068778 (+249976us) service_pool.cc:224] Handling call
0808 01:51:40.895375 (+826597us) spinlock_profiling.cc:243] Waited 42.5 ms on lock 0x7b54001013d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:40.895579 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":97709440}
I20260808 01:51:40.926226  9551 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260808 01:51:40.843683 14844 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:40.934538 14844 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:40.936007 14899 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:51:40.924559  9637 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:51:40.931810  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.933676  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:40.942126  9472 raft_consensus.cc:2410] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 13 because replica has last-logged OpId of term: 12 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
W20260808 01:51:40.948076  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:40.909821 14715 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20260808 01:51:40.915961 14866 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:40.908656 14809 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:40.940896  9633 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:40.990635  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:40.921279 14852 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.613s	user 0.015s	sys 0.003s
I20260808 01:51:40.989887  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:40.999701  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:41.009338  9632 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260808 01:51:41.009886  9632 pending_rounds.cc:85] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 7
I20260808 01:51:41.010110  9632 pending_rounds.cc:107] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260808 01:51:41.010375  9632 raft_consensus.cc:2889] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:41.015013  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:41.017081  9472 raft_consensus.cc:2410] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260808 01:51:41.017755  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:41.018086  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:41.019196  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:40.996282 14809 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:41.012722  9632 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260808 01:51:41.020437  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:41.020066  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:41.022605  9472 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:41.023844  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:41.025202  9472 raft_consensus.cc:2410] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
W20260808 01:51:41.028762  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:41.029407  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:51:41.032356 14821 raft_consensus.cc:3055] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:41.033010 14821 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:41.041195  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:41.044282  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:41.043876  9472 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:41.045646  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2557) took 1142 ms. Trace:
I20260808 01:51:41.002837  9475 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:41.054570  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2441) took 1096 ms. Trace:
I20260808 01:51:41.055292  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:41.106132  9554 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:41.103897  9629 rpcz_store.cc:276] 0808 01:51:39.958229 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:39.959937 (+  1708us) service_pool.cc:224] Handling call
0808 01:51:41.054184 (+1094247us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b540032c654. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:41.054461 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":881153280}
I20260808 01:51:41.109397  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:41.058473  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2946) took 2367 ms. Trace:
I20260808 01:51:41.116885  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:41.124986  9479 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260808 01:51:41.106012  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:41.041435 10361 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:41.127012  9474 raft_consensus.cc:2410] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
W20260808 01:51:41.092597  9560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 2939) took 2415 ms (client timeout 1683 ms). Trace:
I20260808 01:51:41.072666  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2575) took 1008 ms. Trace:
I20260808 01:51:41.162294  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2565) took 1138 ms. Trace:
I20260808 01:51:41.162716  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2580) took 1082 ms. Trace:
I20260808 01:51:41.151520  9475 rpcz_store.cc:276] 0808 01:51:40.064130 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:40.202499 (+138369us) service_pool.cc:224] Handling call
0808 01:51:41.072386 (+869887us) spinlock_profiling.cc:243] Waited 655 ms on lock 0x7b54002dc3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:41.072578 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1507229568}
I20260808 01:51:41.046173  9477 rpcz_store.cc:276] 0808 01:51:39.905203 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:39.930015 (+ 24812us) service_pool.cc:224] Handling call
0808 01:51:41.043864 (+1113849us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b5800005498. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:51:41.045521 (+  1657us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24695680}
I20260808 01:51:41.167435  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:41.091372 14707 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:41.162159  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2570) took 1173 ms. Trace:
I20260808 01:51:41.164409  9474 rpcz_store.cc:276] 0808 01:51:40.080132 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.108975 (+1028843us) service_pool.cc:224] Handling call
0808 01:51:41.162607 (+ 53632us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:51:41.150398  9560 rpcz_store.cc:269] 0808 01:51:38.677313 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:38.679450 (+  2137us) service_pool.cc:224] Handling call
0808 01:51:41.092125 (+2412675us) spinlock_profiling.cc:243] Waited 7.38 ms on lock 0x7b54003147d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:41.092313 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16964992}
I20260808 01:51:41.162987  9479 rpcz_store.cc:276] 0808 01:51:39.987176 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.123946 (+1136770us) service_pool.cc:224] Handling call
0808 01:51:41.124401 (+   455us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:41.125441 (+  1040us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:41.125669 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:41.188090 10361 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:51:41.000794 14875 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.729s	user 0.014s	sys 0.000s
I20260808 01:51:41.197542 14875 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:41.174471  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:41.184286  9473 rpcz_store.cc:276] 0808 01:51:39.989154 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.105490 (+1116336us) service_pool.cc:224] Handling call
0808 01:51:41.162051 (+ 56561us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:51:41.082461 14922 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.209s	user 0.001s	sys 0.000s
I20260808 01:51:41.195923  9556 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260808 01:51:41.121770 14284 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:41.223055  9552 consensus_meta.cc:287] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.838s	user 0.018s	sys 0.004s
I20260808 01:51:41.223580  9552 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:41.191959  9635 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
W20260808 01:51:41.085533 14879 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.348s	user 0.012s	sys 0.001s
I20260808 01:51:41.196420  9560 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260808 01:51:41.213475  9599 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:41.213338  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:41.209954  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:41.226276  9551 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:41.226749  9472 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:41.110585  9550 rpcz_store.cc:276] 0808 01:51:38.690481 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:38.739350 (+ 48869us) service_pool.cc:224] Handling call
0808 01:51:38.739864 (+   514us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:38.947755 (+207891us) raft_consensus.cc:1534] Early marking committed up to index 10
0808 01:51:39.640455 (+692700us) log.cc:844] Serialized 14 byte log entry
0808 01:51:39.793150 (+152695us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:39.976770 (+183620us) log.cc:844] Serialized 26 byte log entry
0808 01:51:39.977002 (+   232us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:51:39.977463 (+   461us) log.cc:844] Serialized 14 byte log entry
0808 01:51:39.977753 (+   290us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:41.057742 (+1079989us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b54000fee54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:41.057828 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:41.058036 (+   208us) raft_consensus.cc:1685] finished
0808 01:51:41.058058 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:41.058316 (+   258us) inbound_call.cc:177] Queueing success response
0808 01:51:41.067243 (+  8927us) spinlock_profiling.cc:243] Waited 2.71 ms on lock 0x7b48000c03d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
Metrics: {"prepare.queue_time_us":300336,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":108,"raft.queue_time_us":52659,"raft.run_cpu_time_us":7452,"raft.run_wall_time_us":1245529,"spinlock_wait_cycles":483452928,"thread_start_us":814746,"threads_started":2,"wal-append.queue_time_us":214408,"wal-append.run_cpu_time_us":5188,"wal-append.run_wall_time_us":1182375}
I20260808 01:51:41.255942  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:41.257913  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:41.259311  9481 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 13: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:51:41.233997  9444 connection.cc:256] RPC call timeout handler was delayed by -2.6013087619794533s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1688s for a response
W20260808 01:51:41.260456  9444 connection.cc:256] RPC call timeout handler was delayed by -2.5959087619849015s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1694s for a response
W20260808 01:51:41.260798  9444 connection.cc:256] RPC call timeout handler was delayed by -2.2926087619853206s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20310000000000003s for a response
W20260808 01:51:41.261157  9444 connection.cc:256] RPC call timeout handler was delayed by -2.2359087619988713s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2094s for a response
I20260808 01:51:41.263100  9556 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260808 01:51:41.263291 14830 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:41.264015 14830 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:41.264879  9556 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260808 01:51:41.267288  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:41.260922 14907 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.412s	user 0.001s	sys 0.001s
I20260808 01:51:41.059286 14821 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:41.249505 14824 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:41.274622  9560 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:51:41.277199  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2712) took 1046 ms. Trace:
I20260808 01:51:41.268811 14591 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:51:41.280299  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2710) took 1201 ms. Trace:
I20260808 01:51:41.283740 14632 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:41.288415  9444 connection.cc:256] RPC call timeout handler was delayed by -2.1567087619914673s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2182s for a response
W20260808 01:51:41.298394  9444 connection.cc:256] RPC call timeout handler was delayed by -1.9588783239887562s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1501s for a response
W20260808 01:51:41.298823  9444 connection.cc:256] RPC call timeout handler was delayed by -1.9318783239868935s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1531s for a response
W20260808 01:51:41.299125  9444 connection.cc:256] RPC call timeout handler was delayed by -1.7095783239929006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1778s for a response
W20260808 01:51:41.299396  9444 connection.cc:256] RPC call timeout handler was delayed by -1.60247832396999s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1897s for a response
W20260808 01:51:41.299779  9444 connection.cc:256] RPC call timeout handler was delayed by -1.5835783239745069s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1918s for a response
W20260808 01:51:41.300063  9444 connection.cc:256] RPC call timeout handler was delayed by -1.5754783239681274s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1927s for a response
W20260808 01:51:41.300334  9444 connection.cc:256] RPC call timeout handler was delayed by -1.3297783239686396s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22000000000000003s for a response
W20260808 01:51:41.300727  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1577087619807571s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3292s for a response
I20260808 01:51:41.291127 14729 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:41.272980 14813 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:51:41.295475  9475 rpcz_store.cc:276] 0808 01:51:40.078638 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.208432 (+1129794us) service_pool.cc:224] Handling call
0808 01:51:41.252894 (+ 44462us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:41.279929 (+ 27035us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:41.279961 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:41.280011 (+    50us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:51:41.280041 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:41.280111 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:41.280233 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:41.282634  9480 rpcz_store.cc:276] 0808 01:51:40.231093 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.250075 (+1018982us) service_pool.cc:224] Handling call
0808 01:51:41.268828 (+ 18753us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:41.276795 (+  7967us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:41.276825 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:41.276881 (+    56us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:51:41.276918 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:41.276991 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:41.277128 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:41.293937  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:41.307333  9560 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:41.280216  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:41.304672  9554 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:51:41.308733  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 682) took 1104 ms. Trace:
I20260808 01:51:41.284250 14830 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:41.276324 14676 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:41.309278  9554 rpcz_store.cc:276] 0808 01:51:40.204612 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.054852 (+850240us) service_pool.cc:224] Handling call
0808 01:51:41.308641 (+253789us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:41.313642  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:41.321169 14534 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:51:41.322127 14591 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:41.323570 14591 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:41.327649  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:41.328588  9475 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 14: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:51:41.328919 14849 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.868s	user 0.013s	sys 0.000s
I20260808 01:51:41.329336 14849 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:41.330183 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:41.312968 14824 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
I20260808 01:51:41.331137 10364 raft_consensus.cc:2368] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12.
I20260808 01:51:41.331792 14875 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:41.332255  9481 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260808 01:51:41.332381 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:41.332894  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2719) took 1067 ms. Trace:
I20260808 01:51:41.333329 10367 raft_consensus.cc:2393] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 16: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:41.333629  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:41.334314  9554 raft_consensus.cc:2376] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12. Re-sending same reply.
I20260808 01:51:41.335807 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:41.333288  9481 rpcz_store.cc:276] 0808 01:51:40.265526 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.313053 (+1047527us) service_pool.cc:224] Handling call
0808 01:51:41.331909 (+ 18856us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:41.332640 (+   731us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:41.332835 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:41.340504 10332 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:41.281396 14833 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.060s
I20260808 01:51:41.342825  9599 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:41.341463  9524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.095s	sys 0.126s
W20260808 01:51:41.346756  9524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.096s	sys 0.126s
W20260808 01:51:41.347160  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 739670us
I20260808 01:51:41.347802  9524 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:41.347941  9560 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:51:41.352739  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:41.363452 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:41.319546 14821 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:41.342944 14284 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:41.357064 14868 raft_consensus.cc:2764] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260808 01:51:41.380540  9524 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:41.381165 14824 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:41.390834 14813 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:51:41.391855  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:41.404524 10334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.012s	sys 0.016s
I20260808 01:51:41.307762 14822 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.053s
I20260808 01:51:41.395627  9524 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:41.351311 14534 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:41.306784 14632 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:41.357561 14676 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260808 01:51:41.396970 14681 consensus_meta.cc:287] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.153s	user 0.026s	sys 0.022s
I20260808 01:51:41.364111 14911 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:51:41.391678 14876 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:41.412853 10361 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260808 01:51:41.338156 14763 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:41.416843  9524 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:41.404937 10334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.012s	sys 0.016s
W20260808 01:51:41.405174 14806 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.017s	sys 0.012s
W20260808 01:51:41.417470 14853 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.943s	user 0.021s	sys 0.010s
W20260808 01:51:41.422994  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 414388us
I20260808 01:51:41.425652 10362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 691) took 1383 ms. Trace:
I20260808 01:51:41.413854 14534 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:41.437081 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 682) took 1804 ms. Trace:
W20260808 01:51:41.427120 14884 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.010s	sys 0.020s
W20260808 01:51:41.444542  9558 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.004s	sys 0.030s
I20260808 01:51:41.448940  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:41.346323 14729 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:41.406349 14707 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:41.479075  9477 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 15: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:51:41.446489 14805 consensus_meta.cc:287] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.632s	user 0.011s	sys 0.000s
W20260808 01:51:41.476469  9558 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.004s	sys 0.030s
I20260808 01:51:41.394536 14830 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.043s
I20260808 01:51:41.458352 10361 rpcz_store.cc:276] 0808 01:51:39.632850 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:39.658911 (+ 26061us) service_pool.cc:224] Handling call
0808 01:51:39.727228 (+ 68317us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:41.040775 (+1313547us) spinlock_profiling.cc:243] Waited 84.5 ms on lock 0x7b5400255420. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4491144f 00007f5d4498a340 00007f5d4498bbbe 00007f5d44995f29 00007f5d449967f6 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2
0808 01:51:41.436660 (+395885us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:41.436991 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":194300288}
W20260808 01:51:41.481768 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:41.417886 14676 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:51:41.447538 14806 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.018s	sys 0.012s
W20260808 01:51:41.487519 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:41.457036 14853 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.983s	user 0.022s	sys 0.010s
W20260808 01:51:41.489830  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 159855us
W20260808 01:51:41.490199  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 169659us
I20260808 01:51:41.492611 10334 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:41.457311 10362 rpcz_store.cc:276] 0808 01:51:40.035033 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:40.041414 (+  6381us) service_pool.cc:224] Handling call
0808 01:51:40.041994 (+   580us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:40.042721 (+   727us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:40.042763 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:40.043446 (+   683us) log.cc:844] Serialized 26 byte log entry
0808 01:51:40.179922 (+136476us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:40.180186 (+   264us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:40.180456 (+   270us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:41.418122 (+1237666us) raft_consensus.cc:1685] finished
0808 01:51:41.418156 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:41.418374 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1772,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":133,"thread_start_us":103744,"threads_started":1,"wal-append.queue_time_us":307630}
W20260808 01:51:41.459306  9638 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.999s	user 0.014s	sys 0.011s
I20260808 01:51:41.437860 14857 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:41.456732 14708 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:41.498126  9638 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
W20260808 01:51:41.478269 14895 consensus_meta.cc:287] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.213s	user 0.020s	sys 0.006s
I20260808 01:51:41.450721 14824 raft_consensus.cc:697] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:41.483194 14652 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.094s	user 0.016s	sys 0.010s
I20260808 01:51:41.477957 14911 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:41.482929 14707 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:41.507320  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2438) took 1055 ms. Trace:
W20260808 01:51:41.508663  9601 leader_election.cc:336] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 2.187s (SENT)
W20260808 01:51:41.509164 10331 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.011s	sys 0.018s
W20260808 01:51:41.536494 10331 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.011s	sys 0.019s
I20260808 01:51:41.509852 14682 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260808 01:51:41.513702 14695 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.008s	user 0.021s	sys 0.008s
I20260808 01:51:41.487048 14830 consensus_queue.cc:1243] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 11.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260808 01:51:41.515291 14904 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.265s	user 0.009s	sys 0.024s
I20260808 01:51:41.544525 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:41.516251 14913 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
W20260808 01:51:41.516762 14647 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.378s	user 0.015s	sys 0.002s
I20260808 01:51:41.486786 14849 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:41.518110  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:41.466969 14534 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:41.520658  9551 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:41.491729 14676 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:41.520443  9552 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:41.523017  9555 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260808 01:51:41.527159  9521 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:41.527943  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:41.433393 14681 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:41.475142 14884 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.010s	sys 0.020s
W20260808 01:51:41.535528  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 118816us
I20260808 01:51:41.490235 14805 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:41.524398 14817 consensus_meta.cc:287] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.674s	user 0.008s	sys 0.010s
I20260808 01:51:41.539975  9476 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:41.540123  9638 rpcz_store.cc:276] 0808 01:51:40.452215 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:40.455812 (+  3597us) service_pool.cc:224] Handling call
0808 01:51:41.507213 (+1051401us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:41.549968  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:41.551170  9478 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:41.569288  9477 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:41.572554  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:41.581259 14755 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:41.581739 14755 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:41.582131 14755 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:41.538424  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:41.542450 14682 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:41.583449  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 45182.9us
I20260808 01:51:41.535511 14707 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:41.528738 14543 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:41.575917  9523 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:41.586563  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:41.587548  9478 raft_consensus.cc:2368] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12.
I20260808 01:51:41.591599 10366 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
W20260808 01:51:41.591948  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 64897.8us
I20260808 01:51:41.528774 14857 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:41.529706 14895 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:41.597748 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:41.601220  9521 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:41.602568  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:41.583081 14682 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:41.614833  9600 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:41.608601  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:41.593427 14543 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:41.585076 14935 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:41.552099 14695 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:41.611347  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:41.596223 14885 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.344s	user 0.000s	sys 0.041s
W20260808 01:51:41.610426  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:41.682188  9477 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:41.588143 14632 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:41.624010  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 220947us
I20260808 01:51:41.591144 14833 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:41.662370 10360 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:41.585841 14817 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:41.619995 14890 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:41.572165 14647 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:41.685583  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 151501us
W20260808 01:51:41.692435  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 116625us
I20260808 01:51:41.622310 14591 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:41.661396 14729 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:41.675932  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 74845.7us
I20260808 01:51:41.701699  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:41.701961  9554 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12. Ops: []
W20260808 01:51:41.821600  9524 connection.cc:256] RPC call timeout handler was delayed by -1.034452424995834s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1759s for a response
W20260808 01:51:41.825134  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:41.662925 14849 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260808 01:51:41.685120 14885 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.441s	user 0.000s	sys 0.041s
I20260808 01:51:41.668043 14935 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:41.896775  9477 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:41.897842  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 802) took 1460 ms. Trace:
I20260808 01:51:41.898692  9477 rpcz_store.cc:276] 0808 01:51:40.437155 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.695095 (+1257940us) service_pool.cc:224] Handling call
0808 01:51:41.833551 (+138456us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:41.897527 (+ 63976us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:41.897778 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:51:41.736812 14669 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
I20260808 01:51:41.901677  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:41.907629  9631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2442) took 1364 ms (client timeout 1572 ms). Trace:
W20260808 01:51:41.799283 14927 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
W20260808 01:51:41.902853  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 54570.7us
W20260808 01:51:41.748095 14936 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260808 01:51:41.899983  9521 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:41.904737  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 801) took 1422 ms. Trace:
W20260808 01:51:41.848377 14844 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.913s	user 0.004s	sys 0.010s
W20260808 01:51:41.799654 14917 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.170s	user 0.000s	sys 0.003s
I20260808 01:51:41.798345 14763 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:41.940026 14844 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:41.919486  9481 rpcz_store.cc:276] 0808 01:51:40.436792 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.346856 (+910064us) service_pool.cc:224] Handling call
0808 01:51:41.357420 (+ 10564us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:41.403902 (+ 46482us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:41.403939 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:41.430434 (+ 26495us) log.cc:844] Serialized 52 byte log entry
0808 01:51:41.587012 (+156578us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:41.587053 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:41.587124 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:41.858308 (+271184us) raft_consensus.cc:1685] finished
0808 01:51:41.858346 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:41.858633 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":23827,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":10978,"thread_start_us":96632,"threads_started":1,"wal-append.queue_time_us":288668}
I20260808 01:51:41.943939  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:41.917688  9637 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
W20260808 01:51:41.908236  9631 rpcz_store.cc:269] 0808 01:51:40.545485 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:40.545734 (+   249us) service_pool.cc:224] Handling call
0808 01:51:41.907367 (+1361633us) spinlock_profiling.cc:243] Waited 86.6 ms on lock 0x7b540008cd54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:41.907558 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":199174528}
W20260808 01:51:41.748963 14659 consensus_meta.cc:287] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.911s	user 0.009s	sys 0.005s
I20260808 01:51:41.752540 14695 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:41.947922  9481 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 12: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:41.948478 14647 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:41.946568 14659 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:41.750990 14805 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:41.899049  9601 leader_election.cc:336] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.524s (SENT)
I20260808 01:51:41.929383  9559 raft_consensus.cc:2368] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 15. Current term is 16.
W20260808 01:51:41.929730 14938 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.231s	user 0.000s	sys 0.001s
I20260808 01:51:41.952860  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2971) took 1476 ms. Trace:
I20260808 01:51:41.701789 14911 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:41.931748  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:41.748297 14830 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:41.748553 14833 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:41.936231  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2718) took 1671 ms. Trace:
I20260808 01:51:41.911749  9478 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:41.937450  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:41.937003 14729 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:51:41.939136  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:41.961764  9521 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:41.915884  9523 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:41.950368  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 803) took 1512 ms. Trace:
W20260808 01:51:41.950852 14809 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.929s	user 0.020s	sys 0.006s
I20260808 01:51:41.902204 14935 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:41.964892  9521 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:41.953315  9550 rpcz_store.cc:276] 0808 01:51:40.476018 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.268148 (+792130us) service_pool.cc:224] Handling call
0808 01:51:41.285435 (+ 17287us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:41.285753 (+   318us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:41.285791 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:41.302764 (+ 16973us) log.cc:844] Serialized 26 byte log entry
0808 01:51:41.530498 (+227734us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:41.530547 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:41.530618 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:41.952595 (+421977us) raft_consensus.cc:1685] finished
0808 01:51:41.952619 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:41.952793 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":227389,"threads_started":1,"wal-append.queue_time_us":603298}
I20260808 01:51:41.938582 14708 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:41.953863  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 54889.1us
I20260808 01:51:41.957060  9478 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:41.962741  9481 rpcz_store.cc:276] 0808 01:51:40.437536 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.949439 (+1511903us) service_pool.cc:224] Handling call
0808 01:51:41.949808 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:41.950022 (+   214us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:41.950046 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:41.950089 (+    43us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:41.950117 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:41.950167 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:41.950285 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:41.974807  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 805) took 1536 ms. Trace:
W20260808 01:51:41.840431 14892 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.976s	user 0.020s	sys 0.000s
I20260808 01:51:41.711576 14824 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:41.954663  9480 rpcz_store.cc:276] 0808 01:51:40.265139 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.308111 (+1042972us) service_pool.cc:224] Handling call
0808 01:51:41.339177 (+ 31066us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:41.358370 (+ 19193us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:41.358434 (+    64us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:41.457351 (+ 98917us) log.cc:844] Serialized 52 byte log entry
0808 01:51:41.898470 (+441119us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:41.921496 (+ 23026us) log.cc:844] Serialized 14 byte log entry
0808 01:51:41.923057 (+  1561us) log.cc:844] Serialized 14 byte log entry
0808 01:51:41.923390 (+   333us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:41.923498 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:41.935809 (+ 12311us) raft_consensus.cc:1685] finished
0808 01:51:41.935836 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:41.936110 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":16540,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":21358,"spinlock_wait_cycles":4731136,"thread_start_us":307310,"threads_started":2,"wal-append.queue_time_us":435325}
I20260808 01:51:41.833166 14543 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:41.963384  9477 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:41.834733 14849 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:41.937918 14950 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260808 01:51:41.971520  9472 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.722s	user 0.011s	sys 0.002s
W20260808 01:51:41.982534 14750 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.160s	user 0.008s	sys 0.004s
I20260808 01:51:41.964089 14681 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:41.975425  9478 rpcz_store.cc:276] 0808 01:51:40.438331 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.956454 (+1518123us) service_pool.cc:224] Handling call
0808 01:51:41.956859 (+   405us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:41.974347 (+ 17488us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:41.974706 (+   359us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:41.969996 14809 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:41.984751  9472 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:42.009713  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2970) took 1555 ms. Trace:
I20260808 01:51:42.010301  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2714) took 1747 ms. Trace:
W20260808 01:51:41.986052 14821 consensus_meta.cc:287] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.591s	user 0.012s	sys 0.000s
I20260808 01:51:41.985888 14950 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:41.983633 14892 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:41.988334  9477 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:41.991744 14849 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:41.973610 14647 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260808 01:51:41.999368 14918 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.143s	user 0.023s	sys 0.000s
I20260808 01:51:41.965963 14830 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.029999  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 808) took 1441 ms. Trace:
I20260808 01:51:42.030576  9477 rpcz_store.cc:276] 0808 01:51:40.580850 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.987527 (+1406677us) service_pool.cc:224] Handling call
0808 01:51:41.988078 (+   551us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:42.022383 (+ 34305us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:42.022641 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:41.971130 14695 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:41.975145 14954 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:41.970861 14805 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:42.003288 14534 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:41.986404 14833 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:42.014910 14924 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260808 01:51:41.984643  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 47458.7us
I20260808 01:51:42.047343  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:42.011103  9472 rpcz_store.cc:276] 0808 01:51:40.263378 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.046061 (+782683us) service_pool.cc:224] Handling call
0808 01:51:42.010211 (+964150us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:42.049464  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:42.037576  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 167255us
I20260808 01:51:42.051695  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.052380  9477 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
W20260808 01:51:42.053313  9601 leader_election.cc:336] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.535s (SENT)
W20260808 01:51:42.034193 14866 consensus_meta.cc:287] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.961s	user 0.005s	sys 0.018s
I20260808 01:51:42.057221 14892 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:42.010243  9556 rpcz_store.cc:276] 0808 01:51:40.454656 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.265815 (+811159us) service_pool.cc:224] Handling call
0808 01:51:41.266168 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:41.266550 (+   382us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:51:41.608726 (+342176us) log.cc:844] Serialized 14 byte log entry
0808 01:51:42.009333 (+400607us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:42.009391 (+    58us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:42.009441 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:42.009521 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:42.009650 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":251213,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":41,"raft.queue_time_us":36261,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":130,"thread_start_us":646620,"threads_started":3,"wal-append.queue_time_us":308890}
W20260808 01:51:42.028352 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 201967us
I20260808 01:51:42.029686  9521 leader_election.cc:400] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12.
I20260808 01:51:41.968664 14652 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:42.027128  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.036845  9560 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260808 01:51:42.019793 14950 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:51:42.059860  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47232.2us
I20260808 01:51:42.037525  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2958) took 1613 ms. Trace:
I20260808 01:51:42.040639 14534 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:42.045413 14809 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:42.069033 14676 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.072914 14676 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:42.060704 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:42.062204  9478 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:42.035673  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.062577  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:42.030177 14647 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:42.064464 14707 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.087124 14647 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:42.087263  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.089112 14707 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:42.089250  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.091172  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.092414  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.092486  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.093569  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.094187  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.094651 14818 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:42.058986 14729 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:42.097043 14729 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:42.098667  9478 raft_consensus.cc:2410] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260808 01:51:42.008133 14750 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:42.062783  9558 rpcz_store.cc:276] 0808 01:51:40.423625 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:40.720278 (+296653us) service_pool.cc:224] Handling call
0808 01:51:40.802320 (+ 82042us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:40.837313 (+ 34993us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:40.837354 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:40.838087 (+   733us) log.cc:844] Serialized 26 byte log entry
0808 01:51:41.494835 (+656748us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:41.494906 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:41.495016 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:42.037266 (+542250us) raft_consensus.cc:1685] finished
0808 01:51:42.037289 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:42.037448 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":19290,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":121,"thread_start_us":644735,"threads_started":1,"wal-append.queue_time_us":703497}
I20260808 01:51:42.073137 14830 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:42.076977 10366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 870) took 1622 ms. Trace:
W20260808 01:51:42.080127  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:42.082247 14852 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:42.085289 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:42.060894  9523 leader_election.cc:400] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12.
I20260808 01:51:42.110776  9523 leader_election.cc:403] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
W20260808 01:51:42.111163 10332 leader_election.cc:336] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Timed out: RequestConsensusVote RPC to 127.7.165.194:36095 timed out after 1.560s (SENT)
I20260808 01:51:42.096805  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2732) took 1323 ms. Trace:
W20260808 01:51:42.074164 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:42.103255 10366 rpcz_store.cc:276] 0808 01:51:40.454994 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:40.455173 (+   179us) service_pool.cc:224] Handling call
0808 01:51:40.455548 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:41.592072 (+1136524us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:42.076700 (+484628us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b540041f0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:42.076907 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2612163840}
W20260808 01:51:42.112584 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:42.111908  9481 rpcz_store.cc:276] 0808 01:51:40.773562 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:42.095425 (+1321863us) service_pool.cc:224] Handling call
0808 01:51:42.095769 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:42.096424 (+   655us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:42.096459 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:42.096510 (+    51us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:42.096543 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:42.096617 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:42.096743 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":143,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":110}
I20260808 01:51:42.109544  9478 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20260808 01:51:42.113778  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 60499.5us
I20260808 01:51:42.057546 14866 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:42.116192  9601 leader_election.cc:336] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.539s (SENT)
I20260808 01:51:42.110261  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2727) took 1631 ms. Trace:
I20260808 01:51:42.119876  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 809) took 1499 ms. Trace:
I20260808 01:51:42.120553  9472 rpcz_store.cc:276] 0808 01:51:40.620697 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:42.050293 (+1429596us) service_pool.cc:224] Handling call
0808 01:51:42.065394 (+ 15101us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:42.108019 (+ 42625us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:42.108066 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:42.108137 (+    71us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:51:42.119570 (+ 11433us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:42.119673 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:42.119798 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:42.120492  9480 rpcz_store.cc:276] 0808 01:51:40.479000 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.976265 (+1497265us) service_pool.cc:224] Handling call
0808 01:51:41.989194 (+ 12929us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:42.109738 (+120544us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:42.109790 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:42.109862 (+    72us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:42.109917 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:42.110020 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:42.110169 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17390,"raft.queue_time_us":19538,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":6823,"thread_start_us":102515,"threads_started":1}
I20260808 01:51:42.126555  9481 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260808 01:51:42.127810  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 67074.3us
I20260808 01:51:42.127955  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2450) took 1284 ms. Trace:
I20260808 01:51:42.035310 14918 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.128819  9637 rpcz_store.cc:276] 0808 01:51:40.843563 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.006594 (+163031us) service_pool.cc:224] Handling call
0808 01:51:41.212459 (+205865us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:41.944922 (+732463us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:42.127652 (+182730us) spinlock_profiling.cc:243] Waited 705 ms on lock 0x7b540008dc54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:42.127889 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1621068672}
I20260808 01:51:42.127416  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2734) took 1223 ms. Trace:
I20260808 01:51:42.133338  9478 rpcz_store.cc:276] 0808 01:51:40.890366 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:42.108593 (+1218227us) service_pool.cc:224] Handling call
0808 01:51:42.109067 (+   474us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:42.113585 (+  4518us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:42.113879 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:42.134752  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.135553  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2736) took 1134 ms. Trace:
I20260808 01:51:42.135985  9479 rpcz_store.cc:276] 0808 01:51:41.001195 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:42.108630 (+1107435us) service_pool.cc:224] Handling call
0808 01:51:42.134940 (+ 26310us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:42.135150 (+   210us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:42.135180 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:42.135225 (+    45us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:42.135254 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:42.135308 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:42.135435 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:42.138312  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.141175  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:42.136662  9601 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.645s (SENT) [suppressed 33 similar messages]
W20260808 01:51:42.143139  9601 leader_election.cc:336] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.683s (SENT)
I20260808 01:51:42.143642  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:42.147193 10363 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:42.145372  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.152945  9481 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 14: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:42.154076 14959 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:42.154613  9550 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:42.150449 14676 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.007s
I20260808 01:51:42.158548  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:42.159435  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.136900 14892 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:42.163410  9558 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:42.163657  9556 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: []
W20260808 01:51:42.164497  9601 leader_election.cc:336] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.743s (SENT)
I20260808 01:51:42.165711  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.100603 14729 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.166934  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.169970  9479 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 12: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:51:42.159627 14284 consensus_meta.cc:287] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.744s	user 0.008s	sys 0.005s
I20260808 01:51:42.120338 14852 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.179724  9550 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260808 01:51:42.150887 14809 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:42.183537  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:42.182900  9481 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260808 01:51:42.183681  9479 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:42.183611  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:42.185588  9479 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:42.187443  9558 raft_consensus.cc:2368] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16.
I20260808 01:51:42.171948 14954 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:42.183501 14818 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260808 01:51:42.193303  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 127516us
I20260808 01:51:42.194195  9472 raft_consensus.cc:3055] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:42.194561  9472 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:42.195641  9472 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.196897  9472 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:42.198805 14835 consensus_queue.cc:1059] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.053s
I20260808 01:51:42.188221 14809 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.200843 14954 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.202800 14765 consensus_queue.cc:1059] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.072s
I20260808 01:51:42.203035  9479 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.154652 14959 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.211618  9550 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: []
W20260808 01:51:42.217078 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 55345.5us
I20260808 01:51:42.218863 14866 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:42.227650  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:42.221901 14750 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:42.229472  9479 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:42.222159 14708 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:42.223655 14857 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:42.223685 14918 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:42.226819  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:42.191496 14932 consensus_queue.cc:1059] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
I20260808 01:51:42.212711  9480 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260808 01:51:42.239580  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.255843  9480 pending_rounds.cc:85] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 7
I20260808 01:51:42.256340  9480 pending_rounds.cc:107] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260808 01:51:42.257351  9480 raft_consensus.cc:2889] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:42.270715  9481 raft_consensus.cc:3055] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:42.271237  9481 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:42.256242  9474 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:51:42.271780  9478 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20260808 01:51:42.273182  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 108742us
W20260808 01:51:42.273826  9601 leader_election.cc:336] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.750s (SENT)
I20260808 01:51:42.274185  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:42.274969  9472 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260808 01:51:42.278822 14809 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:42.278919 14682 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.597s	user 0.005s	sys 0.006s
I20260808 01:51:42.280869 14852 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:42.203600 14954 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:42.287106 14682 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:42.302073 14682 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:42.303006 10361 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 7. Current term is 8. Ops: []
W20260808 01:51:42.288801 14755 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.706s	user 0.012s	sys 0.002s
I20260808 01:51:42.291453  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:42.307343 14932 raft_consensus.cc:3055] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:42.307744 14932 raft_consensus.cc:740] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:42.308238  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:42.308817 14932 consensus_queue.cc:260] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:42.309692  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:42.310195 14932 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:42.310454  9552 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 9: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:42.291993  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:42.297410  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:42.315083  9558 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:42.316179  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:42.316429  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:42.317046  9558 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:42.318205  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:42.292429  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:42.294157  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:42.294459  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:42.295356  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:42.296180  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:42.297103  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:42.297880  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:42.318259  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:42.349817  9638 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 13: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:42.289906  9550 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:42.306026 14755 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.345790  9479 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260808 01:51:42.355020  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.356407 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:42.277801 14830 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:42.358454  9633 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:42.357717  9634 raft_consensus.cc:2376] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12. Re-sending same reply.
I20260808 01:51:42.360196 10363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 625) took 1577 ms. Trace:
I20260808 01:51:42.358278  9558 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:42.329696 14821 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:42.370018  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:42.391023 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:42.391927  9632 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:42.362176  9481 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.388903 10363 rpcz_store.cc:276] 0808 01:51:40.782392 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:40.782573 (+   181us) service_pool.cc:224] Handling call
0808 01:51:40.801846 (+ 19273us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:42.152622 (+1350776us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:42.359942 (+207320us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b5400646c54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:42.360138 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3080350976}
I20260808 01:51:42.398495  9555 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 7. Current term is 8. Ops: []
I20260808 01:51:42.286314 14950 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.286653 14892 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12.
I20260808 01:51:42.408398  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:42.410447 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:42.416215  9481 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:42.361330 14284 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:51:42.399729  9550 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260808 01:51:42.403803  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.421048 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:42.421554  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.344558 14647 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.388142 14959 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:42.428931  9556 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:42.429572  9558 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 10 index: 12. Preceding OpId from leader: term: 12 index: 13. (index mismatch)
I20260808 01:51:42.433853 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:42.437525  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:42.419869 14284 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:42.441618  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 147684us
I20260808 01:51:42.404881 14849 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.449582  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:42.450057  9550 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:42.406327 14821 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:42.453723  9634 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:42.457207 14631 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:51:42.459426 14631 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:42.465261 14631 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:42.484861  9559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 681) took 2456 ms (client timeout 2085 ms). Trace:
W20260808 01:51:42.485460  9559 rpcz_store.cc:269] 0808 01:51:40.028352 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:40.030062 (+  1710us) service_pool.cc:224] Handling call
0808 01:51:42.484535 (+2454473us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b540002cbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:42.484752 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4208411520}
I20260808 01:51:42.487703 14755 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:42.489033 14851 raft_consensus.cc:993] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:51:42.489707 14851 raft_consensus.cc:1081] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Signalling peer 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:51:42.492933 14950 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:42.494737  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2974) took 1957 ms. Trace:
I20260808 01:51:42.495532 10360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 869) took 2084 ms. Trace:
I20260808 01:51:42.495975 10360 rpcz_store.cc:276] 0808 01:51:40.410856 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:40.411098 (+   242us) service_pool.cc:224] Handling call
0808 01:51:40.436805 (+ 25707us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:41.685373 (+1248568us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:42.495280 (+809907us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b5400690354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:42.495465 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2818061568}
I20260808 01:51:42.497061 14924 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:42.497614 14924 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:42.501616 14821 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:42.502620 14935 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.503116 14817 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:42.505059  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:42.505313 14935 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:42.446017 14284 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:42.457044 14782 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:42.495235  9554 rpcz_store.cc:276] 0808 01:51:40.537172 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.344399 (+807227us) service_pool.cc:224] Handling call
0808 01:51:41.344794 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:41.824786 (+479992us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:42.494426 (+669640us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b5400313dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:42.494651 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":820884736}
I20260808 01:51:42.511269  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.512217  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:42.436970 14765 consensus_queue.cc:1059] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
I20260808 01:51:42.513093  9630 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:42.513854 14765 raft_consensus.cc:3055] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:51:42.514354 14765 raft_consensus.cc:740] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:42.515918 14805 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:42.516646 14805 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:42.517398 14961 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260808 01:51:42.482311 14659 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:42.521330 14849 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:42.522372  9477 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:42.524235 14537 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:42.526963 14851 raft_consensus.cc:3055] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:42.527480 14851 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:42.528831 14851 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:42.530241 14851 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:51:42.531134  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:42.531641  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:42.532088  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:42.532398  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:42.538764  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:42.541222 14656 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:42.541687 14706 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:42.542062 14656 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.542373 14706 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.544697 14656 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:42.545673  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:42.546614 14706 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:42.515666 14765 consensus_queue.cc:260] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:42.548018 14951 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.510s	user 0.005s	sys 0.000s
I20260808 01:51:42.549840  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:42.550010 14647 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:42.550750  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:42.552356  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:42.552667  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:42.591504  9632 raft_consensus.cc:2410] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 14 because replica has last-logged OpId of term: 12 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 5.
I20260808 01:51:42.553398  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:42.554477  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:42.592700  9635 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:42.555626  9631 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260808 01:51:42.556048 14669 raft_consensus.cc:3055] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:51:42.556809  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:42.576869 14631 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:42.603246  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:42.606568  9637 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:42.557134 10362 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:42.558213  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2730) took 1936 ms. Trace:
I20260808 01:51:42.607728  9477 rpcz_store.cc:276] 0808 01:51:40.622365 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:42.052939 (+1430574us) service_pool.cc:224] Handling call
0808 01:51:42.055100 (+  2161us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:42.522917 (+467817us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:42.557962 (+ 35045us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x7b54004042d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:42.558151 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1073924864}
I20260808 01:51:42.560695 14844 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:42.560308 14833 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:42.560343 14817 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:51:42.565694  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.566223  9556 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260808 01:51:42.566709  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.570328  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.570812  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.572813 10334 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:42.474745  9444 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:42.624826  9474 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:42.578110  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:42.625310 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52616.9us
I20260808 01:51:42.625674  9472 raft_consensus.cc:2376] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13. Re-sending same reply.
W20260808 01:51:42.566267  9601 leader_election.cc:336] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.761s (SENT)
I20260808 01:51:42.579579  9521 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:42.578588  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:42.627902  9477 raft_consensus.cc:3055] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:51:42.628227  9552 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 12: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:42.628297  9477 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:42.587695  9554 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:42.624004 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:42.624840  9478 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
W20260808 01:51:42.628254  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:42.615047  9475 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260808 01:51:42.617086  9524 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:42.619283 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:42.596145 14669 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:42.629515  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:42.644539  9638 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:42.629379  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:42.620848  9476 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2569) took 2616 ms (client timeout 1759 ms). Trace:
I20260808 01:51:42.645256  9552 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:51:42.646977 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:42.647109  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:42.638504  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:42.638836  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:42.639057  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59540.3us
I20260808 01:51:42.639333 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:42.650810  9554 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:51:42.645287  9476 rpcz_store.cc:269] 0808 01:51:39.988790 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:40.045030 (+ 56240us) service_pool.cc:224] Handling call
0808 01:51:42.604706 (+2559676us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54000212d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:42.604841 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3434726144}
I20260808 01:51:42.646234 14632 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:42.652781  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 86595.3us
I20260808 01:51:42.587735 14656 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.036s
I20260808 01:51:42.653223  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:42.653980  9601 leader_election.cc:336] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.853s (SENT)
I20260808 01:51:42.654623  9629 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 12: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:42.660267  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:42.655434 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:42.657372  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:42.657773 10367 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
I20260808 01:51:42.662120  9559 raft_consensus.cc:2393] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:42.663578  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:42.624146 14647 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:42.668213  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.670192  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.681332  9481 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:42.695678  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.696466  9479 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:42.621694 14706 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260808 01:51:42.698637  9552 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:42.659551 14849 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:42.735082  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.666229 14957 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.068s
I20260808 01:51:42.663573 14959 consensus_queue.cc:1059] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.073s
I20260808 01:51:42.697400 14895 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:42.735859  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 690) took 1408 ms. Trace:
W20260808 01:51:42.740125  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 265742us
I20260808 01:51:42.664268 14632 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:42.672372 14935 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:42.665630 14806 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:51:42.672715  9477 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:42.727217  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:42.731173  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 77215.6us
W20260808 01:51:42.769155  9601 leader_election.cc:336] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.885s (SENT)
I20260808 01:51:42.762693  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.682947 14954 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:51:42.674347 14844 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.080s
I20260808 01:51:42.770764  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.783416  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.784614  9479 raft_consensus.cc:2393] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:42.773393  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:42.778611  9477 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:42.780939  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.765201  9551 rpcz_store.cc:276] 0808 01:51:41.327859 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.575714 (+247855us) service_pool.cc:224] Handling call
0808 01:51:41.589443 (+ 13729us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:41.596932 (+  7489us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:41.596981 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:41.610111 (+ 13130us) log.cc:844] Serialized 26 byte log entry
0808 01:51:41.968279 (+358168us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:41.968346 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:41.968446 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:42.547346 (+578900us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:42.685030 (+137684us) raft_consensus.cc:1685] finished
0808 01:51:42.685055 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:42.735775 (+ 50720us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":29443840,"thread_start_us":357636,"threads_started":1,"wal-append.queue_time_us":415722}
I20260808 01:51:42.764163  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:42.786540  9552 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:42.752030 14849 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:42.787988  9554 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:42.794940  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:42.795869  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:42.798523  9635 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:51:42.798695  9637 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:42.809700  9552 raft_consensus.cc:2368] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 15. Current term is 16.
W20260808 01:51:42.812992  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 43886.2us
W20260808 01:51:42.814007  9601 leader_election.cc:336] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.895s (SENT)
I20260808 01:51:42.813846  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.817966  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.818830  9479 raft_consensus.cc:2376] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12. Re-sending same reply.
I20260808 01:51:42.700177 14647 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:42.780051 14852 raft_consensus.cc:3055] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:51:42.868794  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:42.864886  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:42.865651  9630 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:42.871411  9559 raft_consensus.cc:2376] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11. Re-sending same reply.
I20260808 01:51:42.889197  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.879918  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:42.889338  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.890220  9479 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:42.786371 14935 raft_consensus.cc:697] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:51:42.900727  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 86791.2us
W20260808 01:51:42.901465  9601 leader_election.cc:336] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.904s (SENT)
I20260808 01:51:42.882534  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.901858  9601 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:42.808660 14849 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:42.899784  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 105015us
I20260808 01:51:42.937592  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:42.937862 14974 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:42.866081 14959 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:42.895618  9551 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:42.903273  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.905306 14647 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:42.728139 14669 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:42.913686  9557 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260808 01:51:42.913278 14935 consensus_queue.cc:237] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:42.915382 14632 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:42.905331 14608 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:51:42.919896 14784 raft_consensus.cc:3055] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:42.868640 14852 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:42.799700 14805 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:42.922365  9601 leader_election.cc:336] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.922s (SENT)
I20260808 01:51:42.927471  9524 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:42.929108  9558 pending_rounds.cc:85] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 6
I20260808 01:51:42.932844 14853 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:51:42.933089 14895 consensus_queue.cc:1059] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:51:42.946233 14899 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:42.935730  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.977136  9472 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:42.932886 10332 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:42.981537 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 48741.1us
I20260808 01:51:42.939749  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:42.988554 10332 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:42.956597 10363 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:42.966415  9558 pending_rounds.cc:107] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260808 01:51:42.967952  9479 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:42.903273  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:42.990002  9558 raft_consensus.cc:2889] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:42.940213 14652 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:51:42.940905 14959 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:42.972595  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50296.2us
W20260808 01:51:43.009634  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:42.973922  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46528.2us
I20260808 01:51:43.012271  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:43.013667  9524 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:43.013109  9472 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:51:43.015619 14806 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:43.016136 14806 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:43.016556 14806 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:42.974745 14743 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:42.972666  9477 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260808 01:51:42.995170  9476 raft_consensus.cc:2410] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 10 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260808 01:51:42.996840 10332 leader_election.cc:400] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 15. Current term is 16.
I20260808 01:51:43.021952  9477 pending_rounds.cc:85] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 7
I20260808 01:51:43.022182 10332 leader_election.cc:403] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:51:43.022256  9477 pending_rounds.cc:107] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260808 01:51:43.022622  9477 raft_consensus.cc:2889] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:42.998287  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.955025 14784 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:43.024358  9479 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:43.015691  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:42.938653 14974 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:43.028180  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:43.028527  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:43.032711  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 93101.6us
I20260808 01:51:43.034430  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:43.036149  9524 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:43.038151  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:43.039106  9477 raft_consensus.cc:2410] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260808 01:51:43.038977  9524 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:42.983938 14669 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:42.993848 14868 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:43.046187  9524 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:43.020661 14833 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:42.992190 14647 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:43.047901  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:43.059985  9477 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:51:43.051486  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:43.052026  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:43.056569  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:43.064939  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:43.019495 14729 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:51:43.066785  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:43.067796  9481 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:43.071439  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:43.068918  9524 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:43.035257 14895 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 16
I20260808 01:51:43.076903  9473 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260808 01:51:43.083513  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 10 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:43.087605  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:43.088382  9479 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:43.040395 14952 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:42.985743 14899 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:43.095805  9472 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260808 01:51:43.061393 14868 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:43.054282 14954 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260808 01:51:43.101735  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:43.064482 14918 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:43.073866 14957 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:51:43.103663 14868 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:43.104068 14647 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:43.083292 14950 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:51:43.130266  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:43.124692 14652 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:43.138921 14632 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:43.144241 14652 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 11.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:51:43.113273 14824 raft_consensus.cc:3055] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Stepping down as leader of term 7
I20260808 01:51:43.141697 14805 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:43.146411  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:43.146576  9481 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:51:43.065176 14852 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:43.138984 14835 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.014s
I20260808 01:51:43.151530  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2978) took 2123 ms. Trace:
I20260808 01:51:43.105803 14833 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.038s
I20260808 01:51:43.158147  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:43.100271  9477 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:43.156388  9560 rpcz_store.cc:276] 0808 01:51:41.027711 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.357816 (+330105us) service_pool.cc:224] Handling call
0808 01:51:41.358208 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:42.061940 (+703732us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:43.150296 (+1088356us) spinlock_profiling.cc:243] Waited 678 ms on lock 0x7b540002c6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:43.150582 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1560078080}
I20260808 01:51:43.167960  9477 raft_consensus.cc:3055] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Stepping down as leader of term 12
W20260808 01:51:43.174306  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:43.173370  9560 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260808 01:51:43.175278  9477 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:43.181834 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:43.148492 14824 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:43.185192  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:43.154419 14632 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Stepping down as leader of term 12
W20260808 01:51:43.194216 14851 consensus_meta.cc:287] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.664s	user 0.012s	sys 0.000s
I20260808 01:51:43.113955 14784 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:43.196219  9477 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:43.177495 14924 consensus_meta.cc:287] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.679s	user 0.012s	sys 0.003s
I20260808 01:51:43.166973 14659 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:51:43.202335  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 44266.4us
W20260808 01:51:43.183796 14969 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.215s	user 0.002s	sys 0.000s
W20260808 01:51:43.182863 14782 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.673s	user 0.010s	sys 0.004s
I20260808 01:51:43.125653 14959 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.5, Last appended by leader: 5, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:43.173669 14976 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:43.174225 14768 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:43.219653  9477 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:43.197098 14632 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:43.208945 14852 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:43.219138 14882 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:43.271673  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 200940us
W20260808 01:51:43.276803  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 65532.6us
I20260808 01:51:43.276892  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:43.280622 14950 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:51:43.282593  9524 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:43.214008 14924 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:43.284472 14969 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260808 01:51:43.272102 14706 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260808 01:51:43.297834 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:43.272389 14768 raft_consensus.cc:697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:43.273183 14824 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:43.297225  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 680) took 3269 ms. Trace:
I20260808 01:51:43.325152  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2726) took 2912 ms. Trace:
I20260808 01:51:43.292502 14974 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:43.287842 14882 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:43.322525 14632 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:43.326911  9599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.018s	sys 0.004s
W20260808 01:51:43.343528  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 61046.9us
I20260808 01:51:43.347373  9475 rpcz_store.cc:276] 0808 01:51:40.407472 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:41.329310 (+921838us) service_pool.cc:224] Handling call
0808 01:51:42.611392 (+1282082us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:42.634413 (+ 23021us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:43.297652 (+663239us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54000212d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:43.320120 (+ 22468us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2947583360}
W20260808 01:51:43.360021  9599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.747s	user 0.019s	sys 0.004s
I20260808 01:51:43.352195  9477 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
I20260808 01:51:43.347110  9557 rpcz_store.cc:276] 0808 01:51:40.027707 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:40.027950 (+   243us) service_pool.cc:224] Handling call
0808 01:51:40.028578 (+   628us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:42.945890 (+2917312us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:43.295247 (+349357us) spinlock_profiling.cc:243] Waited 2.88 s on lock 0x7b540002e254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:43.297135 (+  1888us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6633930496}
I20260808 01:51:43.360839  9477 pending_rounds.cc:85] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 6
I20260808 01:51:43.361243  9477 pending_rounds.cc:107] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260808 01:51:43.361517  9477 raft_consensus.cc:2889] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:51:43.360489  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 747819us
I20260808 01:51:43.351446  9551 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:51:43.343056 14950 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260808 01:51:43.327685 14972 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
W20260808 01:51:43.356782  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 224730us
I20260808 01:51:43.324939 14656 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260808 01:51:43.345364  9555 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:53208
I20260808 01:51:43.345419 14824 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:43.381264  9555 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:51:43.381772  9555 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:51:43.366583 14971 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.239s	user 0.001s	sys 0.001s
I20260808 01:51:43.385172  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:43.386112  9556 raft_consensus.cc:2368] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13.
I20260808 01:51:43.343384 14743 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:43.341979 14924 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:43.373706 14981 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.220s	user 0.000s	sys 0.001s
I20260808 01:51:43.380966  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:43.369477 14632 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:43.391863  9636 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:51:43.341014 14782 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.062s
I20260808 01:51:43.330148 14851 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20260808 01:51:43.330298 14817 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260808 01:51:43.344527 14768 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:43.417879 14918 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:43.329983 14833 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260808 01:51:43.340292 14806 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:43.341208 14849 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:43.411975 14284 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.905s	user 0.012s	sys 0.003s
I20260808 01:51:43.422046 14284 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:43.423089 14849 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:43.424281 14284 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:43.413625 14821 raft_consensus.cc:3055] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:43.426487 14821 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:43.428166 14821 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:43.393023 14950 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:43.361451 14978 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260808 01:51:43.361250 14977 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.041s
I20260808 01:51:43.431726  9557 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:43.392505 14608 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.040s
I20260808 01:51:43.390836 14743 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:43.433790 14895 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 15. Current term is 16.
I20260808 01:51:43.438301  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:43.445822  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 58201.8us
I20260808 01:51:43.395586 14830 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:43.508394 14830 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:43.510810  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 68571.2us
I20260808 01:51:43.494207 14950 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:43.511660 14978 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:43.512135 14978 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:43.513558 14950 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:43.495426  9601 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:43.454396  9555 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:43.493656 14977 consensus_queue.cc:1243] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 14.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260808 01:51:43.507758  9559 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:43.513094 14978 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:43.505313  9556 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260808 01:51:43.513108  9444 leader_election.cc:400] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16.
I20260808 01:51:43.550278  9444 leader_election.cc:403] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
W20260808 01:51:43.470371  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:43.552910  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:43.434792 14806 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:43.557438  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 44505.4us
I20260808 01:51:43.560704  9555 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:43.563953  9557 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:43.514603 14659 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:43.570179  9556 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
W20260808 01:51:43.571574  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 76275.3us
I20260808 01:51:43.571784  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:43.576553  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:43.576968  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:43.578254  9478 raft_consensus.cc:2368] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13.
I20260808 01:51:43.554244 14859 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:43.559995 14830 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:43.582783  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:43.565280 14632 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16.
I20260808 01:51:43.584524  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:43.584738  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:43.586067  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:43.568739 14608 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:43.516345 14868 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:43.589600  9477 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260808 01:51:43.572938 14833 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:43.582755 14978 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:43.594748  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:43.599762  9630 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:43.595022  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:43.596308  9472 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260808 01:51:43.591512  9521 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:43.602520  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:43.601195  9521 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:43.603704  9521 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:43.605932  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:43.608402  9555 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260808 01:51:43.585197 14859 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:43.601895 14987 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:51:43.581293 14806 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
W20260808 01:51:43.654951  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 71217.7us
I20260808 01:51:43.657262  9599 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:43.630236  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:43.598348 14817 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:43.663753  9630 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:51:43.591933 14959 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
W20260808 01:51:43.666967  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:43.591066 14852 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
I20260808 01:51:43.599571 14833 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:43.616184 14935 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:43.607657 14632 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:43.676739  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:43.690363  9444 connection.cc:256] RPC call timeout handler was delayed by -1.8777513670211192s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1572s for a response
W20260808 01:51:43.695621  9444 connection.cc:256] RPC call timeout handler was delayed by -1.8264513670292217s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16290000000000002s for a response
W20260808 01:51:43.696046  9444 connection.cc:256] RPC call timeout handler was delayed by -1.7922513670055196s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16670000000000001s for a response
I20260808 01:51:43.687332  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:43.615506 15002 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:43.701063  9599 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:43.624267 14918 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:43.681618  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:43.697651  9632 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
W20260808 01:51:43.696369  9444 connection.cc:256] RPC call timeout handler was delayed by -1.7121513670135755s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1756s for a response
W20260808 01:51:43.701704  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:43.702601 14835 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:43.702355 14659 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260808 01:51:43.703505  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:43.666314 14830 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:51:43.704952  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:43.732476  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:43.747120  9599 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:43.703720 14851 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260808 01:51:43.747421 14859 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:43.748477 14996 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260808 01:51:43.748922 14859 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:43.749643 14989 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260808 01:51:43.598170 14608 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:43.751510  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2990) took 1179 ms. Trace:
I20260808 01:51:43.752749 14608 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:43.751885  9558 rpcz_store.cc:276] 0808 01:51:42.572027 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:42.928434 (+356407us) service_pool.cc:224] Handling call
0808 01:51:42.928856 (+   422us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:42.994949 (+ 66093us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:42.995093 (+   144us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:42.995873 (+   780us) log.cc:844] Serialized 52 byte log entry
0808 01:51:43.299074 (+303201us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:43.395148 (+ 96074us) log.cc:844] Serialized 14 byte log entry
0808 01:51:43.397643 (+  2495us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:43.397726 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:43.751265 (+353539us) raft_consensus.cc:1685] finished
0808 01:51:43.751288 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:43.751447 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5511,"prepare.queue_time_us":53501,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":16976,"raft.run_cpu_time_us":325,"raft.run_wall_time_us":45526,"thread_start_us":282480,"threads_started":2,"wal-append.queue_time_us":536193}
I20260808 01:51:43.754426  9558 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
W20260808 01:51:43.732821  9444 connection.cc:256] RPC call timeout handler was delayed by -1.7067513670190237s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17620000000000002s for a response
I20260808 01:51:43.667004 14987 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:43.737685  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:43.757766  9444 connection.cc:256] RPC call timeout handler was delayed by -1.6626513670198619s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1811s for a response
W20260808 01:51:43.758119  9444 connection.cc:256] RPC call timeout handler was delayed by -1.6527513670152985s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18220000000000003s for a response
W20260808 01:51:43.758471  9444 connection.cc:256] RPC call timeout handler was delayed by -1.6086513670161366s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18710000000000002s for a response
W20260808 01:51:43.758900  9444 connection.cc:256] RPC call timeout handler was delayed by -1.5456513670214918s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19410000000000002s for a response
W20260808 01:51:43.759243  9444 connection.cc:256] RPC call timeout handler was delayed by -1.5114513670268934s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19790000000000002s for a response
W20260808 01:51:43.759701  9444 connection.cc:256] RPC call timeout handler was delayed by -1.3431513670075219s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21660000000000001s for a response
W20260808 01:51:43.760088  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1361513670126442s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2396s for a response
I20260808 01:51:43.761262  9629 raft_consensus.cc:2393] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:43.766850  9555 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
W20260808 01:51:43.768172 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:43.740489  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:43.701602 14982 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
I20260808 01:51:43.691377 14833 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:43.746042  9601 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:43.773036  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:43.739907  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:43.741367 14918 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:43.704730 14935 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:43.776618  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:43.742668 14284 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:43.785980  9601 leader_election.cc:400] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13.
I20260808 01:51:43.790570  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:43.794797  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:43.742822 14990 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:43.799116  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:43.800741  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:43.807662  9629 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:43.737205 14632 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:43.755326 14974 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.054s
I20260808 01:51:43.819890  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:43.793778 14608 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:43.793231 14851 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:43.828931 14851 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:43.828614  9599 leader_election.cc:400] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13.
I20260808 01:51:43.830524  9599 leader_election.cc:403] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260808 01:51:43.831022 14851 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:43.793679 14659 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:43.732321 15002 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:43.799333 14782 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260808 01:51:43.820219 15000 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.222s	user 0.001s	sys 0.000s
I20260808 01:51:43.793313 14284 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:43.793974 14817 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:51:43.863328  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 818) took 1362 ms. Trace:
I20260808 01:51:43.801968 14990 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:43.867779  9473 rpcz_store.cc:276] 0808 01:51:42.501178 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:42.503283 (+  2105us) service_pool.cc:224] Handling call
0808 01:51:42.503682 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:43.085215 (+581533us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:43.859434 (+774219us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x7b540009a6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:43.863247 (+  3813us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1244660096}
I20260808 01:51:43.789649 14976 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.068s
W20260808 01:51:43.855311 14995 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.284s	user 0.000s	sys 0.002s
I20260808 01:51:43.873301  9631 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
W20260808 01:51:43.880901  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 61148.7us
I20260808 01:51:43.801221 14835 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.064s
I20260808 01:51:43.856714 14959 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260808 01:51:43.864166 14998 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.280s	user 0.001s	sys 0.001s
I20260808 01:51:43.900815 14950 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:43.896703  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 102009us
I20260808 01:51:43.904430  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:43.918570  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 57528.5us
W20260808 01:51:43.920491  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:43.867705 14817 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:43.922030 10362 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 696) took 1510 ms (client timeout 1635 ms). Trace:
I20260808 01:51:43.923840  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:43.924124 10362 rpcz_store.cc:269] 0808 01:51:42.394358 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:42.394557 (+   199us) service_pool.cc:224] Handling call
0808 01:51:43.904327 (+1509770us) spinlock_profiling.cc:243] Waited 90.1 ms on lock 0x7b540064aad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:43.904465 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":207149952}
I20260808 01:51:43.864444 15001 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.038s
I20260808 01:51:43.872735 14990 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:43.886904 14782 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:43.880220 14992 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:43.937083 14631 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:43.938259 14632 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:43.933478 14784 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:43.940717  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 695) took 1242 ms. Trace:
I20260808 01:51:43.944806  9560 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 15. Ops: []
I20260808 01:51:43.954746  9551 rpcz_store.cc:276] 0808 01:51:42.698424 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:42.795461 (+ 97037us) service_pool.cc:224] Handling call
0808 01:51:43.940476 (+1145015us) spinlock_profiling.cc:243] Waited 8.93 ms on lock 0x7b54000fee54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:43.940638 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20534528}
W20260808 01:51:43.957963  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 53642us
I20260808 01:51:43.901717 14950 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:43.922446 14868 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:43.887512 14743 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:43.982965  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:43.983994 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:43.995160  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2591) took 1346 ms. Trace:
I20260808 01:51:43.994202 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:43.996815  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:44.001307  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:44.003165 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:44.003505 15004 log.cc:927] Time spent T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260808 01:51:44.004175 10362 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:43.997416 14988 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:44.006208 14988 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:44.009827 14988 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:43.884985 14987 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:43.995759  9476 rpcz_store.cc:276] 0808 01:51:42.648449 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:43.087704 (+439255us) service_pool.cc:224] Handling call
0808 01:51:43.110161 (+ 22457us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:43.125037 (+ 14876us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:43.125074 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:43.155452 (+ 30378us) log.cc:844] Serialized 26 byte log entry
0808 01:51:43.354338 (+198886us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:51:43.354390 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:43.354521 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:43.994781 (+640260us) raft_consensus.cc:1685] finished
0808 01:51:43.994809 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:43.995074 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8704,"raft.queue_time_us":7937,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":14721,"thread_start_us":176028,"threads_started":1,"wal-append.queue_time_us":363454}
W20260808 01:51:44.014756 15009 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260808 01:51:44.016563 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:44.021647 14950 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:43.922761 14935 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:43.927516 14918 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.5, Last appended by leader: 5, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:43.982044 15003 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.240s	user 0.000s	sys 0.002s
I20260808 01:51:44.052222 14632 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260808 01:51:44.076668  9636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2428) took 5123 ms (client timeout 3236 ms). Trace:
W20260808 01:51:44.077247  9636 rpcz_store.cc:269] 0808 01:51:38.952857 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:39.229281 (+276424us) service_pool.cc:224] Handling call
0808 01:51:44.076351 (+4847070us) spinlock_profiling.cc:243] Waited 3.93 s on lock 0x7b54003fa554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:44.076522 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9034078080}
I20260808 01:51:44.080055 14706 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:44.080797 14706 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:44.084429 14743 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:44.028410 14824 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.638s	user 0.013s	sys 0.000s
W20260808 01:51:44.051973  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 50799.7us
I20260808 01:51:44.055689  9480 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:51:43.981639 15024 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13.
I20260808 01:51:44.092856  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:44.034623 15019 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:44.025758 15002 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:44.041025  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:44.067600 14849 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260808 01:51:44.102805  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:44.107656  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 109896us
I20260808 01:51:44.113381  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:44.114233  9556 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:44.098227 14729 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:44.093680 14892 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:44.117046  9480 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260808 01:51:44.118674  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:44.117017 10332 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:44.128182  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:44.128532 10332 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:44.129204  9477 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 11: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:43.988152 14284 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:44.133065  9481 pending_rounds.cc:85] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 8
I20260808 01:51:44.134531  9478 pending_rounds.cc:85] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 6
I20260808 01:51:44.137161  9481 pending_rounds.cc:107] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20260808 01:51:44.132931  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:44.137670  9481 raft_consensus.cc:2889] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:44.137212  9478 pending_rounds.cc:107] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260808 01:51:44.108343 14765 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:44.138258  9478 raft_consensus.cc:2889] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:44.123975  9637 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:51:44.143411  9473 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:44.022598 14817 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:44.124912 14632 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:44.124402 14729 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:44.165262  9476 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: []
W20260808 01:51:44.151160  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:44.172500  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:44.173521  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:44.140185 14659 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:44.122576 14950 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.034s
I20260808 01:51:44.195292  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:44.140828 14868 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:51:44.196463 15020 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260808 01:51:44.121847 15021 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.056s
I20260808 01:51:44.140599 14849 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:44.141067 14835 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:44.154054 14632 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:44.139216 14892 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:44.169116 14729 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:51:44.207089 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 78673.8us
I20260808 01:51:44.199348 14743 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:51:44.155287 14852 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260808 01:51:44.215997 14743 raft_consensus.cc:697] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:44.217227 14852 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260808 01:51:44.218351  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:44.217646 14743 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:44.238291  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:44.211426 14632 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:51:44.244197  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:44.245028  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:44.239274  9556 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:44.246598  9445 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:44.282581 14849 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:44.283643  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:44.283775 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:44.285166  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:44.285867  9476 raft_consensus.cc:2376] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13. Re-sending same reply.
I20260808 01:51:44.240248 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:44.241803  9524 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:44.301002  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:44.303162  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 65704.4us
I20260808 01:51:44.307112 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:51:44.308022  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66277.8us
W20260808 01:51:44.308988  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 113813us
I20260808 01:51:44.309386  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:44.302547 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62392.3us
I20260808 01:51:44.313517  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:44.317513 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:44.318750 10364 raft_consensus.cc:2368] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13.
I20260808 01:51:44.205479 14990 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:44.225142 14835 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:44.327411  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:44.320406  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:44.293112 14849 raft_consensus.cc:697] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:44.286794 15033 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:44.348869  9445 leader_election.cc:400] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13.
I20260808 01:51:44.331919  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:44.313558 14868 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:44.352114  9635 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:44.317035 14987 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260808 01:51:44.362835  9445 leader_election.cc:403] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:51:44.332544  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:44.353055 14935 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260808 01:51:44.377590  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 45804.9us
I20260808 01:51:44.350042 15039 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:44.358556 14852 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:44.321954 15024 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.044s
W20260808 01:51:44.396214  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 86989.4us
W20260808 01:51:44.390460  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 460e1059f5504707b891c4b0e0ac6a09: Illegal state: Replica 6e7856bed1be43acaa821290d506ee97 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 14 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:44.322789 14659 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:44.326010 15030 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.062s
I20260808 01:51:44.393260 14631 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 13
I20260808 01:51:44.361406 15037 consensus_queue.cc:1059] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260808 01:51:44.332401 15002 consensus_queue.cc:1059] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.045s
W20260808 01:51:44.387852  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:44.401188  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:44.404563  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:44.412968  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:44.385978 15033 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:44.420579  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:44.405671 15023 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.201s	user 0.001s	sys 0.001s
I20260808 01:51:44.427636 14950 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:44.406376 15038 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260808 01:51:44.428268 14950 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:44.429875 14833 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.630s	user 0.013s	sys 0.000s
W20260808 01:51:44.408273 15025 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
W20260808 01:51:44.415289 15027 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.220s	user 0.002s	sys 0.000s
W20260808 01:51:44.426993 15035 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260808 01:51:44.414746 15037 raft_consensus.cc:3055] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:44.434064  9476 raft_consensus.cc:2410] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260808 01:51:44.435611  9524 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:44.431519 15036 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260808 01:51:44.444176 15033 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:44.444928 14892 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:44.445911 14990 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:44.447227 14978 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.840s	user 0.016s	sys 0.000s
I20260808 01:51:44.396801 15039 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:44.430358 14833 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:44.461776  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 60673.6us
I20260808 01:51:44.459525 15037 raft_consensus.cc:740] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:44.475106  9599 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:44.468369  9524 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:44.486894  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 124696us
I20260808 01:51:44.490036  9524 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:44.490357  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:44.467216 15024 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:44.382014 14849 consensus_queue.cc:237] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:44.469551 14935 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:44.504276  9638 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:44.511647  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:44.474053 14950 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:44.491374 15033 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:44.515483  9557 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:44.495546 15024 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:44.491600 15047 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:51:44.536727  9550 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260808 01:51:44.517693 14868 raft_consensus.cc:697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:44.511291 14892 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:44.544524 14868 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:44.546885  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:44.529138 15033 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:44.558391 15037 consensus_queue.cc:260] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:44.559729 14899 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:44.560683 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:44.566306  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:44.567121  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:44.568909  9472 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 10. (term mismatch)
I20260808 01:51:44.569484  9472 pending_rounds.cc:85] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 9
I20260808 01:51:44.569000 14706 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:44.569726  9472 pending_rounds.cc:107] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260808 01:51:44.570001  9472 raft_consensus.cc:2889] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:44.539341 15024 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:44.526197 14852 raft_consensus.cc:697] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:44.560520 15037 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:44.514578 14659 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.036s
I20260808 01:51:44.575819 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:44.578502 10360 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260808 01:51:44.603310 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:44.604753  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:44.605726  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:44.572410 15033 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:44.599910 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:44.608889  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 118941us
I20260808 01:51:44.611192  9524 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:44.611373 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:44.609454 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: true dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:51:44.610934  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 98453.9us
I20260808 01:51:44.633459  9523 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:44.634971  9630 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:44.637619  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:44.642656  9630 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: []
W20260808 01:51:44.643468 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 55520.2us
I20260808 01:51:44.645390  9630 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:44.591892 14682 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.039s
I20260808 01:51:44.645895  9524 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:44.628194 14935 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:51:44.648329 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:44.651592  9636 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260808 01:51:44.651712 10367 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:44.648788  9633 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:44.648552  9523 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:44.652691 10364 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:44.649832  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:44.653904  9630 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:44.656706  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:44.657202  9523 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:44.662017  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:44.663225  9524 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:44.663789  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:44.651111 14537 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:44.659839 14647 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:44.654556 15047 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:44.684226  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:44.671185  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:44.685199 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:44.694559 10367 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:51:44.712152  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:44.666301 14950 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:51:44.715191 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:44.669391 14706 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:44.684406 14935 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:44.687180 14892 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:51:44.732676 10364 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 11: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:51:44.737181  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 407cb6b113ce432587849bc840ba6872: Illegal state: Replica 259bcb3f0d254e8b9effac3277e654e0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 14 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:44.666337 14652 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20260808 01:51:44.687964 14647 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:44.667644 14918 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.064s
I20260808 01:51:44.745589  9600 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:44.752269  9636 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260808 01:51:44.758808  9630 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
W20260808 01:51:44.732981  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 61816.3us
I20260808 01:51:44.766009  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:44.728322 14537 raft_consensus.cc:697] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:44.770115  9630 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: []
W20260808 01:51:44.773211  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:44.744825 14817 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:44.731690 15065 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:44.746076 15047 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election won for term 14
W20260808 01:51:44.781446 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 62237us
I20260808 01:51:44.684576 14852 consensus_queue.cc:237] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:44.764681 14647 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:51:44.799360  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 118397us
I20260808 01:51:44.763239 15002 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:44.733985 15039 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:44.762867 14892 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:44.807195  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:44.834420 15063 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
I20260808 01:51:44.890527  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:44.891599  9551 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:44.893229  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:44.837623 15070 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260808 01:51:44.809890 15078 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:44.895556 15078 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:44.832067 14537 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:44.831969 15081 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:51:44.840330 14682 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:51:44.900373 15078 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:44.901981 15078 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:44.841303  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:44.902335 14918 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:51:44.902544 15078 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:44.903112  9472 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:51:44.843981  9474 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:44.845180 14821 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:44.905522 15049 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.313s	user 0.003s	sys 0.000s
I20260808 01:51:44.849861 14835 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:44.909539 10334 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:44.921643  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:44.790565 14818 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:44.750770 15053 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.054s
I20260808 01:51:44.855731  9634 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:51:44.795531 15051 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.061s
I20260808 01:51:44.780635 15071 consensus_queue.cc:1059] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.096s
W20260808 01:51:44.867188 14632 consensus_meta.cc:287] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.543s	user 0.009s	sys 0.004s
I20260808 01:51:44.867988 15079 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:44.956445 10366 raft_consensus.cc:3055] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Stepping down as leader of term 14
W20260808 01:51:44.868093 14729 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.634s	user 0.008s	sys 0.005s
I20260808 01:51:44.794401 15065 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:44.872895 14631 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13.
I20260808 01:51:44.875661 14892 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:44.886097 15061 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.201s	user 0.000s	sys 0.001s
I20260808 01:51:44.807246 15047 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:51:44.833149 15052 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
I20260808 01:51:44.835331 15080 consensus_queue.cc:1059] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260808 01:51:44.899238  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:44.898718  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 90088us
I20260808 01:51:44.984014  9524 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:44.905176 15078 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:44.909510 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:44.852396 14765 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.709s	user 0.015s	sys 0.000s
I20260808 01:51:44.988116  9600 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:44.923154  9551 raft_consensus.cc:2393] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 12: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:51:44.923388  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 157471us
I20260808 01:51:44.990906  9600 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:44.992007 15051 raft_consensus.cc:2764] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260808 01:51:44.957103 10366 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:44.938206 14818 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:44.938607 14988 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:44.986706  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:44.988447  9555 raft_consensus.cc:2393] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 8: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:51:44.988487  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 354059us
W20260808 01:51:44.975344 15032 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.731s	user 0.001s	sys 0.004s
I20260808 01:51:44.974299 15075 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:44.995236  9634 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14. Ops: []
W20260808 01:51:44.993567  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:44.963737 14632 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:45.018186  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:45.018023  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:45.003159  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:45.019173  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:45.019325  9634 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:51:45.023789  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:45.024617 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 380024us
I20260808 01:51:44.968642 14729 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.001483 15080 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election won for term 15
W20260808 01:51:45.050352  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 63771.6us
I20260808 01:51:45.017242  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:45.026602  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:45.055037 10361 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:44.988080 14918 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:51:45.022990 10366 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:45.001654 15051 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:45.066766  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:45.079957  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:45.087210 10366 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:45.090173  9633 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:45.091676 15085 raft_consensus.cc:3055] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:45.092129 15085 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:45.092170  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:45.093164 15085 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.093734  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:45.100612  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:45.101235  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:45.101635  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:45.093873 14652 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:44.966523 15079 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.091154  9478 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:51:45.018042 14988 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:45.110122  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:45.094413 14729 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:45.163468 10363 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 8.
I20260808 01:51:45.052557 14899 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260808 01:51:45.133675  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:45.113747  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:45.078354  9524 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:45.138593  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:45.171928  9475 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:51:45.176683  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 98433us
I20260808 01:51:45.177917  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:45.100389 15085 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:45.184947  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:45.188331  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:45.099934 15080 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:45.109354 14849 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:51:45.194599  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:45.190541  9472 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
W20260808 01:51:45.198982  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 179928us
I20260808 01:51:45.084488 14991 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.099s
I20260808 01:51:45.203253  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:45.156376 14988 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:45.183647 14669 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:45.208940  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:45.212893  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:45.189059 14918 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:45.191361 14706 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:45.049518 15075 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:45.054519 14818 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:45.221803  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:45.218473 15082 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
I20260808 01:51:45.133281 14652 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:45.191612 14729 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:45.234098  9476 raft_consensus.cc:2376] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16. Re-sending same reply.
I20260808 01:51:45.247044  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:45.250051 14991 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:45.189640 15047 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.250751 14991 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:45.251543 15079 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:45.252940 15065 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:45.255928  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:45.223286 14892 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:45.264079 14729 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.223860 15090 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:45.271690  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:45.270751  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:45.224395 15100 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:45.275197  9551 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:51:45.271354 14782 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:45.298661 14818 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:45.306399 14782 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:45.308710  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:45.291283  9636 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:45.227419 14669 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:45.302446  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:45.273049 15096 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:51:45.305241 15090 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.313848  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:45.295338 14844 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:45.328486 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:45.329388 10362 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:45.307605 14782 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.316630 14632 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:45.299974 15080 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:45.321046  9476 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:45.317261  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:45.332938  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:45.324239  9551 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
W20260808 01:51:45.274008 15033 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.643s	user 0.009s	sys 0.004s
I20260808 01:51:45.297001 15075 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:45.343767 14632 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260808 01:51:45.344301 15073 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.339s	user 0.000s	sys 0.002s
I20260808 01:51:45.344007 15075 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:51:45.344934 15075 consensus_queue.cc:1243] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 14.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260808 01:51:45.345475 15075 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:45.346745 15096 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:45.347201 15096 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:45.347689  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:45.348227 15096 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.349361  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:45.349666 15096 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:45.351061  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:45.271767 14647 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.352543  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:45.356387  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:45.278766 14729 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:45.269866 14892 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:45.346096 15075 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.300899 15100 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.370250 15090 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:45.390504 15090 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:45.391103 15090 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.391683 14652 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:45.330950 15065 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:45.393307 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:45.393873 15065 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.394356 10361 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 8: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:45.334513 14844 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.397790 14844 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:45.398527 14844 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:45.412086 14844 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:45.376721  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:45.383680  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:45.418586  9550 raft_consensus.cc:2393] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 12: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:45.268141 14991 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:45.388322 15075 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:45.389552 14892 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:45.365589  9636 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:45.376883 14782 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:45.431701  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:45.433007  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:45.433638  9636 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
W20260808 01:51:45.436128 14983 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 2.042s	user 0.015s	sys 0.007s
I20260808 01:51:45.437529  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2989) took 2866 ms. Trace:
I20260808 01:51:45.437918  9552 rpcz_store.cc:276] 0808 01:51:42.571656 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:42.906725 (+335069us) service_pool.cc:224] Handling call
0808 01:51:42.907102 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:42.907522 (+   420us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:42.907553 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:42.908096 (+   543us) log.cc:844] Serialized 26 byte log entry
0808 01:51:43.268787 (+360691us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:43.268854 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:43.268954 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:45.435297 (+2166343us) spinlock_profiling.cc:243] Waited 2.55 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:45.437214 (+  1917us) raft_consensus.cc:1685] finished
0808 01:51:45.437244 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:45.437461 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2254,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":111,"spinlock_wait_cycles":5859200,"thread_start_us":360209,"threads_started":1,"wal-append.queue_time_us":464413}
I20260808 01:51:45.440321  9552 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:45.441318 14647 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:45.441521  9554 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 13. Ops: []
I20260808 01:51:45.431984  9550 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:45.444160  9550 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:45.444324  9558 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260808 01:51:45.444675 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:45.445313 15100 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:45.450387 14817 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.667s	user 0.013s	sys 0.003s
I20260808 01:51:45.441608  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2483) took 1784 ms. Trace:
I20260808 01:51:45.451885  9632 rpcz_store.cc:276] 0808 01:51:43.657618 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:43.792646 (+135028us) service_pool.cc:224] Handling call
0808 01:51:43.814631 (+ 21985us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:43.823078 (+  8447us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:43.823127 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:43.823207 (+    80us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:43.823249 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:45.441349 (+1618100us) spinlock_profiling.cc:243] Waited 8.18 ms on lock 0x7b54000dfa54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:45.441404 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:45.441544 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18813696}
I20260808 01:51:45.452934 14631 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:45.453372 14631 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:45.409400  9555 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15. Ops: [12.7-14.8]
I20260808 01:51:45.330125 15081 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:45.455633 15081 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:45.410914 14784 raft_consensus.cc:3055] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:45.370437  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2797) took 1261 ms. Trace:
I20260808 01:51:45.464867  9480 rpcz_store.cc:276] 0808 01:51:44.109514 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:44.109738 (+   224us) service_pool.cc:224] Handling call
0808 01:51:44.110211 (+   473us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:44.126738 (+ 16527us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:45.370131 (+1243393us) spinlock_profiling.cc:243] Waited 6.47 ms on lock 0x7b54001013d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:45.370357 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14887808}
I20260808 01:51:45.467034  9480 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:45.415942 10363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 633) took 1155 ms. Trace:
I20260808 01:51:45.430402  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:45.468789  9550 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260808 01:51:45.457754 10366 raft_consensus.cc:2410] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260808 01:51:45.468840 10363 rpcz_store.cc:276] 0808 01:51:44.260283 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:44.260716 (+   433us) service_pool.cc:224] Handling call
0808 01:51:45.415624 (+1154908us) spinlock_profiling.cc:243] Waited 695 ms on lock 0x7b5400499654. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:45.415844 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1598926592}
I20260808 01:51:45.461128  9552 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260808 01:51:45.421821  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:45.474292  9475 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:45.476565  9475 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:45.474120 14892 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:45.479530 14892 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:45.486472 14997 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 2.078s	user 0.002s	sys 0.004s
I20260808 01:51:45.482049  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:45.495275  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:45.495612 14975 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:45.496316 14975 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:45.497530 14975 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.487331 15065 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:45.454262 14631 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.500334 15065 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:51:45.499424 14806 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:45.501052  9637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 625) took 1906 ms (client timeout 2166 ms). Trace:
I20260808 01:51:45.501332 14806 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.501840 14892 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:45.502482 14706 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:45.503105 14706 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:45.500562 14631 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:45.501518  9637 rpcz_store.cc:269] 0808 01:51:43.594619 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:43.594778 (+   159us) service_pool.cc:224] Handling call
0808 01:51:45.500830 (+1906052us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b540003c0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:45.500981 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1139780352}
I20260808 01:51:45.393428 15090 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:45.496326  9552 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:45.465165 14935 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:45.503996 14806 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:45.519752  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:45.360697 15033 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:45.524161 15101 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.249s	user 0.002s	sys 0.000s
I20260808 01:51:45.464726 14784 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:45.527839 15098 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.234s	user 0.000s	sys 0.001s
W20260808 01:51:45.577176 10333 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.7.165.254:43565, user_credentials={real_user=slave}} blocked reactor thread for 46620.1us
I20260808 01:51:45.553450  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:45.581271  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2593) took 2932 ms. Trace:
I20260808 01:51:45.581883  9479 rpcz_store.cc:276] 0808 01:51:42.649007 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:43.089040 (+440033us) service_pool.cc:224] Handling call
0808 01:51:43.105860 (+ 16820us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:43.114623 (+  8763us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:43.114656 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:43.150659 (+ 36003us) log.cc:844] Serialized 26 byte log entry
0808 01:51:43.354075 (+203416us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:51:43.354136 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:43.354230 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:45.479666 (+2125436us) spinlock_profiling.cc:243] Waited 35.7 ms on lock 0x7b24000ef228. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41713594 00007f5d4171169b 00007f5d418cd6d9 00007f5d418cd365 00007f5d418cc9ad 00007f5d445162fe 00007f5d4452da7c 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411
0808 01:51:45.580821 (+101155us) raft_consensus.cc:1685] finished
0808 01:51:45.580854 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:45.581150 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4141,"raft.queue_time_us":138,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":6293,"spinlock_wait_cycles":82185472,"thread_start_us":165871,"threads_started":1,"wal-append.queue_time_us":231975}
I20260808 01:51:45.587075  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:45.587381  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:45.588320  9551 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:45.488801 14817 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
I20260808 01:51:45.596784 14844 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:45.597105 15033 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:45.528791 14935 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:45.599536 10394 heartbeater.cc:646] Failed to heartbeat to 127.7.165.254:43565 (1 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.7.165.254:43565 timed out after 15.000s (SENT)
W20260808 01:51:45.600425  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:45.600883  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 179297us
I20260808 01:51:45.616423  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:45.576629 14784 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.621119 14784 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:45.629032 15085 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:45.629710 15085 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.631492  9480 raft_consensus.cc:3055] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:45.632005  9480 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:45.632165  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:45.626295 14729 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:45.632871  9480 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 13.6, Last appended by leader: 6, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.646456  9551 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260808 01:51:45.651425  9637 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15. Ops: []
W20260808 01:51:45.637298 15024 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.036s	user 0.018s	sys 0.004s
W20260808 01:51:45.655405  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:45.664379  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:45.665089  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:45.667450  9601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.011s	sys 0.052s
I20260808 01:51:45.668771  9551 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
W20260808 01:51:45.670022  9601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.515s	user 0.011s	sys 0.052s
I20260808 01:51:45.672477  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:45.671569  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 516775us
I20260808 01:51:45.654111 14844 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:45.644320 14631 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:45.648630 15090 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:45.654407 15033 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:45.684113  9524 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:45.690569  9524 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:45.675283 14844 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.651786 15100 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
W20260808 01:51:45.694612  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:45.695917  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:45.680475 15105 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:45.679112 14729 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:45.698516 14806 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:45.712993  9551 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260808 01:51:45.688835 14817 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:45.705955 15051 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.595s	user 0.014s	sys 0.000s
I20260808 01:51:45.719712  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:45.720742  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:45.695091 14706 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:45.706881 14631 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:45.707134 14859 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:45.699671 15094 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:45.673347 15024 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:45.685680 15090 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.741567  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:45.717589 14806 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:45.699173 15033 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:45.748775  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:45.755760 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 478361us
I20260808 01:51:45.709723 14647 consensus_queue.cc:1059] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
W20260808 01:51:45.739190 14988 consensus_meta.cc:287] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.515s	user 0.013s	sys 0.002s
I20260808 01:51:45.730105 15051 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:45.756487  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2489) took 1467 ms. Trace:
I20260808 01:51:45.766847  9635 rpcz_store.cc:276] 0808 01:51:44.289577 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:44.289744 (+   167us) service_pool.cc:224] Handling call
0808 01:51:44.290171 (+   427us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:44.367230 (+ 77059us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:45.756163 (+1388933us) spinlock_profiling.cc:243] Waited 61.5 ms on lock 0x7b54000db1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:45.756406 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":141518080}
I20260808 01:51:45.768569  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:45.714838 15105 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:45.760721  9472 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.011s	sys 0.007s
W20260808 01:51:45.762060  9474 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.000s	sys 0.046s
W20260808 01:51:45.777633  9474 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.676s	user 0.000s	sys 0.047s
I20260808 01:51:45.735235 15100 raft_consensus.cc:3055] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:45.729835 14817 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.771246  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:45.781502  9637 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:45.757649 14806 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:45.775146 10332 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:45.776834  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:45.811987 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:45.812351  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:45.813416  9632 raft_consensus.cc:2410] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260808 01:51:45.814749 14821 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260808 01:51:45.815222  9632 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260808 01:51:45.815577 14821 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:45.817265  9632 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260808 01:51:45.820082  9632 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 11 index: 5. Preceding OpId from leader: term: 14 index: 6. (index mismatch)
I20260808 01:51:45.820962  9372 catalog_manager.cc:5697] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:51:45.777853  9472 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.012s	sys 0.007s
I20260808 01:51:45.822717  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 825) took 1259 ms. Trace:
I20260808 01:51:45.823313  9478 rpcz_store.cc:276] 0808 01:51:44.562972 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:44.565973 (+  3001us) service_pool.cc:224] Handling call
0808 01:51:45.822455 (+1256482us) spinlock_profiling.cc:243] Waited 409 ms on lock 0x7b54002c1354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:45.822608 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":940972800}
I20260808 01:51:45.824642 10366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 704) took 1248 ms. Trace:
I20260808 01:51:45.825059 10366 rpcz_store.cc:276] 0808 01:51:44.576607 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:44.591450 (+ 14843us) service_pool.cc:224] Handling call
0808 01:51:45.824442 (+1232992us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b540041f0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:45.824581 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":788831616}
I20260808 01:51:45.780783 15084 raft_consensus.cc:3055] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:45.835780 15084 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:45.837206 15084 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:43.976859  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:51:45.741909 14631 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.755136 15106 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:45.779455 15100 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:45.770377 14988 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:45.786722 14882 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:45.788331 14817 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:45.764405 15085 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:45.790561 15024 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:45.788355 15090 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:45.796370  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:45.747051 15094 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.870843 10361 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:45.801151  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2301) took 1142 ms. Trace:
I20260808 01:51:45.872997  9554 rpcz_store.cc:276] 0808 01:51:44.659165 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:44.670173 (+ 11008us) service_pool.cc:224] Handling call
0808 01:51:45.441317 (+771144us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:45.442106 (+   789us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:45.800847 (+358741us) spinlock_profiling.cc:243] Waited 771 ms on lock 0x7b5400409a50. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:45.801061 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1772618496}
I20260808 01:51:45.802747  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 2993) took 3229 ms. Trace:
I20260808 01:51:45.874517  9560 rpcz_store.cc:276] 0808 01:51:42.572963 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:43.186012 (+613049us) service_pool.cc:224] Handling call
0808 01:51:43.944542 (+758530us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:43.955004 (+ 10462us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:45.802418 (+1847414us) spinlock_profiling.cc:243] Waited 758 ms on lock 0x7b54000fee50. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:45.802636 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1743427968}
I20260808 01:51:45.807756  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2464) took 1151 ms. Trace:
I20260808 01:51:45.808848 15105 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:45.821503 15081 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:45.825099  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:45.863905 15096 consensus_meta.cc:287] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.514s	user 0.009s	sys 0.009s
I20260808 01:51:45.785178  9636 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260808 01:51:45.769016 15055 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:45.883823 15055 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:45.780537 15095 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.203s	user 0.000s	sys 0.002s
I20260808 01:51:45.853487 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:45.855522  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:45.857321 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:45.861617  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 405189us
I20260808 01:51:45.822598  9372 catalog_manager.cc:5697] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 9 to 12, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 12 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:45.859220 14935 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:45.876863  9633 rpcz_store.cc:276] 0808 01:51:44.656373 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:44.712402 (+ 56029us) service_pool.cc:224] Handling call
0808 01:51:44.712871 (+   469us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:45.090649 (+377778us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:45.807483 (+716834us) spinlock_profiling.cc:243] Waited 377 ms on lock 0x7b540003c5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:45.807684 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":867217152}
I20260808 01:51:45.879549  9478 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:45.845705 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:45.903721 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 58164.1us
I20260808 01:51:45.904266  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2482) took 2246 ms. Trace:
I20260808 01:51:45.904793  9638 rpcz_store.cc:276] 0808 01:51:43.657380 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:43.788203 (+130823us) service_pool.cc:224] Handling call
0808 01:51:43.802854 (+ 14651us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:44.505387 (+702533us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:45.903946 (+1398559us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x7b54003fa054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:45.904189 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1612620544}
I20260808 01:51:45.905646 14602 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:45.906338 14602 raft_consensus.cc:697] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:45.907368 14602 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:45.909298 10363 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260808 01:51:45.856705 14659 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:45.912127 14659 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:45.912657 14659 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:45.854440 15089 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260808 01:51:45.914268  9480 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:45.860199 14959 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:45.765559  9634 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.003s	sys 0.014s
W20260808 01:51:45.919157  9634 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.709s	user 0.003s	sys 0.015s
I20260808 01:51:45.921260  9599 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:45.921721  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:45.922236  9599 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:45.922394  9551 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:45.858274 15084 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:45.862414 15106 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:51:45.896525 15119 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.142s	user 0.000s	sys 0.002s
I20260808 01:51:45.922945  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:45.925693  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:45.928625  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 206679us
I20260808 01:51:45.928530 10332 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:45.933691  9476 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260808 01:51:45.941843  9476 pending_rounds.cc:85] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 7
I20260808 01:51:45.942324  9476 pending_rounds.cc:107] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260808 01:51:45.942704  9476 raft_consensus.cc:2889] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:45.941882  9554 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:45.944423  9480 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 14 index: 6. (term mismatch)
I20260808 01:51:45.926085  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:45.944999  9480 pending_rounds.cc:85] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 5
I20260808 01:51:45.945386  9480 pending_rounds.cc:107] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 13.6
I20260808 01:51:45.945827  9480 raft_consensus.cc:2889] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:45.778478 14729 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:45.933492  9599 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:45.949314  9551 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:45.949857  9478 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:45.869109 15085 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
W20260808 01:51:45.923547 15053 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.668s	user 0.027s	sys 0.032s
I20260808 01:51:45.979178  9635 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:51:45.982378  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:45.931782 14959 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:45.983454  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 9949664c54584858ae8fc74c9dc07a4a: Illegal state: Leader has not yet committed an operation in its own term
I20260808 01:51:45.983341 15108 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:51:45.985148 15108 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260808 01:51:45.985558 14959 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:45.986546 15053 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.028s	sys 0.032s
W20260808 01:51:45.987703  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 54281.3us
W20260808 01:51:45.989128  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:45.947518 15059 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:45.932206 15106 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:45.949581 14987 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:45.962776 14844 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:46.001472 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:45.963582  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:45.948200 14851 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:51:45.999818 15018 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:45.991572 15062 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:46.003516 15113 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:46.004055 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:45.964531 14988 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:45.966157 15056 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.399s	user 0.002s	sys 0.002s
I20260808 01:51:45.966574 15079 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:45.920626 14647 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:45.969082 15103 log.cc:927] Time spent T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.695s	user 0.002s	sys 0.000s
I20260808 01:51:45.970175  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 706) took 1163 ms. Trace:
I20260808 01:51:45.948398 15024 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:45.976699 10366 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:45.951805 15089 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:45.956805 14631 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:45.952702 15096 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:45.955031 14859 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260808 01:51:45.957839  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:45.990906 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:46.004758 10364 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:51:46.024175 10367 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 13
I20260808 01:51:45.992329 10332 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:45.995178 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:45.996538  9524 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:46.026562 14935 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:46.026755 15117 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:45.997995 14784 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:45.951417  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:46.007572  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:45.985612 14833 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:46.029582  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:46.034557  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:46.034614  9481 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:46.036448  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:46.036594  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:46.038214  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:46.039358  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:46.039701  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:46.039325  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:46.001715 15033 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:45.996374 14706 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:46.051573  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:46.020855 15018 raft_consensus.cc:697] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:46.031087 15108 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:46.100747 15108 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:46.100469  9372 catalog_manager.cc:5697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 10 to 12, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 12 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:46.018203 15113 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.004567 14987 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:46.047271 14656 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:46.103206  9372 catalog_manager.cc:5697] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 7 to 10, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 10 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:46.104368 14656 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.040445  9550 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 12: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:46.105299  9372 catalog_manager.cc:5697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:46.106727  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:46.107496  9550 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:46.109808 15094 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:46.113916  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:46.034907 15055 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:46.040448  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:46.040774  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:46.121076  9559 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:46.042466  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:46.122169  9480 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 9: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:46.023954  9558 rpcz_store.cc:276] 0808 01:51:44.807322 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:45.445238 (+637916us) service_pool.cc:224] Handling call
0808 01:51:45.445556 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:45.445993 (+   437us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:45.446031 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:45.458933 (+ 12902us) log.cc:844] Serialized 26 byte log entry
0808 01:51:45.753804 (+294871us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:51:45.753865 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:45.753942 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:45.969893 (+215951us) raft_consensus.cc:1685] finished
0808 01:51:45.969921 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:45.970100 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2455,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"thread_start_us":225538,"threads_started":1,"wal-append.queue_time_us":248851}
I20260808 01:51:46.043428  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:46.055039  9600 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:46.123608  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 68658us
I20260808 01:51:46.058162 14706 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.018285 15029 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:46.067721 14631 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:46.068050 15106 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.070920  9521 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:46.073340 15058 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.502s	user 0.003s	sys 0.000s
I20260808 01:51:46.073597 10367 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:46.031572 14784 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:46.027418 15117 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:46.086762 10362 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 632) took 1844 ms (client timeout 1696 ms). Trace:
W20260808 01:51:46.151310 10362 rpcz_store.cc:269] 0808 01:51:44.242781 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:44.242973 (+   192us) service_pool.cc:224] Handling call
0808 01:51:46.086432 (+1843459us) spinlock_profiling.cc:243] Waited 983 ms on lock 0x7b54006108d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:46.086628 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2261114240}
I20260808 01:51:46.089038  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:46.121071  9635 raft_consensus.cc:2376] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8. Re-sending same reply.
I20260808 01:51:46.026499 15089 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:46.045569  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:45.996973  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:46.158447  9478 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:51:46.159878  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:46.160305  9550 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:46.161880  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:46.163623 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:46.164669 10362 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 13: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:46.164961  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:46.165809  9478 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:51:46.001825 15059 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.027377 14935 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:46.028488 15024 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:45.970191 14806 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.137497  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:46.193951  9480 raft_consensus.cc:2410] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 12 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260808 01:51:46.085179  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:46.141353  9600 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:46.143741  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 72917us
I20260808 01:51:46.196604  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:46.041805 14853 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:46.153187  9477 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:46.105213 15113 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:46.154656 14784 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:46.155666 14987 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:46.163046 15089 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:46.028498 14647 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:46.135998 15001 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:51:46.004067 15062 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.175341  9550 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:46.217200  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:46.139271 15051 consensus_queue.cc:1059] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.063s
I20260808 01:51:46.218146  9551 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:46.137336  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:46.189582  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:46.101549 15018 consensus_queue.cc:237] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.197661  9480 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:51:46.196988  9521 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:46.222219  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2811) took 1116 ms. Trace:
I20260808 01:51:46.224086  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:46.224596  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:46.224852  9480 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:46.131425  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:46.150144 15047 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:46.223065  9524 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:46.047983 15096 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.195114  9473 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
W20260808 01:51:46.230490  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 89234.3us
W20260808 01:51:46.204843  9555 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.715s	user 0.016s	sys 0.025s
I20260808 01:51:46.206132  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:46.236846  9477 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:46.208379  9481 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:46.162871 14892 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:46.201850 15108 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:46.210793 14669 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:46.211359 15094 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:46.212175 14656 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:46.212538 15117 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:46.193740 15024 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:46.226728  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:46.244782  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:46.245780  9480 raft_consensus.cc:2410] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 16 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260808 01:51:46.244860  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:46.242666 14924 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20260808 01:51:46.249267 15051 raft_consensus.cc:3055] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:46.250048 15051 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:46.250946 14833 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:46.251386  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 114167us
I20260808 01:51:46.252347 14656 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:46.253054 14656 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:46.253245 15059 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:46.253650 14656 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:46.254175 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:46.255751  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:46.255765  9633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:46.256824 14647 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:46.256880  9633 raft_consensus.cc:606] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:46.257746  9633 raft_consensus.cc:614] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:51:46.258731  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 487114a18943467b8d3a6db94b17a84d: Illegal state: Replica 52ccf4dcca1843f7bb29e9172e2d4ab4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 9 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:46.262225  9560 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:51:46.269116 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:46.270087 10366 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 8.
I20260808 01:51:46.217989 15001 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:46.170176  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:46.149068 14631 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:46.229722  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:46.277510  9550 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 15: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:46.193010 14659 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.230742  9551 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:46.278095 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:46.278852 10367 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:51:46.265713 10362 raft_consensus.cc:2410] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260808 01:51:46.266458  9601 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:46.228904 15113 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260808 01:51:46.285367  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 2c617cb141a34007b6c1b8af0d911923 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:46.273358  9478 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:51:46.166807 15029 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.284821  9635 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 9. (term mismatch)
I20260808 01:51:46.294459  9635 pending_rounds.cc:85] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 8
I20260808 01:51:46.294917  9635 pending_rounds.cc:107] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20260808 01:51:46.295261  9635 raft_consensus.cc:2889] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:46.274004  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:46.222815  9472 rpcz_store.cc:276] 0808 01:51:45.105460 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:45.201423 (+ 95963us) service_pool.cc:224] Handling call
0808 01:51:45.202884 (+  1461us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:45.203144 (+   260us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:51:45.232153 (+ 29009us) log.cc:844] Serialized 14 byte log entry
0808 01:51:46.221865 (+989712us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:46.221924 (+    59us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:46.221967 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:46.222037 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:46.222153 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22954,"prepare.queue_time_us":28538,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":4511,"raft.queue_time_us":34966,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":7164,"thread_start_us":964408,"threads_started":2,"wal-append.queue_time_us":731013}
I20260808 01:51:46.288494 10334 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:46.290977  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:46.292186  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:46.239359  9555 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.749s	user 0.016s	sys 0.026s
I20260808 01:51:46.283667  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:46.309088  9600 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:46.310554  9480 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 7.
I20260808 01:51:46.312183  9480 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:46.313972  9472 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260808 01:51:46.263903 15115 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:46.271962 14924 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260808 01:51:46.317126  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 457792us
I20260808 01:51:46.317714 14978 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:46.285624 15053 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:46.285687 14991 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:46.246486 14987 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.062s
I20260808 01:51:46.309789  9601 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:46.293853 15108 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:46.296094 15113 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:46.264845 15094 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:46.354110  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:46.358340  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:46.245615 14892 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.275229 15107 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260808 01:51:46.359074  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:46.359592  9554 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:46.316033 14868 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:51:46.313512 15122 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:46.360261 14868 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:46.360481 15122 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:46.360903 14868 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:46.361073 15122 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:46.286641 15001 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:46.362171 14852 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:51:46.335287 14978 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:46.363133 14852 consensus_queue.cc:1243] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 13.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260808 01:51:46.336576 14706 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:46.364802  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:46.339002  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:46.340164  9480 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:46.314976 15117 consensus_queue.cc:1059] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.020s
I20260808 01:51:46.352202  9638 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3759209338e14437aff5f92a2fff36da"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:46.316109 14935 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.353040  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:46.354154  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 44453.3us
I20260808 01:51:46.354799 15113 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:46.355371 15059 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:46.396386  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:46.348767 14991 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:46.329051  9472 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260808 01:51:46.354509 14990 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:51:46.363157 14978 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:46.315518 14844 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:51:46.371374 15117 raft_consensus.cc:3055] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:46.352993 14729 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:46.392814  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:46.332763 14924 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:46.404335  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:46.405988  9637 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:51:46.344023 15053 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:46.340888 15115 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:46.409518 15105 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:51:46.430335  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:46.431896  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:46.506531  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 76317.2us
I20260808 01:51:46.509407  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:46.453969  9552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.850s	user 0.013s	sys 0.049s
I20260808 01:51:46.422511 15117 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:46.511909  9552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.908s	user 0.014s	sys 0.049s
I20260808 01:51:46.412741 14844 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:46.415102 15106 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:46.522850 15121 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
I20260808 01:51:46.523419 14659 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:46.524523 15124 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260808 01:51:46.526898 14959 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:51:46.515127 15105 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:46.538930 14631 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:46.491987 14729 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:46.497038 15053 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:46.606592  9445 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:46.479385 14852 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:46.489778 14990 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:46.491189 14924 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:46.553510  9559 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:51:46.562606 15096 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:46.548491  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 326628us
I20260808 01:51:46.627815  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:46.330839 15051 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:46.576210 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:46.548430 14844 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:46.588575 15105 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:51:46.581650 15120 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.263s	user 0.001s	sys 0.001s
I20260808 01:51:46.553273 15079 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:46.613464 10332 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:46.617966  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:46.648520  9559 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:51:46.648602 10332 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:46.649991  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:46.408419 15059 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.595649 15108 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:46.582227 15125 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:46.602262 15062 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:46.616286 14631 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:46.616693 14990 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:51:46.659771  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:46.670194  9559 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:46.647560 15096 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.021s
I20260808 01:51:46.615808 15094 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:46.635553 14647 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:51:46.713311  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 711) took 1652 ms. Trace:
I20260808 01:51:46.669519 14602 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:51:46.680908 15024 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:46.683341 15084 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.741s	user 0.011s	sys 0.004s
I20260808 01:51:46.687577 14656 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.687680  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:46.642748 14859 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.049s
I20260808 01:51:46.692233 15115 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:46.654090 14959 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:46.690205  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:46.694310 14892 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:46.658515 15079 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:46.659037 15055 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260808 01:51:46.696689  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2634) took 1416 ms. Trace:
I20260808 01:51:46.659691 15029 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:46.698120 14806 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:46.647878 14987 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.055s
I20260808 01:51:46.702453  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2488) took 2413 ms. Trace:
I20260808 01:51:46.704468 15083 raft_consensus.cc:3055] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:46.704591 15110 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:46.707482  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2635) took 1395 ms. Trace:
I20260808 01:51:46.808482  9636 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:46.709553  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:46.677232 15100 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.810532  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:46.811180 15122 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:46.729110  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 506097us
I20260808 01:51:46.813472 15122 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:46.760857 14782 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:46.815189 14782 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.816648  9524 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:46.739732  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:46.821179  9599 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:46.821525 14656 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:46.756997  9558 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:46.760740 14602 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:46.823604 15100 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:46.824307  9558 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260808 01:51:46.825006 14602 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:46.715523 15071 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:46.726729  9555 rpcz_store.cc:276] 0808 01:51:45.061161 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:45.455166 (+394005us) service_pool.cc:224] Handling call
0808 01:51:45.455551 (+   385us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:45.455959 (+   408us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:45.455990 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:45.456488 (+   498us) log.cc:844] Serialized 26 byte log entry
0808 01:51:46.312293 (+855805us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:51:46.712248 (+399955us) log.cc:844] Serialized 14 byte log entry
0808 01:51:46.712764 (+   516us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:46.712900 (+   136us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:46.712989 (+    89us) raft_consensus.cc:1685] finished
0808 01:51:46.713011 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:46.713220 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12866,"raft.queue_time_us":3707,"raft.run_cpu_time_us":351,"raft.run_wall_time_us":15463,"thread_start_us":1176214,"threads_started":2,"wal-append.queue_time_us":966559}
I20260808 01:51:46.782148  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:46.726799 14659 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:46.794782  9475 rpcz_store.cc:276] 0808 01:51:45.280452 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:45.477639 (+197187us) service_pool.cc:224] Handling call
0808 01:51:45.478154 (+   515us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:45.627204 (+149050us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:45.627244 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:45.628014 (+   770us) log.cc:844] Serialized 52 byte log entry
0808 01:51:46.028784 (+400770us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:46.459288 (+430504us) log.cc:844] Serialized 14 byte log entry
0808 01:51:46.496404 (+ 37116us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:46.496537 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:46.696302 (+199765us) raft_consensus.cc:1685] finished
0808 01:51:46.696336 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:46.696576 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":190399,"prepare.queue_time_us":671509,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":186046,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":44108,"thread_start_us":791825,"threads_started":2,"wal-append.queue_time_us":617072}
I20260808 01:51:46.800880  9631 rpcz_store.cc:276] 0808 01:51:44.289211 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:44.289349 (+   138us) service_pool.cc:224] Handling call
0808 01:51:44.289747 (+   398us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:44.439651 (+149904us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:44.439702 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:44.461389 (+ 21687us) log.cc:844] Serialized 26 byte log entry
0808 01:51:44.967935 (+506546us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:44.968001 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:46.701964 (+1733963us) spinlock_profiling.cc:243] Waited 70.9 ms on lock 0x7b540032d554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:46.702030 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:46.702089 (+    59us) raft_consensus.cc:1685] finished
0808 01:51:46.702105 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:46.702379 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1474039,"raft.queue_time_us":31844,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":1459662,"spinlock_wait_cycles":163135232,"thread_start_us":551534,"threads_started":2,"wal-append.queue_time_us":626643,"wal-append.run_cpu_time_us":2411,"wal-append.run_wall_time_us":1101743}
I20260808 01:51:46.826332  9558 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260808 01:51:46.709578 15059 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:46.827009 15110 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.829844  9599 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:46.872123  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 49407.4us
I20260808 01:51:46.807368  9479 rpcz_store.cc:276] 0808 01:51:45.312022 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:45.591242 (+279220us) service_pool.cc:224] Handling call
0808 01:51:45.614083 (+ 22841us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:45.633432 (+ 19349us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:45.633477 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:45.634003 (+   526us) log.cc:844] Serialized 26 byte log entry
0808 01:51:46.127701 (+493698us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:46.127746 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:46.127843 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:46.707211 (+579368us) raft_consensus.cc:1685] finished
0808 01:51:46.707242 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:46.707405 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52460,"raft.queue_time_us":22824,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":7316,"thread_start_us":451944,"threads_started":1,"wal-append.queue_time_us":663498}
I20260808 01:51:46.830765 15107 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:46.832409 15062 consensus_queue.cc:1059] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:51:46.833792 14859 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260808 01:51:46.800670 15123 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260808 01:51:46.794238 14959 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:46.781386 14784 consensus_meta.cc:287] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.565s	user 0.010s	sys 0.004s
W20260808 01:51:46.844085 15089 consensus_meta.cc:287] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.614s	user 0.008s	sys 0.008s
I20260808 01:51:46.807053 15083 raft_consensus.cc:740] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:46.903421 15083 consensus_queue.cc:260] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:46.838593  9524 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:46.870391  9555 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
W20260808 01:51:46.912526  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 74041.8us
I20260808 01:51:46.864974 14656 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:51:46.881047  9559 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
W20260808 01:51:46.852774 14833 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.601s	user 0.014s	sys 0.000s
I20260808 01:51:46.869400 15071 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:46.887212 15107 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:46.880190 15092 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:46.800232 14991 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:46.889259 15138 raft_consensus.cc:3055] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:46.897967 14959 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:46.898470 14784 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:46.899539 15113 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.526s	user 0.013s	sys 0.000s
I20260808 01:51:46.901679 15024 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:46.903331 14924 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.860880  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:46.905385 15108 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260808 01:51:46.905921  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:46.917385  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 330782us
W20260808 01:51:46.943364 14868 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.582s	user 0.010s	sys 0.006s
I20260808 01:51:46.922809  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:46.970147  9559 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:46.888306 14284 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:46.928238 14656 raft_consensus.cc:697] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:46.934863 14782 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:46.834617 14978 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:46.909698 15062 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:46.908704 14729 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:46.916421 15083 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:46.877043 14659 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:47.003368 15071 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:46.922770 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:46.928865 14833 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:46.962797  9479 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:46.964370 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:46.964761 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:51:46.965118  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 455776us
I20260808 01:51:46.930178 15092 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:51:46.910185 15001 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.547s	user 0.008s	sys 0.013s
I20260808 01:51:46.970335  9555 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:46.921325 14987 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:46.983326 14784 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:46.983582 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:46.985852 14859 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:46.987437 15047 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:51:46.988688  9523 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:46.990834  9638 raft_consensus.cc:606] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:47.026851  9638 raft_consensus.cc:614] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:46.908479 15089 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:46.996793 15062 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:46.999502 14978 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:47.000757 15107 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:46.944397 15138 raft_consensus.cc:740] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:46.956300  9560 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:51:47.006371 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:46.928010 14892 consensus_queue.cc:1059] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260808 01:51:47.011497  9558 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:46.938004 15084 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.049s
I20260808 01:51:47.012555  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:47.036278  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 47721.3us
I20260808 01:51:47.041601  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:47.010968 15139 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:47.045960  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:47.084314  9479 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
W20260808 01:51:47.084352  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 72012.2us
I20260808 01:51:46.967350 15113 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:47.085457  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:47.051124 14706 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:51:47.085812  9523 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:47.023648 15092 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:46.993520 14868 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:47.025339 14987 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:47.030339 15001 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:47.090245  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:47.040510 14859 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:47.087198 14959 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:47.083832 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:47.080660 14935 raft_consensus.cc:3055] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:47.036132 14782 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:47.102355 14656 consensus_queue.cc:237] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:47.107686  9551 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.028s	sys 0.037s
I20260808 01:51:47.088092 14729 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:47.109030  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2814) took 2002 ms. Trace:
I20260808 01:51:47.109122 14987 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:47.055496 10366 raft_consensus.cc:2368] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13.
I20260808 01:51:47.134263 10334 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:47.107625 15037 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.034s	sys 0.032s
W20260808 01:51:47.150403 15037 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.035s	sys 0.032s
I20260808 01:51:47.097236 15059 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:47.137594 10361 raft_consensus.cc:2368] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9.
I20260808 01:51:47.096487 14924 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:47.101066 14978 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:47.101387 15107 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260808 01:51:47.037911 15071 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:51:47.135984 15110 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:47.139242  9630 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:51:47.144021  9555 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
W20260808 01:51:47.142335  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 52239.3us
I20260808 01:51:47.139600 14892 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:47.138924  9551 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.029s	sys 0.037s
I20260808 01:51:47.091087 15139 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:47.107610 14706 consensus_queue.cc:1059] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.054s
I20260808 01:51:47.149075 15138 consensus_queue.cc:260] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:47.133702 15040 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:47.167116 15138 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:47.167200 15040 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:47.168000 15040 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:47.169631 14959 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:47.141057 15078 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:47.149946 14991 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:47.156728 15071 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:47.156605 14978 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:47.175426  9559 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 13 index: 6. (term mismatch)
I20260808 01:51:47.323189  9559 pending_rounds.cc:85] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 5
I20260808 01:51:47.323618  9559 pending_rounds.cc:107] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 12.6
I20260808 01:51:47.323933  9559 raft_consensus.cc:2889] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:47.138082 14935 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:47.149065 14782 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:47.140965  9476 rpcz_store.cc:276] 0808 01:51:45.106450 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:45.363786 (+257336us) service_pool.cc:224] Handling call
0808 01:51:45.364167 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:45.953795 (+589628us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:47.108618 (+1154823us) spinlock_profiling.cc:243] Waited 569 ms on lock 0x7b54000933d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:47.108922 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1308645632}
I20260808 01:51:47.182755  9555 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 11. Current term is 12. Ops: []
W20260808 01:51:47.177450 14844 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.524s	user 0.013s	sys 0.000s
I20260808 01:51:47.142876 14833 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:47.367547 15139 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:47.368185 14782 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:47.368371 15139 raft_consensus.cc:3055] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:47.368773 15139 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:47.369313  9476 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:47.370031  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2649) took 1461 ms. Trace:
I20260808 01:51:47.369629 15139 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:47.370401  9476 rpcz_store.cc:276] 0808 01:51:45.909119 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:47.327051 (+1417932us) service_pool.cc:224] Handling call
0808 01:51:47.369116 (+ 42065us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:47.369741 (+   625us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:47.369966 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:47.372565 14959 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:47.372710  9476 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260808 01:51:47.373857  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2650) took 1464 ms. Trace:
I20260808 01:51:47.320255 14991 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:47.374347  9476 rpcz_store.cc:276] 0808 01:51:45.909409 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:47.371560 (+1462151us) service_pool.cc:224] Handling call
0808 01:51:47.372170 (+   610us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:47.373416 (+  1246us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:47.373762 (+   346us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:47.375593 14991 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:47.379707  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2651) took 1396 ms. Trace:
I20260808 01:51:47.325043 15078 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:47.387465 14935 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:47.389272 14935 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:47.382735  9476 rpcz_store.cc:276] 0808 01:51:45.983050 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:47.377221 (+1394171us) service_pool.cc:224] Handling call
0808 01:51:47.378209 (+   988us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:47.378984 (+   775us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:47.379040 (+    56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:47.379117 (+    77us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:47.379184 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:47.379315 (+   131us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:47.379580 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:51:47.393792 15100 consensus_meta.cc:287] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.569s	user 0.014s	sys 0.000s
I20260808 01:51:47.147857 14729 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:47.397266 14729 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:47.404167 14868 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:47.400158  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 843) took 1407 ms. Trace:
I20260808 01:51:47.406216 15089 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:47.408785 15105 consensus_meta.cc:287] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.748s	user 0.004s	sys 0.012s
I20260808 01:51:47.409523 15105 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:47.418130 14978 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:47.406111  9476 rpcz_store.cc:276] 0808 01:51:45.992169 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:47.392200 (+1400031us) service_pool.cc:224] Handling call
0808 01:51:47.395544 (+  3344us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:47.397090 (+  1546us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:47.397234 (+   144us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:47.397513 (+   279us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:51:47.397784 (+   271us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:47.398260 (+   476us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:47.399630 (+  1370us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:51:47.429854 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:47.142953 14784 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:47.322000 14806 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:51:47.320714 15071 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:51:47.327703 14659 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:47.521507  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:47.522321  9476 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:47.162428 14892 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:47.447713 15084 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:47.439126 14833 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:47.437348 14844 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:47.529764 14987 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:47.467612 15113 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:47.509855 15115 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.024s	sys 0.049s
W20260808 01:51:47.514261 14602 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.035s	sys 0.074s
W20260808 01:51:47.512967 14284 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.534s	user 0.014s	sys 0.001s
W20260808 01:51:47.626155 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:47.467996 14784 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:47.660099 15151 log.cc:927] Time spent T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.330s	user 0.001s	sys 0.000s
I20260808 01:51:47.653401  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:47.654204  9550 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.005s	sys 0.056s
W20260808 01:51:47.669867  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.021s	sys 0.050s
I20260808 01:51:47.530241 14991 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:47.641345  9523 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.027s	sys 0.020s
I20260808 01:51:47.688910  9551 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:51:47.689122  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.021s	sys 0.051s
W20260808 01:51:47.688898  9550 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.005s	sys 0.056s
W20260808 01:51:47.670038 15115 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.721s	user 0.025s	sys 0.050s
I20260808 01:51:47.692287  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3044) took 1536 ms. Trace:
I20260808 01:51:47.693533  9550 rpcz_store.cc:276] 0808 01:51:46.156357 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:46.278208 (+121851us) service_pool.cc:224] Handling call
0808 01:51:46.309557 (+ 31349us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:46.309784 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:51:46.727361 (+417577us) log.cc:844] Serialized 14 byte log entry
0808 01:51:47.134954 (+407593us) log.cc:844] Serialized 14 byte log entry
0808 01:51:47.691826 (+556872us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:47.691886 (+    60us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:47.691938 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:47.692020 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:47.692217 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":990,"prepare.queue_time_us":538045,"prepare.run_cpu_time_us":147,"prepare.run_wall_time_us":151,"raft.queue_time_us":46701,"spinlock_wait_cycles":411335168,"thread_start_us":1361262,"threads_started":3,"wal-append.queue_time_us":937429}
W20260808 01:51:47.690120  9523 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.028s	sys 0.020s
W20260808 01:51:47.695320  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 601323us
I20260808 01:51:47.696022  9523 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:47.697659  9523 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:47.680667 15040 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.512s	user 0.014s	sys 0.004s
W20260808 01:51:47.672003 15079 consensus_meta.cc:287] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.853s	user 0.012s	sys 0.004s
W20260808 01:51:47.672688 14602 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.036s	sys 0.074s
I20260808 01:51:47.701926 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:47.702524 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:47.702869 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:47.665699 15084 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:47.784610  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:47.774092  9476 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:51:47.689941 15157 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:47.759006 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:47.757252  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3047) took 1555 ms. Trace:
I20260808 01:51:47.782532  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:47.854773  9477 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:47.756834  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3052) took 1370 ms. Trace:
I20260808 01:51:47.863096  9558 rpcz_store.cc:276] 0808 01:51:46.386076 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:46.915162 (+529086us) service_pool.cc:224] Handling call
0808 01:51:46.941294 (+ 26132us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:47.092371 (+151077us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:47.756575 (+664204us) spinlock_profiling.cc:243] Waited 69.9 ms on lock 0x7b54000b4ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:47.756760 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":160697216}
I20260808 01:51:47.855191 14882 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:47.789130  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 643) took 1746 ms. Trace:
I20260808 01:51:47.865351  9630 rpcz_store.cc:276] 0808 01:51:46.043073 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:46.045236 (+  2163us) service_pool.cc:224] Handling call
0808 01:51:47.788835 (+1743599us) spinlock_profiling.cc:243] Waited 981 ms on lock 0x7b54003fa554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:47.789037 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2255927168}
I20260808 01:51:47.864503  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:47.867758  9559 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:47.686946 15042 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:47.871017  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:47.904433  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:47.764272 14990 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.037s	user 0.015s	sys 0.003s
W20260808 01:51:47.781592 15158 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.390s	user 0.001s	sys 0.000s
W20260808 01:51:47.757620 15150 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260808 01:51:47.985011  9558 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:47.910475  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34802 (request call id 62) took 1570 ms. Trace:
I20260808 01:51:47.985762  9551 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:47.888409  9476 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
I20260808 01:51:47.756693 14987 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:47.763494 15160 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
W20260808 01:51:47.768767 15059 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.617s	user 0.011s	sys 0.000s
W20260808 01:51:47.779834 15131 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.086s	user 0.008s	sys 0.013s
W20260808 01:51:48.051725 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:47.769122 15159 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260808 01:51:47.858937 14859 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:47.760433 14844 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:47.776175 14991 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:48.213201 14656 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.818s	user 0.060s	sys 0.044s
I20260808 01:51:47.778224 14892 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:47.787142 15047 raft_consensus.cc:3055] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:48.242636  9476 pending_rounds.cc:85] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 6
W20260808 01:51:48.284778 14656 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.889s	user 0.061s	sys 0.051s
I20260808 01:51:48.056713  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:48.252007 15059 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260808 01:51:47.856551 15064 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:47.758854 14988 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:47.905597 14729 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.508s	user 0.010s	sys 0.005s
I20260808 01:51:47.795205 15139 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:47.795150 15105 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:47.702476 15062 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:47.976723  9554 rpcz_store.cc:276] 0808 01:51:46.201389 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:46.812872 (+611483us) service_pool.cc:224] Handling call
0808 01:51:46.813217 (+   345us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:46.835681 (+ 22464us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:46.835717 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:46.837168 (+  1451us) log.cc:844] Serialized 52 byte log entry
0808 01:51:47.318986 (+481818us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:47.319037 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:47.319124 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:47.757013 (+437889us) raft_consensus.cc:1685] finished
0808 01:51:47.757036 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:47.757197 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19927,"raft.queue_time_us":21271,"raft.run_cpu_time_us":387,"raft.run_wall_time_us":386,"thread_start_us":480948,"threads_started":1,"wal-append.queue_time_us":835534}
I20260808 01:51:48.263768 14987 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:47.886442 15138 consensus_meta.cc:287] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.719s	user 0.008s	sys 0.006s
I20260808 01:51:48.214869  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 841) took 2363 ms. Trace:
W20260808 01:51:48.262049 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:47.765937 14833 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:48.070469  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2824) took 2313 ms. Trace:
I20260808 01:51:48.306944  9554 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:48.056499 14990 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:48.308050  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 721) took 1340 ms. Trace:
I20260808 01:51:48.308624  9554 rpcz_store.cc:276] 0808 01:51:46.967400 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:48.302497 (+1335097us) service_pool.cc:224] Handling call
0808 01:51:48.302944 (+   447us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:48.307699 (+  4755us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:48.307983 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:47.885574 15055 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:48.265972 10366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 641) took 1301 ms (client timeout 1696 ms). Trace:
I20260808 01:51:47.885365 14882 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:47.907841 15071 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:48.273183 15083 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.273s	user 0.012s	sys 0.004s
I20260808 01:51:48.273933  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2506) took 2009 ms. Trace:
I20260808 01:51:48.275240  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 833) took 3261 ms. Trace:
I20260808 01:51:48.262023  9638 rpcz_store.cc:276] 0808 01:51:46.340110 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:46.340246 (+   136us) service_pool.cc:224] Handling call
0808 01:51:47.910153 (+1569907us) spinlock_profiling.cc:243] Waited 619 ms on lock 0x7b5400421dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4498dc33 00007f5d4998d41a 00007f5d42ee1934 00007f5d42ee18c2 00007f5d42ee17f1 00007f5d42ee176c 00007f5d42ee0a22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:47.910339 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1424833792}
I20260808 01:51:48.252737  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:47.891939 15113 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:47.910821 14784 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:47.775256 15132 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.075s	user 0.003s	sys 0.019s
I20260808 01:51:47.983480 14659 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:48.347119 14659 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:48.349653 14659 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:48.283764  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:47.893832 15084 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:47.752097 15079 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:48.286332  9476 pending_rounds.cc:107] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260808 01:51:48.148725  9477 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:51:48.352771  9476 raft_consensus.cc:2889] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:48.061725 15169 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:51:48.062134 15171 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:48.359287 14882 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:48.057710  9478 raft_consensus.cc:2410] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260808 01:51:47.858402 15040 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:48.338289 15055 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:48.293856  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:48.362015 15079 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:48.301192  9479 rpcz_store.cc:276] 0808 01:51:45.851179 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:47.095631 (+1244452us) service_pool.cc:224] Handling call
0808 01:51:47.095994 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:47.373459 (+277465us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:47.373507 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:47.373570 (+    63us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:47.373616 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:48.214403 (+840787us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b540005fd54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:48.214486 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:48.214756 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":637548928}
I20260808 01:51:47.890650 15157 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:48.311105  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 722) took 1331 ms. Trace:
I20260808 01:51:48.366186  9554 rpcz_store.cc:276] 0808 01:51:46.979304 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:48.309843 (+1330539us) service_pool.cc:224] Handling call
0808 01:51:48.310316 (+   473us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:48.310588 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:48.310620 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:48.310676 (+    56us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:48.310764 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:48.310839 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:48.311033 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:47.777846 15117 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:48.306214  9472 rpcz_store.cc:276] 0808 01:51:45.756920 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:46.404231 (+647311us) service_pool.cc:224] Handling call
0808 01:51:46.404715 (+   484us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:46.426045 (+ 21330us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:51:46.426090 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:46.426712 (+   622us) log.cc:844] Serialized 26 byte log entry
0808 01:51:46.904934 (+478222us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:46.904978 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:46.905058 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:48.070021 (+1164963us) raft_consensus.cc:1685] finished
0808 01:51:48.070061 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:48.070350 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20721,"raft.queue_time_us":76320,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":15098,"thread_start_us":357741,"threads_started":1,"wal-append.queue_time_us":604919}
I20260808 01:51:48.286724 15047 raft_consensus.cc:740] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:48.378086 15047 consensus_queue.cc:260] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:48.379626 15047 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:51:48.331259 10366 rpcz_store.cc:269] 0808 01:51:46.964345 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:46.964517 (+   172us) service_pool.cc:224] Handling call
0808 01:51:48.265696 (+1301179us) spinlock_profiling.cc:243] Waited 43.1 ms on lock 0x7b54006108d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:48.265893 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":99167488}
W20260808 01:51:48.317282 15053 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.684s	user 0.015s	sys 0.001s
I20260808 01:51:48.383597 15053 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:48.385991 15053 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:48.300002 15064 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:48.363554 15169 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:48.389227 15169 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
W20260808 01:51:48.308727 15134 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 1.619s	user 0.005s	sys 0.016s
I20260808 01:51:48.339721  9633 rpcz_store.cc:276] 0808 01:51:46.264291 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:46.264441 (+   150us) service_pool.cc:224] Handling call
0808 01:51:46.264782 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:46.325774 (+ 60992us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:46.325818 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:46.325901 (+    83us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:46.325947 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:48.273653 (+1947706us) spinlock_profiling.cc:243] Waited 60.7 ms on lock 0x7b54000dfa54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:48.273704 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:48.273861 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":139599744}
I20260808 01:51:48.339795  9474 rpcz_store.cc:276] 0808 01:51:45.013537 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:45.013844 (+   307us) service_pool.cc:224] Handling call
0808 01:51:45.014457 (+   613us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:45.852368 (+837911us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:45.852418 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:46.707227 (+854809us) spinlock_profiling.cc:243] Waited 3.86 ms on lock 0x7b2c00134ff8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:51:46.707762 (+   535us) log.cc:844] Serialized 26 byte log entry
0808 01:51:47.140071 (+432309us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:51:47.140827 (+   756us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:47.140945 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:48.274786 (+1133841us) raft_consensus.cc:1685] finished
0808 01:51:48.274829 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:48.275150 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51378,"raft.queue_time_us":166873,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":7108,"spinlock_wait_cycles":8872192,"thread_start_us":1110566,"threads_started":2,"wal-append.queue_time_us":871337}
I20260808 01:51:48.346099  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 715) took 2427 ms. Trace:
I20260808 01:51:48.353567  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2828) took 2357 ms. Trace:
I20260808 01:51:48.357491  9558 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:48.358832 15171 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:48.287288  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2491) took 3149 ms. Trace:
I20260808 01:51:48.362396  9550 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:48.362813 14632 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:47.755787 14975 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:48.297403 14729 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:48.371167  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 3759209338e14437aff5f92a2fff36da (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
W20260808 01:51:48.418335 10332 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.716s	user 0.019s	sys 0.048s
W20260808 01:51:48.419059 10332 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.019s	sys 0.048s
I20260808 01:51:48.371686  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2831) took 2342 ms. Trace:
I20260808 01:51:48.406087  9634 rpcz_store.cc:276] 0808 01:51:45.138210 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:45.196103 (+ 57893us) service_pool.cc:224] Handling call
0808 01:51:45.196538 (+   435us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:45.932462 (+735924us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:45.932507 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:45.956806 (+ 24299us) log.cc:844] Serialized 52 byte log entry
0808 01:51:46.344175 (+387369us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:46.758939 (+414764us) log.cc:844] Serialized 14 byte log entry
0808 01:51:46.759295 (+   356us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:46.759436 (+   141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:47.779139 (+1019703us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:48.277555 (+498416us) raft_consensus.cc:1685] finished
0808 01:51:48.286537 (+  8982us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:48.287180 (+   643us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":664309,"prepare.queue_time_us":332449,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":44470,"raft.run_cpu_time_us":431,"raft.run_wall_time_us":637136,"spinlock_wait_cycles":36934528,"thread_start_us":1480982,"threads_started":3,"wal-append.queue_time_us":715862}
I20260808 01:51:48.419739  9479 rpcz_store.cc:276] 0808 01:51:46.028927 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:48.365640 (+2336713us) service_pool.cc:224] Handling call
0808 01:51:48.371127 (+  5487us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:48.371328 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:48.371352 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:48.371396 (+    44us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:51:48.371439 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:48.371506 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:48.371624 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:48.362100 15040 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:48.299539 14988 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:48.354055 15092 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.252s	user 0.007s	sys 0.008s
I20260808 01:51:48.400439  9476 rpcz_store.cc:276] 0808 01:51:45.995844 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:47.870767 (+1874923us) service_pool.cc:224] Handling call
0808 01:51:47.887976 (+ 17209us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:48.353316 (+465340us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:48.353503 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:48.402081  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2507) took 2137 ms. Trace:
I20260808 01:51:48.434216  9636 rpcz_store.cc:276] 0808 01:51:46.264637 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:46.264764 (+   127us) service_pool.cc:224] Handling call
0808 01:51:46.265026 (+   262us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:46.847824 (+582798us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:48.401754 (+1553930us) spinlock_profiling.cc:243] Waited 535 ms on lock 0x7b540003b1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:48.401996 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1229806464}
I20260808 01:51:48.355319 15083 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:48.400066  9556 rpcz_store.cc:276] 0808 01:51:45.918212 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:45.931892 (+ 13680us) service_pool.cc:224] Handling call
0808 01:51:45.954599 (+ 22707us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:45.958389 (+  3790us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:45.958439 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:45.959171 (+   732us) log.cc:844] Serialized 26 byte log entry
0808 01:51:46.424559 (+465388us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:51:46.424610 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:46.425234 (+   624us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:47.774416 (+1349182us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:48.345721 (+571305us) raft_consensus.cc:1685] finished
0808 01:51:48.345749 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:48.346024 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5911,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94,"spinlock_wait_cycles":24130304,"thread_start_us":437741,"threads_started":1,"wal-append.queue_time_us":713063}
W20260808 01:51:48.419431 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 717554us
I20260808 01:51:48.406023 15117 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:48.306634 14987 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:48.425861  9479 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260808 01:51:48.484936  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2652) took 2364 ms. Trace:
I20260808 01:51:48.485487  9479 rpcz_store.cc:276] 0808 01:51:46.120794 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:48.421289 (+2300495us) service_pool.cc:224] Handling call
0808 01:51:48.425384 (+  4095us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:48.484549 (+ 59165us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:48.484843 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:48.421790 14632 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:48.441421 15128 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 1.742s	user 0.011s	sys 0.011s
I20260808 01:51:48.443835 15092 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:48.444984  9523 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.743s	user 0.006s	sys 0.069s
I20260808 01:51:48.342260 14602 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:48.489655  9523 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.788s	user 0.006s	sys 0.070s
I20260808 01:51:48.467182 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:48.491201 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:48.473889 10366 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260808 01:51:48.491026  9550 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:48.474170  9636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7c508218361c446898fede7c5682fa9e"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:48.439534 14729 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:48.483472 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:48.489579 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:48.497601 10365 raft_consensus.cc:2368] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
I20260808 01:51:48.499711  9479 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:48.500412  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 845) took 2254 ms. Trace:
I20260808 01:51:48.500711  9479 rpcz_store.cc:276] 0808 01:51:46.245926 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:48.486608 (+2240682us) service_pool.cc:224] Handling call
0808 01:51:48.499549 (+ 12941us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:48.500157 (+   608us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:48.500358 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:48.493373  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:48.493757  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:48.503372  9550 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
W20260808 01:51:48.503711  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:48.505072  9523 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:48.468132 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:48.493841  9636 raft_consensus.cc:606] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:48.493938  9524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.048s	sys 0.071s
I20260808 01:51:48.502454  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:48.503788 10363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 888) took 2520 ms. Trace:
I20260808 01:51:48.508018  9556 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:51:48.508466 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:48.494796 14602 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:48.509181  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:48.509939  9556 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:48.510078 15084 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:48.508015 10363 rpcz_store.cc:276] 0808 01:51:45.975407 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:45.985853 (+ 10446us) service_pool.cc:224] Handling call
0808 01:51:45.986218 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:45.986636 (+   418us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:45.986664 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:45.987189 (+   525us) log.cc:844] Serialized 26 byte log entry
0808 01:51:46.408284 (+421095us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:51:46.408328 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:46.408398 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:48.276563 (+1868165us) spinlock_profiling.cc:243] Waited 28.5 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:48.490732 (+214169us) raft_consensus.cc:1685] finished
0808 01:51:48.490761 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:48.495494 (+  4733us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5272,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":2150,"spinlock_wait_cycles":65596288,"thread_start_us":420688,"threads_started":1,"wal-append.queue_time_us":647691}
W20260808 01:51:48.514806  9524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.048s	sys 0.071s
I20260808 01:51:48.514902  9523 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:48.521123  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2637) took 2864 ms. Trace:
I20260808 01:51:48.492046 15055 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:48.528138  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2834) took 2272 ms. Trace:
I20260808 01:51:48.528604  9479 rpcz_store.cc:276] 0808 01:51:46.256326 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:48.501400 (+2245074us) service_pool.cc:224] Handling call
0808 01:51:48.527389 (+ 25989us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:48.527635 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:48.527657 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:48.527689 (+    32us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:48.527709 (+    20us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:48.527764 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:48.527999 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:48.474378 15083 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:48.518059 15113 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:48.532794 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:48.575101  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:48.579322 10368 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:48.581360  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2486) took 2240 ms. Trace:
I20260808 01:51:48.581844  9637 rpcz_store.cc:276] 0808 01:51:46.341077 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:46.341243 (+   166us) service_pool.cc:224] Handling call
0808 01:51:48.581129 (+2239886us) spinlock_profiling.cc:243] Waited 46 ms on lock 0x7b540032d054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:48.581257 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":105856768}
I20260808 01:51:48.611830  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2816) took 2860 ms. Trace:
I20260808 01:51:48.612686 15083 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:48.612499  9481 rpcz_store.cc:276] 0808 01:51:45.751366 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:46.365259 (+613893us) service_pool.cc:224] Handling call
0808 01:51:46.365642 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:46.365894 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:51:46.753452 (+387558us) log.cc:844] Serialized 14 byte log entry
0808 01:51:47.142171 (+388719us) log.cc:844] Serialized 14 byte log entry
0808 01:51:48.611063 (+1468892us) spinlock_profiling.cc:243] Waited 28.7 ms on lock 0x7b18004c0de0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667 00007f5d449ae5e8
0808 01:51:48.611288 (+   225us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:48.611364 (+    76us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:48.611435 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:48.611548 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:48.611741 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7308,"prepare.queue_time_us":340897,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":102,"raft.queue_time_us":7476,"raft.run_wall_time_us":135213,"spinlock_wait_cycles":77221504,"thread_start_us":774986,"threads_started":2,"wal-append.queue_time_us":1015601}
I20260808 01:51:48.616393  9479 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260808 01:51:48.616338  9481 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260808 01:51:48.617046 10364 raft_consensus.cc:2393] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 14: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:48.618059  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 847) took 2250 ms. Trace:
I20260808 01:51:48.618592  9481 rpcz_store.cc:276] 0808 01:51:46.368196 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:48.615353 (+2247157us) service_pool.cc:224] Handling call
0808 01:51:48.615800 (+   447us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:48.617705 (+  1905us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:48.617979 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:48.518055 15084 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:48.523212 15064 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:48.521637  9473 rpcz_store.cc:276] 0808 01:51:45.657085 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:46.250266 (+593181us) service_pool.cc:224] Handling call
0808 01:51:46.250724 (+   458us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:46.356973 (+106249us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:51:46.577648 (+220675us) log.cc:844] Serialized 14 byte log entry
0808 01:51:47.011513 (+433865us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:47.011577 (+    64us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:47.011637 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:48.520796 (+1509159us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b540009a1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:48.520854 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:48.521044 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":343587,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":1819,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":5500,"spinlock_wait_cycles":243772288,"thread_start_us":651346,"threads_started":2,"wal-append.queue_time_us":1481252}
I20260808 01:51:48.624203  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 846) took 2352 ms. Trace:
I20260808 01:51:48.620803  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2819) took 2867 ms. Trace:
W20260808 01:51:48.625044  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 1.46665e+06us
I20260808 01:51:48.643097  9479 rpcz_store.cc:276] 0808 01:51:46.272127 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:48.529429 (+2257302us) service_pool.cc:224] Handling call
0808 01:51:48.615987 (+ 86558us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:48.623861 (+  7874us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:48.624112 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:48.644337  9556 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
W20260808 01:51:48.403635 14974 consensus_queue.cc:1175] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" responder_term: 13 status { last_received { term: 12 index: 8 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260808 01:51:48.656251 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:48.651113  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:48.658833  9599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.049s	sys 0.101s
I20260808 01:51:48.540534 14988 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:48.614156 15113 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:48.529968 14892 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.123s
W20260808 01:51:48.659482  9599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.049s	sys 0.101s
I20260808 01:51:48.645535  9481 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:48.585721 15055 consensus_queue.cc:1059] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.130s
W20260808 01:51:48.682476  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 814186us
I20260808 01:51:48.682906  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2838) took 2153 ms. Trace:
I20260808 01:51:48.696724  9481 rpcz_store.cc:276] 0808 01:51:46.529119 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:48.644868 (+2115749us) service_pool.cc:224] Handling call
0808 01:51:48.645342 (+   474us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:48.682570 (+ 37228us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:48.682840 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:48.701913  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:48.702906  9637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:48.703855  9636 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:48.706364  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:48.707139  9636 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:51:48.691823 15053 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:48.712515  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:48.713277  9633 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:51:48.716240  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:48.717321  9633 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:48.722210  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:48.726100  9638 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: []
W20260808 01:51:48.727175 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 228501us
I20260808 01:51:48.729279  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:48.739980  9629 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:48.697494 14659 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:48.701043 14784 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:48.745946  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:48.712952 15157 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:48.751724 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:48.716259 15092 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:48.758638  9638 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260808 01:51:48.759121  9638 pending_rounds.cc:85] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 7
I20260808 01:51:48.735219 15171 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:48.759866  9638 pending_rounds.cc:107] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260808 01:51:48.762255  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:48.762717  9638 raft_consensus.cc:2889] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:48.762938  9631 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:51:48.765036  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:48.765496  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:48.721390 15053 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:48.782866 15053 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:48.756336 14659 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:48.785126  9634 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260808 01:51:48.785763 14659 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:48.785979 14656 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:48.786635 14656 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:48.789103 14656 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:48.779672  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:48.789947 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:48.744419 15177 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:48.796361  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:48.797355  9630 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 15: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:48.800169  9638 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
W20260808 01:51:48.778033 15173 log.cc:927] Time spent T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.275s	user 0.001s	sys 0.001s
I20260808 01:51:48.758836 14991 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:48.805161  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:48.808213  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:48.808802  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 720) took 1845 ms. Trace:
I20260808 01:51:48.813212  9555 rpcz_store.cc:276] 0808 01:51:46.963271 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:47.173714 (+210443us) service_pool.cc:224] Handling call
0808 01:51:47.174180 (+   466us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:47.327574 (+153394us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:48.808526 (+1480952us) spinlock_profiling.cc:243] Waited 8.21 ms on lock 0x7b540002ec54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:48.808737 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18878336}
I20260808 01:51:48.810901  9635 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:48.765265 14987 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:48.792582 15174 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:48.762053 14844 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260808 01:51:48.842515 15174 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:48.843058 15174 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:48.846895 15177 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:48.848304 15177 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:48.849783 15171 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:48.850395 15171 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:48.851079 15113 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:48.851704 15113 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:48.852193 15113 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:48.852586 15171 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:48.854259 15113 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:48.856201 15071 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:48.818608 14991 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:48.840768  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3067) took 1037 ms. Trace:
I20260808 01:51:48.765107 14784 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:48.854712 15092 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:48.857936 14935 consensus_meta.cc:287] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.468s	user 0.010s	sys 0.005s
W20260808 01:51:48.864892 15139 consensus_meta.cc:287] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.568s	user 0.010s	sys 0.002s
I20260808 01:51:48.867893 14784 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:48.868607 15092 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:48.864995  9554 rpcz_store.cc:276] 0808 01:51:47.803478 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:48.367577 (+564099us) service_pool.cc:224] Handling call
0808 01:51:48.368094 (+   517us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:48.368651 (+   557us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:48.368695 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:48.375782 (+  7087us) log.cc:844] Serialized 26 byte log entry
0808 01:51:48.709557 (+333775us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:48.709619 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:48.709717 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:48.840501 (+130784us) raft_consensus.cc:1685] finished
0808 01:51:48.840526 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:48.840692 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":200,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":140,"thread_start_us":284306,"threads_started":1,"wal-append.queue_time_us":360587}
I20260808 01:51:48.870399  9637 raft_consensus.cc:606] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:48.910140  9637 raft_consensus.cc:614] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:48.911027  9550 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:51:48.874925  9633 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:51:48.886353 14632 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:48.912573  9630 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:48.915627  9601 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.951s (SENT) [suppressed 18 similar messages]
I20260808 01:51:48.916328  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:48.885618  9635 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260808 01:51:48.883700  9599 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:48.643219  9480 rpcz_store.cc:276] 0808 01:51:45.753391 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:46.377870 (+624479us) service_pool.cc:224] Handling call
0808 01:51:46.378248 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:46.378522 (+   274us) raft_consensus.cc:1534] Early marking committed up to index 10
0808 01:51:46.763231 (+384709us) log.cc:844] Serialized 14 byte log entry
0808 01:51:47.113748 (+350517us) log.cc:844] Serialized 14 byte log entry
0808 01:51:47.796631 (+682883us) spinlock_profiling.cc:243] Waited 29.3 ms on lock 0x7b180048d3c0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188ce62 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:51:48.620321 (+823690us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:48.620392 (+    71us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:51:48.620447 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:48.620553 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:48.620711 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2971461,"prepare.queue_time_us":322580,"prepare.run_cpu_time_us":145,"prepare.run_wall_time_us":1973,"raft.queue_time_us":1527302,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":4367,"spinlock_wait_cycles":100037376,"thread_start_us":689608,"threads_started":2,"wal-append.queue_time_us":1122146}
W20260808 01:51:48.902693 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 151024us
I20260808 01:51:48.908589  9558 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
I20260808 01:51:48.919766  9558 pending_rounds.cc:85] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 9
W20260808 01:51:48.888068 15136 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 2.190s	user 0.014s	sys 0.012s
I20260808 01:51:48.920107  9558 pending_rounds.cc:107] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260808 01:51:48.924886  9558 raft_consensus.cc:2889] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:48.922914  9554 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260808 01:51:48.921030  9599 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:48.894057 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:48.920034 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:48.930030 10332 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:48.904258 15107 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:48.931206 15107 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:48.880704 15169 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:48.932235  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:48.932996  9555 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:48.933873 15169 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:48.934224  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:48.936367 15092 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:48.874576 14602 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:48.936563  9555 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:48.931619 15187 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:51:48.937747 15187 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:48.938189 15187 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:48.934952 15089 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:48.935765  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:48.937366  9551 raft_consensus.cc:2368] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17.
I20260808 01:51:48.901743 15047 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:48.903640 15117 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
W20260808 01:51:48.931582 15126 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 2.239s	user 0.009s	sys 0.022s
I20260808 01:51:48.862977 14833 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:48.944463  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:48.976851  9559 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:48.981968  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:48.928366  9599 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:48.972114 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:48.984853 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:48.991442 10363 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:51:48.980991 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:48.992623  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:48.981356  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:48.995296  9555 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:48.993956  9554 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:48.998615 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:48.947933 14784 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:49.000877 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:48.924566 15139 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:49.000427 10361 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 17: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:49.003386 10363 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 14: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:49.005057 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:48.942055 14924 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:48.950091 14844 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:48.942397 15171 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:49.008206  9630 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:49.009840 15064 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:49.015650 15064 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:48.991575 15089 raft_consensus.cc:697] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:49.023348  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3035) took 2986 ms. Trace:
I20260808 01:51:48.996600 15113 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:49.002058 15117 consensus_queue.cc:1243] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 14.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260808 01:51:49.025259  9557 rpcz_store.cc:276] 0808 01:51:46.036974 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:46.041563 (+  4589us) service_pool.cc:224] Handling call
0808 01:51:46.098147 (+ 56584us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:46.098716 (+   569us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:46.098752 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:46.099425 (+   673us) log.cc:844] Serialized 26 byte log entry
0808 01:51:46.476468 (+377043us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:46.476544 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:46.476682 (+   138us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:48.887239 (+2410557us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:49.018064 (+130825us) raft_consensus.cc:1685] finished
0808 01:51:49.018100 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:49.023247 (+  5147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":23284,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":9011,"spinlock_wait_cycles":27982080,"thread_start_us":376252,"threads_started":1,"wal-append.queue_time_us":597840}
I20260808 01:51:49.034466 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:49.007319 15107 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:49.007095 14784 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:51:49.037539  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 109312us
I20260808 01:51:49.040488  9555 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
W20260808 01:51:49.038383  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 122788us
I20260808 01:51:48.981475 14806 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260808 01:51:49.019017 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:48.997151 14833 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:49.001202 14935 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260808 01:51:48.981969 15047 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:49.046298  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:49.051249  9633 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:51:49.043107 14784 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:49.047608  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:49.047801 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:49.048985  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:49.050348  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:49.054260  9638 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
W20260808 01:51:49.052162 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 106949us
W20260808 01:51:49.052497  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:49.008594 15175 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260808 01:51:49.008936 15139 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:49.018352 14924 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:49.040863 15117 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260808 01:51:49.065958  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:49.054447  9629 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:49.066869  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 713) took 3839 ms. Trace:
I20260808 01:51:49.068202 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:49.070211  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:49.027808 15171 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:49.074080 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:49.072118  9552 rpcz_store.cc:276] 0808 01:51:45.227111 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:45.589441 (+362330us) service_pool.cc:224] Handling call
0808 01:51:45.590011 (+   570us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:45.590644 (+   633us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:45.590718 (+    74us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:45.602803 (+ 12085us) log.cc:844] Serialized 52 byte log entry
0808 01:51:46.513292 (+910489us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:51:46.948364 (+435072us) log.cc:844] Serialized 14 byte log entry
0808 01:51:46.967941 (+ 19577us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:46.968083 (+   142us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:48.907525 (+1939442us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:49.057430 (+149905us) raft_consensus.cc:1685] finished
0808 01:51:49.066591 (+  9161us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:49.066798 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":332328,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":10827,"raft.run_cpu_time_us":910,"raft.run_wall_time_us":31279,"spinlock_wait_cycles":40144640,"thread_start_us":1279529,"threads_started":2,"wal-append.queue_time_us":1089409}
I20260808 01:51:49.075060 15186 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:49.075569 15186 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:49.075960 15186 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:49.062317 15113 raft_consensus.cc:3055] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:49.091010  9631 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:49.104199  9631 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:49.091354  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:49.092401  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:49.116998  9633 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 10: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:49.094121  9599 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:49.095006 15185 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260808 01:51:49.095808 15139 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:49.163177  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 69142.4us
I20260808 01:51:49.105448  9631 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:49.164811  9631 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:49.165642  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:49.161768  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:49.167327  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:49.066169 14844 raft_consensus.cc:697] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:49.071739 15117 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:49.177346  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:49.088174 14806 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:51:49.178491  9629 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:49.183446  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:49.188722  9638 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:51:49.079243 15182 consensus_queue.cc:1059] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.094s
I20260808 01:51:49.098410 15113 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:49.191031 10332 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:49.077934 14602 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.117s
I20260808 01:51:49.179883 15117 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:49.191447 15126 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260808 01:51:49.195310  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:49.208539  9560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 717) took 2636 ms (client timeout 2186 ms). Trace:
W20260808 01:51:49.209084  9560 rpcz_store.cc:269] 0808 01:51:46.572110 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:46.781858 (+209748us) service_pool.cc:224] Handling call
0808 01:51:49.208326 (+2426468us) spinlock_profiling.cc:243] Waited 96.3 ms on lock 0x7b54000ff854. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:49.208458 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":221407616}
I20260808 01:51:49.193419 14806 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:49.198875  9599 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:49.205711 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:49.196095  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:49.218011  9638 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:49.219334  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:49.220597  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:49.221128  9638 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:49.221876  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:49.208367  9631 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260808 01:51:49.210104  9633 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:49.195605  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:49.201334 14647 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:49.191942 15194 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:49.086367 15089 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:49.170902 15195 consensus_queue.cc:1059] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260808 01:51:49.223290  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:49.224401  9633 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13. Ops: []
W20260808 01:51:49.225369 15189 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
I20260808 01:51:49.227305 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:49.227955 14868 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:51:49.229033 15174 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:49.246946 14868 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:49.247509 14868 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:49.230628  9629 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:49.208652 15188 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:49.219533  9599 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:49.239112 14656 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:49.254757  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:49.241460 15184 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.231s	user 0.002s	sys 0.000s
I20260808 01:51:49.257722  9633 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:51:49.230438 14647 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:51:49.264272  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:49.258472 15197 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260808 01:51:49.259092 14656 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:51:49.274019  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:49.260118 15071 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:51:49.303933  9600 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:49.264540 15188 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:49.240407 14844 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:49.304662 14656 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:49.317447 14924 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:49.319217 15001 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:49.327466 15161 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.579s	user 0.008s	sys 0.000s
I20260808 01:51:49.310590 15047 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:49.316116  9600 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:49.310956 15194 raft_consensus.cc:697] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:49.313679 15195 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:49.313864 15071 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:49.262778 15113 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:49.347805 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 95545.2us
I20260808 01:51:49.348884 15204 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:51:49.349466 15204 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:49.349817 15204 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:49.345806 15001 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:49.363220 10364 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:49.341198 15195 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:49.368500 15195 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:49.367306 15194 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:49.369975 14990 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:49.373730 15209 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:51:49.375368 14990 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:49.375491 15174 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:49.375909 14990 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:49.343294 14974 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:49.377152 14974 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:49.377825  9551 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3068) took 1574 ms (client timeout 1946 ms). Trace:
W20260808 01:51:49.378407  9551 rpcz_store.cc:269] 0808 01:51:47.803789 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:48.252328 (+448539us) service_pool.cc:224] Handling call
0808 01:51:49.377575 (+1125247us) spinlock_profiling.cc:243] Waited 597 ms on lock 0x7b540002cbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:49.377754 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1372421504}
I20260808 01:51:49.380157 14974 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:49.381121 15115 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 14 is higher.
I20260808 01:51:49.381229 14990 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:49.383096 14990 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:51:49.385667  9478 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2836) took 3128 ms (client timeout 2229 ms). Trace:
W20260808 01:51:49.386166  9478 rpcz_store.cc:269] 0808 01:51:46.256992 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:46.290414 (+ 33422us) service_pool.cc:224] Handling call
0808 01:51:49.385448 (+3095034us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b54002bb6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:49.385591 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4040047488}
I20260808 01:51:49.389839 14656 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:49.391302  9631 raft_consensus.cc:2393] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 15: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:49.357823 14784 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:49.396263  9638 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:49.397513 14784 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:49.403764  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 214436us
I20260808 01:51:49.406869 15169 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:49.408098 15169 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:49.408702 15169 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:49.410337 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:49.412506  9521 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:49.420153  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:49.421106  9560 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:51:49.415033 15051 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:49.415856 15064 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:49.436187 15117 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:49.438956 15117 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:49.316497 14647 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:49.424038 15071 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:49.424319 15187 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:49.353837 14868 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:49.425606 15169 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:49.425184 15188 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:49.427702 15186 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:49.429050 15113 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:49.432458 14991 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:49.436806 14892 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:51:49.499168 14892 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:49.500506 14892 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:49.439350 15182 raft_consensus.cc:3055] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:51:49.439657 14833 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:49.440227 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:51:49.440713  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 82043e5428784a1eb53a98811ecbbbb4 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:49.441736 15062 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:49.444947 14935 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:49.446297 14806 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:49.446827  9521 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:49.417390  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:49.418455  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:49.518498  9554 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:49.447510 14988 raft_consensus.cc:3055] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:49.523028 14988 raft_consensus.cc:740] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:49.524636 14988 consensus_queue.cc:260] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:49.448077 15177 raft_consensus.cc:3055] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:51:49.527021 15177 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:49.528371 15177 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:49.529863 15177 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:51:49.451413  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:49.453047  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:49.531442  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:49.532029  9631 raft_consensus.cc:2410] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260808 01:51:49.449965 15055 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.012s
I20260808 01:51:49.535290 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:49.540539 10332 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:49.545063 14988 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:49.478437  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:49.493818 14647 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:51:49.502429  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 282915us
I20260808 01:51:49.551326  9637 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:51:49.552430  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:49.574184  9554 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
W20260808 01:51:49.584830  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 138069us
I20260808 01:51:49.586153  9521 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:49.594238 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 53914.2us
I20260808 01:51:49.587805  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:49.597371  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45078.3us
I20260808 01:51:49.596663 10332 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:49.616760 15096 consensus_queue.cc:1059] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:51:49.622586  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:49.504387 15182 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:49.630878  9521 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:49.631029  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:49.628835  9523 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:49.631699  9481 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:49.631637  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:49.632611  9480 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:49.634244  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2647) took 3823 ms. Trace:
I20260808 01:51:49.629076  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:49.635231  9478 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:49.605700 14892 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:49.584054 14806 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:51:49.626194 14987 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:49.634724  9475 rpcz_store.cc:276] 0808 01:51:45.811117 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:46.961434 (+1150317us) service_pool.cc:224] Handling call
0808 01:51:46.961868 (+   434us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:46.962114 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 12
0808 01:51:46.996825 (+ 34711us) log.cc:844] Serialized 14 byte log entry
0808 01:51:49.633785 (+2636960us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:49.633849 (+    64us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:51:49.633914 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:49.633994 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:49.634177 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2170741,"prepare.queue_time_us":36060,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"thread_start_us":479811,"threads_started":1,"wal-append.queue_time_us":766886,"wal-append.run_cpu_time_us":1355,"wal-append.run_wall_time_us":1003381}
I20260808 01:51:49.640136 15176 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:49.605829 15169 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260808 01:51:49.653798  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 55446us
I20260808 01:51:49.640324 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:49.659508  9481 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:49.661320  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 725) took 1170 ms. Trace:
I20260808 01:51:49.663342  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:49.663228 10362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 643) took 1200 ms. Trace:
I20260808 01:51:49.605821 14602 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:49.652297  9523 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:49.656925  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:49.657704  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:49.605506 14647 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:49.618573 15096 raft_consensus.cc:3055] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:51:49.678638  9480 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:49.684402  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:49.928586  9481 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:49.883425  9372 catalog_manager.cc:5697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 01:51:49.636337  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:49.890115  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 1.23879e+06us
I20260808 01:51:49.943264  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:49.894145  9445 leader_election.cc:400] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9.
I20260808 01:51:49.945767  9445 leader_election.cc:403] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
W20260808 01:51:49.947902  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 54333.3us
I20260808 01:51:49.949599  9445 leader_election.cc:400] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13.
I20260808 01:51:49.950037  9445 leader_election.cc:403] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260808 01:51:49.691140  9473 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:49.902243  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2833) took 3872 ms. Trace:
I20260808 01:51:49.951146  9474 rpcz_store.cc:276] 0808 01:51:46.029817 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:48.402765 (+2372948us) service_pool.cc:224] Handling call
0808 01:51:48.403219 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:48.403439 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:51:48.688931 (+285492us) log.cc:844] Serialized 14 byte log entry
0808 01:51:49.900939 (+1212008us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:49.901136 (+   197us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:49.901309 (+   173us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:49.901527 (+   218us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:49.901874 (+   347us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1162236,"prepare.queue_time_us":303046,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":52,"thread_start_us":301021,"threads_started":2,"wal-append.queue_time_us":252867}
I20260808 01:51:49.955523  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:49.956862  9474 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:49.903810  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:49.958880  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:49.959389  9475 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:49.960237  9474 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:49.683957  9479 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:49.963879  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:49.965392  9479 raft_consensus.cc:2376] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9. Re-sending same reply.
I20260808 01:51:49.965929  9445 leader_election.cc:400] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
I20260808 01:51:49.742986  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:49.937541  9372 catalog_manager.cc:5697] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 8 to 11. New cstate: current_term: 11 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:49.961200  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:49.911160  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2829) took 3911 ms. Trace:
W20260808 01:51:49.756196  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 103984us
I20260808 01:51:49.683445  9556 rpcz_store.cc:276] 0808 01:51:48.491164 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:48.645474 (+154310us) service_pool.cc:224] Handling call
0808 01:51:48.646128 (+   654us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:48.660377 (+ 14249us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:48.660410 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:48.660464 (+    54us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:51:48.660494 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:49.661057 (+1000563us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b54000fe954. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:49.661113 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:49.661252 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":32188672}
I20260808 01:51:49.682879  9599 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:49.747941  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 87948.2us
I20260808 01:51:49.748427  9637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:49.966641  9445 leader_election.cc:403] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:51:49.968782  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:50.017982  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 52346.8us
I20260808 01:51:50.018400  9479 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:49.969242  9477 rpcz_store.cc:276] 0808 01:51:45.999393 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:48.357079 (+2357686us) service_pool.cc:224] Handling call
0808 01:51:48.357547 (+   468us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:48.357855 (+   308us) raft_consensus.cc:1534] Early marking committed up to index 9
0808 01:51:48.667527 (+309672us) log.cc:844] Serialized 14 byte log entry
0808 01:51:49.906066 (+1238539us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:49.906224 (+   158us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:51:49.910472 (+  4248us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:49.910633 (+   161us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:49.910962 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1141088,"prepare.queue_time_us":341996,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":3147,"thread_start_us":405666,"threads_started":2,"wal-append.queue_time_us":302758}
I20260808 01:51:49.970148  9475 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
W20260808 01:51:50.020277 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 379932us
I20260808 01:51:50.021631  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:49.683866 10362 rpcz_store.cc:276] 0808 01:51:48.462558 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:48.463704 (+  1146us) service_pool.cc:224] Handling call
0808 01:51:48.464098 (+   394us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:48.514706 (+ 50608us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:48.514742 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:48.519954 (+  5212us) log.cc:844] Serialized 26 byte log entry
0808 01:51:48.758651 (+238697us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:51:48.758715 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:49.662704 (+903989us) spinlock_profiling.cc:243] Waited 40.7 ms on lock 0x7b540056fed4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:49.662759 (+    55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:49.662819 (+    60us) raft_consensus.cc:1685] finished
0808 01:51:49.662833 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:49.663138 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9463,"raft.queue_time_us":105582,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"spinlock_wait_cycles":93513984,"thread_start_us":144651,"threads_started":1,"wal-append.queue_time_us":295369}
I20260808 01:51:49.689800 14602 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:50.020308  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:50.024411 14602 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:50.024953  9479 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 14
W20260808 01:51:50.027037  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:49.992776  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 309898us
I20260808 01:51:50.027951  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.029402  9477 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
W20260808 01:51:49.995465  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:50.031109 14284 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:50.031663 14284 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:50.031827  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.033334  9477 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 10: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:49.967619  9480 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:49.999600  9637 raft_consensus.cc:606] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:50.039336  9637 raft_consensus.cc:614] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:50.040380  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:49.984637  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:50.042424  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:50.042325  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 8c49fd321f8b4f89868246239530e78b (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:50.043064  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:50.043980  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 1
W20260808 01:51:50.044061  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:50.044682  9480 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:49.969604  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:49.769065 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 118797us
W20260808 01:51:50.048298  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 79146.7us
I20260808 01:51:50.050057 10334 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:49.971704  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:50.051410  9474 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:49.968334  9372 catalog_manager.cc:5697] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 7 to 10, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 10 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:50.032188 14284 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:50.051755  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:50.052767  9560 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:50.053227 14632 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:50.054396 14632 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:50.056921  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
W20260808 01:51:49.995487  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:49.999408  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2518) took 1205 ms. Trace:
I20260808 01:51:50.058233  9632 rpcz_store.cc:276] 0808 01:51:48.793817 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:48.793956 (+   139us) service_pool.cc:224] Handling call
0808 01:51:48.794244 (+   288us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:48.804101 (+  9857us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:48.804142 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:48.804667 (+   525us) log.cc:844] Serialized 26 byte log entry
0808 01:51:49.389421 (+584754us) spinlock_profiling.cc:243] Waited 18.2 ms on lock 0x7b1800387e20. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:51:49.606514 (+217093us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:51:49.606566 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:49.606643 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:49.998432 (+391789us) raft_consensus.cc:1685] finished
0808 01:51:49.998484 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:49.998979 (+   495us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11480,"raft.queue_time_us":10037,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":5414,"spinlock_wait_cycles":41972480,"thread_start_us":216857,"threads_started":1,"wal-append.queue_time_us":971064}
I20260808 01:51:50.030336  9599 leader_election.cc:400] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17.
I20260808 01:51:50.022872  9475 raft_consensus.cc:2376] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13. Re-sending same reply.
I20260808 01:51:50.062618  9599 leader_election.cc:403] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260808 01:51:50.064334  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.067185  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:50.071296 14632 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:50.072271 14632 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:49.741729 14647 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:49.641141 14656 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:49.749966 14924 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:51:49.804566 14868 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:49.654295 14935 consensus_queue.cc:1059] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.115s
I20260808 01:51:49.638813 14959 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:49.812038 14978 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:49.656327 14987 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:49.783981 14991 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:49.805586 15001 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:50.250718  9473 raft_consensus.cc:2410] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 12 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260808 01:51:50.265126 15079 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13.
I20260808 01:51:49.626775 15055 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.126s
I20260808 01:51:49.584405 15062 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:49.786784 15078 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:49.650897 15089 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:50.302749 14853 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:51:50.303741 15089 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:50.304315 15089 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:49.743778 15096 raft_consensus.cc:740] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:50.308564  9445 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50140 dropped due to backpressure: service queue is full with 50 items
I20260808 01:51:49.653689 15138 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:49.647320 15139 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.116s
I20260808 01:51:50.310820 14784 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.067s
I20260808 01:51:49.659077 15169 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:50.313117 15169 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:49.816103 15171 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:49.627310 15177 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:49.659286 15176 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:50.321822 15176 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:49.775962 15182 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:50.324895 15182 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:49.789031 15187 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:49.803929 15195 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:49.755549 15203 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.046s
I20260808 01:51:49.751122 15204 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:50.329231  9478 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.693s	user 0.014s	sys 0.017s
I20260808 01:51:50.330053  9478 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
W20260808 01:51:49.803504 15207 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.312s	user 0.001s	sys 0.001s
I20260808 01:51:49.780817 15209 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
W20260808 01:51:49.802085 15214 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20260808 01:51:50.121796  9560 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
W20260808 01:51:50.134182  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:50.375361 14991 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:50.050410  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:50.149392 14632 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:50.370397 15188 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.070s	sys 0.095s
W20260808 01:51:50.379472 15188 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.070s	sys 0.095s
I20260808 01:51:50.379664 14632 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:50.153489  9372 catalog_manager.cc:5697] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 9 to 11, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 11 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:50.381721  9372 catalog_manager.cc:5697] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 4 to 5, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 5 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:51:50.383893  9372 catalog_manager.cc:5697] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 9 to 14, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 14 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:51:50.175226  9377 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50084 (request call id 233) took 37445 ms (client timeout 15000 ms). Trace:
I20260808 01:51:50.184504  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:51:50.386336  9372 catalog_manager.cc:5697] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 8 to 9, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 9 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:50.388845  9372 catalog_manager.cc:5697] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:50.189899  9474 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:50.191103  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:50.228224  9479 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:50.194059 14924 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:50.392139 14924 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:50.395977 14924 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:50.379534 15139 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:50.398209  9372 catalog_manager.cc:5697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:51:50.399089 15139 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:50.405246 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:50.409097 15139 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:50.234552  9481 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:50.194074 15083 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:50.190414 14284 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:49.834630 15186 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:50.411563 15083 raft_consensus.cc:697] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:50.248471 14987 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:50.067168  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.254483  9554 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: []
I20260808 01:51:50.413012 15083 consensus_queue.cc:237] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:50.413601 14284 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:50.413614  9475 raft_consensus.cc:2376] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9. Re-sending same reply.
I20260808 01:51:50.264158 15075 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 8. Current term is 9.
W20260808 01:51:50.385547  9377 rpcz_store.cc:269] 0808 01:51:12.954501 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:12.987500 (+ 32999us) service_pool.cc:224] Handling call
0808 01:51:16.511197 (+3523697us) spinlock_profiling.cc:243] Waited 2.94 s on lock 0x7b540005c3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fd470 00007f5d4499c31a 00007f5d4a78c38c 00007f5d4a888bfe 00007f5d48d424a4 00007f5d48d42432 00007f5d48d42361 00007f5d48d422dc 00007f5d48d41592 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:50.174797 (+33663600us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:51:36.728940 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:51:36.859685 (+130745us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:51:36.859769 (+    84us) write_op.cc:435] Acquired schema lock
0808 01:51:36.859801 (+    32us) tablet.cc:662] Decoding operations
0808 01:51:36.918864 (+ 59063us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:51:36.921907 (+  3043us) write_op.cc:641] Partition lock acquired for write op
0808 01:51:36.921967 (+    60us) tablet.cc:685] Acquiring locks for 17 operations
0808 01:51:36.931051 (+  9084us) tablet.cc:701] Row locks acquired
0808 01:51:36.931075 (+    24us) write_op.cc:260] PREPARE: finished
0808 01:51:36.931232 (+   157us) write_op.cc:270] Start()
0808 01:51:36.931363 (+   131us) write_op.cc:276] Timestamp: P: 1786153896931185 usec, L: 0
0808 01:51:36.931384 (+    21us) op_driver.cc:348] REPLICATION: starting
0808 01:51:37.035203 (+103819us) log.cc:844] Serialized 6741 byte log entry
0808 01:51:38.387725 (+1352522us) spinlock_profiling.cc:243] Waited 9.48 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:39.365720 (+977995us) op_driver.cc:464] REPLICATION: finished
0808 01:51:39.711499 (+345779us) write_op.cc:301] APPLY: starting
0808 01:51:39.711721 (+   222us) tablet.cc:1367] starting BulkCheckPresence
0808 01:51:39.981007 (+269286us) tablet.cc:1370] finished BulkCheckPresence
0808 01:51:39.981046 (+    39us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:51:40.401032 (+419986us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:51:40.595120 (+194088us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=34,key_file_lookups=34,delta_file_lookups=34,mrs_lookups=0
0808 01:51:40.623231 (+ 28111us) write_op.cc:312] APPLY: finished
0808 01:51:40.871135 (+247904us) log.cc:844] Serialized 156 byte log entry
0808 01:51:41.590556 (+719421us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:51:41.701508 (+110952us) write_op.cc:454] Released schema lock
0808 01:51:41.799633 (+ 98125us) write_op.cc:341] FINISH: Updating metrics
0808 01:51:41.800325 (+   692us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:51:41.800368 (+    43us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1222400,"reported_tablets":51,"spinlock_wait_cycles":6768692608,"child_traces":[["op",{"apply.queue_time_us":342932,"apply.run_cpu_time_us":67639,"apply.run_wall_time_us":2091129,"cfile_cache_hit":70,"cfile_cache_hit_bytes":75289,"num_ops":17,"prepare.queue_time_us":311544,"prepare.run_cpu_time_us":36586,"prepare.run_wall_time_us":598630,"raft.queue_time_us":911551,"raft.run_cpu_time_us":20531,"raft.run_wall_time_us":343288,"replication_time_us":2434097,"spinlock_wait_cycles":21811456,"thread_start_us":2266818,"threads_started":5,"wal-append.queue_time_us":1112852,"wal-append.run_cpu_time_us":48025,"wal-append.run_wall_time_us":3924142}]]}
I20260808 01:51:50.262449 15059 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:51:50.268414 15105 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:50.256423 14882 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17.
I20260808 01:51:50.265671 14859 raft_consensus.cc:3055] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:50.013468  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:50.260768 14988 consensus_queue.cc:1059] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.032s
I20260808 01:51:50.263676 15042 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:51:50.272701 14729 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:50.276583  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.257973 15044 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:50.447469 15044 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:50.271842 15152 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:50.266064 15085 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.016s
I20260808 01:51:50.279066 14852 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:50.298698 14659 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:51:50.298925 15040 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:50.308148 15096 consensus_queue.cc:260] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:49.657279 15107 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:50.463204 14659 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:50.464099 15107 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:50.465540 15107 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:50.330587 15204 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:50.297731 14602 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:50.248369 14978 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:50.352725  9631 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.015s	sys 0.030s
I20260808 01:51:50.375180 15195 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:50.171051 14647 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:51:50.378731  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 328712us
I20260808 01:51:50.409269 15203 raft_consensus.cc:3055] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Stepping down as leader of term 11
W20260808 01:51:50.415254 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:50.227097 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 153881us
I20260808 01:51:50.415247  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2541) took 1214 ms. Trace:
I20260808 01:51:50.481158  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.267450 15100 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 14
I20260808 01:51:50.419132  9632 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:51:50.431336  9554 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260808 01:51:50.438859 14859 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:50.485260  9477 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:50.377281 14991 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:50.455189  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 442409us
I20260808 01:51:50.465214 14659 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:50.311728 14959 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:50.451166 15042 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:50.333402  9445 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50140 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2675)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2674)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2864)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2665)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2679)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 862)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2676)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 857)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2663)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2873)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2854)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2682)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2858)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2656)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2668)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2667)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2680)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2852)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2662)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2681)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2677)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2671)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2855)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2868)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 854)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2666)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2657)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2655)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2658)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 852)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2669)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2866)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2871)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 851)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 853)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 855)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 858)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2659)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2660)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2661)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 863)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2664)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2670)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2672)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2673)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2678)
I20260808 01:51:50.460040  9480 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:51:50.478730 14924 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:50.489055  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:50.495311  9631 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.899s	user 0.016s	sys 0.030s
I20260808 01:51:50.510476 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:51:50.515059  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:50.483042  9629 rpcz_store.cc:276] 0808 01:51:49.200307 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:49.255643 (+ 55336us) service_pool.cc:224] Handling call
0808 01:51:49.258410 (+  2767us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:49.625942 (+367532us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:49.626010 (+    68us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:49.629470 (+  3460us) log.cc:844] Serialized 26 byte log entry
0808 01:51:50.358347 (+728877us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:50.414614 (+ 56267us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:50.414775 (+   161us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:50.414850 (+    75us) raft_consensus.cc:1685] finished
0808 01:51:50.414863 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:50.415042 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4155,"raft.queue_time_us":80783,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":9758,"spinlock_wait_cycles":1938304,"thread_start_us":889799,"threads_started":2,"wal-append.queue_time_us":619102}
I20260808 01:51:50.486049 15203 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:50.520618  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.573462  9477 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:50.523171  9560 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:50.523708 15195 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:50.524670 14959 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:50.517145 15042 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:51:50.529240 14859 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:50.523741 15044 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:50.531149  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.531908 14924 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:50.534765 14852 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:50.536058  9550 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260808 01:51:50.536866 10362 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 16: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:50.537142  9601 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:50.539263  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:50.540174 15078 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:51:50.541241 15096 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:51:50.542902 14806 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.823s	user 0.020s	sys 0.002s
I20260808 01:51:50.543641 15122 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:50.543725 15062 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:50.544605  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:50.544723  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:50.545460  9635 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:51:50.599893 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:50.546895 15113 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.042s	user 0.008s	sys 0.012s
I20260808 01:51:50.552476  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:50.557430 15177 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:50.559873  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2654) took 4343 ms. Trace:
I20260808 01:51:50.543030  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:51:50.534790  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:50.542520  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:50.609831  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:50.568384  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2832) took 4489 ms. Trace:
I20260808 01:51:50.569659 15139 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:50.571300  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.611341  9481 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 9.
I20260808 01:51:50.566469  9445 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50124 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2663)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2873)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2865)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2854)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2682)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2858)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2656)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2668)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2684)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2686)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2667)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2867)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 856)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2680)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2852)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2687)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2683)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2662)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2681)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2689)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2677)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2671)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2855)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2685)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2868)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 854)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2666)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2657)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2655)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2658)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 852)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2669)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2688)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2690)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2691)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2866)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2871)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 851)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 853)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 855)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 858)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2659)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2660)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2661)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 863)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2664)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2670)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2672)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2673)
I20260808 01:51:50.463116 15040 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:50.458885 14729 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:50.608973  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:50.616482  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:50.619841  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 82851us
I20260808 01:51:50.508004 15152 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:50.623988  9552 raft_consensus.cc:2410] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260808 01:51:50.624176  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:50.624966  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:50.620136  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:50.610565  9472 rpcz_store.cc:276] 0808 01:51:46.029455 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:48.372414 (+2342959us) service_pool.cc:224] Handling call
0808 01:51:48.375042 (+  2628us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:49.692261 (+1317219us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:49.692296 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:49.901144 (+208848us) log.cc:844] Serialized 26 byte log entry
0808 01:51:50.227186 (+326042us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:50.483455 (+256269us) log.cc:844] Serialized 14 byte log entry
0808 01:51:50.491890 (+  8435us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:50.504822 (+ 12932us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:50.513819 (+  8997us) raft_consensus.cc:1685] finished
0808 01:51:50.513848 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:50.518355 (+  4507us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1352613,"prepare.queue_time_us":288297,"raft.queue_time_us":1324808,"raft.run_cpu_time_us":534,"raft.run_wall_time_us":33127,"spinlock_wait_cycles":9798528,"thread_start_us":513722,"threads_started":2,"wal-append.queue_time_us":365920}
W20260808 01:51:50.636225 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:50.637604  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.639171  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.639998  9480 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
W20260808 01:51:50.640478 14892 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.986s	user 0.010s	sys 0.009s
I20260808 01:51:50.641045  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.641750  9480 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:50.628993 14988 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election won for term 16
W20260808 01:51:50.626652 15230 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.168s	user 0.000s	sys 0.002s
W20260808 01:51:50.636433 15229 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260808 01:51:50.644097  9560 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:50.608987  9476 rpcz_store.cc:276] 0808 01:51:46.216198 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:48.427641 (+2211443us) service_pool.cc:224] Handling call
0808 01:51:48.437747 (+ 10106us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:49.749732 (+1311985us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:49.749770 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:49.750413 (+   643us) log.cc:844] Serialized 26 byte log entry
0808 01:51:50.558774 (+808361us) spinlock_profiling.cc:243] Waited 63.2 ms on lock 0x7b54001269a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fd470 00007f5d4490aa98 00007f5d44993d0b 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c
0808 01:51:50.558892 (+   118us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:51:50.558951 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:50.559209 (+   258us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:50.559270 (+    61us) raft_consensus.cc:1685] finished
0808 01:51:50.559299 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:50.559658 (+   359us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1652739,"raft.queue_time_us":1316005,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":11217,"spinlock_wait_cycles":145447680,"thread_start_us":492805,"threads_started":1,"wal-append.queue_time_us":516787}
I20260808 01:51:50.646075  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.649513  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.634375 14859 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:50.643261 14988 raft_consensus.cc:697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:50.648200 15044 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:50.651031  9475 raft_consensus.cc:2410] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260808 01:51:50.651038 15209 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:50.652261 15040 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:50.652707 15177 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:50.695845 14988 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:50.702801  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:50.704847 14892 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:50.705654 14892 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:50.654049 14602 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:50.707939  9480 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:50.708345 15152 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:50.655830 14924 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:50.657578  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 657) took 1613 ms. Trace:
I20260808 01:51:50.709112  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.631721 15188 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20260808 01:51:50.709438  9635 rpcz_store.cc:276] 0808 01:51:49.044858 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:49.045056 (+   198us) service_pool.cc:224] Handling call
0808 01:51:50.657262 (+1612206us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b54003faf54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:50.657444 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3344414592}
I20260808 01:51:50.660451 14852 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:50.661989 14729 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:50.710750  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.709729 14892 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:50.711678  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2854) took 1058 ms. Trace:
I20260808 01:51:50.662451  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.712085  9481 rpcz_store.cc:276] 0808 01:51:49.658228 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:50.681465 (+1023237us) service_pool.cc:224] Handling call
0808 01:51:50.711608 (+ 30143us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:50.712738  9473 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:50.663559 14978 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:50.664052  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 666) took 1121 ms. Trace:
I20260808 01:51:50.664620 15203 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:50.666913 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 155091us
I20260808 01:51:50.667553  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:50.668049  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.627128 15122 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:50.636137 15139 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:50.673369  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.727694  9474 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:50.686787 15209 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:50.728880  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.729691  9474 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 16: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:50.730026  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2856) took 1070 ms. Trace:
I20260808 01:51:50.643689 14987 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:50.730221  9474 rpcz_store.cc:276] 0808 01:51:49.660080 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:50.728514 (+1068434us) service_pool.cc:224] Handling call
0808 01:51:50.729978 (+  1464us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:50.654444 14647 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:50.728987 15209 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:50.731102  9551 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:50.732694 14647 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:50.646610  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:50.680274  9601 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:50.685067  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.733701  9554 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:50.734009  9479 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:50.734933  9476 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:51:50.735385  9557 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 15: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:50.687383  9560 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:50.689008  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2864) took 1026 ms. Trace:
I20260808 01:51:50.689090  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:50.736102  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:50.687842  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 144922us
I20260808 01:51:50.736532  9555 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:50.737423  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:50.683290 15044 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:50.738328 15044 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:50.622118 15195 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:50.692704 15227 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
I20260808 01:51:50.709955  9480 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:51:50.661404 15071 raft_consensus.cc:3055] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:51:50.662451 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:50.714080  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:50.713582  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:50.722261  9477 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:50.721894  9631 rpcz_store.cc:276] 0808 01:51:49.542632 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:49.542882 (+   250us) service_pool.cc:224] Handling call
0808 01:51:49.543369 (+   487us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:49.595841 (+ 52472us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:49.595873 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:49.596477 (+   604us) log.cc:844] Serialized 26 byte log entry
0808 01:51:50.567522 (+971045us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:51:50.567625 (+   103us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:50.567774 (+   149us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:50.663749 (+ 95975us) raft_consensus.cc:1685] finished
0808 01:51:50.663776 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:50.663981 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":68331,"raft.queue_time_us":80735,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":15260,"spinlock_wait_cycles":2816,"thread_start_us":922538,"threads_started":1,"wal-append.queue_time_us":688462}
I20260808 01:51:50.735164  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.694923  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:50.735992  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.743098  9476 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:50.735826  9475 rpcz_store.cc:276] 0808 01:51:49.662405 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:50.643464 (+981059us) service_pool.cc:224] Handling call
0808 01:51:50.688924 (+ 45460us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:50.752350  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2852) took 1099 ms. Trace:
I20260808 01:51:50.752727  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.753417  9480 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:50.753197  9476 rpcz_store.cc:276] 0808 01:51:49.652589 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:50.735727 (+1083138us) service_pool.cc:224] Handling call
0808 01:51:50.752240 (+ 16513us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:50.706123 15169 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:50.759383  9559 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 14: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:50.759529  9557 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260808 01:51:50.760756 14647 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:50.761111 15169 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:50.761471 14647 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:50.762403 15195 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:50.714239 15177 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:50.769927  9551 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260808 01:51:50.748432 15071 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:50.771307 14647 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:50.784300 15071 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:50.786123 15071 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:50.771220 15122 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:50.787477  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.788444  9472 raft_consensus.cc:2393] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:50.789726  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.772622  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:50.792040  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:50.792730  9475 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:50.750339 15078 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:50.764482 10362 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:50.799001 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:50.800102 10366 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: [15.13-15.13]
I20260808 01:51:50.800915  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:50.801841  9551 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:50.802088 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:50.779512  9479 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:51:50.775213  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.804422  9474 raft_consensus.cc:2410] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 13 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260808 01:51:50.804639  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.805424  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:50.805612  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.806226  9474 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:50.808175 15139 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:50.805428  9479 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:51:50.810134  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:50.810747  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 107915us
I20260808 01:51:50.811019  9479 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:50.812191  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.813007  9480 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:50.808315  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:50.790570  9472 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:50.808175  9600 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:50.820443  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 189843us
I20260808 01:51:50.822099  9554 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
I20260808 01:51:50.823243  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:50.826738  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.823854  9477 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:51:50.829660  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.829836  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:50.828754  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.804215 15042 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:50.839025 15051 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:50.839617 15051 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:50.839988 15051 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:50.843781  9476 raft_consensus.cc:2393] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 9: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:50.845042  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.845757  9475 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:51:50.824587 15078 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:50.846658  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:50.864491  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:50.867908  9474 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:50.870045  9475 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:51:50.870909  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 62979.5us
I20260808 01:51:50.873179  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.873646 10334 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:50.874872  9601 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:50.878882  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:50.881495  9477 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:51:50.874907  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:50.884024  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.885353  9473 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:51:50.881969  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:50.885462  9479 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:50.886710  9480 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:51:50.888033  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 60646.3us
I20260808 01:51:50.888880 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:50.889750  9472 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 14: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:50.895138  9551 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:51:50.898092  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:50.899953  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.900619  9478 raft_consensus.cc:2410] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
W20260808 01:51:50.908074 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 75624.4us
I20260808 01:51:50.899366  9551 raft_consensus.cc:2368] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
I20260808 01:51:50.900933  9476 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
W20260808 01:51:50.915480  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:50.919668  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.906999 15100 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
I20260808 01:51:50.929016  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.924258 10332 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:50.902320 14852 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:50.933401  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:50.940064  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:50.921258 15188 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:50.947255 15042 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:50.948223 15231 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:51:50.951304 15195 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:51:50.894034 15152 consensus_queue.cc:1059] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.051s
I20260808 01:51:50.955253 15226 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:50.957351 15157 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.056s
I20260808 01:51:50.958877  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:50.959264  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:50.960111  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:50.962327  9475 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:50.969813  9479 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 12: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:50.963478  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:50.952905 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:50.930948 15182 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:50.984169  9632 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:50.984407 15182 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:50.984421  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:50.988164  9476 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 15
I20260808 01:51:50.991837  9554 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260808 01:51:50.984481  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:50.958879 14833 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:50.969117 15188 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:51:50.995390  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 165977us
W20260808 01:51:50.988898 15089 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.676s	user 0.013s	sys 0.010s
I20260808 01:51:51.007869  9555 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:51:50.986820 15001 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.058s
I20260808 01:51:51.018293  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:51.019213  9478 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:51.012358  9631 raft_consensus.cc:2368] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
W20260808 01:51:51.024601 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:50.989420 14852 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:50.994971 15231 raft_consensus.cc:697] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:50.999933 14833 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:51:51.012794 15176 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.690s	user 0.012s	sys 0.009s
I20260808 01:51:51.009773 15188 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:51.014086 14647 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:51.044941  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:51.045907  9478 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:51.047214  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:51.036480  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:51.038223  9601 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:51.016396 14987 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:51.026384 15100 raft_consensus.cc:3055] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:51:51.048887  9635 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
W20260808 01:51:51.056793  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:51.069296  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:51.084867  9474 raft_consensus.cc:2410] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260808 01:51:51.088228  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2855) took 1429 ms. Trace:
I20260808 01:51:51.088475  9629 raft_consensus.cc:2368] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15.
I20260808 01:51:51.088832  9474 rpcz_store.cc:276] 0808 01:51:49.658648 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:50.820850 (+1162202us) service_pool.cc:224] Handling call
0808 01:51:51.088155 (+267305us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:50.992228 15226 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.086946  9599 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:51.097102 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 153267us
I20260808 01:51:51.102324  9480 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:51.043046 14833 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:51.019111 15089 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:51.117117  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:51.118116  9478 raft_consensus.cc:2376] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12. Re-sending same reply.
I20260808 01:51:51.121177  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:51.122457  9550 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260808 01:51:51.083784 15100 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:51.122431  9474 raft_consensus.cc:2410] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260808 01:51:51.131065  9637 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
W20260808 01:51:51.137583  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:51.052446 14647 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.154652  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:51.135646  9554 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:51.156391  9601 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:51.121968 15247 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
W20260808 01:51:51.166230 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:51.051465 15176 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:51.170593  9550 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260808 01:51:51.170984  9478 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:51:51.185084  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:51.159375 15177 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:51.132803 14975 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:51.191810 14844 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:51.192755 14844 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:51.193979  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 80464.3us
I20260808 01:51:51.195075  9524 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:51.196264  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:51.197392 14924 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:51.198029 14924 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:51.198422 14924 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:51:51.199330  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 89344.4us
W20260808 01:51:51.199890  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 143595us
I20260808 01:51:51.200204 14844 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:51.200917 15241 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.227s	user 0.001s	sys 0.000s
I20260808 01:51:51.169601 15157 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:51.207674  9632 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:51:51.155238 15139 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:51.211529  9556 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 13: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:51.212275  9474 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260808 01:51:51.219340  9474 pending_rounds.cc:85] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 7
I20260808 01:51:51.213567  9476 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:51:51.187439 14659 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:51:51.189234 15177 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:51.219705  9474 pending_rounds.cc:107] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260808 01:51:51.223309  9478 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260808 01:51:51.202359 15157 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:51.225349  9474 raft_consensus.cc:2889] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:51.207736 14975 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:51.217260 15195 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:51.227175  9599 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:51.132797 15051 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.231702  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2871) took 1331 ms. Trace:
I20260808 01:51:51.236663 14975 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:51.240792 14647 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:51.241367 15157 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:51.252884  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:51.245993  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 851) took 1300 ms. Trace:
I20260808 01:51:51.249397  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2866) took 1351 ms. Trace:
I20260808 01:51:51.230293 15177 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:51.245116  9478 rpcz_store.cc:276] 0808 01:51:49.895348 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:51.185923 (+1290575us) service_pool.cc:224] Handling call
0808 01:51:51.216096 (+ 30173us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:51.226255 (+ 10159us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:51.226537 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:51.171598 14852 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.254949  9474 rpcz_store.cc:276] 0808 01:51:49.892246 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:51.167036 (+1274790us) service_pool.cc:224] Handling call
0808 01:51:51.210665 (+ 43629us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:51.242203 (+ 31538us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:51.242424 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:51.253633  9476 rpcz_store.cc:276] 0808 01:51:49.937237 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:51.220583 (+1283346us) service_pool.cc:224] Handling call
0808 01:51:51.237282 (+ 16699us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:51.237435 (+   153us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:51.237454 (+    19us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:51.237483 (+    29us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:51:51.237502 (+    19us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:51.237537 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:51.237626 (+    89us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:51:51.256094 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:51.221590 15139 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.257239 15096 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:51.262318 15001 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election won for term 14
W20260808 01:51:51.263010  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66887.9us
I20260808 01:51:51.243459 15195 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:51.183773 15231 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:51.226560 15209 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.259429 14647 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260808 01:51:51.281294  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 80959.9us
I20260808 01:51:51.277806  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:51.284166  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:51.284694 15176 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:51.285269  9634 raft_consensus.cc:2368] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
I20260808 01:51:51.285943 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:51.287144  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:51.280030 14784 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:51.288244 15096 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.288681 14784 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.290083  9524 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:51.272720 15195 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:51.287838  9474 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:51:51.281747 14647 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:51.295929  9524 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:51.297345  9637 raft_consensus.cc:2368] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
I20260808 01:51:51.290642 15096 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:51.335299  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 853) took 1397 ms. Trace:
I20260808 01:51:51.336006  9476 rpcz_store.cc:276] 0808 01:51:49.938317 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:51.275242 (+1336925us) service_pool.cc:224] Handling call
0808 01:51:51.291348 (+ 16106us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:51.333171 (+ 41823us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:51.334738 (+  1567us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:51.334836 (+    98us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:51:51.334896 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:51.335011 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:51.335207 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:51.286338 15042 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:51.305788 14647 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:51.259195 15100 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:51.299959 15089 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:51.357874  9472 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:51:51.358338  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 855) took 1419 ms. Trace:
I20260808 01:51:51.358798  9474 rpcz_store.cc:276] 0808 01:51:49.939495 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:51.338430 (+1398935us) service_pool.cc:224] Handling call
0808 01:51:51.348428 (+  9998us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:51.357954 (+  9526us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:51.357993 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:51.358043 (+    50us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:51:51.358077 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:51.358140 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:51.358275 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:51:51.369812 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 109430us
W20260808 01:51:51.361860 15044 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.623s	user 0.013s	sys 0.001s
W20260808 01:51:51.371598 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:51.369264  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 73810.6us
I20260808 01:51:51.372040  9474 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: []
I20260808 01:51:51.348385 14784 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:51.357508 15226 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:51.350162 15255 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:51.364959 15001 raft_consensus.cc:697] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:51.378106  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 858) took 1433 ms. Trace:
I20260808 01:51:51.380126  9474 rpcz_store.cc:276] 0808 01:51:49.941224 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:51.359583 (+1418359us) service_pool.cc:224] Handling call
0808 01:51:51.371798 (+ 12215us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:51.374207 (+  2409us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:51.374526 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:51.360116 15042 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:51.387125  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 91340.1us
I20260808 01:51:51.407557 15139 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:51.411855 15252 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.135s	user 0.000s	sys 0.000s
I20260808 01:51:51.412657  9477 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260808 01:51:51.413627  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2660) took 1391 ms. Trace:
I20260808 01:51:51.376262 15044 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.415153 15051 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:51.416165 15044 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:51.416399 14924 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.417135  9481 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 15: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:51.416831 15122 raft_consensus.cc:3055] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:51:51.418212 15122 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:51.419431  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2667) took 1260 ms. Trace:
I20260808 01:51:51.419966  9481 rpcz_store.cc:276] 0808 01:51:50.159265 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:50.713267 (+554002us) service_pool.cc:224] Handling call
0808 01:51:51.419164 (+705897us) spinlock_profiling.cc:243] Waited 667 ms on lock 0x7b5400392bd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:51.419358 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1534495488}
I20260808 01:51:51.416175  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2659) took 1394 ms. Trace:
I20260808 01:51:51.414266  9477 rpcz_store.cc:276] 0808 01:51:50.022342 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:51.411815 (+1389473us) service_pool.cc:224] Handling call
0808 01:51:51.412284 (+   469us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:51.413237 (+   953us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:51.413529 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:51.421720  9481 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:51.421584  9474 rpcz_store.cc:276] 0808 01:51:50.021973 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:51.381151 (+1359178us) service_pool.cc:224] Handling call
0808 01:51:51.415095 (+ 33944us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:51.415782 (+   687us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:51.415813 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:51.415882 (+    69us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:51.415918 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:51.415981 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:51.416113 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":131,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":88}
I20260808 01:51:51.422468  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2661) took 1399 ms. Trace:
I20260808 01:51:51.431238  9481 rpcz_store.cc:276] 0808 01:51:50.107575 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:51.421165 (+1313590us) service_pool.cc:224] Handling call
0808 01:51:51.421543 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:51.422173 (+   630us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:51.422405 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:51.380584 15255 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:51.443027 15042 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:51.443991 15248 log.cc:927] Time spent T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
W20260808 01:51:51.445101 15251 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.103s	user 0.002s	sys 0.000s
I20260808 01:51:51.420068 15122 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.447306 15001 consensus_queue.cc:237] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.476665 14833 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:51.435792 15188 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:51.457926  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:51.427894 15100 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:51.384203 15226 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:51.466382  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:51.433355 10334 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:51.503234 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 70018.4us
I20260808 01:51:51.427253 14924 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:51.468623  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:51.464269 15255 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:51.479544  9632 raft_consensus.cc:2368] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15.
I20260808 01:51:51.497254  9524 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:51.475625 15117 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.101s
I20260808 01:51:51.501729 15226 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:51.529634  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:51.505679 15042 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:51.536644  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 57671.1us
I20260808 01:51:51.538805  9481 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:51.539521  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2672) took 1366 ms. Trace:
I20260808 01:51:51.539834  9481 rpcz_store.cc:276] 0808 01:51:50.173566 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:51.531281 (+1357715us) service_pool.cc:224] Handling call
0808 01:51:51.538597 (+  7316us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:51.539258 (+   661us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:51.539464 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:51.509189 15209 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:51.521944 15157 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:51.523444 15188 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:51.528534 15051 raft_consensus.cc:2764] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260808 01:51:51.547363  9444 leader_election.cc:400] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15.
I20260808 01:51:51.533356 15226 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:51.537786 15122 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:51.563663  9475 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260808 01:51:51.564296 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:51.542760 14924 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:51:51.524608 14975 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.546759 15157 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:51.568766 14924 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:51.568769  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:51.577932  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:51.571277  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 229059us
I20260808 01:51:51.558799 15044 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:51.588654 15044 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:51.589159 15044 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:51.596877 14975 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:51.599466  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2867) took 1706 ms. Trace:
I20260808 01:51:51.600358  9473 rpcz_store.cc:276] 0808 01:51:49.893184 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:50.713719 (+820535us) service_pool.cc:224] Handling call
0808 01:51:51.599230 (+885511us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b540023aad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:51.599397 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":311660800}
I20260808 01:51:51.601644  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:51.585829  9634 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:51.603170  9474 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
W20260808 01:51:51.607275  9521 leader_election.cc:336] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.598s (SENT)
I20260808 01:51:51.587988 15182 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:51.608201  9521 leader_election.cc:336] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.635s (SENT)
I20260808 01:51:51.602349  9473 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 15
I20260808 01:51:51.608861  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 852) took 1671 ms. Trace:
I20260808 01:51:51.610798  9472 rpcz_store.cc:276] 0808 01:51:49.937863 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:50.916368 (+978505us) service_pool.cc:224] Handling call
0808 01:51:51.608657 (+692289us) spinlock_profiling.cc:243] Waited 24.9 ms on lock 0x7b54002a6cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:51.608794 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":57208064}
W20260808 01:51:51.609704  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:51.610263  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 73936.7us
I20260808 01:51:51.613461  9444 leader_election.cc:400] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
I20260808 01:51:51.613831  9444 leader_election.cc:403] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
W20260808 01:51:51.615696  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 51074.9us
I20260808 01:51:51.616732  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:51.618480  9524 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:51.615793 15176 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:51.619647 15176 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:51.620024 15176 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:51.622797 15078 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:51.623036  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 242058us
I20260808 01:51:51.623196 15139 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:51.624362 15139 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:51:51.623415 15078 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:51.579681  9599 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:51.608243  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:51.634343  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:51.635417  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:51.599619  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:51.636791  9638 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:51.637130 15096 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:51.637703 15096 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.638839  9473 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:51:51.640080 15096 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:51.645957 15096 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
W20260808 01:51:51.642030  9556 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 742) took 1590 ms (client timeout 1801 ms). Trace:
I20260808 01:51:51.644253 14647 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:51.646745  9556 rpcz_store.cc:269] 0808 01:51:50.051714 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:50.121013 (+ 69299us) service_pool.cc:224] Handling call
0808 01:51:51.641758 (+1520745us) spinlock_profiling.cc:243] Waited 812 ms on lock 0x7b540002ec54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:51.641930 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1867868416}
I20260808 01:51:51.688946 14647 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:51.640426 15246 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.424s	user 0.002s	sys 0.000s
I20260808 01:51:51.690423  9473 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260808 01:51:51.690546 15044 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.579962 15157 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:51.636310  9629 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:51.649927 10334 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:51.701174 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 51406.5us
W20260808 01:51:51.654203 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:51.702952 14988 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:51:51.703660 14988 raft_consensus.cc:697] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:51.705039 14988 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:51.706921 15188 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:51:51.709872  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:51.636288  9479 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260808 01:51:51.717775  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:51.719166 15044 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:51.719485  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:51.720459  9629 raft_consensus.cc:2410] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260808 01:51:51.721716  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:51.722642  9629 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 17: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:51:51.715519  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:51.739652  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:51.740053 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:51.740556  9551 raft_consensus.cc:2368] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13.
I20260808 01:51:51.571885 15203 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:51.546157 15042 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.747262  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:51.785084  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:51.787252  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:51.788563  9634 raft_consensus.cc:2368] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13.
W20260808 01:51:51.789700 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:51.792945 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:51:51.747390  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 167871us
I20260808 01:51:51.572804 14833 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:51.748294  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:51.750588  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2664) took 1580 ms. Trace:
I20260808 01:51:51.751426  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:51.752270  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 133911us
I20260808 01:51:51.755412  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:51.756018 15169 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:51.757030  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 121744us
I20260808 01:51:51.759872  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:51.625676 14924 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:51.761344  9638 raft_consensus.cc:2410] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260808 01:51:51.759176 15001 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:51.801539  9474 rpcz_store.cc:276] 0808 01:51:50.157743 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:51.429169 (+1271426us) service_pool.cc:224] Handling call
0808 01:51:51.602868 (+173699us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:51.607361 (+  4493us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:51.737741 (+130380us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b540023aad0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:51.737905 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":370337152}
I20260808 01:51:51.801931  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:51.801765  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:51.807552  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:51.808014 10362 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 13: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:51.808674  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:51.819607  9556 raft_consensus.cc:2368] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
I20260808 01:51:51.820564  9629 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:51:51.823086  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:51.823479 15117 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:51.824091 15117 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:51.824637  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:51.827113  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:51.827406  9554 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 15: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:51.636821 15182 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.169s
I20260808 01:51:51.746594 14647 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:51.717844 15195 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.847129 15001 raft_consensus.cc:697] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:51.848618 15001 consensus_queue.cc:237] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.849548 15195 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:51.850816 15169 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:51.853205 15169 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:51.853776 15169 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:51.856374  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47750.7us
I20260808 01:51:51.857270  9524 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:51.858131 15107 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:51:51.858927 15107 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:51.859524 15107 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:51:51.859700 15139 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:51.860332 15139 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.847025 15265 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:51:51.863194  9631 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 15: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:51.863878 15266 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:51.863103 15265 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:51.864288 15042 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:51.864478 15265 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:51.847268 14647 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:51.865738  9601 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:51.867832 14647 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:51.868580 14833 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:51.841698  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:51.871191  9558 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 10: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:51.872470  9524 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:51.873370 14987 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:51.873878 14987 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:51.874301 14987 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:51.877120 15117 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:51.862589 15139 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:51.883512 15139 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:51.883606  9559 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 6.
I20260808 01:51:51.885504  9637 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:51:51.854287 15169 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:51.864552 15266 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:51.890501 15266 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:51.893388 15169 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:51.826462 15176 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:51.897509 15176 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:51.900460 10362 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:51.901783 15188 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:51.902477 15188 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:51.903637 15210 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260808 01:51:51.897692 14987 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.874430  9524 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:51.904732 15188 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:51.905849  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:51.905853 15265 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:51.909335 15176 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:51.909833 15176 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:51.910259 15176 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:51.910369 15265 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:51.884035 15139 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:51:51.919737  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:51.921065 14987 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:51.921020 15078 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:51.927107  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:51.928074  9637 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:51.929123 15078 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:51.929345 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:51.938879  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:51.940649  9551 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 10: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:51.940881  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:51.930897  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:51.931226  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:51.942343  9554 raft_consensus.cc:2368] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15.
I20260808 01:51:51.931701  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:51.921963 15139 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:51.932161  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:51.934589  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 68990.5us
I20260808 01:51:51.930658  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:51.953042  9558 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:51:51.960318  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 347034us
W20260808 01:51:51.942961 15177 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.673s	user 0.013s	sys 0.000s
I20260808 01:51:51.959168  9599 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:51.982363  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 865) took 1050 ms. Trace:
I20260808 01:51:51.990401  9478 rpcz_store.cc:276] 0808 01:51:50.931452 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:51.624462 (+693010us) service_pool.cc:224] Handling call
0808 01:51:51.624836 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:51.981949 (+357113us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:51.981992 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:51.982039 (+    47us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:51:51.982074 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:51.982149 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:51.982280 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":62793,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":124,"thread_start_us":356629,"threads_started":1}
I20260808 01:51:51.995990  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 863) took 1909 ms. Trace:
W20260808 01:51:51.998200  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 92238.2us
I20260808 01:51:52.000180  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:51.996687  9477 rpcz_store.cc:276] 0808 01:51:50.072632 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:51.422177 (+1349545us) service_pool.cc:224] Handling call
0808 01:51:51.422993 (+   816us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:51.423581 (+   588us) raft_consensus.cc:1534] Early marking committed up to index 13
0808 01:51:51.532420 (+108839us) log.cc:844] Serialized 14 byte log entry
0808 01:51:51.981818 (+449398us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:51.981886 (+    68us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:51:51.981940 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:51.982021 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:51.982160 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3589,"prepare.queue_time_us":275428,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":7932,"thread_start_us":517418,"threads_started":2}
W20260808 01:51:52.008577  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 49496.4us
I20260808 01:51:51.935907 14852 consensus_queue.cc:1059] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.055s
I20260808 01:51:52.013228 15265 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:51:52.014271 14924 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:52.026885  9600 consensus_peers.cc:597] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c508218361c446898fede7c5682fa9e. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:51:52.028776 15270 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:52.031154 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:52.031392  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:52.046118  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 44428.9us
W20260808 01:51:52.035117  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 53845.8us
I20260808 01:51:52.049021  9444 leader_election.cc:400] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
I20260808 01:51:52.036296  9558 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:52.036931 14784 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:52.037696  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:52.039091  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:52.050304  9554 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:52.039868  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:52.051541  9555 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: []
I20260808 01:51:52.040937  9560 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260808 01:51:51.979095 15258 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.068s
I20260808 01:51:52.047780  9551 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:52.045327  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:52.065129 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:52.034279  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:52.050796  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:52.069422  9558 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 16: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:51:52.046857 15231 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.016s	sys 0.057s
I20260808 01:51:51.982075 15177 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:52.072533  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:52.073203 15117 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.007s
W20260808 01:51:52.083330 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 180944us
I20260808 01:51:52.083606  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:52.109663  9551 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:52.068410 15157 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:52.080878  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:52.086022 15051 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:52.073416 14784 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:52.116114 15051 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:52.117033  9552 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260808 01:51:52.077507 14975 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20260808 01:51:52.119153 15157 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:52.093014  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2673) took 1919 ms. Trace:
W20260808 01:51:52.074055 15231 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.016s	sys 0.058s
W20260808 01:51:52.100970 15255 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.556s	user 0.007s	sys 0.004s
I20260808 01:51:52.064711 15265 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:52.103665  9630 raft_consensus.cc:2368] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15.
I20260808 01:51:52.105108 10364 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:52.105383 15177 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:52.085052 15226 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.516s	user 0.012s	sys 0.000s
W20260808 01:51:52.107199  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 58401.9us
I20260808 01:51:52.113101 14924 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:52.115630 15100 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.611s	user 0.007s	sys 0.006s
I20260808 01:51:52.120743  9481 rpcz_store.cc:276] 0808 01:51:50.174105 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:51.540614 (+1366509us) service_pool.cc:224] Handling call
0808 01:51:51.557241 (+ 16627us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:51.607841 (+ 50600us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:51.607877 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:51.608487 (+   610us) log.cc:844] Serialized 26 byte log entry
0808 01:51:51.959854 (+351367us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:51.959918 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:51.960032 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:52.092723 (+132691us) raft_consensus.cc:1685] finished
0808 01:51:52.092750 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:52.092936 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6475,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":11516,"thread_start_us":350855,"threads_started":1,"wal-append.queue_time_us":481493}
I20260808 01:51:52.124069 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:52.128355  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:52.140487  9554 raft_consensus.cc:2410] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260808 01:51:52.119724 14975 raft_consensus.cc:3055] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:52.145208  9444 leader_election.cc:400] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
W20260808 01:51:52.149091 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:52.118494 14784 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:52.152258  9444 leader_election.cc:403] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
W20260808 01:51:52.152150 15122 consensus_meta.cc:287] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.580s	user 0.015s	sys 0.000s
I20260808 01:51:52.159557  9635 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:51:52.160873 10364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 725) took 1291 ms. Trace:
I20260808 01:51:52.139356 14987 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:51:52.167433  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:52.161440 10364 rpcz_store.cc:276] 0808 01:51:50.859620 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:50.860050 (+   430us) service_pool.cc:224] Handling call
0808 01:51:52.150638 (+1290588us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400681d54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:52.150836 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2795047680}
I20260808 01:51:52.126792 15255 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:52.177107  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 45584.3us
W20260808 01:51:52.191738  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 181026us
I20260808 01:51:52.151952 14975 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:52.235325 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 67352.4us
I20260808 01:51:52.144755 15226 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:52.231277 15122 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:52.236282 15105 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.004s
I20260808 01:51:52.239392 15265 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:52.240399 15096 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:52.259534 15277 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 14
I20260808 01:51:52.260432 14833 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:52.193019  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:52.165078 15051 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:52.265179 15278 raft_consensus.cc:3055] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:52.319787  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:52.322199  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2670) took 2149 ms. Trace:
W20260808 01:51:52.326150  9475 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.016s	sys 0.065s
I20260808 01:51:52.330135  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:52.343226  9475 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.016s	sys 0.066s
I20260808 01:51:52.338953 15278 raft_consensus.cc:740] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:52.328701 15096 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:52.344204 15273 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260808 01:51:52.346628 15096 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:52.350651  9554 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 15: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:52.339221 15078 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:52.354890  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 376949us
I20260808 01:51:52.352264  9634 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
W20260808 01:51:52.388896  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 195989us
I20260808 01:51:52.397764  9524 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:52.347028 14852 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:51:52.408432  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:52.422196 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:52.425684 15226 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:52.428380  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 108704us
I20260808 01:51:52.430209  9444 leader_election.cc:400] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13.
I20260808 01:51:52.412815 15096 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:51:52.340291 15122 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:52.430610  9444 leader_election.cc:403] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:51:52.408944 15195 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:52.441522 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:52.412061 14924 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election won for term 15
W20260808 01:51:52.445314  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:52.433959 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 363129us
I20260808 01:51:52.450114  9523 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:52.412842 14852 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:52.416483 15266 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:52.454175  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:52.489002  9550 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:52.466800 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:52.449064  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 376902us
I20260808 01:51:52.490352  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:52.339807 14975 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:52.489852 10366 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:52.454401 15100 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:51:52.459702  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:52.478772  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:52.481534 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:52.459345  9444 leader_election.cc:400] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15.
I20260808 01:51:52.502269  9444 leader_election.cc:403] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:51:52.452652 15226 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260808 01:51:52.486860 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:52.491053  9478 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:51:52.493216  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:52.392227 15078 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:52.495441  9476 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20260808 01:51:52.482453 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:52.501482 10364 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:52.509651  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:52.511214  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:52.512447  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:52.517021  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:52.517930  9554 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:52.518134  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:52.521888  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 100567us
I20260808 01:51:52.481112 15195 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:52.522632  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:52.485266 15266 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:52.515552 15096 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15.
I20260808 01:51:52.516650 14975 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:52.540689  9560 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260808 01:51:52.518790 14833 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260808 01:51:52.536156  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 77041.7us
I20260808 01:51:52.526049 15255 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:52.542733  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:52.543259 14833 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:52.550290 15258 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:51:52.550668 14833 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:52.548641  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:52.536480 15282 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.081s	user 0.001s	sys 0.001s
I20260808 01:51:52.553007  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:52.553469  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:52.549202 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:52.562907  9550 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 13: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:52.569584  9444 leader_election.cc:400] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15.
I20260808 01:51:52.570089  9444 leader_election.cc:403] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
W20260808 01:51:52.572504  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:52.546397 15291 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:52.582130  9550 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
W20260808 01:51:52.584514 15280 log.cc:927] Time spent T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260808 01:51:52.554772 15276 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:52.592713 15276 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:51:52.592962  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 195322us
I20260808 01:51:52.593303 15276 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:52.594651  9524 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:52.596400  9524 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:52.564183 15096 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:51:52.600561 15295 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:51:52.600854 15294 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:52.603752 15203 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:51:52.606652 15195 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:52.608340  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 68582.9us
W20260808 01:51:52.608994 15176 consensus_meta.cc:287] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.698s	user 0.011s	sys 0.002s
I20260808 01:51:52.609620  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:52.610121 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:52.558913 15188 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:52.578526 15284 log.cc:927] Time spent T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260808 01:51:52.615763  9550 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260808 01:51:52.580021 15255 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:52.525084 14852 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:52.579751 15105 raft_consensus.cc:2764] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260808 01:51:52.585273 15292 ts_tablet_manager.cc:933] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:52.586004 15231 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:52.639156 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:52.583134 14924 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:52.592981 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:52.555426 15001 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
I20260808 01:51:52.588201 15270 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Stepping down as leader of term 15
W20260808 01:51:52.650965 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 73101.4us
I20260808 01:51:52.636091 15122 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:52.628854 15255 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:52.644917 15270 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:52.668845  9478 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:52.639334 15231 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:51:52.670356  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 74088.5us
I20260808 01:51:52.609658 15176 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:52.687143  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 77686.9us
I20260808 01:51:52.620674 15083 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 12.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.042s
I20260808 01:51:52.631824 15188 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:52.655630 15195 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:52.685186  9524 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:52.652032 15105 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15.
W20260808 01:51:52.687966  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:52.686003 15255 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:52.701542  9556 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 13. Current term is 14. Ops: []
W20260808 01:51:52.700682  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 131355us
I20260808 01:51:52.686765 15231 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:52.705901  9559 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:52.706184  9444 leader_election.cc:400] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
I20260808 01:51:52.675734 15157 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.029s
W20260808 01:51:52.703074 15283 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.220s	user 0.002s	sys 0.000s
I20260808 01:51:52.700742 14852 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:52.722745 15270 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:52.723598  9524 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:51:52.724200 15287 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.205s	user 0.000s	sys 0.002s
I20260808 01:51:52.724329  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:52.725234 15293 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.135s	user 0.001s	sys 0.000s
W20260808 01:51:52.725656 15139 consensus_meta.cc:287] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.748s	user 0.009s	sys 0.005s
I20260808 01:51:52.706736 15078 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:52.725881 15266 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:52.726862 14987 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.551s	user 0.015s	sys 0.000s
I20260808 01:51:52.729059 15188 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:52.729487 15278 consensus_queue.cc:260] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:52.700142 15096 raft_consensus.cc:697] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:52.711702 15107 consensus_meta.cc:287] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.851s	user 0.018s	sys 0.000s
I20260808 01:51:52.745043  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:52.707572 15195 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:52.754513  9524 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:52.762822 15302 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:52.763650 15302 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:52.764163 15302 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:52.764400 15051 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:52.750648 14987 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:51:52.769917  9558 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260808 01:51:52.774963 15122 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Leader election won for term 9
I20260808 01:51:52.776288 15122 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:52.777503 15122 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:52.779502 10334 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:52.785779 15226 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:52.780206 15278 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:52.781468  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:52.783030 14833 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:52.785064  9524 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:52.785431  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 61218.2us
I20260808 01:51:52.792500  9473 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:51:52.751868 15277 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
I20260808 01:51:52.795209 15195 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:52.748721 15139 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:52.799690 15083 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:51:52.740872 15300 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:52.738891 15303 consensus_queue.cc:1059] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.057s
I20260808 01:51:52.786541 15226 raft_consensus.cc:697] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:52.815420 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:52.820397 15096 consensus_queue.cc:237] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:52.805428 15304 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:52.824653  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:52.828100  9560 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:52.830647 15304 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:52.831413 15304 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:52.833760 15304 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:52.711675 14924 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:52.838961 15270 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:52.842226  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:52.846268  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:51:52.824550 14833 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:52.852869 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:51:52.857800  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:52.862071 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 48134.6us
I20260808 01:51:52.786959 15107 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:52.867488  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 132074us
I20260808 01:51:52.869304 15265 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:51:52.869901  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:52.839866 14987 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:52.844127 15296 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:52.876667  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:52.854938 15083 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:52.855444 15177 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260808 01:51:52.882890  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:52.883033  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 98066.7us
I20260808 01:51:52.887180  9524 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:52.891597  9551 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17. Ops: []
I20260808 01:51:52.872375 15286 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:51:52.889963 15177 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:52.877781 15304 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:52.908181  9444 leader_election.cc:400] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13.
I20260808 01:51:52.895718 15294 consensus_queue.cc:1059] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260808 01:51:52.896884 15309 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
I20260808 01:51:52.873150 15265 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:52.870961 15226 consensus_queue.cc:237] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:52.902427  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:52.911249  9444 leader_election.cc:403] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260808 01:51:52.925065  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:52.926921 15078 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13.
I20260808 01:51:52.927821 15078 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:51:52.930898  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:52.931945  9472 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:52.933516  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:52.934538  9479 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:52.895360 15139 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:52.898396 15274 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.510s	user 0.002s	sys 0.002s
I20260808 01:51:52.926399  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:52.900447 15286 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:52.897445 15277 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260808 01:51:52.900475 15176 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:52.908043 15177 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:52.907763 15300 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260808 01:51:52.948977  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 106929us
I20260808 01:51:52.951758  9444 leader_election.cc:400] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
I20260808 01:51:52.956430  9477 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:52.964371  9524 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:52.965468  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:52.946400 15291 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260808 01:51:52.967881  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:52.891517 15296 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:52.974716 10362 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:52.948601 15286 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:51:52.342360  9480 rpcz_store.cc:276] 0808 01:51:50.172299 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:51.478401 (+1306102us) service_pool.cc:224] Handling call
0808 01:51:51.478801 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:51.497432 (+ 18631us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:51:51.571217 (+ 73785us) log.cc:844] Serialized 14 byte log entry
0808 01:51:52.321711 (+750494us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:52.321785 (+    74us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:52.321838 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:52.321916 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:52.322107 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":873,"prepare.queue_time_us":201864,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":13244,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":12163,"thread_start_us":509958,"threads_started":2}
I20260808 01:51:52.951207 15300 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:52.979565 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:52.979347  9523 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:52.981280  9523 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:52.966024 15301 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260808 01:51:52.982504 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:52.984449 15299 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:52.985224 15299 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:52.986078  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:52.977108 15083 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:52.987224 15139 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:52.975533 15176 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:52.991225  9481 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:52.992990 10362 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:51:52.994540  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:52.995299  9472 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:51:52.998311  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46821.5us
I20260808 01:51:52.999332  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:53.000571  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:53.003808  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:53.004500  9480 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:53.008414  9445 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:53.010239  9524 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:53.016832  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:53.017186  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:53.011144  9524 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:53.017920  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:52.990314 15294 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:53.021471  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:52.980773 15300 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:52.987716 15139 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:53.029618  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:52.993989 15176 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:53.012260 15078 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:53.041009  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:53.042574  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:53.046511  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:53.043462  9481 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:53.022279 15266 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:53.019379 15105 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260808 01:51:53.067427  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:53.029446 15294 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:52.993815 15313 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.067s
I20260808 01:51:53.035020 15291 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:53.041481 15176 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:53.078969  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:53.034612 15139 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:53.040413 15301 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260808 01:51:53.084867 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:53.079941  9472 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:53.087582  9481 raft_consensus.cc:2393] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 13: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:53.057260 15266 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:53.089892  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:53.092221 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:53.039353 15051 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20260808 01:51:53.069216 15105 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:53.023872 15276 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:53.055392 14784 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
I20260808 01:51:53.055678 15258 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260808 01:51:53.089084 15122 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:53.089759 15195 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260808 01:51:53.115788 15312 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260808 01:51:53.012562 15265 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:53.048413 14975 consensus_queue.cc:1059] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260808 01:51:53.078714 15294 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:53.085335 15107 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:53.102337  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:53.089658 14924 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election won for term 14
W20260808 01:51:53.097028  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 79998.4us
I20260808 01:51:53.093848  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:53.091711 15301 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:53.044145  9634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:53.296277  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:53.103796 15105 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:51:53.120026 15299 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:53.120090 15308 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.366s	user 0.002s	sys 0.000s
W20260808 01:51:53.286013 15100 consensus_meta.cc:287] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.778s	user 0.018s	sys 0.010s
W20260808 01:51:53.138319 15279 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.617s	user 0.002s	sys 0.002s
I20260808 01:51:53.140022  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:53.305887  9477 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:53.307170  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:53.095944  9472 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:51:53.124375 14988 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:51:53.115285 15298 raft_consensus.cc:3055] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 LEADER]: Stepping down as leader of term 13
W20260808 01:51:53.272921  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 226541us
I20260808 01:51:53.268553 15295 raft_consensus.cc:3055] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 LEADER]: Stepping down as leader of term 13
W20260808 01:51:53.268416 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 121310us
I20260808 01:51:53.340337  9550 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13. Ops: []
W20260808 01:51:53.290205  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 258665us
I20260808 01:51:53.290931  9480 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:53.343108  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:53.343852  9480 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:53.292567  9479 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:53.296669  9481 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:53.298431  9444 leader_election.cc:400] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15.
I20260808 01:51:53.097693 15266 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:53.286407 15182 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:53.312247  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:53.346979  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 48774.5us
I20260808 01:51:53.300102 15301 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:53.360803  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:53.311404 15100 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13.
W20260808 01:51:53.332031 15292 thread.cc:641] tablet-download-pool [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.008s	sys 0.025s
I20260808 01:51:53.361536  9472 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
W20260808 01:51:53.362396  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:53.335511 15295 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:53.356736  9551 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
W20260808 01:51:53.415589 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66662.8us
I20260808 01:51:53.363641  9557 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:53.416049  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:53.342729 15122 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:51:53.374795  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:53.328433 15298 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:53.343353 15277 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:53.360607  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:53.373270  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 362133us
I20260808 01:51:53.430114  9478 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
W20260808 01:51:53.421290 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 56141.1us
I20260808 01:51:53.346655 14924 raft_consensus.cc:697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:51:53.430811  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 70322.8us
I20260808 01:51:53.431597  9478 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:53.431716  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:53.430733  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:53.420152  9480 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:51:53.425137  9477 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
W20260808 01:51:53.380759 15292 thread.cc:608] tablet-download-pool [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.008s	sys 0.026s
W20260808 01:51:53.438337  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 48152.4us
I20260808 01:51:53.426403  9472 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:51:53.423352 15122 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:53.358152 15182 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:53.437187  9481 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:51:53.456521  9479 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:53.414921 15258 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:53.460286 14975 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:53.482900  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:53.483796  9474 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17. Ops: []
I20260808 01:51:53.489578  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 867) took 2555 ms. Trace:
I20260808 01:51:53.490151  9475 rpcz_store.cc:276] 0808 01:51:50.934655 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:51.631861 (+697206us) service_pool.cc:224] Handling call
0808 01:51:51.690137 (+ 58276us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:51.690609 (+   472us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:51.690637 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:51.691242 (+   605us) log.cc:844] Serialized 26 byte log entry
0808 01:51:52.343826 (+652584us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:51:52.343882 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:52.347722 (+  3840us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:53.137386 (+789664us) spinlock_profiling.cc:243] Waited 3.29 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:53.474577 (+337191us) raft_consensus.cc:1685] finished
0808 01:51:53.474599 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:53.489465 (+ 14866us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1421,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":85,"spinlock_wait_cycles":7564672,"thread_start_us":652098,"threads_started":1,"wal-append.queue_time_us":785693}
W20260808 01:51:53.492916  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 111234us
I20260808 01:51:53.455863 15100 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:51:53.504058  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 51719us
I20260808 01:51:53.425979 15157 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 12.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.046s
W20260808 01:51:53.506325  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:53.454797 15296 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:53.481107 15096 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:53.489174 15276 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:53.473102 15337 raft_consensus.cc:3055] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:53.534137  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:53.496573 14975 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:53.553066  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 92010.6us
I20260808 01:51:53.508922 15292 tablet_copy_client.cc:323] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.195:34565
I20260808 01:51:53.524904 15334 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260808 01:51:53.549305 15302 consensus_meta.cc:287] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.784s	user 0.015s	sys 0.004s
W20260808 01:51:53.554996 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:53.578066  9524 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:53.546664 15304 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.640s	user 0.013s	sys 0.001s
W20260808 01:51:53.546198 15278 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.746s	user 0.011s	sys 0.007s
I20260808 01:51:53.504040 15258 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:53.551874 14852 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.818s	user 0.012s	sys 0.025s
I20260808 01:51:53.597041  9524 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:53.551841 15231 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.819s	user 0.013s	sys 0.005s
I20260808 01:51:53.605032  9524 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:53.537809 15326 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:53.614951  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:53.598692 15255 consensus_meta.cc:287] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.889s	user 0.013s	sys 0.013s
I20260808 01:51:53.550088 15001 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260808 01:51:53.624346 15322 log.cc:927] Time spent T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
I20260808 01:51:53.624771 14987 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:53.543851 15295 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:53.540280 15337 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:53.618692 15231 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:53.574579 14924 consensus_queue.cc:237] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:53.638811 15331 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260808 01:51:53.639663 15305 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.767s	user 0.004s	sys 0.001s
W20260808 01:51:53.644177 15329 log.cc:927] Time spent T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.214s	user 0.001s	sys 0.000s
I20260808 01:51:53.645357 15299 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:53.646603 14975 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:53.648084 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:53.651773 15327 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.213s	user 0.000s	sys 0.001s
I20260808 01:51:53.626453 15258 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:53.656586 15177 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.688s	user 0.011s	sys 0.002s
I20260808 01:51:53.657442 15107 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:51:53.729478 15107 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:53.729959 15107 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:51:53.658639 15291 raft_consensus.cc:2955] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Committing config change with OpId 14.9: config changed from index -1 to 9, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:51:53.661262 15188 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:51:53.731284 15188 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:53.731823  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:53.667337 15321 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.364s	user 0.001s	sys 0.001s
W20260808 01:51:53.647980  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:53.622629  9524 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:53.632109 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:53.547372 15338 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.149s
I20260808 01:51:53.619051 15304 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260808 01:51:53.707767 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:53.712011  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 197432us
I20260808 01:51:53.528750 15298 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:53.580413 15139 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:53.606417 15302 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:53.683923 15334 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:53.655040 10366 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:53.714463 15299 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:53.735786 10364 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 11: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:51:53.747575  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 457938us
W20260808 01:51:53.752120  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 129616us
I20260808 01:51:53.769683  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:53.689733  9550 raft_consensus.cc:2955] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Committing config change with OpId 14.9: config changed from index -1 to 9, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
W20260808 01:51:53.724047 15286 consensus_meta.cc:287] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.741s	user 0.012s	sys 0.002s
W20260808 01:51:53.757539 15320 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.327s	user 0.002s	sys 0.000s
I20260808 01:51:53.712428 14987 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:53.758200 15314 consensus_queue.cc:1059] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.004s
I20260808 01:51:53.728849 15177 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:53.762933  9554 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:53.764603 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:53.682950 15255 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:53.611881 15122 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:53.780602 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 271892us
I20260808 01:51:53.807610  9558 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
W20260808 01:51:53.794133  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:53.791494  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 359826us
I20260808 01:51:53.801563  9477 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:53.804803  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:53.804141 10362 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:53.770200 15139 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:53.810990  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:53.814651  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:53.771237 15299 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:53.820912  9634 raft_consensus.cc:606] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:53.821494  9634 raft_consensus.cc:614] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:51:53.822079  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52505.2us
I20260808 01:51:53.823033  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:53.822219  9521 leader_election.cc:336] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.785s (SENT)
I20260808 01:51:53.780862 15345 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election won for term 10
W20260808 01:51:53.827047 15176 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.731s	user 0.015s	sys 0.000s
W20260808 01:51:53.875276  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 102898us
I20260808 01:51:53.818452 15334 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:53.820443 15304 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260808 01:51:53.878364  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 61039.6us
W20260808 01:51:53.878160  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 55224.6us
W20260808 01:51:53.894511  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:53.824982 15001 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260808 01:51:53.894145 15341 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.183s	user 0.001s	sys 0.000s
I20260808 01:51:53.829699 15347 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:51:53.801105 15286 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:53.915812 15348 consensus_queue.cc:1059] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.017s
I20260808 01:51:53.870589 15226 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:53.878172 15296 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260808 01:51:53.912957  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:53.938839 15231 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:53.922254 15203 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:53.930509 14987 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:53.905048 15345 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:51:53.925817 15335 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.199s	user 0.000s	sys 0.000s
W20260808 01:51:53.971119  9521 leader_election.cc:336] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.535s (SENT)
I20260808 01:51:53.954214 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:53.946290 15255 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:53.971746  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:53.973201 10332 leader_election.cc:336] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.593s (SENT)
I20260808 01:51:53.894395 15301 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:53.921144 15176 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:53.961007  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 51571.6us
I20260808 01:51:53.962924 15304 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260808 01:51:53.960531  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 47673us
I20260808 01:51:53.993955  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:53.999964  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 770) took 1140 ms. Trace:
I20260808 01:51:54.000531  9478 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260808 01:51:54.000471  9555 rpcz_store.cc:276] 0808 01:51:52.859574 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:52.872042 (+ 12468us) service_pool.cc:224] Handling call
0808 01:51:52.887321 (+ 15279us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:52.907690 (+ 20369us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:52.907725 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:52.917823 (+ 10098us) log.cc:844] Serialized 52 byte log entry
0808 01:51:53.350004 (+432181us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:51:53.350054 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:53.350124 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:53.999668 (+649544us) raft_consensus.cc:1685] finished
0808 01:51:53.999701 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:53.999867 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6892,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"thread_start_us":383188,"threads_started":1,"wal-append.queue_time_us":761367}
I20260808 01:51:53.931108 15337 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.002629  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:51:54.003319 15300 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.945s	user 0.012s	sys 0.003s
I20260808 01:51:54.003880 15337 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:54.003773 15300 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:54.006093 15333 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260808 01:51:54.006263 15300 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:54.006714 15255 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:54.008800 15286 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:54.009789  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:54.013937 10332 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.049s (SENT) [suppressed 42 similar messages]
I20260808 01:51:54.017160  9478 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:51:53.975608 15203 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:54.025110 15203 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:54.025652 15353 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:54.026100  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:54.026896  9472 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:53.925192 15334 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:53.989007  9524 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:54.036728  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47850.3us
I20260808 01:51:54.042596 15255 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:51:54.044305 15296 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:51:54.044832 15296 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:54.045084  9556 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: []
I20260808 01:51:54.045912 15296 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.6, Last appended by leader: 6, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.047511 15301 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:54.048831  9601 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:54.050596  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:54.051717 14784 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:54.052294 14784 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:54.051803 15255 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260808 01:51:54.052745 14784 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:54.053824  9473 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 6. Current term is 7. Ops: []
I20260808 01:51:54.037075 15177 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:54.037168  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:54.062644  9551 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:54.037520 15300 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:51:54.039001 15277 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:54.055446 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:54.053893  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:54.013620 15107 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.058043 15363 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:54.060393 15176 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:54.061017  9557 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:54.063863  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:54.089920  9551 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:54.065028 10332 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:54.092490  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:54.095257 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:53.972347 15226 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:54.083429 15100 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.575s	user 0.004s	sys 0.009s
I20260808 01:51:54.079308  9478 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
W20260808 01:51:54.086580  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 115508us
I20260808 01:51:54.088110  9524 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:54.061856 15270 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:54.102564  9601 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:54.103251 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:54.104854  9524 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:54.085217 15363 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:54.110666  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2722) took 1147 ms. Trace:
I20260808 01:51:54.111303  9480 rpcz_store.cc:276] 0808 01:51:52.963382 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:53.435503 (+472121us) service_pool.cc:224] Handling call
0808 01:51:53.436118 (+   615us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:53.436463 (+   345us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:53.436505 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:53.452859 (+ 16354us) log.cc:844] Serialized 26 byte log entry
0808 01:51:53.830081 (+377222us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:53.830135 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:53.830210 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:54.110280 (+280070us) raft_consensus.cc:1685] finished
0808 01:51:54.110315 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:54.110565 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":287431,"threads_started":1,"wal-append.queue_time_us":407648}
W20260808 01:51:54.104914 15105 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.767s	user 0.014s	sys 0.000s
I20260808 01:51:54.080535 15117 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.035s
I20260808 01:51:54.104385 15100 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:54.107931 15176 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:51:54.135629 10364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 918) took 1497 ms. Trace:
I20260808 01:51:54.136108  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:54.115561  9521 leader_election.cc:336] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.607s (SENT)
I20260808 01:51:54.137648 10364 rpcz_store.cc:276] 0808 01:51:52.638500 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:52.638851 (+   351us) service_pool.cc:224] Handling call
0808 01:51:54.135335 (+1496484us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54004c3454. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:54.135531 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2519785216}
I20260808 01:51:54.138711  9521 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:54.107753 15302 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:54.121753 15291 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:45650 (request call id 20) took 3754 ms. Trace:
I20260808 01:51:54.117393 15301 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:54.118413 15105 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:54.067937 10362 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:54.085008 15277 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.156595 15117 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:54.157223  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:54.157485 15270 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:54.159198  9633 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260808 01:51:54.182366  9523 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:54.160153 15280 log.cc:927] Time spent T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.448s	user 0.002s	sys 0.000s
I20260808 01:51:54.160739  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:54.160840  9630 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 11. Current term is 12. Ops: []
I20260808 01:51:54.162568  9637 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:54.163750  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:54.164100 15344 raft_consensus.cc:3055] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:51:54.129680 15296 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:51:54.165902 15343 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.408s	user 0.000s	sys 0.002s
I20260808 01:51:54.166788  9636 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:54.171392 15326 raft_consensus.cc:2764] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260808 01:51:54.174985  9555 raft_consensus.cc:2393] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 13: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:51:54.175464  9521 leader_election.cc:336] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.630s (SENT)
I20260808 01:51:54.176920  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:54.129957 15294 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.830s	user 0.005s	sys 0.009s
W20260808 01:51:54.133459 15319 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.840s	user 0.004s	sys 0.000s
W20260808 01:51:54.133375 15266 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.769s	user 0.013s	sys 0.000s
I20260808 01:51:54.164758  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:54.123170 15370 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:51:54.136353 15177 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:51:54.136181 15278 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:54.133656 15304 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:54.142997 15300 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:54.167858  9629 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
W20260808 01:51:54.211071  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 106317us
I20260808 01:51:54.192932 15302 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:54.193075 15348 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:54.218166 15117 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:54.237015  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34802 (request call id 66) took 1203 ms. Trace:
I20260808 01:51:54.237476  9634 rpcz_store.cc:276] 0808 01:51:53.033273 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:53.033473 (+   200us) service_pool.cc:224] Handling call
0808 01:51:54.236773 (+1203300us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x7b54003fb454. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4498dc33 00007f5d4998d41a 00007f5d42ee1934 00007f5d42ee18c2 00007f5d42ee17f1 00007f5d42ee176c 00007f5d42ee0a22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:54.236908 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1205152128}
I20260808 01:51:54.139122 14784 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.239667  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:54.240568 14784 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:54.240928  9475 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 12: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:54.241627 15051 raft_consensus.cc:3055] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:54.242092 15051 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:54.242980 15051 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.229038 15266 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.244251 15051 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:54.245743  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2723) took 1281 ms. Trace:
I20260808 01:51:54.245991 15266 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:54.220400 15100 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:54.247521 15100 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:54.247900 15100 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:54.249544  9600 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:54.250984  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3133) took 1235 ms. Trace:
I20260808 01:51:54.251566  9556 rpcz_store.cc:276] 0808 01:51:53.015841 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:53.016493 (+   652us) service_pool.cc:224] Handling call
0808 01:51:53.029213 (+ 12720us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:54.045657 (+1016444us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:54.250517 (+204860us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540008acd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:54.250905 (+   388us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2335671680}
I20260808 01:51:54.252518  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:54.253075  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 886) took 1689 ms. Trace:
I20260808 01:51:54.253201  9479 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:54.253381  9476 rpcz_store.cc:276] 0808 01:51:52.564013 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:52.575729 (+ 11716us) service_pool.cc:224] Handling call
0808 01:51:52.590645 (+ 14916us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:52.590918 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:52.590946 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:52.638851 (+ 47905us) log.cc:844] Serialized 26 byte log entry
0808 01:51:52.717573 (+ 78722us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:51:52.717636 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:52.717751 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:53.327320 (+609569us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:54.248099 (+920779us) spinlock_profiling.cc:243] Waited 28.7 ms on lock 0x7b24000ef198. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41713594 00007f5d4171169b 00007f5d418cd6d9 00007f5d418cd365 00007f5d418cc8c0 00007f5d4455a638 00007f5d4455ad98 00007f5d44532588 00007f5d4452d95a 00007f5d4452cda6
0808 01:51:54.252868 (+  4769us) raft_consensus.cc:1685] finished
0808 01:51:54.252887 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:54.253022 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":108458112,"thread_start_us":40946,"threads_started":1,"wal-append.queue_time_us":130430}
I20260808 01:51:54.254585  9631 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:54.223323 15107 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:54.108601 15345 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.257575  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:54.193435 15270 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.230779 10366 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:51:54.223546 15294 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:54.281075 15294 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:54.281260 15270 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:54.281992  9560 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:54.282130  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3129) took 1274 ms. Trace:
W20260808 01:51:54.232899  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:54.282876 15300 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.232641 14987 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:54.131210 15363 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.186502 15344 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:54.146298 15291 rpcz_store.cc:276] 0808 01:51:50.415579 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:50.416332 (+   753us) service_pool.cc:224] Handling call
0808 01:51:50.527810 (+111478us) log.cc:844] Serialized 506 byte log entry
0808 01:51:50.809258 (+281448us) spinlock_profiling.cc:243] Waited 2.68 ms on lock 0x7b18000cfb20. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d448eceeb 00007f5d44971f43 00007f5d44999b32 00007f5d4998bb39
0808 01:51:50.956176 (+146918us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b54000396a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490919e 00007f5d44913d10 00007f5d44913cb8 00007f5d44913c38 00007f5d44913bf3 00007f5d44912e7c 00007f5d4a837996 00007f5d4a832148 00007f5d4494fbab 00007f5d449526a0
0808 01:51:50.995906 (+ 39730us) spinlock_profiling.cc:243] Waited 48.3 ms on lock 0x7b54000396a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44913f06 00007f5d4490d87a 00007f5d4490caf4 00007f5d448ed28a 00007f5d448f4133 00007f5d448f40b9 00007f5d448f4049 00007f5d448f4011 00007f5d448f31dd 00000000003aa554
0808 01:51:54.121642 (+3125736us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32065,"raft.queue_time_us":214307,"raft.run_cpu_time_us":9134,"raft.run_wall_time_us":662860,"spinlock_wait_cycles":121066496,"thread_start_us":72369,"threads_started":1,"wal-append.queue_time_us":118635,"wal-append.run_cpu_time_us":7932,"wal-append.run_wall_time_us":1308251}
I20260808 01:51:54.221848  9638 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 16: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:51:54.222750 10332 leader_election.cc:336] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.801s (SENT)
I20260808 01:51:54.306600  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:54.316104 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:54.242491  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:54.255998  9477 rpcz_store.cc:276] 0808 01:51:52.963783 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:53.436671 (+472888us) service_pool.cc:224] Handling call
0808 01:51:53.436978 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:53.809503 (+372525us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:54.245088 (+435585us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x7b54002bb1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:54.245329 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":837816192}
I20260808 01:51:54.317299  9475 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:54.258319  9472 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:54.230625  9636 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:54.238550 15117 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:54.260421 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:54.260835 15105 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:54.336295  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:54.261377 15340 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:54.262320 10364 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:54.263017 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:54.263751  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:54.264496  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:54.229991 15176 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:54.267526 15226 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:54.268600  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:54.271721 15188 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:54.273319 15304 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.273862  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3134) took 1257 ms. Trace:
W20260808 01:51:54.275645 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:54.260303  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:54.276648 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:54.359144 10367 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:51:54.198438 15301 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.282699 15277 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:54.288836 15370 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:54.233884 15278 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.282546  9554 rpcz_store.cc:276] 0808 01:51:53.007541 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:53.007719 (+   178us) service_pool.cc:224] Handling call
0808 01:51:53.013282 (+  5563us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:53.810091 (+796809us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:54.281863 (+471772us) spinlock_profiling.cc:243] Waited 749 ms on lock 0x7b540008a7d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:54.282052 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1723454848}
I20260808 01:51:54.372072  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:54.234464  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:54.282155  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:54.304383 15348 raft_consensus.cc:697] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:51:54.308837  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 73543.8us
I20260808 01:51:54.312305  9479 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:54.387637  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:54.314383  9635 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:51:54.416910  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:54.314622  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:54.320998 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 98303.1us
I20260808 01:51:54.324343  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:54.328538  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:54.415573  9629 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:54.419822  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:54.304382 15332 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.836s	user 0.005s	sys 0.000s
I20260808 01:51:54.296993 14784 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:54.334323  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:54.327118  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:54.444101  9633 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:54.338316  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:54.308035 15344 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.6, Last appended by leader: 6, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.307346 14987 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:54.358301  9558 rpcz_store.cc:276] 0808 01:51:53.016172 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:53.056170 (+ 39998us) service_pool.cc:224] Handling call
0808 01:51:53.056707 (+   537us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:53.813587 (+756880us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:54.273541 (+459954us) spinlock_profiling.cc:243] Waited 749 ms on lock 0x7b5400382f54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:54.273788 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1722339584}
I20260808 01:51:54.370898 15340 consensus_queue.cc:1243] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 7201605bb2f54c949a4ebe6e9b3b4b2a log is divergent from this leader: its last log entry 10.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260808 01:51:54.337898 14833 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:54.367831  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:54.483567  9476 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 15: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:54.369218  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:54.483815  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:54.335587 15318 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:51:54.369508  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:54.326445 15376 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:54.486944 15376 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.292104 15314 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:54.306824 15299 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.390034  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:54.491986  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:54.491652 14987 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:54.493592  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:54.392421 15301 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:54.450855 15294 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.050s
I20260808 01:51:54.496398  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:54.497344  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 771) took 1637 ms. Trace:
I20260808 01:51:54.397603 15304 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:54.497857  9552 rpcz_store.cc:276] 0808 01:51:52.859950 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:52.860234 (+   284us) service_pool.cc:224] Handling call
0808 01:51:52.860573 (+   339us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:52.860801 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:52.860824 (+    23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:52.868908 (+  8084us) log.cc:844] Serialized 26 byte log entry
0808 01:51:53.273459 (+404551us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:51:53.273528 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:53.273640 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:54.303547 (+1029907us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:51:54.496560 (+193013us) raft_consensus.cc:1685] finished
0808 01:51:54.496586 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:54.497227 (+   641us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23810816,"thread_start_us":362934,"threads_started":1,"wal-append.queue_time_us":470779}
W20260808 01:51:54.398494  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:54.400156 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:51:54.500991  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:54.501407 10361 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:51:54.501637  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:54.398053 15348 consensus_queue.cc:237] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.393393 15231 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:51:54.401289  9481 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:51:54.508232  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3130) took 1493 ms. Trace:
I20260808 01:51:54.508666  9560 rpcz_store.cc:276] 0808 01:51:53.014853 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:53.016572 (+  1719us) service_pool.cc:224] Handling call
0808 01:51:54.507991 (+1491419us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54000b40d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:54.508165 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2907488512}
I20260808 01:51:54.398053 15278 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:54.407181 15176 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:54.410115  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:54.320185 15265 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
W20260808 01:51:54.388912  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:54.402601 15370 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.334666  9638 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260808 01:51:54.420368  9477 raft_consensus.cc:2368] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17.
I20260808 01:51:54.421164  9475 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
W20260808 01:51:54.423067 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:54.485780  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:54.424461 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 54240.3us
I20260808 01:51:54.421216  9554 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:54.440837 10367 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:51:54.385658 15177 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:51:54.447855  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:54.366483  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:54.500034  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:54.512246  9555 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 17: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:54.510651  9560 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13. Ops: []
W20260808 01:51:54.516319 10332 leader_election.cc:336] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.846s (SENT)
I20260808 01:51:54.514900 10360 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 13: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:54.479357 15344 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:54.484153  9634 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:51:54.521088 15177 raft_consensus.cc:3055] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Stepping down as leader of term 12
W20260808 01:51:54.422719  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:54.520718  9637 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:54.527226  9481 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
W20260808 01:51:54.521085 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:54.527616 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:54.518666  9521 leader_election.cc:336] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.719s (SENT)
I20260808 01:51:54.522192  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:54.446326 15277 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:54.515153  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:54.484073  9632 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:54.525483  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:54.529713  9631 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:51:54.526124  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:54.532644  9479 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:54.496259  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:54.548429  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:54.548825  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:54.541672  9521 leader_election.cc:336] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.658s (SENT)
I20260808 01:51:54.546180  9638 raft_consensus.cc:2376] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16. Re-sending same reply.
I20260808 01:51:54.549888  9521 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:54.550940  9560 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:54.552899  9629 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:51:54.553081  9637 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:51:54.553463  9478 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:54.555035  9479 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:54.555198  9481 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260808 01:51:54.556672  9630 raft_consensus.cc:2393] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 15: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:54.556902  9481 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260808 01:51:54.567437  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:54.560407  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:54.572444  9635 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:51:54.513734 15265 raft_consensus.cc:3055] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:51:54.558358  9558 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 13: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:51:54.577768 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:54.555908  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:54.566277  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:54.514747 14833 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:54.576191 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 58241.2us
I20260808 01:51:54.515515 15299 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:54.611791  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 51460.9us
I20260808 01:51:54.614557  9478 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
W20260808 01:51:54.575748  9521 leader_election.cc:336] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.840s (SENT)
I20260808 01:51:54.614779 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:54.577365  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:54.616310  9479 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260808 01:51:54.612973  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:54.580484  9481 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260808 01:51:54.589998  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:54.617331  9481 pending_rounds.cc:85] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 7
I20260808 01:51:54.618026  9481 pending_rounds.cc:107] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260808 01:51:54.618309  9481 raft_consensus.cc:2889] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:54.583830  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:54.613116  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:54.583148  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:54.520210  9648 tablet_copy_service.cc:140] P 259bcb3f0d254e8b9effac3277e654e0: Received BeginTabletCopySession request for tablet 7c508218361c446898fede7c5682fa9e from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:45658)
I20260808 01:51:54.536499 15177 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:54.559141 15278 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.081s
I20260808 01:51:54.558509 15277 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:54.576354 14784 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.079s
W20260808 01:51:54.621898  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:54.579367  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:54.629623  9633 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:51:54.725798  9634 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:51:54.559337 15345 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:51:54.580289 15340 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.200s
I20260808 01:51:54.757428 15105 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:54.580574 15318 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:51:54.695113 15369 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:54.695791 15182 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 12.13, Last appended by leader: 13, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:54.779768  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:54.782230  9634 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260808 01:51:54.782694  9637 raft_consensus.cc:2376] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12. Re-sending same reply.
I20260808 01:51:54.782965  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:54.641409  9478 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: []
I20260808 01:51:54.659392  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:54.623339  9555 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 5. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
W20260808 01:51:54.733520  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:54.623629  9481 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260808 01:51:54.694156 15313 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election won for term 17
W20260808 01:51:54.557495 15337 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.553s	user 0.014s	sys 0.000s
W20260808 01:51:54.697166 15360 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.762s	user 0.003s	sys 0.000s
I20260808 01:51:54.640887  9479 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:54.781292 15105 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.807927 15313 raft_consensus.cc:697] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:54.779796  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:54.675467 15353 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.649s	user 0.013s	sys 0.004s
I20260808 01:51:54.784341  9637 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260808 01:51:54.590538 15203 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.222s
I20260808 01:51:54.807864 15177 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:54.784214  9634 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260808 01:51:54.809680 15337 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:54.577297 15265 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:54.650945 15299 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:54.622622  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:54.681753  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 03b17f21031f4ff4828c85b7332fcc2d (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:54.574891 15304 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:54.590828 15301 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:51:54.834800  9630 raft_consensus.cc:2368] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17.
I20260808 01:51:54.623000 15051 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:54.810927  9633 raft_consensus.cc:3055] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:54.811525  9478 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 9. (term mismatch)
I20260808 01:51:54.844887  9478 pending_rounds.cc:85] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 8
I20260808 01:51:54.845341  9478 pending_rounds.cc:107] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20260808 01:51:54.845690  9478 raft_consensus.cc:2889] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:51:54.847620  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 439040d168824496acc076b1c280bde3: Illegal state: Replica 259bcb3f0d254e8b9effac3277e654e0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 17 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:54.848007  9478 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:54.536566 15100 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.651731  9648 tablet_copy_service.cc:161] P 259bcb3f0d254e8b9effac3277e654e0: Beginning new tablet copy session on tablet 7c508218361c446898fede7c5682fa9e from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:45658: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-7c508218361c446898fede7c5682fa9e
I20260808 01:51:54.853130 15100 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:54.784293  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:54.659798  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:54.854779  9629 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
W20260808 01:51:54.854974  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 70751.9us
I20260808 01:51:54.858844 14833 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:54.859233  9633 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:54.860582  9633 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:54.862916 15100 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:51:54.863662 15100 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:54.864719 15100 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:54.850359  9478 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260808 01:51:54.806484  9479 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:54.815019  9481 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260808 01:51:54.696516  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:54.872421  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:54.818089  9632 raft_consensus.cc:2393] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 15: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:54.873548  9629 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:51:54.874102 15376 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:54.809321 15353 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260808 01:51:54.874832 15379 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.171s	user 0.001s	sys 0.000s
I20260808 01:51:54.875746 15353 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 01:51:54.876662  9474 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:54.816879  9560 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:54.781904 15369 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:54.829422  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:54.813921 15105 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:54.665259 10332 leader_election.cc:336] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.428s (SENT)
W20260808 01:51:55.273020 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 607789us
I20260808 01:51:55.273020  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:55.274655 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:54.858330 15296 consensus_meta.cc:287] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.662s	user 0.013s	sys 0.000s
I20260808 01:51:54.880132  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:54.703051  9559 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:55.275352 15411 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:54.883140  9633 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:54.870200  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.283109  9630 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:55.284322  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.285748  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:54.826436 15278 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:51:54.874712  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:54.897631 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 282961us
W20260808 01:51:54.924340  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 4bce6ea2a8244811bf007638181a417a: Illegal state: Replica 6e7856bed1be43acaa821290d506ee97 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 16 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:51:54.880398 15299 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:54.927839  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:55.290725 15411 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:55.291828  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:54.933218  9558 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:55.294912  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:54.809303 15313 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:55.295994  9636 raft_consensus.cc:2376] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17. Re-sending same reply.
I20260808 01:51:55.297333  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:54.944633 15051 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:54.945403 15177 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:55.299665  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:55.299866 15182 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:55.301193  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:55.302707  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.303383  9636 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:51:55.305383  9633 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260808 01:51:54.945880 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:54.968024 15369 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:54.987090  9556 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:51:54.930562 15105 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.307984 15105 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.309157 15411 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:55.306872 15117 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
W20260808 01:51:55.021456  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:55.311878  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:55.312245  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:55.313287  9600 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:54.971968 15389 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.035226  9554 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17. Ops: []
I20260808 01:51:54.828471 15337 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.001775 15353 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:55.037320 15176 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:55.048009  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.049054  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3139) took 1360 ms. Trace:
I20260808 01:51:55.003657 15265 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.056283 15100 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.324074 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:55.072427  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:55.324081 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:55.075026  9552 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:55.075389  9481 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:55.090435 10366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 735) took 1146 ms. Trace:
I20260808 01:51:55.099185  9524 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:55.100157  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3138) took 1421 ms. Trace:
I20260808 01:51:55.326799  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:55.327344  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 228270us
I20260808 01:51:55.113169 15298 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:55.131405  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:55.135005  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:55.326838 10366 rpcz_store.cc:276] 0808 01:51:53.944612 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:53.944856 (+   244us) service_pool.cc:224] Handling call
0808 01:51:55.090247 (+1145391us) spinlock_profiling.cc:243] Waited 82.3 ms on lock 0x7b5400681d54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:55.090367 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":189198080}
I20260808 01:51:55.327219  9557 rpcz_store.cc:276] 0808 01:51:53.678605 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:53.678840 (+   235us) service_pool.cc:224] Handling call
0808 01:51:53.707504 (+ 28664us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:54.089431 (+381927us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:55.099605 (+1010174us) spinlock_profiling.cc:243] Waited 353 ms on lock 0x7b5400313dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:55.100051 (+   446us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":812479232}
W20260808 01:51:55.135464  9601 service_pool.cc:136] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:42824 dropped due to backpressure: service queue is full with 50 items [suppressed 1 similar messages]
I20260808 01:51:55.140578 15398 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:55.141819 15390 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.070s	user 0.002s	sys 0.001s
I20260808 01:51:55.352650  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:55.143519  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:55.354229  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:55.145164 15399 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.149166  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:55.153097  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.182152  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:55.359400  9555 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:55.185401  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:55.187088  9475 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:55.362049  9476 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 14: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:55.362865  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2875) took 1081 ms. Trace:
I20260808 01:51:55.362962  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2882) took 1039 ms. Trace:
I20260808 01:51:55.363791  9476 rpcz_store.cc:276] 0808 01:51:54.323781 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:55.184959 (+861178us) service_pool.cc:224] Handling call
0808 01:51:55.362863 (+177904us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:55.363511  9475 rpcz_store.cc:276] 0808 01:51:54.281226 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:55.186499 (+905273us) service_pool.cc:224] Handling call
0808 01:51:55.186877 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:55.362365 (+175488us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:55.362766 (+   401us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:55.187417  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:55.200330 15402 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.183524 15400 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.200501 15406 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:55.202811 15404 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:55.199386 15401 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:55.381728 15401 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.208997 15314 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:55.339562 15369 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:55.214483 14833 raft_consensus.cc:3055] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:55.215349 15363 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:55.225538 15304 raft_consensus.cc:697] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:51:55.236490 15396 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.166s	user 0.003s	sys 0.000s
I20260808 01:51:55.242365 15188 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:51:55.249178  9472 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260808 01:51:55.394189  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2732) took 1158 ms. Trace:
I20260808 01:51:55.252177  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2874) took 1119 ms. Trace:
I20260808 01:51:55.244633 15295 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:55.254508 15403 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.264068 15410 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:55.270963  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:55.394955  9480 rpcz_store.cc:276] 0808 01:51:54.132938 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:54.280934 (+147996us) service_pool.cc:224] Handling call
0808 01:51:54.281328 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:54.402164 (+120836us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:51:54.405649 (+  3485us) log.cc:844] Serialized 14 byte log entry
0808 01:51:54.485614 (+ 79965us) log.cc:844] Serialized 14 byte log entry
0808 01:51:54.485839 (+   225us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:54.485905 (+    66us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:54.485980 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:55.251785 (+765805us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b5400399254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:55.251868 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:55.252075 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":835,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":82,"raft.queue_time_us":85,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":113,"spinlock_wait_cycles":276915968,"thread_start_us":78916,"threads_started":1,"wal-append.queue_time_us":147334}
I20260808 01:51:55.271175  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.272908 15226 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:51:55.264683 15301 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:55.280236  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:55.389006 15314 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:54.808570 15277 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:55.287437  9631 raft_consensus.cc:2410] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260808 01:51:54.902084  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.306726  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.408396  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:55.419739  9473 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:51:55.420779  9633 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:51:55.420894  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2894) took 1069 ms. Trace:
I20260808 01:51:55.307286  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.294929  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:55.421484  9473 rpcz_store.cc:276] 0808 01:51:54.351636 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:55.148722 (+797086us) service_pool.cc:224] Handling call
0808 01:51:55.420804 (+272082us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:55.321921  9551 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:51:55.323557 15411 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.420802  9476 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:51:55.336512  9523 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:55.337061  9524 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:55.310411 15105 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:55.320988  9550 rpcz_store.cc:276] 0808 01:51:53.688391 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:53.688618 (+   227us) service_pool.cc:224] Handling call
0808 01:51:53.689202 (+   584us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:53.689539 (+   337us) raft_consensus.cc:1534] Early marking committed up to index 9
0808 01:51:53.975722 (+286183us) log.cc:844] Serialized 14 byte log entry
0808 01:51:55.048385 (+1072663us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b180041a3a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4453a6ac 00007f5d4186fe3b 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667
0808 01:51:55.048538 (+   153us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:55.048615 (+    77us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:51:55.048684 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:55.048785 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:55.048969 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12893,"raft.queue_time_us":65394,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":21931,"spinlock_wait_cycles":26708352,"thread_start_us":219902,"threads_started":2,"wal-append.queue_time_us":145654,"wal-append.run_cpu_time_us":1288,"wal-append.run_wall_time_us":1005726}
I20260808 01:51:55.340560  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:55.325976 10360 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:51:55.428011  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:55.337764  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:55.428325  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 91408.7us
I20260808 01:51:55.428761  9550 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:51:55.366205  9478 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:55.430233  9445 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:55.367044  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:55.429486  9557 raft_consensus.cc:2410] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 16 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260808 01:51:55.372079  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2527) took 1049 ms. Trace:
I20260808 01:51:55.431552  9638 rpcz_store.cc:276] 0808 01:51:54.322973 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:54.638348 (+315375us) service_pool.cc:224] Handling call
0808 01:51:55.152886 (+514538us) spinlock_profiling.cc:243] Waited 1.98 ms on lock 0x7b5800231698. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:51:55.371965 (+219079us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4561664}
I20260808 01:51:55.384490 15383 consensus_queue.cc:1059] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260808 01:51:55.432664 15411 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.433418 15325 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:55.433549 15383 raft_consensus.cc:3055] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Stepping down as leader of term 6
I20260808 01:51:55.434015 15325 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:55.434083 15383 raft_consensus.cc:740] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:55.434911 15304 consensus_queue.cc:237] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.435146 15325 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.376478  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.435987  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.437283  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:55.376510  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:55.320528 14784 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:55.437310 15301 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 14.6, Last appended by leader: 6, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.426685 15410 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:55.444169  9479 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:55.419129 15277 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:55.446018 15277 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:55.401633 15226 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:55.448343 15226 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.449487 15226 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:55.449621  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:55.398603 15401 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:55.324842 15100 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:55.381067 15391 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.212268  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:55.468828  9474 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:55.377374 15402 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.392346  9555 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:55.473227  9552 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:55.389993 15278 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:55.394435  9472 rpcz_store.cc:276] 0808 01:51:54.236162 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:55.248212 (+1012050us) service_pool.cc:224] Handling call
0808 01:51:55.248734 (+   522us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:55.393886 (+145152us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:55.394140 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:55.325194 15176 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:55.370534 15398 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:55.352937 15353 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:54.821031  9521 leader_election.cc:336] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.070s (SENT)
W20260808 01:51:55.496563  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 675529us
I20260808 01:51:55.456632 15105 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:55.457929 15414 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260808 01:51:55.388939 14833 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:55.326413 15389 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.387766 15404 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:55.407325 15403 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.389391 15369 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:55.422019  9636 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:51:55.422736  9558 raft_consensus.cc:2410] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260808 01:51:55.321921  9635 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:51:55.420250  9634 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:51:55.492403 15299 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:55.073995 15122 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:51:55.425583  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 89236.2us
I20260808 01:51:55.435227 15383 consensus_queue.cc:260] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:55.429493  9524 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:55.438498  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:55.373538  9599 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:55.375663  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 244391us
I20260808 01:51:55.462651  9479 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:51:55.467164  9478 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:55.418391 15294 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:55.389915 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:55.353283 15300 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:55.475242 15391 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:55.471936  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:55.450875 15376 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election won for term 12
W20260808 01:51:55.480400  9556 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2366) took 1339 ms (client timeout 1731 ms). Trace:
I20260808 01:51:55.387351 15400 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.483872  9472 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260808 01:51:55.435173 15411 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:55.494418  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:55.554672  9481 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:51:55.407661  9559 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:55.334978  9601 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:42824 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2577)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2564)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2578)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2591)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2583)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2596)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2566)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2562)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2584)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2563)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2592)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2556)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2593)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2581)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2586)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2575)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2568)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2551)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2570)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2585)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2560)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2594)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2580)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2573)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2590)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2561)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2567)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2559)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2582)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2553)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34802 (request call id 67)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2546)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2547)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2548)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2549)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2565)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2587)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2589)
Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:45650 (request call id 22)
I20260808 01:51:55.493530  9648 tablet_copy_source_session.cc:215] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:51:55.483853 15412 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260808 01:51:55.525234  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 95866.9us
W20260808 01:51:55.520504  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 147132us
I20260808 01:51:55.583968  9475 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260808 01:51:55.585655  9475 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260808 01:51:55.521865  9478 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260808 01:51:55.520104  9631 raft_consensus.cc:2393] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 13: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:55.527418  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.589664  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.378459 15399 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.536464 14784 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:55.533963  9474 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:55.605325 14784 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:55.536043  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.606384 14784 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.607074  9636 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:51:55.608165  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:55.608989  9636 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 15: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:55.538662  9630 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:55.611227  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.612157 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:55.612670  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.614018  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:55.543695  9556 rpcz_store.cc:269] 0808 01:51:54.141139 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:54.252634 (+111495us) service_pool.cc:224] Handling call
0808 01:51:55.480109 (+1227475us) spinlock_profiling.cc:243] Waited 596 ms on lock 0x7b540002d5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:55.480304 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1370305408}
I20260808 01:51:55.615669  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:55.616859  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:55.617580  9556 raft_consensus.cc:2368] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
I20260808 01:51:55.507846 15105 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:55.618620 15353 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:55.517913 15369 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:55.620493 15105 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:55.596585  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:55.545701 15294 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:55.553691  9477 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 15: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:55.624027 15391 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:55.384944 15406 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:55.626015  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:55.626930 15406 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:55.568760  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 65580.2us
I20260808 01:51:55.627096 15386 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260808 01:51:55.628120  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:55.628273 15386 consensus_queue.cc:1243] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 13.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
W20260808 01:51:55.628603  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:55.629051  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:55.629360 15386 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:51:55.629866 15386 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:55.630858 15291 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:55.630805 15278 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.6, Last appended by leader: 6, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.631371  9473 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260808 01:51:55.632212 15325 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:55.633018 15325 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:51:55.633910 15294 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:55.635746 15402 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:55.580902 15376 raft_consensus.cc:697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:55.636446 15404 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:55.637235 15376 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:55.640084 15176 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:55.567972 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:55.595050  9480 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:55.618423  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:55.641762  9556 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:51:55.572923  9551 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2361) took 1534 ms (client timeout 1692 ms). Trace:
I20260808 01:51:55.569199  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 785) took 1578 ms. Trace:
I20260808 01:51:55.599859  9472 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260808 01:51:55.642858  9554 rpcz_store.cc:276] 0808 01:51:53.991199 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:54.519432 (+528233us) service_pool.cc:224] Handling call
0808 01:51:54.519811 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:55.315714 (+795903us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:55.568826 (+253112us) spinlock_profiling.cc:243] Waited 515 ms on lock 0x7b5400217ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:55.569107 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1184697216}
I20260808 01:51:55.532639  9558 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:55.601541  9648 rpcz_store.cc:275] Call kudu.tserver.TabletCopyService.BeginTabletCopySession from 127.0.0.1:45658 (request call id 691) took 1100 ms. Trace:
I20260808 01:51:55.534539 15139 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:55.602245 15051 consensus_queue.cc:1059] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 01:51:55.604063 15389 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:55.604655 15415 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.237s	user 0.000s	sys 0.002s
I20260808 01:51:55.537559 15304 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.630502 15386 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:51:55.611454 10361 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260808 01:51:55.622542  9559 raft_consensus.cc:2393] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 13: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:55.624152  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2896) took 1256 ms. Trace:
I20260808 01:51:55.665506  9477 rpcz_store.cc:276] 0808 01:51:54.367712 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:55.272691 (+904979us) service_pool.cc:224] Handling call
0808 01:51:55.623899 (+351208us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b54000212d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:55.624081 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":628408576}
I20260808 01:51:55.631422 15291 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:55.671931  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 693) took 1150 ms. Trace:
I20260808 01:51:55.672335  9635 rpcz_store.cc:276] 0808 01:51:54.521860 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:54.573298 (+ 51438us) service_pool.cc:224] Handling call
0808 01:51:55.671766 (+1098468us) spinlock_profiling.cc:243] Waited 682 ms on lock 0x7b540008c854. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:55.671869 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1568574848}
I20260808 01:51:55.673710  9474 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:55.550808  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:55.645569  9630 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:55.597687  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.585935 15412 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:55.614244 15403 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:55.586378  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:55.648169  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:55.544582 15337 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:55.629904 15399 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:55.650965 10332 leader_election.cc:336] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.673s (SENT)
I20260808 01:51:55.645030  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:55.642719  9551 rpcz_store.cc:269] 0808 01:51:54.038632 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:54.092092 (+ 53460us) service_pool.cc:224] Handling call
0808 01:51:55.572690 (+1480598us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400325ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:55.572841 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2721163008}
I20260808 01:51:55.640681  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.543038 15300 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.659344 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:55.659149 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:55.660614  9632 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:55.661752  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:55.637270 14833 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.715221  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.715446  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:55.675989  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.571740  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3147) took 1052 ms. Trace:
I20260808 01:51:55.716215  9631 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:55.650188 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:55.716501  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:55.598361 15411 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.626931 15340 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.032s
I20260808 01:51:55.707239 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:55.705502 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:55.728596  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 43502.3us
I20260808 01:51:55.682319  9634 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 15: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:51:55.682253  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 99546.8us
I20260808 01:51:55.729190  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:55.681242  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.731098  9635 raft_consensus.cc:2410] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260808 01:51:55.675616 15353 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.716428  9556 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:55.681125 15389 raft_consensus.cc:3055] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:55.719627  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.745568  9637 raft_consensus.cc:2410] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
W20260808 01:51:55.678792  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:55.745869  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:55.716501  9552 rpcz_store.cc:276] 0808 01:51:54.519601 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:55.445865 (+926264us) service_pool.cc:224] Handling call
0808 01:51:55.473032 (+ 27167us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:55.476032 (+  3000us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:55.571435 (+ 95403us) spinlock_profiling.cc:243] Waited 26.8 ms on lock 0x7b5400258350. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:55.571649 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":61627648}
W20260808 01:51:55.710861  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:55.667266 15139 raft_consensus.cc:697] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:51:55.732676 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 81754.3us
I20260808 01:51:55.730223  9524 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:55.748553  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:55.749399 10332 leader_election.cc:336] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.735s (SENT)
I20260808 01:51:55.750540  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:55.739799  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:55.751037 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:55.740679  9550 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:55.718542 15416 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.717736  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.708755 15294 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election won for term 17
W20260808 01:51:55.801970  9521 leader_election.cc:336] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.623s (SENT)
W20260808 01:51:55.805738 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 53612.6us
I20260808 01:51:55.719197 15328 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.807073  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:55.806730  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:55.816601  9372 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55058 (request call id 129) took 26204 ms (client timeout 15000 ms). Trace:
I20260808 01:51:55.809242 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:55.679849 15412 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:51:55.761792  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:55.810487  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:55.818174  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:55.813495  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 83370.3us
I20260808 01:51:55.819768  9638 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
W20260808 01:51:55.811223  9521 leader_election.cc:336] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.702s (SENT)
I20260808 01:51:55.671559 15304 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.822566  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:55.829986  9631 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 13: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:51:55.822415  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 80055.6us
I20260808 01:51:55.816385  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:55.723795 15325 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:55.653112  9648 rpcz_store.cc:276] 0808 01:51:54.501382 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:54.519893 (+ 18511us) service_pool.cc:224] Handling call
0808 01:51:55.426135 (+906242us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b5400420ed4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4499d7d7 00007f5d49964e46 00007f5d4996e84e 00007f5d411e261b 00007f5d4996e7ca 00007f5d499698dc 00007f5d49964a8b 00007f5d499596e1 00007f5d48a0c5d4 00007f5d48a0c562
0808 01:51:55.601436 (+175301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":566120832}
I20260808 01:51:55.828469  9636 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:55.831770  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:55.835074  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:55.826040  9524 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:55.724627 15177 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.092s
I20260808 01:51:55.753930 15401 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.854291 15401 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.858028 15401 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:55.845997  9634 raft_consensus.cc:2393] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 13: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:55.859766  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:55.860589  9634 raft_consensus.cc:2376] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14. Re-sending same reply.
I20260808 01:51:55.753361 15117 raft_consensus.cc:3055] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:55.841292  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.848317  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:55.844420  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:55.745025 15389 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:55.746176 15083 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:55.753634 15399 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.850618  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.861811  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.865554 10367 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:51:55.817989 15334 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.867971  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:55.877722 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:55.736091 15403 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.115s
I20260808 01:51:55.882462  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:55.889969  9638 raft_consensus.cc:2376] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16. Re-sending same reply.
W20260808 01:51:55.817032  9372 rpcz_store.cc:269] 0808 01:51:29.601382 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:29.601968 (+   586us) service_pool.cc:224] Handling call
0808 01:51:55.805537 (+26203569us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:51:50.683270 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:51:50.698643 (+ 15373us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:51:50.698734 (+    91us) write_op.cc:435] Acquired schema lock
0808 01:51:50.698758 (+    24us) tablet.cc:662] Decoding operations
0808 01:51:50.700169 (+  1411us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:51:50.700300 (+   131us) write_op.cc:641] Partition lock acquired for write op
0808 01:51:50.700328 (+    28us) tablet.cc:685] Acquiring locks for 14 operations
0808 01:51:50.701838 (+  1510us) tablet.cc:701] Row locks acquired
0808 01:51:50.705223 (+  3385us) write_op.cc:260] PREPARE: finished
0808 01:51:50.705451 (+   228us) write_op.cc:270] Start()
0808 01:51:50.705644 (+   193us) write_op.cc:276] Timestamp: P: 1786153910705374 usec, L: 0
0808 01:51:50.705709 (+    65us) op_driver.cc:348] REPLICATION: starting
0808 01:51:50.706627 (+   918us) log.cc:844] Serialized 5512 byte log entry
0808 01:51:51.386401 (+679774us) op_driver.cc:464] REPLICATION: finished
0808 01:51:51.711800 (+325399us) write_op.cc:301] APPLY: starting
0808 01:51:51.712004 (+   204us) tablet.cc:1367] starting BulkCheckPresence
0808 01:51:51.937856 (+225852us) tablet.cc:1370] finished BulkCheckPresence
0808 01:51:51.942468 (+  4612us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:51:53.025410 (+1082942us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:51:53.131290 (+105880us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=28,mrs_lookups=0
0808 01:51:53.131406 (+   116us) write_op.cc:312] APPLY: finished
0808 01:51:53.132109 (+   703us) log.cc:844] Serialized 129 byte log entry
0808 01:51:53.771936 (+639827us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:51:53.805724 (+ 33788us) write_op.cc:454] Released schema lock
0808 01:51:53.818903 (+ 13179us) write_op.cc:341] FINISH: Updating metrics
0808 01:51:53.927655 (+108752us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:51:53.927695 (+    40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4324870,"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":273588,"apply.run_cpu_time_us":64517,"apply.run_wall_time_us":2172425,"cfile_cache_hit":58,"cfile_cache_hit_bytes":62731,"num_ops":14,"prepare.queue_time_us":185380,"prepare.run_cpu_time_us":18444,"prepare.run_wall_time_us":146311,"raft.queue_time_us":253225,"raft.run_cpu_time_us":37102,"raft.run_wall_time_us":185447,"replication_time_us":669147,"thread_start_us":937953,"threads_started":5,"wal-append.queue_time_us":797332,"wal-append.run_cpu_time_us":39456,"wal-append.run_wall_time_us":2458531}]]}
I20260808 01:51:55.891072  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.820881 15416 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.892131 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:55.744787 15347 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.121s
I20260808 01:51:55.850981 15337 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:51:55.833726 15176 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:51:55.732792 15411 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.749835 15353 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.894001  9638 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 15: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:55.865096 15370 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election won for term 14
W20260808 01:51:55.894531  9521 leader_election.cc:336] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.788s (SENT)
I20260808 01:51:55.876812  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:55.895049  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.845759 15294 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:55.873842  9634 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
W20260808 01:51:55.898603  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46962.2us
W20260808 01:51:55.910476  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 67806us
I20260808 01:51:55.871716 15177 consensus_queue.cc:1243] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 14.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260808 01:51:55.901808  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:55.819116 15291 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.883368 15313 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.873044 15100 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:55.848213 15299 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260808 01:51:55.908790  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.831218 15328 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:55.866716 15117 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:55.916630  9521 leader_election.cc:336] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.813s (SENT)
I20260808 01:51:55.914594 15400 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:55.906306 15188 consensus_meta.cc:287] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.504s	user 0.010s	sys 0.002s
I20260808 01:51:55.887131 15399 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.883061 15325 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260808 01:51:55.900746 15176 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:55.926612  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.913645  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:55.936210  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:55.971692  9481 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 14
W20260808 01:51:55.938568  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:55.941473 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:55.925153 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:55.896543 15334 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:55.951560  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 74882.5us
I20260808 01:51:55.986310  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:55.917481  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:55.918393 15404 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:51:55.952174 15388 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.953795  9477 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:55.921564 15398 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:55.905133 15139 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.006492  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:55.961328 15363 consensus_meta.cc:287] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.565s	user 0.012s	sys 0.000s
I20260808 01:51:55.962944 15416 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:55.963325 15295 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.566s	user 0.006s	sys 0.006s
I20260808 01:51:55.926335 15313 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:55.964478 15417 log.cc:927] Time spent T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.231s	user 0.001s	sys 0.000s
W20260808 01:51:55.953384 15226 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.503s	user 0.010s	sys 0.005s
I20260808 01:51:55.964973 15291 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:51:55.965291 15422 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.210s	user 0.000s	sys 0.000s
W20260808 01:51:55.966024 15425 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.140s	user 0.000s	sys 0.001s
I20260808 01:51:56.038285 15410 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:56.038791 14924 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:56.039484 14924 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:55.966995 10366 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:55.910147 15051 consensus_queue.cc:1059] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
I20260808 01:51:55.974789  9638 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 12: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:56.042749  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:56.042883  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:56.043638  9638 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 13: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:56.043593  9480 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:51:56.044785  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:55.984436  9521 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:55.988999 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:55.994231  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:55.998765  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:55.957491 15100 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:55.968837 15231 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:55.973371 15176 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:56.031318  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:56.029687 15296 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:56.033977 15426 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
I20260808 01:51:56.036003 15334 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:56.046357  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62055.9us
I20260808 01:51:56.064093  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:56.069816  9638 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:51:56.070245  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:56.049247  9480 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:56.057734  9481 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:56.061074  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:56.062198  9634 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:56.065528  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:56.065541  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 71431.5us
I20260808 01:51:56.073544  9477 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:51:56.010118 15403 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:51:56.076414  9601 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:56.076902  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:56.072093  9521 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:56.074965  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 163753us
I20260808 01:51:56.084185  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:56.089027  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:56.089010  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2553) took 1056 ms. Trace:
I20260808 01:51:56.090850  9638 rpcz_store.cc:276] 0808 01:51:55.032823 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:56.083820 (+1050997us) service_pool.cc:224] Handling call
0808 01:51:56.088915 (+  5095us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:56.092470  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:56.097625  9638 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:56.079969 15226 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:51:56.076834 15414 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:56.074896 15300 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:56.104557  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:56.109941  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:56.050781 15363 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.040s
I20260808 01:51:56.112274  9524 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:56.080210 15270 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:56.112417  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:56.006297 15294 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.084295 15443 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:56.079806 15304 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:56.097496 15353 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:56.064874 15231 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:56.100562 15370 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:56.007882 15388 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.068130 15296 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.084769 15298 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260808 01:51:56.161821  9629 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:51:56.089859 15434 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260808 01:51:56.154419  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:56.068344 15182 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
I20260808 01:51:56.119305  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:56.153959  9524 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:56.101193  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:56.158188 15300 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election won for term 14
W20260808 01:51:56.171145 15437 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
I20260808 01:51:56.171294 15398 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:56.171453  9634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "244e080726394210b5467bf688813cf0"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:56.172171  9634 raft_consensus.cc:606] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:56.163501 15083 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.009s
I20260808 01:51:56.172592  9634 raft_consensus.cc:614] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:56.172930 15083 consensus_queue.cc:1243] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 13.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260808 01:51:56.122872 15325 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20260808 01:51:56.174044  9634 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 13. Current term is 14. Ops: []
W20260808 01:51:56.174834  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 244e080726394210b5467bf688813cf0 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:56.159292 15286 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.021s
I20260808 01:51:56.094694 15435 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260808 01:51:56.045301 15117 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.111630 15100 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:56.112212 15122 raft_consensus.cc:3055] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:56.117379 15347 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:56.096786 15278 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.062s
I20260808 01:51:56.097033 15318 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.063s
I20260808 01:51:56.153168 15411 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:56.178181 15399 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:56.162161 15298 raft_consensus.cc:3055] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:51:56.155196 15401 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election won for term 13
W20260808 01:51:56.170113  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 51121.7us
I20260808 01:51:56.267117  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:56.179427  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2546) took 1213 ms. Trace:
W20260808 01:51:56.156782  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 80466.9us
I20260808 01:51:56.183190 15300 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:56.183883 15226 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:51:56.184804  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:56.272627  9478 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
W20260808 01:51:56.185362  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:56.273960  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:56.152444 15270 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:56.158380 15295 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
W20260808 01:51:56.209259  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:56.282467  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:56.223835  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:56.283725  9521 leader_election.cc:336] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.694s (SENT)
I20260808 01:51:56.265463 15298 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:56.255559 15188 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:51:56.297794 15188 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:51:56.269824  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:56.300595 15270 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:56.303122 15446 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:56.303797 15446 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:56.263492 15429 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.339s	user 0.001s	sys 0.001s
I20260808 01:51:56.268074  9634 rpcz_store.cc:276] 0808 01:51:54.961875 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:56.173536 (+1211661us) service_pool.cc:224] Handling call
0808 01:51:56.173864 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:56.174453 (+   589us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:56.174628 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:56.269781  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:56.322731  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 53033.7us
I20260808 01:51:56.323734  9601 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:56.196622 15122 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:56.271342 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 154946us
I20260808 01:51:56.161038 15443 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.276715  9444 leader_election.cc:400] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
W20260808 01:51:56.284817  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:56.156080  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:56.271452 15401 raft_consensus.cc:697] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:56.286404  9524 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:56.272192 15226 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:56.292155  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:56.294872  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:56.326607 10332 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:51:56.304438 15386 consensus_meta.cc:287] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.619s	user 0.010s	sys 0.001s
I20260808 01:51:56.283116 15411 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:56.330302  9444 leader_election.cc:403] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:51:56.328794  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:56.278358 15363 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:56.255229 15347 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:56.273268 15370 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.325088  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:56.511629 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:56.360823  9634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:56.454769  9445 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:51:56.473762  9630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2557) took 1396 ms (client timeout 1630 ms). Trace:
W20260808 01:51:56.338832  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:56.495646 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:56.496987 10362 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
W20260808 01:51:56.379886  9632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2588) took 1279 ms (client timeout 1683 ms). Trace:
I20260808 01:51:56.331466  9638 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:56.520349 10366 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:56.398168  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:56.519655  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 65049.1us
I20260808 01:51:56.410766 10363 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260808 01:51:56.459781 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:56.519441  9630 rpcz_store.cc:269] 0808 01:51:55.077286 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:55.285430 (+208144us) service_pool.cc:224] Handling call
0808 01:51:56.473450 (+1188020us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54000dfa54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:56.473659 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":580234624}
I20260808 01:51:56.395380  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3150) took 1872 ms. Trace:
I20260808 01:51:56.524408  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2547) took 1562 ms. Trace:
I20260808 01:51:56.335846  9637 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
W20260808 01:51:56.518368  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:56.525098  9638 rpcz_store.cc:276] 0808 01:51:54.962440 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:56.330846 (+1368406us) service_pool.cc:224] Handling call
0808 01:51:56.331210 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:56.524129 (+192919us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:56.524348 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:56.330302  9477 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:56.393143  9631 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:56.531291  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2548) took 1568 ms. Trace:
W20260808 01:51:56.520807  9632 rpcz_store.cc:269] 0808 01:51:55.100650 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:55.352280 (+251630us) service_pool.cc:224] Handling call
0808 01:51:56.379608 (+1027328us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b540032c654. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:56.379795 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":574513024}
I20260808 01:51:56.531749  9631 rpcz_store.cc:276] 0808 01:51:54.962884 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:56.392327 (+1429443us) service_pool.cc:224] Handling call
0808 01:51:56.392875 (+   548us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:56.531029 (+138154us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:56.531223 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:56.533056  9632 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13. Ops: []
W20260808 01:51:56.516083  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 239570us
I20260808 01:51:56.533927  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2565) took 1442 ms. Trace:
I20260808 01:51:56.534621  9632 rpcz_store.cc:276] 0808 01:51:55.091302 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:56.532336 (+1441034us) service_pool.cc:224] Handling call
0808 01:51:56.532802 (+   466us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:56.533656 (+   854us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:56.533876 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:56.525321  9480 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
W20260808 01:51:56.537554  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 212540us
I20260808 01:51:56.538336  9601 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:56.539758  9601 leader_election.cc:400] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17.
I20260808 01:51:56.540058  9601 leader_election.cc:403] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260808 01:51:56.321058 15100 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:56.329985 15226 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:51:56.328410 15399 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:56.388473 15443 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:56.405913 14924 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:56.542194  9631 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260808 01:51:56.524428  9550 rpcz_store.cc:276] 0808 01:51:54.526628 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:55.451498 (+924870us) service_pool.cc:224] Handling call
0808 01:51:55.451821 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:55.806105 (+354284us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:56.395114 (+589009us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x7b54001acf54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:56.395305 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":663679872}
W20260808 01:51:56.386735 15450 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260808 01:51:56.324520 15270 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:56.382010 15176 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.563074 15270 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.564798 15176 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:56.565529 15270 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:56.548878 15100 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:56.392261 15314 consensus_meta.cc:287] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.965s	user 0.006s	sys 0.007s
I20260808 01:51:56.567123 15314 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:56.567812 15100 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:56.342597 15188 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:56.568609 15454 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:51:56.569778 15314 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:56.570799  9559 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: []
I20260808 01:51:56.523360 15363 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.329991 15291 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:56.573104 15363 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:56.566390 15453 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:51:56.360489 15334 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:51:56.522974 15386 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.574187 15334 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:56.409961 15449 raft_consensus.cc:3055] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:51:56.574807  9634 raft_consensus.cc:606] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:56.575229 15449 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:56.574333 15453 raft_consensus.cc:697] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:51:56.406147 15369 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.787s	user 0.012s	sys 0.000s
I20260808 01:51:56.576030 15386 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:56.576259 15369 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.553547 15399 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:56.396575 15298 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:56.577732  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:56.333264 15391 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:56.578661 15298 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:56.404460 15446 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:56.579077 15391 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:56.495726  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:56.387364 15436 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:56.433378 15231 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:56.584329 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 257798us
W20260808 01:51:56.585851  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:56.386819 15277 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:56.325100 15398 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:56.552460  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2589) took 1450 ms. Trace:
I20260808 01:51:56.553721  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:56.528332 15304 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:56.594398  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:56.386147 15448 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260808 01:51:56.386735 15122 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.544305 15177 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260808 01:51:56.544525 15451 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:56.606643 15122 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:56.607098 15391 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:56.607503 15292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c508218361c446898fede7c5682fa9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:51:56.575340  9634 raft_consensus.cc:614] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:56.575738  9633 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:56.608631  9636 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 17: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:56.576699 15453 consensus_queue.cc:237] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:56.580274  9474 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:56.592026 15436 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.611297  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:56.613497 15436 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:56.579097 15399 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:56.387032 15444 log.cc:927] Time spent T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260808 01:51:56.608069 15454 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.007s
I20260808 01:51:56.615199 15454 consensus_queue.cc:1243] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260808 01:51:56.583803 15369 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:56.556874  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2587) took 1458 ms. Trace:
I20260808 01:51:56.549202  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:56.574718 15334 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:51:56.546777  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:56.621462  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:56.581053  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:56.592589  9632 rpcz_store.cc:276] 0808 01:51:55.102245 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:56.535846 (+1433601us) service_pool.cc:224] Handling call
0808 01:51:56.551701 (+ 15855us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:56.552106 (+   405us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:56.552134 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:56.552181 (+    47us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:56.552200 (+    19us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:56.552246 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:56.552394 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1843,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":14796}
I20260808 01:51:56.522709 15278 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
I20260808 01:51:56.550361 15435 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17.
I20260808 01:51:56.596571  9478 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:51:56.594425  9601 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:56.595568 15291 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260808 01:51:56.618314  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 69209.2us
I20260808 01:51:56.617929  9631 rpcz_store.cc:276] 0808 01:51:55.098361 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:56.533214 (+1434853us) service_pool.cc:224] Handling call
0808 01:51:56.533546 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:56.554933 (+ 21387us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:56.556805 (+  1872us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:51:56.617779 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:56.389833 15389 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.644526 15389 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:56.394363 15401 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.633831  9633 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:56.650578  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2607) took 1032 ms. Trace:
W20260808 01:51:56.602609 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:56.651854 10334 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:56.382514 15300 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.598089 15304 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:56.641889 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:56.642302  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:56.643040  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:56.645634  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2737) took 1456 ms. Trace:
I20260808 01:51:56.657437  9473 rpcz_store.cc:276] 0808 01:51:55.189064 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:55.423683 (+234619us) service_pool.cc:224] Handling call
0808 01:51:55.630983 (+207300us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:55.632008 (+  1025us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:56.645455 (+1013447us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b54000212d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:56.645579 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":475274624}
I20260808 01:51:56.659137  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:56.660199  9473 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:56.661506  9473 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:56.637840 15432 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:56.593461 15314 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.042s
I20260808 01:51:56.662433 15432 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:56.663182  9473 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260808 01:51:56.663399 15432 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.648972  9632 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260808 01:51:56.665277  9473 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260808 01:51:56.665681  9632 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260808 01:51:56.667418  9632 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 7. (term mismatch)
I20260808 01:51:56.586790 15386 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.040s
I20260808 01:51:56.638316 15177 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260808 01:51:56.627303 15451 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:56.603686 15231 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.043s
I20260808 01:51:56.628775 15369 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:56.650992  9633 rpcz_store.cc:276] 0808 01:51:55.617832 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:56.609409 (+991577us) service_pool.cc:224] Handling call
0808 01:51:56.633561 (+ 24152us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:56.650266 (+ 16705us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:56.650504 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:56.597180 15398 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:56.649658  9601 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:56.637182 15328 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:56.646660 15412 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.795s	user 0.014s	sys 0.001s
I20260808 01:51:56.667977  9473 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260808 01:51:56.681131  9632 pending_rounds.cc:85] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 6
I20260808 01:51:56.687219  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:56.705008  9632 pending_rounds.cc:107] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 13.7
I20260808 01:51:56.670765 15304 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:56.733441  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 791) took 1399 ms. Trace:
W20260808 01:51:56.733572  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 44250.1us
I20260808 01:51:56.733826  9632 raft_consensus.cc:2889] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:56.707186  9633 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260808 01:51:56.668401 15386 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:51:56.735903  9555 rpcz_store.cc:276] 0808 01:51:55.333842 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:55.473318 (+139476us) service_pool.cc:224] Handling call
0808 01:51:55.473674 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:55.624819 (+151145us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:55.624858 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:55.624911 (+    53us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:51:55.624961 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:56.733143 (+1108182us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b540015ac54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:56.733213 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:56.733381 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":347056512}
I20260808 01:51:56.739954  9474 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:56.674360 15231 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:56.744962  9521 leader_election.cc:336] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.356s (SENT)
I20260808 01:51:56.684478 15369 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:56.750989  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:56.684129 15457 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:51:56.704090 15448 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260808 01:51:56.736020 15456 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
W20260808 01:51:56.756245 15376 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.672s	user 0.008s	sys 0.022s
I20260808 01:51:56.669561 15451 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:56.752821 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 99497.2us
I20260808 01:51:56.673111 15226 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.775074 15398 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:56.753123 15391 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260808 01:51:56.742451 15340 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260808 01:51:56.778293 15226 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:56.750084 15177 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
I20260808 01:51:56.779590 15451 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:56.733337 15412 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:56.772817 15432 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.011s
I20260808 01:51:56.784997 15412 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:56.789235 15412 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:56.789840 15412 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.790428 15347 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:56.790462  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:56.793552 15412 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:56.794947  9472 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []
I20260808 01:51:56.795082 15188 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:56.773190 15270 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260808 01:51:56.799335  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2603) took 1182 ms. Trace:
I20260808 01:51:56.799739  9636 rpcz_store.cc:276] 0808 01:51:55.616807 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:55.915298 (+298491us) service_pool.cc:224] Handling call
0808 01:51:56.799134 (+883836us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b5400307fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:56.799273 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1428576000}
W20260808 01:51:56.802039  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:56.740361 15435 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260808 01:51:56.760375 15434 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:51:56.755609 15457 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:56.740787 15386 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:56.772195  9633 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260808 01:51:56.755529 15369 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:51:56.810580  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:56.769598  9632 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
W20260808 01:51:56.813254  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 43938.6us
I20260808 01:51:56.814203  9555 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:56.817461  9552 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []
W20260808 01:51:56.770304 15376 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.009s	sys 0.022s
I20260808 01:51:56.837234 15270 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:56.839211  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:56.841809  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:56.842710  9554 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:56.755905 15231 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:56.834542  9601 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:56.773871  9474 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:51:56.823381 15412 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:56.848475  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:56.849983  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:56.850983  9555 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:56.852348  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:56.853919 15432 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:56.823804 15457 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:51:56.872131 15334 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.878592 15386 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.859068  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:56.879701 15383 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 6 FOLLOWER]: Advancing to term 7
I20260808 01:51:56.884218  9555 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:56.856638  9601 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:56.888202  9550 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: []
I20260808 01:51:56.895256  9638 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 14. Ops: []
I20260808 01:51:56.898312  9557 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:56.899160  9601 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:56.901237  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:56.844142 15296 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:56.907240 15177 consensus_queue.cc:1059] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260808 01:51:56.910996 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:56.915611 15122 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260808 01:51:56.916559 15122 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:51:56.917210 15122 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:56.920045  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:56.921095 15391 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:51:56.923859 15391 consensus_queue.cc:1243] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 17.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260808 01:51:56.924603 15433 raft_consensus.cc:3055] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:56.925076 15433 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:56.916563 15411 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:51:56.927968 15334 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:56.922698 15435 raft_consensus.cc:3055] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:51:56.933636 15435 raft_consensus.cc:740] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:56.934111  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2738) took 1744 ms. Trace:
I20260808 01:51:56.887337 15416 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:56.935130 15435 consensus_queue.cc:260] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.937120 15313 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:56.937364 15435 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:56.934741  9476 rpcz_store.cc:276] 0808 01:51:55.189377 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:55.465739 (+276362us) service_pool.cc:224] Handling call
0808 01:51:55.483587 (+ 17848us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:55.514233 (+ 30646us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:51:55.646139 (+131906us) log.cc:844] Serialized 14 byte log entry
0808 01:51:55.990419 (+344280us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:55.990479 (+    60us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:55.990538 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:56.933621 (+943083us) spinlock_profiling.cc:243] Waited 30.4 ms on lock 0x7b540021d754. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:56.933863 (+   242us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:56.934037 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2268,"prepare.queue_time_us":238938,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":61,"raft.queue_time_us":123,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":18815,"spinlock_wait_cycles":69885312,"thread_start_us":460851,"threads_started":2,"wal-append.queue_time_us":382521}
I20260808 01:51:56.940891  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:45650 (request call id 22) took 1628 ms. Trace:
I20260808 01:51:56.941337  9636 rpcz_store.cc:276] 0808 01:51:55.312944 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:56.800663 (+1487719us) service_pool.cc:224] Handling call
0808 01:51:56.940540 (+139877us) spinlock_profiling.cc:243] Waited 6.7 ms on lock 0x7b5400039ba0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44910900 00007f5d44999f22 00007f5d44998be0 00007f5d4998bb39 00007f5d42ed8334 00007f5d42ed82c2 00007f5d42ed81f1 00007f5d42ed816c 00007f5d42ed7422 00007f5d42ccd07c
0808 01:51:56.940800 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15413632}
I20260808 01:51:56.924629 15391 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:51:56.942633 15231 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:56.942826 15391 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:56.950510  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 44161us
I20260808 01:51:56.952045  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:56.957082  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:56.943256 14833 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:51:56.962098  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 50fd024764ea49c7a1f16e992e7599f3: Illegal state: Leader has not yet committed an operation in its own term
I20260808 01:51:56.903558 15370 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:56.975093 15370 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.977129 15265 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:56.964155  9559 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260808 01:51:56.967736  9599 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:56.961491 15389 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:56.980031 15389 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:56.980327  9599 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:51:56.980582  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:56.981356  9474 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:51:56.963711 14784 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:51:56.982887 15455 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:56.983263 15337 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:56.983762 15337 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:56.984187 15337 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:56.985378 15369 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:56.987856  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2550) took 2024 ms. Trace:
I20260808 01:51:56.988162 15369 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:56.935900 15416 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.988412  9638 rpcz_store.cc:276] 0808 01:51:54.963778 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:56.541920 (+1578142us) service_pool.cc:224] Handling call
0808 01:51:56.542330 (+   410us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:56.897131 (+354801us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:56.987611 (+ 90480us) spinlock_profiling.cc:243] Waited 353 ms on lock 0x7b540032d554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:56.987791 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":811004672}
I20260808 01:51:56.965258  9601 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:56.973477 15457 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:56.979130 15432 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:51:57.006985 15432 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:57.001740 15416 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:56.888322 15412 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.926105 15433 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:57.012655 15433 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:57.013487 15412 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:56.906288 15451 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:51:56.918354 15122 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:57.015012  9478 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:57.014854 15451 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:57.016616 15451 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:56.983603 15455 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:56.983592 14784 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:56.972404  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3153) took 1568 ms. Trace:
I20260808 01:51:56.992496 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:56.993363 15443 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:56.980542  9476 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13. Ops: []
I20260808 01:51:57.032533  9473 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260808 01:51:57.029083  9560 rpcz_store.cc:276] 0808 01:51:55.403797 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:55.511710 (+107913us) service_pool.cc:224] Handling call
0808 01:51:55.511977 (+   267us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:55.737511 (+225534us) raft_consensus.cc:1534] Early marking committed up to index 9
0808 01:51:55.904141 (+166630us) log.cc:844] Serialized 14 byte log entry
0808 01:51:56.088414 (+184273us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:56.088471 (+    57us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:51:56.088519 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:56.294396 (+205877us) spinlock_profiling.cc:243] Waited 5.38 ms on lock 0x7b18003bd340. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4185ac56 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229
0808 01:51:56.972113 (+677717us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b540029e5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:56.972167 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:56.972333 (+   166us) inbound_call.cc:177] Queueing success response
0808 01:51:57.005649 (+ 33316us) spinlock_profiling.cc:243] Waited 2.69 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
Metrics: {"mutex_wait_us":10147,"prepare.queue_time_us":103298,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":2286,"raft.queue_time_us":3266,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":7331,"spinlock_wait_cycles":536472448,"thread_start_us":274987,"threads_started":2,"wal-append.queue_time_us":349939}
I20260808 01:51:56.996348  9601 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:56.990345 10334 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:57.004288 15457 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:57.006414 15445 log.cc:927] Time spent T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.722s	user 0.004s	sys 0.000s
I20260808 01:51:56.979323 15370 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:56.971889 15340 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:51:57.024643  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:56.962937 15388 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:57.027667  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:57.035487 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 45204.8us
I20260808 01:51:57.029562 10361 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:51:57.048161  9554 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:57.049005  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:57.049149  9600 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:57.022246 15455 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:56.975183 15286 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:51:57.039786  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:57.049397 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:57.051532  9638 raft_consensus.cc:2376] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17. Re-sending same reply.
I20260808 01:51:56.992312 15389 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:57.052608  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:57.041138  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:57.027058  9552 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: []
I20260808 01:51:57.055302  9636 raft_consensus.cc:2376] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11. Re-sending same reply.
I20260808 01:51:57.045455  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:57.032481 14784 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:57.007555 15432 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:51:57.050545  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:57.065522  9473 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:51:57.036381 15188 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:57.068818  9632 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260808 01:51:57.060734  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:57.038357 15177 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:51:57.053253  9638 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:51:57.066977 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:56.944142 15391 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:57.074460 15370 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:51:57.079150 15340 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:57.079880 15340 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:57.082240 15340 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:57.065135 15294 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:57.084285 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:57.062649 15443 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:51:57.087245 15443 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:57.087680 15443 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:57.089617  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:57.093129  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:57.094911 15370 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:57.051201 15286 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:57.096514 15286 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:57.097886 15286 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:57.062668 15083 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:57.099045 15083 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:57.099104  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:57.099676  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:51:57.100469  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:57.100631  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:57.100971  9524 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:57.101220  9476 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 7.
I20260808 01:51:57.102535  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:57.103186  9480 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:51:57.101223  9638 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 11: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:57.105299  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:57.105801  9473 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:51:57.101974  9524 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:57.074020  9601 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:57.099594 15083 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:57.066409 14784 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:57.116842  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:57.118251  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:57.119732  9632 raft_consensus.cc:2376] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13. Re-sending same reply.
I20260808 01:51:57.120251  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:57.120884  9476 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:57.120841  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:57.121841  9474 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:51:57.098651 15388 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:57.116117 15369 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:57.133581 10334 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:57.117039 15370 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260808 01:51:57.132699  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 58777us
I20260808 01:51:57.150321 10334 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:57.153014 10367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 658) took 1830 ms. Trace:
I20260808 01:51:57.124701 15452 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:57.125205 15300 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:57.162639 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:57.119277 15188 raft_consensus.cc:697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:57.160744 10367 rpcz_store.cc:276] 0808 01:51:55.323034 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:55.323292 (+   258us) service_pool.cc:224] Handling call
0808 01:51:57.152843 (+1829551us) spinlock_profiling.cc:243] Waited 540 ms on lock 0x7b540064aad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:57.152951 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1242888192}
I20260808 01:51:57.166205  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:57.166855  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:57.170854  9632 pending_rounds.cc:85] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 8
I20260808 01:51:57.171219  9632 pending_rounds.cc:107] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260808 01:51:57.171443  9632 raft_consensus.cc:2889] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:51:57.171554  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:57.159191 15455 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:51:57.182210  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:57.133670 15296 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260808 01:51:57.182780  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 3b8e0e8fda6744ad877514b11658b9d9 (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:57.144265 15434 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
I20260808 01:51:57.133607 15391 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:51:57.172294  9476 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
W20260808 01:51:57.125439 15448 consensus_queue.cc:1175] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6e7856bed1be43acaa821290d506ee97" responder_term: 17 status { last_received { term: 13 index: 8 } last_committed_idx: 8 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 01:51:57.139871 15177 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:57.184751  9480 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:51:57.189728  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2906) took 1258 ms. Trace:
I20260808 01:51:57.146803 15369 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:57.231686 15369 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:57.132169 15436 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260808 01:51:57.093101 15304 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:57.162907 15383 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260808 01:51:57.239148 15383 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:57.239648 15383 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:57.161406 15370 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:57.197571 15455 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:57.140334 15453 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.058s
I20260808 01:51:57.168622 15300 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:57.139333 15334 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
I20260808 01:51:57.144027 15376 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.052s
W20260808 01:51:57.174530  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:57.182466 15298 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:57.190996 15452 raft_consensus.cc:697] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:51:57.151855 15388 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:57.192133 15406 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:51:57.229912  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:57.228408  9479 rpcz_store.cc:276] 0808 01:51:55.931346 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:55.931577 (+   231us) service_pool.cc:224] Handling call
0808 01:51:55.959198 (+ 27621us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:55.967174 (+  7976us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:51:55.991273 (+ 24099us) log.cc:844] Serialized 14 byte log entry
0808 01:51:56.102585 (+111312us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:56.103268 (+   683us) log.cc:844] Serialized 26 byte log entry
0808 01:51:56.103452 (+   184us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:57.083888 (+980436us) spinlock_profiling.cc:243] Waited 9.84 ms on lock 0x7b18000ffac0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:51:57.188827 (+104939us) spinlock_profiling.cc:243] Waited 31.1 ms on lock 0x7b18000ffac0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d483aada7 00007f5d4499dd87 00007f5d449ad49f 00007f5d449ad418
0808 01:51:57.189143 (+   316us) log.cc:844] Serialized 14 byte log entry
0808 01:51:57.189377 (+   234us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:57.189462 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:57.189504 (+    42us) raft_consensus.cc:1685] finished
0808 01:51:57.189517 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:57.189661 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":139179,"prepare.queue_time_us":23903,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":12504,"raft.run_cpu_time_us":459,"raft.run_wall_time_us":33438,"spinlock_wait_cycles":94062080,"thread_start_us":64987,"threads_started":1,"wal-append.queue_time_us":119208}
I20260808 01:51:57.215925  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:57.213792 15472 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260808 01:51:57.277175  9477 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:57.166944  9601 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:57.213392 15337 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:57.228665  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:57.282725  9476 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:57.283706  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:57.284614  9479 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:51:57.209275 15296 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:57.256763 15300 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:51:57.261305  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:57.252264 15473 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
I20260808 01:51:57.255867  9475 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:57.247850  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:57.269387 15298 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:57.279007  9480 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:51:57.248845 15370 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:57.312362  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:57.295392  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:57.307739  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 140872us
I20260808 01:51:57.320923  9601 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:57.318011 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:57.258229  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 79440.7us
W20260808 01:51:57.301133 15475 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.246s	user 0.002s	sys 0.000s
I20260808 01:51:57.255676 15231 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.055s
I20260808 01:51:57.299165 15296 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:57.345963  9476 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:57.364516  9476 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260808 01:51:57.350417  9477 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:51:57.319324 15298 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:57.316900 15451 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:51:57.277750 15188 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:57.348852 15294 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260808 01:51:57.349179  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 206527us
I20260808 01:51:57.381274  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2758) took 1196 ms. Trace:
I20260808 01:51:57.395927  9478 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260808 01:51:57.398989  9472 rpcz_store.cc:276] 0808 01:51:56.184841 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:56.797549 (+612708us) service_pool.cc:224] Handling call
0808 01:51:56.797993 (+   444us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:56.798383 (+   390us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:56.798416 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:56.824179 (+ 25763us) log.cc:844] Serialized 26 byte log entry
0808 01:51:57.038364 (+214185us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:57.038412 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:57.038552 (+   140us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:57.380995 (+342443us) raft_consensus.cc:1685] finished
0808 01:51:57.381019 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:57.381205 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1278,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":116,"thread_start_us":180380,"threads_started":1,"wal-append.queue_time_us":297276}
I20260808 01:51:57.356670 15435 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:51:57.358404 15334 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:57.388892  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:57.389520  9552 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 13 index: 6. (term mismatch)
I20260808 01:51:57.391458  9476 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260808 01:51:57.403478  9552 pending_rounds.cc:85] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 5
I20260808 01:51:57.403868  9552 pending_rounds.cc:107] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 12.6
I20260808 01:51:57.404126  9552 raft_consensus.cc:2889] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:51:57.394697  9477 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260808 01:51:57.352754 15443 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:57.414907  9476 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260808 01:51:57.409024  9478 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
W20260808 01:51:57.421397  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:57.370509 15376 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.036s
I20260808 01:51:57.390502 15451 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:57.335403 15452 consensus_queue.cc:237] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:57.425077 15435 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:57.432116  9472 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 13 index: 6. (index mismatch)
I20260808 01:51:57.338815 15455 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:57.434007  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:57.438982  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 86000.8us
I20260808 01:51:57.419382  9479 raft_consensus.cc:2955] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Committing config change with OpId 14.9: config changed from index -1 to 9, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:51:57.397002 15434 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.045s
I20260808 01:51:57.452401  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:57.458496  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:57.427482 15294 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:57.461061  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:57.461279 15294 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:57.465348 15294 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:57.467504  9629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2554) took 2412 ms (client timeout 1719 ms). Trace:
W20260808 01:51:57.468122  9629 rpcz_store.cc:269] 0808 01:51:55.055422 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:55.328324 (+272902us) service_pool.cc:224] Handling call
0808 01:51:57.467169 (+2138845us) spinlock_profiling.cc:243] Waited 725 ms on lock 0x7b540003b6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:57.467334 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1667875072}
I20260808 01:51:57.469751  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:57.476197  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2549) took 2512 ms. Trace:
I20260808 01:51:57.476694  9630 rpcz_store.cc:276] 0808 01:51:54.963339 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:56.541785 (+1578446us) service_pool.cc:224] Handling call
0808 01:51:56.542178 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:56.608717 (+ 66539us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:56.608752 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:56.619022 (+ 10270us) log.cc:844] Serialized 26 byte log entry
0808 01:51:56.659806 (+ 40784us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:56.659843 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:57.475631 (+815788us) spinlock_profiling.cc:243] Waited 66.1 ms on lock 0x7b54004213d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:57.475786 (+   155us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:57.475845 (+    59us) raft_consensus.cc:1685] finished
0808 01:51:57.475866 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:57.476124 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1249,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":16683,"spinlock_wait_cycles":152092928,"thread_start_us":39189,"threads_started":1,"wal-append.queue_time_us":39768}
I20260808 01:51:57.481659 15369 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:57.482564  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2623) took 1577 ms. Trace:
I20260808 01:51:57.483131  9633 rpcz_store.cc:276] 0808 01:51:55.905663 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:56.737689 (+832026us) service_pool.cc:224] Handling call
0808 01:51:56.753947 (+ 16258us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:56.823478 (+ 69531us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:57.482209 (+658731us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b540003c5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:57.482480 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41132928}
I20260808 01:51:57.484840  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:57.487802 15376 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260808 01:51:57.491777 15482 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
W20260808 01:51:57.494120 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:57.497308 15401 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:57.413832 15334 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:57.500751 15334 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:57.505331  9473 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260808 01:51:57.506860 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:57.508533 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:57.512645 15388 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:57.513669 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:57.517119  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:57.519131  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:57.520115  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:57.520749  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:57.521582  9560 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:57.521755  9633 raft_consensus.cc:2410] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 18 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
W20260808 01:51:57.521754  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62565.1us
I20260808 01:51:57.524595  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:57.527972  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:57.529008  9478 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:57.512477 15432 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:57.529520  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:57.531219  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:57.536437  9560 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 17: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:51:57.532946  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 80994.1us
I20260808 01:51:57.438875 15451 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:57.539482  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2761) took 1210 ms. Trace:
I20260808 01:51:57.540832  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:57.540165  9481 rpcz_store.cc:276] 0808 01:51:56.329086 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:56.927702 (+598616us) service_pool.cc:224] Handling call
0808 01:51:56.928226 (+   524us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:56.928746 (+   520us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:56.928793 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:56.929529 (+   736us) log.cc:844] Serialized 26 byte log entry
0808 01:51:57.516743 (+587214us) spinlock_profiling.cc:243] Waited 1.69 ms on lock 0x7b18004193e0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4453a6ac 00007f5d4186fe3b 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1
0808 01:51:57.517272 (+   529us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:57.517333 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:57.517432 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:57.517490 (+    58us) raft_consensus.cc:1685] finished
0808 01:51:57.517503 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:57.539409 (+ 21906us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":15821,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":10462,"spinlock_wait_cycles":3894528,"thread_start_us":34063,"threads_started":1,"wal-append.queue_time_us":105616}
I20260808 01:51:57.544179  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:57.443116 15304 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:57.545967  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:57.450273 15412 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:57.554994  9554 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260808 01:51:57.559996  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:57.552270  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:57.461169 15443 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:57.455468 15478 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:57.501762 15083 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:57.544113 15401 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:51:57.462469 15434 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.113s
I20260808 01:51:57.494134 15383 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:57.565953  9556 raft_consensus.cc:2410] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260808 01:51:57.561233  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:57.574064  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:57.621011  9629 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 17: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:57.575413 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:57.590092 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:57.622182 10366 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:57.576610 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:51:57.623682  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:57.597355  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:57.605247  9480 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260808 01:51:57.570449 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:57.613762  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:57.627151 10362 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:57.596560  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:57.630561  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:57.588927  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 51572.1us
W20260808 01:51:57.634783  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:57.679242  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:57.679232  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:57.685410  9555 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:57.685212  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:57.573478 15265 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.596s	user 0.014s	sys 0.000s
W20260808 01:51:57.613111  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 213421us
I20260808 01:51:57.693152  9445 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:57.701472  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 68449.6us
I20260808 01:51:57.690965  9524 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:57.705003 15376 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:51:57.705266  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:57.706830 15294 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:51:57.706643  9524 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:57.707662 15294 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:57.708580 15492 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:57.707041 15434 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:51:57.709894 15434 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:57.710246 15434 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:51:57.706890 15414 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:57.713792 15414 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:57.714207 15414 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:57.706620 15376 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20260808 01:51:57.716537  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:57.718640  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:57.719542  9559 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:57.594354 15401 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:57.570930 15337 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:57.726415  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:57.733104  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:57.709254  9524 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:57.734411 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 120377us
I20260808 01:51:57.764137  9629 raft_consensus.cc:2410] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 15 because replica has last-logged OpId of term: 13 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260808 01:51:57.767125  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:57.585076 15304 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:57.570300 15389 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:57.635144 15433 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.622s	user 0.011s	sys 0.003s
W20260808 01:51:57.748821  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 46238.1us
I20260808 01:51:57.571834 15432 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:57.759414 10361 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13. Ops: []
W20260808 01:51:57.700901 14784 consensus_meta.cc:287] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.562s	user 0.010s	sys 0.003s
W20260808 01:51:57.768666  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 59517us
I20260808 01:51:57.606614 15486 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.155s
I20260808 01:51:57.738526 10366 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:57.606559 15435 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:57.775141  9556 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 6. Preceding OpId from leader: term: 15 index: 7. (index mismatch)
I20260808 01:51:57.776201  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:57.709029 15492 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:57.566982 15300 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:57.818825  9524 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:57.782126 10362 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:57.768458  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:57.593731 15478 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:57.791594  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 101221us
I20260808 01:51:57.843891  9631 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260808 01:51:57.835840  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3165) took 1011 ms. Trace:
I20260808 01:51:57.851212  9558 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
W20260808 01:51:57.843430  9637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2609) took 2224 ms (client timeout 1694 ms). Trace:
I20260808 01:51:57.818393 15391 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:57.861105  9637 rpcz_store.cc:269] 0808 01:51:55.618483 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:55.750307 (+131824us) service_pool.cc:224] Handling call
0808 01:51:57.843173 (+2092866us) spinlock_profiling.cc:243] Waited 540 ms on lock 0x7b54001b49d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:57.843336 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1241257728}
I20260808 01:51:57.851788 15492 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:57.851920  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:57.708930 15294 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:57.792598  9555 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 6. Preceding OpId from leader: term: 15 index: 7. (index mismatch)
I20260808 01:51:57.781565 15226 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:51:57.852982  9560 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 15: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:57.792133 15188 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:57.783890 15443 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:57.782231 15122 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.071s
I20260808 01:51:57.782974 15304 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:57.782514 15337 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:57.832641 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:57.821669 14784 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:57.850567  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62001.7us
I20260808 01:51:57.855834  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:57.861325  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:57.748342 15177 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.073s
I20260808 01:51:57.861227  9557 rpcz_store.cc:276] 0808 01:51:56.824251 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:56.824438 (+   187us) service_pool.cc:224] Handling call
0808 01:51:56.824918 (+   480us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:56.901202 (+ 76284us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:57.835528 (+934326us) spinlock_profiling.cc:243] Waited 73 ms on lock 0x7b540016a3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:57.835767 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":167894912}
I20260808 01:51:57.862651  9444 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:57.760881 15401 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:51:57.871363  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:57.904896  9637 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:51:57.871614  9629 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:51:57.872083  9635 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:51:57.906139  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:57.906358  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:57.766530 15265 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.076s
I20260808 01:51:57.864087 15433 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.034s
I20260808 01:51:57.883106 15369 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:57.720489 15449 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:57.888111  9550 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:51:57.894165 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:57.854476 15300 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:57.896721  9556 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 6.
I20260808 01:51:57.901024  9558 raft_consensus.cc:2368] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
W20260808 01:51:57.886957 15489 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.350s	user 0.000s	sys 0.001s
I20260808 01:51:57.792757 15389 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:57.956861  9636 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:57.958362 15389 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:57.907115  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:57.875931  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:57.892629 15292 consensus_meta.cc:287] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.939s	user 0.008s	sys 0.008s
I20260808 01:51:57.908419  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:57.907869  9478 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
W20260808 01:51:57.960402  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 53358.2us
W20260808 01:51:57.909464 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:57.907087  9637 raft_consensus.cc:2410] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260808 01:51:57.963003  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:57.963649  9637 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:51:57.964860  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:57.895560 15490 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.211s	user 0.001s	sys 0.000s
I20260808 01:51:57.887847 15347 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:51:57.891685 15226 raft_consensus.cc:697] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:57.879949 15391 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:57.941150  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:57.980726  9558 raft_consensus.cc:2393] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:57.938465 15083 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:57.905966 15270 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:57.896227 15443 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:57.991771 15497 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260808 01:51:57.993459 15443 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:57.875775  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 125591us
I20260808 01:51:57.995132 15292 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:56266 (request call id 916) took 5758 ms. Trace:
I20260808 01:51:57.967087  9637 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:51:57.995672 15292 rpcz_store.cc:276] 0808 01:51:52.241134 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:52.241506 (+   372us) service_pool.cc:224] Handling call
0808 01:51:52.644266 (+402760us) ts_tablet_manager.cc:934] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
0808 01:51:53.135579 (+491313us) spinlock_profiling.cc:243] Waited 34.6 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4186fa5c 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188965a 00007f5d41889582 00007f5d49938f41
0808 01:51:56.605600 (+3470021us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:51:56.870565 (+264965us) tablet_metadata.cc:665] Metadata flushed
0808 01:51:56.895890 (+ 25325us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:51:57.993926 (+1098036us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:51:57.994385 (+   459us) tablet_replica.cc:206] Creating consensus instance
0808 01:51:57.994852 (+   467us) ts_tablet_manager.cc:989] Replica registered.
0808 01:51:57.995019 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":52232,"spinlock_wait_cycles":79673088,"tablet-copy.queue_time_us":275036,"thread_start_us":815917,"threads_started":2}
W20260808 01:51:57.968415 15498 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260808 01:51:57.997841 15292 tablet_copy_client.cc:806] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:51:57.970427  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:57.996203  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:57.999394  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:58.000725  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:57.913427 15265 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:51:58.002429 15265 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:57.973403  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:57.911579 15411 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:58.002722 15501 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:57.982358  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:57.947074  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 87781.7us
I20260808 01:51:57.977317 15412 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:51:58.005893  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:58.006652 15412 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 01:51:58.007718  9558 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260808 01:51:57.939378 15508 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.096s
I20260808 01:51:58.011175 15508 consensus_queue.cc:1243] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 14.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260808 01:51:58.011791 15508 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260808 01:51:58.012456 15508 raft_consensus.cc:3055] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:58.012836 15508 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:58.013442  9476 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260808 01:51:58.013646 15508 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:58.014650  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:58.015945  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:58.019940  9473 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260808 01:51:58.022771 15496 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:58.023306 15226 consensus_queue.cc:237] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:57.947788 15435 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:57.987082 15402 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260808 01:51:58.027031 15435 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:58.028834 15402 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:58.029381 15402 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:58.029901 15435 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:57.953217 15122 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:57.955026 15509 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:57.947361 15383 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:57.969305 15491 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260808 01:51:58.003413 15411 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:57.897806 15337 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:58.000730 15443 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.004s
I20260808 01:51:57.997192  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:58.004654 15499 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260808 01:51:58.023480 15496 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:57.985778 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:58.001415  9556 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:51:57.931563 15479 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.080s
I20260808 01:51:58.038864  9558 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:57.891978 15188 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:57.983659 15505 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260808 01:51:58.040617  9636 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:51:57.940910 15369 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20260808 01:51:58.047847 15402 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:58.040041 15270 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:58.051218 15298 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.645s	user 0.006s	sys 0.008s
W20260808 01:51:58.051931 15177 consensus_queue.cc:1175] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6e7856bed1be43acaa821290d506ee97" responder_term: 14 status { last_received { term: 12 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: [13.7-13.7]" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 01:51:58.026413 15461 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:58.147212  9556 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260808 01:51:58.094416 10366 raft_consensus.cc:2368] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
W20260808 01:51:58.101130  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 137917us
W20260808 01:51:58.107257  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 237129us
I20260808 01:51:58.040467 15347 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:58.057911 15411 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:58.102809 15292 tablet_copy_client.cc:670] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:51:58.139935 15340 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:58.120816 15495 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:58.174602  9559 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:58.056559 15122 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:58.198890 10334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.004s	sys 0.012s
W20260808 01:51:58.201541 10334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.004s	sys 0.013s
I20260808 01:51:58.189837  9445 leader_election.cc:400] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
I20260808 01:51:58.202872  9445 leader_election.cc:403] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260808 01:51:58.092617 15300 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.073s
W20260808 01:51:58.172333 15296 consensus_meta.cc:287] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.813s	user 0.013s	sys 0.000s
I20260808 01:51:58.095088 15486 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.078s
I20260808 01:51:58.042644 15435 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.118s
I20260808 01:51:58.059460 15508 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:58.157150 15461 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:58.220058 15296 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:58.223685  9557 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 15: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:51:58.204993  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 151848us
I20260808 01:51:58.147941 15298 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:58.042896 15501 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:58.205924 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 660537us
W20260808 01:51:58.215559  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 219436us
I20260808 01:51:58.056761 15509 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:58.235415  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 50505.8us
I20260808 01:51:58.204571 14784 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:58.250190  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:58.252893  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:58.260501 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:58.256048  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:58.198545 15481 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.059s
I20260808 01:51:58.214257 15478 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:58.281772  9556 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:58.282541  9560 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:58.200667 15434 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:58.194096 15340 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:58.222977 15461 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:58.218788 15436 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.066s
I20260808 01:51:58.198330 15495 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:58.213044 15414 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:58.298247  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 78770.3us
I20260808 01:51:58.321413  9473 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:51:58.302978  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 152825us
I20260808 01:51:58.324970  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:58.247196 15296 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:58.258324 15517 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:58.265657 15304 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:58.284314 15291 consensus_queue.cc:1059] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.061s
I20260808 01:51:58.264549 15443 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:51:58.336654  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:58.315186 15391 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:58.302907 15451 consensus_meta.cc:287] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.750s	user 0.014s	sys 0.000s
I20260808 01:51:58.348361 15533 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:58.360719 15443 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:58.342808 15337 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:58.358330 15117 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:58.359103  9599 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:51:58.372164 15507 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.195s	user 0.001s	sys 0.001s
I20260808 01:51:58.362807 15501 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:58.363106 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:58.418586 10366 raft_consensus.cc:2368] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15.
I20260808 01:51:58.406522 15533 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:58.364522 15520 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
W20260808 01:51:58.420452  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 61427.7us
I20260808 01:51:58.420720 15533 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:58.364387  9560 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:58.421641 15533 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:51:58.420480  9445 leader_election.cc:400] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15.
I20260808 01:51:58.423166  9445 leader_election.cc:403] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260808 01:51:58.422964  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:58.423893 15495 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:58.424657  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:58.339735  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 88524.4us
I20260808 01:51:58.367220  9473 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:51:58.284958 15347 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:58.426394  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:58.359527  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:58.428407  9630 raft_consensus.cc:2410] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260808 01:51:58.428941  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2552) took 1059 ms. Trace:
I20260808 01:51:58.429243  9630 rpcz_store.cc:276] 0808 01:51:57.369106 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:58.356622 (+987516us) service_pool.cc:224] Handling call
0808 01:51:58.428848 (+ 72226us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:58.382282 10362 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
W20260808 01:51:58.368309 15515 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260808 01:51:58.323902 15478 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:58.331470  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 87608.6us
I20260808 01:51:58.441413  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:58.442268  9601 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:58.442437 15411 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:58.443650  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:58.444538 14833 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 01:51:58.444820 15411 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:58.327737 15516 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:58.445652 15516 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:58.447026 15518 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:51:58.395615 15451 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:58.360631 15122 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:58.449560 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:58.422744 15533 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:51:58.358175 15512 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.230s	user 0.000s	sys 0.001s
I20260808 01:51:58.451750 15533 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:58.399255  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2755) took 2203 ms. Trace:
I20260808 01:51:58.453377  9475 rpcz_store.cc:276] 0808 01:51:56.184020 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:56.577438 (+393418us) service_pool.cc:224] Handling call
0808 01:51:58.386911 (+1809473us) spinlock_profiling.cc:243] Waited 677 ms on lock 0x7b5400399754. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:58.387048 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1557949696}
I20260808 01:51:58.455000  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:58.404286 15535 raft_consensus.cc:3055] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:51:58.367257 15188 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:58.405987 15492 consensus_meta.cc:287] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.518s	user 0.009s	sys 0.002s
I20260808 01:51:58.355129 15496 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:58.407821 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:58.409595 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:58.410457 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:58.368978 15401 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:58.371054 15433 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:58.361627 15414 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:58.366612 15370 consensus_meta.cc:287] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.037s	user 0.005s	sys 0.007s
I20260808 01:51:58.483058 15370 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:51:58.485719 15370 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:58.414258 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:58.486406  9551 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: []
I20260808 01:51:58.368979 15340 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:58.416922 15443 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:58.361555 15509 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:58.417739  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:58.493438  9474 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 7 FOLLOWER]: Advancing to term 8
I20260808 01:51:58.425359  9481 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:58.359207  9523 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:58.435774  9473 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260808 01:51:58.437084  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:58.495633  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:58.496274  9473 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:58.497267  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:58.450285 15451 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:58.446307 14833 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260808 01:51:58.469306  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:58.466361 15535 raft_consensus.cc:740] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:51:58.472136  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:58.476229 15265 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:58.481457 15434 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:58.489708  9524 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:58.465555 15508 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:58.458709  9629 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:58.497422  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:58.501368  9479 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 13: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:51:58.503173  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 144032us
I20260808 01:51:58.519239  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:58.520165  9473 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:58.505970  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:58.474155 15391 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:58.521034  9477 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:51:58.526384  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:58.521921  9481 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:51:58.523566  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:58.464941 15478 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:58.500475 14784 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:58.521358  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:58.501297 14833 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:58.501688 15433 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:58.548260  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:58.538286  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 221451us
I20260808 01:51:58.467617 15492 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:58.508718 15486 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:51:58.541965  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:58.550266  9552 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260808 01:51:58.549058  9556 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:51:58.559126  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:58.560684  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:58.561882  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:58.562568  9560 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 6 FOLLOWER]: Advancing to term 8
W20260808 01:51:58.569145  9601 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:45658 dropped due to backpressure: service queue is full with 50 items
I20260808 01:51:58.570804  9474 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:51:58.569487  9601 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:45658 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2641)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2633)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2643)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2639)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2663)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2635)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2634)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2659)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2648)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2638)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2636)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2640)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2644)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2631)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2637)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2647)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2650)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2658)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2642)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2651)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2652)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2649)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2645)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34802 (request call id 69)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 709)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 710)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 712)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2630)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2541)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2549)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2553)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2558)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2563)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2566)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 713)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2632)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2646)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2653)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2654)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2655)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2656)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2657)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2660)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2661)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2662)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2664)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 714)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 715)
I20260808 01:51:58.530954 15406 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15.
I20260808 01:51:58.576150  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:58.600219  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:58.600481 15386 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:58.601728  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:58.577768  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:58.602440  9552 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:58.604507  9637 raft_consensus.cc:2410] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260808 01:51:58.605108 15291 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:58.605819 15291 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:58.578162  9481 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260808 01:51:58.536757 15513 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:51:58.647363 15513 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:58.648437  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:58.649240  9481 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:58.648758 15513 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:58.650420 15117 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:58.651270 15117 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:58.654090 15117 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:58.654910 15514 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:58.655653 15514 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:58.662817 15177 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:58.663554 15177 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:51:58.664812 15537 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260808 01:51:58.666144 15177 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:58.591303  9444 leader_election.cc:400] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
W20260808 01:51:58.721009  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 129874us
I20260808 01:51:58.722213  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:58.591758  9560 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 8.
I20260808 01:51:58.547587 14833 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:58.593858 15304 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:58.608762  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 62965.1us
I20260808 01:51:58.609674  9555 raft_consensus.cc:2393] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:58.670252 15443 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:58.584949  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:58.711455  9552 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:51:58.522217 15508 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:58.810161  9552 raft_consensus.cc:3055] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:51:58.810815  9552 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:58.776286  9560 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260808 01:51:58.568053 15479 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:58.808400  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2396) took 1123 ms. Trace:
I20260808 01:51:58.823140  9554 rpcz_store.cc:276] 0808 01:51:57.684648 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:57.685146 (+   498us) service_pool.cc:224] Handling call
0808 01:51:57.689288 (+  4142us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:57.707565 (+ 18277us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:57.707600 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:57.708272 (+   672us) log.cc:844] Serialized 52 byte log entry
0808 01:51:57.789665 (+ 81393us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:57.859295 (+ 69630us) log.cc:844] Serialized 14 byte log entry
0808 01:51:58.426509 (+567214us) spinlock_profiling.cc:243] Waited 1.65 ms on lock 0x7b180031f9c0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667 00007f5d449ae5e8
0808 01:51:58.712292 (+285783us) log.cc:844] Serialized 14 byte log entry
0808 01:51:58.713890 (+  1598us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:58.714015 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:58.808011 (+ 93996us) raft_consensus.cc:1685] finished
0808 01:51:58.808039 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:58.808306 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33834,"prepare.queue_time_us":133101,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":6441,"raft.queue_time_us":129871,"raft.run_cpu_time_us":465,"raft.run_wall_time_us":42765,"spinlock_wait_cycles":3804032,"thread_start_us":365347,"threads_started":2,"wal-append.queue_time_us":381680}
W20260808 01:51:58.826014  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 241155us
I20260808 01:51:58.827491  9601 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:58.533880 15391 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:58.599673 15411 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.111s
I20260808 01:51:58.548069 14784 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:58.793985  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2932) took 1205 ms. Trace:
I20260808 01:51:58.831146  9478 rpcz_store.cc:276] 0808 01:51:57.589022 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:57.589197 (+   175us) service_pool.cc:224] Handling call
0808 01:51:57.589663 (+   466us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:57.960804 (+371141us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:58.793633 (+832829us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b54002bb1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:58.793886 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":731011456}
I20260808 01:51:58.832922  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:58.833668  9478 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:58.811777  9552 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:58.836186  9552 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:51:58.847891  9481 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:51:58.851796  9472 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 13. Current term is 14. Ops: []
I20260808 01:51:58.820951 15414 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:58.851882  9474 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: [14.8-15.9]
I20260808 01:51:58.852685  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:58.926867  9481 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
W20260808 01:51:58.900127  9635 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2590) took 3795 ms (client timeout 2112 ms). Trace:
W20260808 01:51:58.927644  9635 rpcz_store.cc:269] 0808 01:51:55.104777 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:55.739367 (+634590us) service_pool.cc:224] Handling call
0808 01:51:58.899911 (+3160544us) spinlock_profiling.cc:243] Waited 1.89 s on lock 0x7b540027e554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:58.900035 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4336455936}
I20260808 01:51:58.928874  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:58.850756 15509 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:58.893214 15291 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:51:58.836225 15434 consensus_queue.cc:1175] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6e7856bed1be43acaa821290d506ee97" responder_term: 13 status { last_received { term: 11 index: 10 } last_committed_idx: 9 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13. Ops: [12.11-12.11]" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 01:51:58.900058 15433 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 12.13, Last appended by leader: 13, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:58.900909  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2564) took 1470 ms. Trace:
I20260808 01:51:58.956208  9637 rpcz_store.cc:276] 0808 01:51:57.430942 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:57.996875 (+565933us) service_pool.cc:224] Handling call
0808 01:51:58.900697 (+903822us) spinlock_profiling.cc:243] Waited 511 ms on lock 0x7b540027e054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:58.900841 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1174641024}
I20260808 01:51:58.901437 15516 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:58.851078 15451 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:51:58.830233 15518 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:58.903458 15536 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.343s	user 0.005s	sys 0.000s
I20260808 01:51:58.823187 15479 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:58.903981 15122 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:58.905372 15507 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260808 01:51:58.905856  9473 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:58.973464  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:58.837186 15177 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:58.906908  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2399) took 1151 ms. Trace:
I20260808 01:51:58.907665 15508 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:58.821555 15443 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:58.821877 15226 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.096s
I20260808 01:51:58.910120 15391 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:58.910988 15401 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:58.821445 15304 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:58.845374 15517 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14.
W20260808 01:51:58.912483 15532 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.399s	user 0.000s	sys 0.001s
I20260808 01:51:58.913573  9559 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: [13.6-13.6]
I20260808 01:51:58.822031 15514 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:58.918530 15448 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:58.919703 15406 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:58.920655  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 925) took 1417 ms. Trace:
I20260808 01:51:59.043550  9472 rpcz_store.cc:276] 0808 01:51:57.503560 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:57.522785 (+ 19225us) service_pool.cc:224] Handling call
0808 01:51:58.851388 (+1328603us) spinlock_profiling.cc:243] Waited 5.45 ms on lock 0x7b5800005498. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:51:58.851609 (+   221us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:58.920259 (+ 68650us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:58.920561 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12541696}
I20260808 01:51:58.921836  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:58.844389  9600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2549) took 1514 ms. Trace:
W20260808 01:51:58.952785  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 230609us
W20260808 01:51:58.953393 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 350608us
I20260808 01:51:58.958328  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:58.974378  9473 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:58.954692 15509 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:58.978435  9555 rpcz_store.cc:276] 0808 01:51:57.755379 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:57.875532 (+120153us) service_pool.cc:224] Handling call
0808 01:51:58.906648 (+1031116us) spinlock_profiling.cc:243] Waited 649 ms on lock 0x7b54002a8354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:58.906840 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1493528320}
I20260808 01:51:58.961999 15451 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:59.046231  9600 rpcz_store.cc:276] 0808 01:51:57.329762 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:58.844323 (+1514561us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260808 01:51:59.033366 15435 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:59.068770 15509 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:59.072594 15451 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:51:58.981392 15117 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:59.100504 15117 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:59.068467 15406 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.103695 15406 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:59.065068 15448 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:59.111397 15554 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:59.112038 15554 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.114064  9637 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:59.035135  9478 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
W20260808 01:51:59.059616 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:59.116513  9478 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260808 01:51:59.063330  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 708) took 2021 ms. Trace:
I20260808 01:51:59.083765 14784 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:59.094079  9600 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:42840 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2663)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2635)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2634)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2659)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2648)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2638)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2636)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2640)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2644)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2631)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2637)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2647)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2650)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2658)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2642)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2651)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2652)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2649)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2567)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2645)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34802 (request call id 69)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 709)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 710)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 712)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2630)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2541)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2553)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2558)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2563)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2566)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 713)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2632)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2646)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2653)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2654)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2655)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2656)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2657)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2660)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2661)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2662)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2664)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 714)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 715)
I20260808 01:51:59.098407 15435 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.119241  9634 rpcz_store.cc:276] 0808 01:51:57.042312 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:57.163674 (+121362us) service_pool.cc:224] Handling call
0808 01:51:57.193822 (+ 30148us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:57.194290 (+   468us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:57.194333 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:58.440674 (+1246341us) spinlock_profiling.cc:243] Waited 2.51 ms on lock 0x7b2c000b67f8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:51:58.455163 (+ 14489us) log.cc:844] Serialized 26 byte log entry
0808 01:51:58.573871 (+118708us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:51:58.979261 (+405390us) log.cc:844] Serialized 14 byte log entry
0808 01:51:58.981183 (+  1922us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:58.982635 (+  1452us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:59.062841 (+ 80206us) raft_consensus.cc:1685] finished
0808 01:51:59.062980 (+   139us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:59.063218 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3799,"prepare.queue_time_us":496333,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":1359,"raft.queue_time_us":20258,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":30732,"spinlock_wait_cycles":5782912,"thread_start_us":493820,"threads_started":2,"wal-append.queue_time_us":500343}
I20260808 01:51:59.090502  9473 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:59.127797  9478 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: [13.6-13.6]
I20260808 01:51:58.983515 15122 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.133052 15122 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:58.985906 15177 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.133256  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.134848  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:59.133733  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2783) took 1121 ms. Trace:
I20260808 01:51:59.135769 15177 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:59.137673  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3178) took 1406 ms. Trace:
I20260808 01:51:59.138191  9551 rpcz_store.cc:276] 0808 01:51:57.732145 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:57.732295 (+   150us) service_pool.cc:224] Handling call
0808 01:51:57.732634 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:58.486906 (+754272us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:59.137424 (+650518us) spinlock_profiling.cc:243] Waited 753 ms on lock 0x7b54000b45d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:59.137614 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1733024000}
I20260808 01:51:59.141669  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:59.092886  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:59.097571  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 270138us
I20260808 01:51:59.100845 15443 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:59.058564  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:51:59.149439  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 90983.4us
I20260808 01:51:59.120118 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 743) took 1614 ms. Trace:
I20260808 01:51:58.985476 15516 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.079s
I20260808 01:51:59.086953 15479 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:51:59.115985 10332 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:45658 dropped due to backpressure: service queue is full with 50 items [suppressed 36 similar messages]
I20260808 01:51:59.156836 10332 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:59.126753  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 803) took 1353 ms. Trace:
I20260808 01:51:59.158094  9557 rpcz_store.cc:276] 0808 01:51:57.772786 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:57.973010 (+200224us) service_pool.cc:224] Handling call
0808 01:51:59.126417 (+1153407us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b54003269d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:59.126626 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":430240640}
I20260808 01:51:58.980320 15518 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:59.095288 15496 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.016s	sys 0.019s
I20260808 01:51:58.988543  9481 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:51:59.145931  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:59.135584  9476 rpcz_store.cc:276] 0808 01:51:58.011828 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:58.016755 (+  4927us) service_pool.cc:224] Handling call
0808 01:51:58.017026 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:58.017258 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:51:58.043257 (+ 25999us) log.cc:844] Serialized 14 byte log entry
0808 01:51:58.480809 (+437552us) spinlock_profiling.cc:243] Waited 53.1 ms on lock 0x7b18002a87a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d4453242a 00007f5d4499de0a
0808 01:51:59.133331 (+652522us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:59.133395 (+    64us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:51:59.133450 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:59.133531 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:59.133669 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":92042,"prepare.queue_time_us":25663,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":122,"raft.queue_time_us":110668,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"spinlock_wait_cycles":122240768,"thread_start_us":652180,"threads_started":2,"wal-append.queue_time_us":871415}
I20260808 01:51:59.151084 10361 rpcz_store.cc:276] 0808 01:51:57.505298 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:57.509619 (+  4321us) service_pool.cc:224] Handling call
0808 01:51:57.509963 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:57.789369 (+279406us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:59.119743 (+1330374us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b5400681d54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:59.119986 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":553182976}
I20260808 01:51:59.149897  9524 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:59.165930  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:59.169713 15479 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:51:59.170424 15479 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:59.170951 15479 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:51:59.171957 15435 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:59.177920  9481 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260808 01:51:59.168900  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:59.161064  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:59.184697  9444 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:51:59.166021 15496 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.016s	sys 0.020s
I20260808 01:51:59.186510  9635 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:51:59.162160 15517 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:59.189256  9481 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260808 01:51:59.196494  9521 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:59.162842 15510 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:51:59.178254  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:59.200664  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 707) took 2158 ms. Trace:
I20260808 01:51:59.201644  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:59.205734  9634 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:51:59.184609 15516 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:51:59.206135  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:59.163453 15369 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.036s
I20260808 01:51:59.190277 15412 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:51:59.190927 15406 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260808 01:51:59.162173 15177 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.036s
I20260808 01:51:59.175223 15401 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
I20260808 01:51:59.162535 15514 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.036s
I20260808 01:51:59.188949 15448 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:59.198869 15517 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:59.208189  9481 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:51:59.195899 15435 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:51:59.207837  9632 rpcz_store.cc:276] 0808 01:51:57.042129 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:57.146482 (+104353us) service_pool.cc:224] Handling call
0808 01:51:57.170610 (+ 24128us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:57.340662 (+170052us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:57.340714 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:57.341554 (+   840us) log.cc:844] Serialized 52 byte log entry
0808 01:51:57.605142 (+263588us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:51:58.822416 (+1217274us) spinlock_profiling.cc:243] Waited 6.79 ms on lock 0x7b18002edce0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4453a6ac 00007f5d4186fe3b 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d41889e05 00007f5d483aada7 00007f5d4499dd87 00007f5d449ad49f
0808 01:51:58.822616 (+   200us) log.cc:844] Serialized 14 byte log entry
0808 01:51:59.192237 (+369621us) log.cc:844] Serialized 14 byte log entry
0808 01:51:59.192511 (+   274us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:59.192595 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:59.200358 (+  7763us) raft_consensus.cc:1685] finished
0808 01:51:59.200385 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:59.200598 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28501,"prepare.queue_time_us":213460,"prepare.run_cpu_time_us":112,"prepare.run_wall_time_us":5846,"raft.queue_time_us":36152,"raft.run_cpu_time_us":529,"raft.run_wall_time_us":65868,"spinlock_wait_cycles":15620864,"thread_start_us":924382,"threads_started":5,"wal-append.queue_time_us":418676}
I20260808 01:51:59.216524  9601 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:42840 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2638)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2636)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2640)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2644)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2631)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2637)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2647)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2650)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2658)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2642)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 719)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2651)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2652)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2671)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2667)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2568)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2665)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2649)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2666)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2645)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2669)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34802 (request call id 69)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 709)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 710)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 712)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2630)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2541)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2553)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2558)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2563)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2566)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 713)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2632)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2646)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2653)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2654)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2655)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2656)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2657)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2660)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2661)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2662)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2664)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 714)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 715)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2668)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2670)
I20260808 01:51:59.198335 15388 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:59.192211 15554 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:59.234201 15412 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:59.247057  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:59.207338  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:59.206372 15547 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:51:59.207108 15556 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:59.352278 15556 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:59.351847  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:59.209465  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:59.236336 15524 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:51:59.210717 15510 raft_consensus.cc:697] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:59.233088 15433 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:51:59.338039  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:59.209263 14784 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:59.347249 15304 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:59.267192 15376 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.084s
I20260808 01:51:59.351342 15432 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:59.351524  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:51:59.355466 14833 consensus_meta.cc:287] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.534s	user 0.010s	sys 0.002s
I20260808 01:51:59.383666  9632 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:51:59.355310  9481 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:51:59.364118  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:51:59.363443  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 115506us
I20260808 01:51:59.244433 15517 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:59.338220 15412 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:51:59.367763  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 160496us
W20260808 01:51:59.389719  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:59.389664  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:59.387526 15432 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.391369  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:59.392758  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:59.393847  9550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2386) took 1871 ms (client timeout 1723 ms). Trace:
I20260808 01:51:59.394075  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:51:59.394343  9550 rpcz_store.cc:269] 0808 01:51:57.530211 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:57.540565 (+ 10354us) service_pool.cc:224] Handling call
0808 01:51:59.393638 (+1853073us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b54000ff854. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:59.393772 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":796798208}
I20260808 01:51:59.395190  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 806) took 1621 ms. Trace:
I20260808 01:51:59.395596  9558 rpcz_store.cc:276] 0808 01:51:57.773690 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:58.104008 (+330318us) service_pool.cc:224] Handling call
0808 01:51:59.394082 (+1290074us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:59.394274 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:59.394300 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:59.394342 (+    42us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:51:59.394372 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:59.394426 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:59.395019 (+   593us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54000ff850. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:59.395130 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2929258880}
I20260808 01:51:59.396613  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:59.397383  9478 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:51:59.397121 15535 consensus_queue.cc:260] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.398561 15554 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:59.398763 15535 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:59.399537 15554 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:51:59.400274 15554 raft_consensus.cc:2764] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260808 01:51:59.367965 15481 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 01:51:59.401540 15451 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.367763 15449 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:51:59.404168  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:59.404637 15492 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:59.404919  9476 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 9.
I20260808 01:51:59.368381 15414 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260808 01:51:59.370005 15558 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.165s	user 0.000s	sys 0.000s
I20260808 01:51:59.371006 15433 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:59.415244 15556 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:59.371491 14784 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.415925 15433 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.377102  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:59.421188  9635 raft_consensus.cc:2368] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12.
I20260808 01:51:59.357194 15510 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:51:59.385623  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:59.398473  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:59.417968  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:59.338972 15411 raft_consensus.cc:3055] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:51:59.422569  9478 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 11: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:51:59.424180  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.426939  9635 raft_consensus.cc:2368] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12.
I20260808 01:51:59.426781  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:59.427601  9481 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260808 01:51:59.429504  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.433032  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.434085  9635 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:51:59.437891  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.438206  9481 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260808 01:51:59.439468  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:59.435554  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.444941  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:59.424279 15406 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:51:59.387218 14833 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.428195 15304 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:51:59.455730  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:59.423542 15411 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:51:59.462426  9638 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:59.456796  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:59.457015  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2929) took 1955 ms. Trace:
I20260808 01:51:59.443727  9635 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:59.464205  9632 raft_consensus.cc:2376] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16. Re-sending same reply.
I20260808 01:51:59.464152  9480 rpcz_store.cc:276] 0808 01:51:57.501485 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:57.513621 (+ 12136us) service_pool.cc:224] Handling call
0808 01:51:57.513925 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:57.625715 (+111790us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:59.456713 (+1830998us) spinlock_profiling.cc:243] Waited 77.8 ms on lock 0x7b54002bc5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:51:59.456943 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":178984320}
I20260808 01:51:59.465294  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.466004  9632 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:51:59.448220 15554 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260808 01:51:59.462865 15561 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20260808 01:51:59.472265  9634 raft_consensus.cc:2368] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15.
I20260808 01:51:59.471067  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:59.437402 15376 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260808 01:51:59.458717 15406 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:59.474969  9632 raft_consensus.cc:2410] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260808 01:51:59.435772 15516 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:51:59.484232  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.485325  9632 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:51:59.486617  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.446978 15564 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.030s
I20260808 01:51:59.490788  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.492040  9632 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 15: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:59.493119  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.493896  9632 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:51:59.495072  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:59.496409  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.497884  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.499277  9632 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:51:59.483923 15406 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:51:59.507747  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3184) took 1281 ms. Trace:
I20260808 01:51:59.513870  9559 rpcz_store.cc:276] 0808 01:51:58.226321 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:58.228351 (+  2030us) service_pool.cc:224] Handling call
0808 01:51:58.913281 (+684930us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:59.030859 (+117578us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:59.507446 (+476587us) spinlock_profiling.cc:243] Waited 684 ms on lock 0x7b54002a8350. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:59.507661 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1574219136}
I20260808 01:51:59.489625 15518 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:59.525224  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:59.520440  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:59.505785  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:59.526098  9554 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:51:59.526228  9550 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:51:59.527360  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.528564  9632 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
W20260808 01:51:59.521684  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:59.522050  9638 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:51:59.522488  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:59.525311  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:59.517397  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:59.530933  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.532274  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.533301  9632 raft_consensus.cc:2393] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 10: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:59.533475  9638 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:51:59.536580  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:59.537147 15557 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:59.537417 14833 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:59.537850 15557 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:59.540627 15557 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:59.541312 15557 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:59.541909 15557 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.514375 15401 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260808 01:51:59.548192  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:59.553434 15479 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:59.555665 15117 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:59.580585 15496 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:51:59.549821  9601 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:42824 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2576)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2575)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2571)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2577)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2572)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2573)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2574)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2649)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2666)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2645)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2669)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2578)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34802 (request call id 69)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 709)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 710)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 712)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2630)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2541)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2550)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2553)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2558)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2563)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2566)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 713)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2632)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2646)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2653)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2654)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2655)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2656)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2657)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2660)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2661)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2662)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2664)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 714)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 715)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2668)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2670)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 721)
I20260808 01:51:59.533403 15304 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:59.578473 15557 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:59.579114 15479 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:59.547266  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2554) took 2175 ms. Trace:
I20260808 01:51:59.503286 15411 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.529650 15554 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:59.593524  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:59.594551  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:59.602476  9629 rpcz_store.cc:276] 0808 01:51:57.373190 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:57.906060 (+532870us) service_pool.cc:224] Handling call
0808 01:51:59.547023 (+1640963us) spinlock_profiling.cc:243] Waited 549 ms on lock 0x7b540003b6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:59.547190 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1261715840}
I20260808 01:51:59.607734  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.592289  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:59.593190 15562 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.204s	user 0.002s	sys 0.000s
I20260808 01:51:59.573622 14784 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:59.615826  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:59.617564  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.624724  9629 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 18: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:59.625670  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:59.627882  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:59.634228  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.635075  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:59.635355  9629 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:51:59.636860  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.638454  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.644500  9629 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:51:59.645998  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.647148  9629 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 17: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:59.642125  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:59.651281  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:59.653100  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:59.653427  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:59.654062  9560 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:51:59.573211 15516 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.600535 15496 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.655834  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:59.657928  9560 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260808 01:51:59.658104  9476 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:59.605255 15304 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:51:59.656596  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.615648 15554 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.662061  9629 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 15: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:51:59.648876 15557 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:59.663268  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2944) took 1078 ms. Trace:
I20260808 01:51:59.671681  9473 rpcz_store.cc:276] 0808 01:51:58.584482 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:59.127802 (+543320us) service_pool.cc:224] Handling call
0808 01:51:59.128183 (+   381us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:59.128561 (+   378us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:59.128593 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:51:59.129150 (+   557us) log.cc:844] Serialized 26 byte log entry
0808 01:51:59.189023 (+ 59873us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:51:59.189073 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:59.189154 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:51:59.662996 (+473842us) raft_consensus.cc:1685] finished
0808 01:51:59.663020 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:51:59.663197 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12848,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":14913,"thread_start_us":59440,"threads_started":1,"wal-append.queue_time_us":118899}
I20260808 01:51:59.674830  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:59.675824  9550 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:51:59.678503  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:59.679906  9473 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 15: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:59.681334  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:51:59.683380  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:59.684456  9473 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:51:59.685824  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:59.687542  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:59.688596  9473 raft_consensus.cc:2410] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 15 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260808 01:51:59.690006  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:59.694243  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2649) took 1177 ms. Trace:
I20260808 01:51:59.694949  9629 rpcz_store.cc:276] 0808 01:51:58.516280 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:59.656350 (+1140070us) service_pool.cc:224] Handling call
0808 01:51:59.694106 (+ 37756us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:59.696333  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:59.697165  9629 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:51:59.702442  9476 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:51:59.668490 15554 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:59.734452  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 83278us
I20260808 01:51:59.734403  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2618) took 3907 ms. Trace:
I20260808 01:51:59.737911  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:59.737744  9631 rpcz_store.cc:276] 0808 01:51:55.824453 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:56.638068 (+813615us) service_pool.cc:224] Handling call
0808 01:51:57.843365 (+1205297us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:57.859893 (+ 16528us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:59.731494 (+1871601us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54001b49d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:51:59.731665 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2764693888}
I20260808 01:51:59.757651  9476 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 10. Current term is 11. Ops: []
I20260808 01:51:59.760629  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:59.762574  9629 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:51:59.724135 15576 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:51:59.698825 15557 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.770467  9633 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:51:59.739219 15554 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260808 01:51:59.756946 15573 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.085s	user 0.003s	sys 0.000s
I20260808 01:51:59.773036  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:59.763983 15414 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:59.776980  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.778628  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.778858  9554 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
W20260808 01:51:59.781337  9637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2663) took 1241 ms (client timeout 1575 ms). Trace:
W20260808 01:51:59.781872  9637 rpcz_store.cc:269] 0808 01:51:58.540434 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:59.778299 (+1237865us) service_pool.cc:224] Handling call
0808 01:51:59.781251 (+  2952us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:59.783358  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:51:59.772754 15570 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.118s	user 0.001s	sys 0.001s
W20260808 01:51:59.774798 15509 consensus_meta.cc:287] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.667s	user 0.009s	sys 0.005s
I20260808 01:51:59.791342  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:59.791425  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.792311  9473 raft_consensus.cc:2410] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260808 01:51:59.792912  9637 raft_consensus.cc:2410] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260808 01:51:59.771205 15576 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:51:59.794637  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2645) took 1279 ms. Trace:
I20260808 01:51:59.774233 15554 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:51:59.779978 15517 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.817253  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.794880 15581 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:51:59.819087  9631 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:51:59.795171  9637 rpcz_store.cc:276] 0808 01:51:58.514986 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:59.791144 (+1276158us) service_pool.cc:224] Handling call
0808 01:51:59.794533 (+  3389us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:59.820365  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:59.821022  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.821640  9637 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:51:59.822048  9554 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 15: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:51:59.823601 15585 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:51:59.824059 15585 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:51:59.824455 15585 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:59.824692  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.825688  9637 raft_consensus.cc:2410] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 6.
I20260808 01:51:59.826727  9637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:51:59.827436  9637 raft_consensus.cc:606] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:51:59.827805  9637 raft_consensus.cc:614] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:51:59.829227  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:51:59.829646 15574 log.cc:927] Time spent T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260808 01:51:59.828226  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34802 (request call id 69) took 2600 ms. Trace:
I20260808 01:51:59.832310  9637 rpcz_store.cc:276] 0808 01:51:57.227936 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:59.826501 (+2598565us) service_pool.cc:224] Handling call
0808 01:51:59.828167 (+  1666us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:59.833086  9480 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 13. Preceding OpId from leader: term: 14 index: 14. (index mismatch)
I20260808 01:51:59.834349 15517 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:59.794291 15509 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:51:59.838721 15575 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
W20260808 01:51:59.839565  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 7d3a904863b545ffa227b964ee3a3f18 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:59.795488 15576 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
W20260808 01:51:59.840853  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:59.818898 15554 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:59.834589  9480 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260808 01:51:59.841624 14784 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:51:59.852342 15571 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.183s	user 0.000s	sys 0.001s
I20260808 01:51:59.828208 15412 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.844115 15509 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:59.822643 15582 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:51:59.845464 15304 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:51:59.846935 15557 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:59.847662 15496 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:51:59.873050  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 81798.3us
I20260808 01:51:59.849938  9632 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 9. Current term is 10. Ops: []
I20260808 01:51:59.875002  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 710) took 2832 ms. Trace:
I20260808 01:51:59.875389  9632 rpcz_store.cc:276] 0808 01:51:57.042804 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:59.849437 (+2806633us) service_pool.cc:224] Handling call
0808 01:51:59.849791 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:59.874651 (+ 24860us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:59.874933 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19712}
I20260808 01:51:59.849794  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:51:59.872502 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:51:59.877979 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:51:59.879985 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:59.880750 10364 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 10.
W20260808 01:51:59.883525 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 726726us
I20260808 01:51:59.886758 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:59.888115  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:51:59.879528  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:59.893019 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:59.893754 10364 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:51:59.895444 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:59.896149 10360 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:51:59.894570  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:59.899461  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:51:59.900385  9551 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:51:59.901425 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:59.868868 15582 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:51:59.904975 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:59.907892 10332 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:59.908890 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:59.912031  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.912863  9632 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:51:59.912961  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:51:59.914793  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:51:59.852391 14784 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:51:59.917423 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:59.920094 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:51:59.921216 10368 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 15: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:51:59.909749 15304 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:51:59.928411 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:59.929939 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:51:59.891772 15452 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:51:59.930308 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:59.932421 10366 raft_consensus.cc:2410] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260808 01:51:59.899176 15557 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:51:59.910508 15582 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:51:59.955943  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:51:59.956779  9555 raft_consensus.cc:2376] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14. Re-sending same reply.
I20260808 01:51:59.956858  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:51:59.957792  9632 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:51:59.979583  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:00.092775  9600 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:00.008203 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48187.4us
I20260808 01:52:00.104296 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:52:00.049438  9636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2561) took 2619 ms (client timeout 2111 ms). Trace:
I20260808 01:51:59.974900  9523 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:51:59.937429 15457 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Leader election won for term 8
I20260808 01:51:59.968029  9551 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
W20260808 01:51:59.960574  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47903us
I20260808 01:52:00.116323  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:00.048882 15406 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.521s	user 0.014s	sys 0.000s
I20260808 01:52:00.117265 15581 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:00.117966 15581 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:00.117882 15406 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:00.118537 15581 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:00.118999 15479 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:52:00.119966 15479 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:00.120389 15479 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:52:00.120858 15406 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:00.112895  9636 rpcz_store.cc:269] 0808 01:51:57.429938 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:57.630260 (+200322us) service_pool.cc:224] Handling call
0808 01:52:00.049138 (+2418878us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b54001261d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:00.049337 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":748804864}
I20260808 01:52:00.122617  9636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:00.123570  9636 raft_consensus.cc:606] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:00.123915  9636 raft_consensus.cc:614] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:00.125336  9636 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:00.125962  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 712) took 3082 ms. Trace:
I20260808 01:52:00.126277  9636 rpcz_store.cc:276] 0808 01:51:57.044017 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:00.124831 (+3080814us) service_pool.cc:224] Handling call
0808 01:52:00.125156 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:00.125716 (+   560us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:00.125907 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:51:59.995107 15509 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:00.128288 15298 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:00.128129 15509 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:00.131283 15509 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:51:59.960960 15500 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:51:59.952643 15557 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:51:59.999353  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:00.134348 15557 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:00.138738  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:00.112530  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:00.016153 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:51:59.983546 10364 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 13. Preceding OpId from leader: term: 14 index: 14. (index mismatch)
I20260808 01:51:59.982522 15412 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:51:59.952839 15587 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:52:00.054160 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:00.144505  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:00.149544  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet affdd24b03ae4ee795a90703613305f7 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:51:59.954744 15504 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:52:00.157195  9480 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260808 01:52:00.157716  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:00.158793 15496 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:52:00.159287  9481 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13. Ops: []
I20260808 01:52:00.160687 15576 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:00.164265 15579 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.384s	user 0.002s	sys 0.000s
I20260808 01:52:00.148911  9632 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:00.150267  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:52:00.152683  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:00.169189  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
W20260808 01:52:00.152689  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 177978us
I20260808 01:52:00.089530 15551 raft_consensus.cc:3055] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:52:00.151400 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:00.170300  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:00.170662  9523 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:00.020335 15304 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:00.134109 15457 raft_consensus.cc:697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:00.177924  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:52:00.184458  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:00.186172 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:00.188372  9632 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260808 01:52:00.190323  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2541) took 2886 ms. Trace:
I20260808 01:52:00.191263  9632 rpcz_store.cc:276] 0808 01:51:57.303435 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:00.168514 (+2865079us) service_pool.cc:224] Handling call
0808 01:52:00.168960 (+   446us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:00.190029 (+ 21069us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:00.190258 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:00.192907  9632 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 6. Preceding OpId from leader: term: 15 index: 7. (index mismatch)
I20260808 01:52:00.193670  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2550) took 2843 ms. Trace:
I20260808 01:52:00.194116  9632 rpcz_store.cc:276] 0808 01:51:57.350001 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:00.192229 (+2842228us) service_pool.cc:224] Handling call
0808 01:52:00.192552 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:00.193401 (+   849us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:00.193611 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:00.189615 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:00.190371  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:00.163807 15581 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:00.180612 15585 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:00.182211 15576 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:00.199322  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:00.167800 15509 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:52:00.200922  9632 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:00.167603 15406 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:00.265854 15585 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:00.266822 15585 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:52:00.132331 15500 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:00.243479  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:51:59.955832 15188 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.221s
I20260808 01:52:00.188310 15517 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:52:00.262437  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2553) took 2891 ms. Trace:
I20260808 01:52:00.267506 15585 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:00.279737 15585 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:00.263159 15509 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:00.188134 15504 raft_consensus.cc:697] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:00.175454 15526 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260808 01:52:00.173761 15551 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:00.277849  9632 rpcz_store.cc:276] 0808 01:51:57.371107 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:00.200413 (+2829306us) service_pool.cc:224] Handling call
0808 01:52:00.200738 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:00.262096 (+ 61358us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:00.262364 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:00.295506  9632 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 6. Preceding OpId from leader: term: 15 index: 7. (index mismatch)
I20260808 01:52:00.296526  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2558) took 2872 ms. Trace:
I20260808 01:52:00.241672 15292 tablet_copy_client.cc:538] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:52:00.205260 15304 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:00.209239 15412 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260808 01:52:00.240091 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 54028.9us
I20260808 01:52:00.210626 15449 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:52:00.304061 15449 raft_consensus.cc:697] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:00.304219 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:00.268252 15461 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:52:00.305244 15461 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:00.305851 15461 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:00.199265 15298 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:00.195823 15587 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:00.308385 15455 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:00.187479 15516 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.099s
I20260808 01:52:00.288662 15509 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:00.267777 15501 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.029s
I20260808 01:52:00.271147 15527 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.038s
I20260808 01:52:00.309152 15412 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:00.306469 15479 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:00.312798 15455 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:00.313140 15581 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:00.328622 15455 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:00.329366 15500 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:00.332571 15554 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:00.302107 15457 consensus_queue.cc:237] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:00.339875 15554 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:00.337502 15551 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:00.341060 15551 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:52:00.349193 15479 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:00.350359 15557 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:00.352594 15557 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:00.327726 15504 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:00.334666 15582 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:00.358773 15582 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:00.394068  9478 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:52:00.297116  9632 rpcz_store.cc:276] 0808 01:51:57.424493 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:00.294720 (+2870227us) service_pool.cc:224] Handling call
0808 01:52:00.295099 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:00.296139 (+  1040us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:00.296450 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:00.436647 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:00.297940 14784 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:00.310571 15304 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:00.557343 15304 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:00.310962 15298 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:00.323271 15412 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:00.552999 15594 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:00.556541 10368 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:52:00.567941 15298 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:00.334808 15449 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:00.580262  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:00.591521  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:00.592168  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:00.593192  9554 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:00.593652  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:00.594264  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:00.599375  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:00.599699  9551 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:00.600013  9479 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: [13.7-14.8]
I20260808 01:52:00.583321 15461 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:00.601671  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:00.591519  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:00.562952  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:00.634775  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 709) took 3592 ms. Trace:
I20260808 01:52:00.640707  9555 raft_consensus.cc:2410] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260808 01:52:00.642755  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:00.640439  9637 rpcz_store.cc:276] 0808 01:51:57.042487 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:59.833187 (+2790700us) service_pool.cc:224] Handling call
0808 01:51:59.833518 (+   331us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:59.836638 (+  3120us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:59.836672 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:59.837314 (+   642us) log.cc:844] Serialized 52 byte log entry
0808 01:52:00.019811 (+182497us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:00.429859 (+410048us) log.cc:844] Serialized 14 byte log entry
0808 01:52:00.617316 (+187457us) log.cc:844] Serialized 14 byte log entry
0808 01:52:00.633896 (+ 16580us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:00.634051 (+   155us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:00.634155 (+   104us) raft_consensus.cc:1685] finished
0808 01:52:00.634167 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:00.634418 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":124189,"raft.run_cpu_time_us":787,"raft.run_wall_time_us":15823,"thread_start_us":638440,"threads_started":3,"wal-append.queue_time_us":340301}
I20260808 01:52:00.644860  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:00.647296  9637 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260808 01:52:00.667460  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2566) took 3218 ms. Trace:
I20260808 01:52:00.666723 15509 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:00.667976  9637 rpcz_store.cc:276] 0808 01:51:57.448389 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:00.644495 (+3196106us) service_pool.cc:224] Handling call
0808 01:52:00.646643 (+  2148us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:00.666736 (+ 20093us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:00.666989 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:00.670172  9637 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260808 01:52:00.670267 15509 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:00.655342  9633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2659) took 2121 ms (client timeout 1702 ms). Trace:
W20260808 01:52:00.670969  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 419229us
I20260808 01:52:00.671809  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 713) took 3174 ms. Trace:
I20260808 01:52:00.696626  9637 rpcz_store.cc:276] 0808 01:51:57.497285 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:00.669119 (+3171834us) service_pool.cc:224] Handling call
0808 01:52:00.669767 (+   648us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:00.671399 (+  1632us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:00.671712 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:00.676389  9599 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:00.681164  9554 raft_consensus.cc:2410] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260808 01:52:00.683145  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:00.683729 15461 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:00.684324  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:00.685017  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:00.662714 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 380788us
I20260808 01:52:00.704910 15457 consensus_queue.cc:1059] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:00.635011  9556 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:00.707867  9556 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:00.709353  9556 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:00.695897  9633 rpcz_store.cc:269] 0808 01:51:58.534084 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:59.145270 (+611186us) service_pool.cc:224] Handling call
0808 01:52:00.636253 (+1490983us) spinlock_profiling.cc:243] Waited 498 ms on lock 0x7b54003a0054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:00.655029 (+ 18776us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1146281088}
I20260808 01:52:00.717101 15605 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:00.717407  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:00.717952 15605 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:00.662482  9521 connection.cc:256] RPC call timeout handler was delayed by -1.0961771619913634s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1796s for a response
I20260808 01:52:00.719105  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:00.719345  9521 connection.cc:256] RPC call timeout handler was delayed by -1.0736771619995125s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1821s for a response
I20260808 01:52:00.719988  9554 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
W20260808 01:52:00.720135  9521 connection.cc:256] RPC call timeout handler was delayed by -1.0286771619867068s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18710000000000002s for a response
I20260808 01:52:00.720687 15605 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:00.721124  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:00.721984  9554 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:52:00.722571  9637 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:00.723416  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2632) took 2232 ms. Trace:
I20260808 01:52:00.723796  9637 rpcz_store.cc:276] 0808 01:51:58.491202 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:00.721559 (+2230357us) service_pool.cc:224] Handling call
0808 01:52:00.722162 (+   603us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:00.723144 (+   982us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:00.723354 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:00.725126  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:00.686911  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:00.672962  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 465584us
I20260808 01:52:00.690258 10360 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260808 01:52:00.710614  9556 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:00.661267 10364 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260808 01:52:00.646629  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:00.660235  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:00.693148 15594 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:00.727255  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2793) took 2203 ms. Trace:
I20260808 01:52:00.731936  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:00.733068 15598 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:52:00.738492 15453 consensus_queue.cc:1059] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:00.740948 15478 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:00.741179 15594 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:00.742211  9635 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
W20260808 01:52:00.742900  9635 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2642) took 2247 ms (client timeout 2207 ms). Trace:
W20260808 01:52:00.743291  9635 rpcz_store.cc:269] 0808 01:51:58.495835 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:59.464803 (+968968us) service_pool.cc:224] Handling call
0808 01:52:00.659769 (+1194966us) spinlock_profiling.cc:243] Waited 8.28 ms on lock 0x7b5800231698. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:52:00.742794 (+ 83025us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19042560}
I20260808 01:52:00.744496  9551 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260808 01:52:00.745353  9635 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 7. (term mismatch)
I20260808 01:52:00.745757  9635 pending_rounds.cc:85] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 6
I20260808 01:52:00.746009  9635 pending_rounds.cc:107] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 13.7
I20260808 01:52:00.746333  9635 raft_consensus.cc:2889] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:00.748387 15414 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:00.748972 15414 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:52:00.751103  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 75130.9us
I20260808 01:52:00.751294 15554 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:52:00.752161  9599 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:00.753577 15554 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:52:00.758488 15451 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:00.758925  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:00.706496 15457 raft_consensus.cc:3055] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:52:00.754789 15582 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260808 01:52:00.759769 15457 raft_consensus.cc:740] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:00.759548 15451 consensus_queue.cc:1059] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:00.761158  9633 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260808 01:52:00.761936 15597 raft_consensus.cc:3055] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:52:00.762387 15597 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:00.763484 15479 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260808 01:52:00.765116 15594 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:52:00.765623 15594 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:00.766908 15594 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 13.6, Last appended by leader: 6, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:00.768672 15594 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:00.776368 15265 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:00.777529 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:00.780566  9474 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:00.759673 15557 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:00.781673 15557 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:00.782054  9559 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260808 01:52:00.755359  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:00.755723 15598 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 01:52:00.757468 15603 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:52:00.758309 15600 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:52:00.730017  9479 rpcz_store.cc:276] 0808 01:51:58.434156 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:00.598377 (+2164221us) service_pool.cc:224] Handling call
0808 01:52:00.599416 (+  1039us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:00.626533 (+ 27117us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:00.637088 (+ 10555us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:00.802636 15500 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:52:00.803336 15500 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:00.803719 15500 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:00.805099  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:00.805238  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:00.805953  9474 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 11
I20260808 01:52:00.806097  9638 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:52:00.808218  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:00.809463  9631 raft_consensus.cc:2410] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260808 01:52:00.810048  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2582) took 1050 ms. Trace:
I20260808 01:52:00.812275  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:00.812731 10363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 672) took 2850 ms. Trace:
I20260808 01:52:00.809120 15527 consensus_queue.cc:1059] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:00.816447  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:00.820734 10368 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:52:00.810412  9631 rpcz_store.cc:276] 0808 01:51:59.759556 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:59.820930 (+ 61374us) service_pool.cc:224] Handling call
0808 01:52:00.807903 (+986973us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x7b5800231698. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d4185589e 000000000038d209 00007f5d49914940 00007f5d499dced4 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:52:00.809938 (+  2035us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2675200}
I20260808 01:52:00.825654  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:00.787976 15451 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260808 01:52:00.843613 15298 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:00.846558  9560 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:00.847707 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:00.850948 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:00.859488 15432 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:00.865990  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2654) took 2307 ms. Trace:
I20260808 01:52:00.787402 15603 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:00.790108 15479 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:00.818274 15598 consensus_queue.cc:1243] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 12.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260808 01:52:00.868814  9635 rpcz_store.cc:276] 0808 01:51:58.529614 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:00.744208 (+2214594us) service_pool.cc:224] Handling call
0808 01:52:00.744751 (+   543us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:00.836153 (+ 91402us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:00.836433 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:00.865584  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:00.874545 15479 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:00.875284  9552 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 11. (term mismatch)
I20260808 01:52:00.875743  9552 pending_rounds.cc:85] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 10
I20260808 01:52:00.876034  9552 pending_rounds.cc:107] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20260808 01:52:00.876304  9552 raft_consensus.cc:2889] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:00.877327  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 816) took 2332 ms. Trace:
I20260808 01:52:00.877483 15479 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:00.878180 15479 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260808 01:52:00.879055 15334 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
I20260808 01:52:00.813117 10363 rpcz_store.cc:276] 0808 01:51:57.962457 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:57.962724 (+   267us) service_pool.cc:224] Handling call
0808 01:51:57.983096 (+ 20372us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:51:58.156617 (+173521us) raft_consensus.cc:1534] Early marking committed up to index 14
0808 01:51:58.241762 (+ 85145us) log.cc:844] Serialized 14 byte log entry
0808 01:51:58.321414 (+ 79652us) log.cc:844] Serialized 14 byte log entry
0808 01:51:58.896333 (+574919us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b18004c4620. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4185ac56 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188ce62 00007f5d4188f121
0808 01:51:58.898156 (+  1823us) spinlock_profiling.cc:243] Waited 1 ms on lock 0x7b18004c4620. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188ce62 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:51:59.663017 (+764861us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b18004c4620. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:51:59.854580 (+191563us) spinlock_profiling.cc:243] Waited 19.2 ms on lock 0x7b18004c4620. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667 00007f5d449ae5e8
0808 01:51:59.856066 (+  1486us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:51:59.856126 (+    60us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:51:59.856179 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:00.812227 (+956048us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b5400694954. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:00.812313 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:00.812550 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1538336,"prepare.queue_time_us":733895,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":112,"raft.queue_time_us":1545765,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":3528,"spinlock_wait_cycles":480243200,"thread_start_us":135146,"threads_started":2,"wal-append.queue_time_us":82336,"wal-append.run_cpu_time_us":2339,"wal-append.run_wall_time_us":1116898}
I20260808 01:52:00.877743  9552 rpcz_store.cc:276] 0808 01:51:58.545051 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:58.779338 (+234287us) service_pool.cc:224] Handling call
0808 01:51:58.809961 (+ 30623us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:00.874803 (+2064842us) spinlock_profiling.cc:243] Waited 90.8 ms on lock 0x7b5400089920. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490b1a0 00007f5d44996167 00007f5d44996850 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c
0808 01:52:00.876957 (+  2154us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:00.877253 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":208813440}
I20260808 01:52:00.783133 15557 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:00.880241  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:00.891566 10360 raft_consensus.cc:2368] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16.
I20260808 01:52:00.896015  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:00.839885 15600 raft_consensus.cc:697] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:00.873490 15603 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:00.905591 10368 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 13. Current term is 14. Ops: []
I20260808 01:52:00.763717 15597 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:00.918869  9560 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:00.931356  9474 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
W20260808 01:52:00.932374  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 211607us
I20260808 01:52:00.944181  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2646) took 2413 ms. Trace:
I20260808 01:52:00.947180  9556 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
I20260808 01:52:00.944936  9633 rpcz_store.cc:276] 0808 01:51:58.515359 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:00.712252 (+2196893us) service_pool.cc:224] Handling call
0808 01:52:00.712853 (+   601us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:00.761626 (+ 48773us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:00.917140 (+155514us) spinlock_profiling.cc:243] Waited 48 ms on lock 0x7b540003bbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:00.928463 (+ 11323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":110329088}
W20260808 01:52:00.950273  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 195116us
I20260808 01:52:00.951620  9599 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:00.947686  9556 pending_rounds.cc:85] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 13
I20260808 01:52:00.961100  9556 pending_rounds.cc:107] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 17.14
I20260808 01:52:00.961445  9556 raft_consensus.cc:2889] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:00.967553  9631 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:00.982102  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:00.988830  9633 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:52:00.989490 15455 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.660s	user 0.022s	sys 0.002s
I20260808 01:52:00.989889  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:00.990129  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:01.001112  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 814) took 2456 ms. Trace:
I20260808 01:52:00.990218 15455 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:00.994380 15557 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:52:00.998127  9521 leader_election.cc:336] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.737s (SENT)
I20260808 01:52:01.001757  9556 rpcz_store.cc:276] 0808 01:51:58.544412 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:58.603488 (+ 59076us) service_pool.cc:224] Handling call
0808 01:51:58.603842 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:00.608328 (+2004486us) spinlock_profiling.cc:243] Waited 386 ms on lock 0x7b54001593a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490b1a0 00007f5d449955a0 00007f5d44996611 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c
0808 01:52:00.987838 (+379510us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.001027 (+ 13189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":888467840}
W20260808 01:52:01.014230 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:01.011330 15412 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:01.017542  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3197) took 1431 ms. Trace:
I20260808 01:52:01.018011  9550 rpcz_store.cc:276] 0808 01:51:59.586118 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:59.625261 (+ 39143us) service_pool.cc:224] Handling call
0808 01:52:01.009930 (+1384669us) spinlock_profiling.cc:243] Waited 49 ms on lock 0x7b54001dfe54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:01.017335 (+  7405us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":112767872}
I20260808 01:52:01.019469 15401 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:00.995898 15565 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Stepping down as leader of term 15
W20260808 01:52:01.008468 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 101766us
I20260808 01:52:01.024657  9476 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: []
W20260808 01:52:01.023198 15608 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.160s	user 0.000s	sys 0.002s
I20260808 01:52:01.048069  9633 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
W20260808 01:52:01.019791 15552 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 1.889s	user 0.006s	sys 0.000s
I20260808 01:52:01.024349 15597 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:01.053886  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 933) took 2752 ms. Trace:
I20260808 01:52:01.023661 15598 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:01.058831 15527 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:01.024570  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:01.054344  9477 rpcz_store.cc:276] 0808 01:51:58.302377 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:58.529189 (+226812us) service_pool.cc:224] Handling call
0808 01:51:58.530092 (+   903us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:51:58.530499 (+   407us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:51:58.530526 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:51:58.557297 (+ 26771us) log.cc:844] Serialized 52 byte log entry
0808 01:51:58.987577 (+430280us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:51:58.987643 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:51:58.987748 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:01.018661 (+2030913us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:01.053277 (+ 34616us) raft_consensus.cc:1685] finished
0808 01:52:01.053300 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:01.053657 (+   357us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1629,"raft.queue_time_us":1565,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":16545,"spinlock_wait_cycles":29820288,"thread_start_us":317151,"threads_started":1,"wal-append.queue_time_us":546759}
W20260808 01:52:01.061801 15553 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.954s	user 0.000s	sys 0.007s
I20260808 01:52:01.063499 15527 raft_consensus.cc:3055] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:01.064100 15527 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:01.065418 15527 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:01.066727  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:01.066927 15527 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:01.069784 15270 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:01.073709  9630 raft_consensus.cc:2410] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 16 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260808 01:52:01.055608  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2563) took 3624 ms. Trace:
I20260808 01:52:01.012233 15479 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:01.075418  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2653) took 2546 ms. Trace:
I20260808 01:52:01.104828  9638 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
W20260808 01:52:01.020568 15296 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 18 current term: 18 is higher.
I20260808 01:52:01.098467  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:01.019979 15412 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:01.024446 15455 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:01.108712  9629 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260808 01:52:01.111114  9633 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:01.066864 15383 raft_consensus.cc:3055] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:01.025434 15298 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:01.113047  9631 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:01.103907  9637 rpcz_store.cc:276] 0808 01:51:58.529097 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:00.726572 (+2197475us) service_pool.cc:224] Handling call
0808 01:52:00.734217 (+  7645us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:00.734789 (+   572us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:00.734855 (+    66us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:00.818972 (+ 84117us) log.cc:844] Serialized 26 byte log entry
0808 01:52:00.994205 (+175233us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:52:00.994249 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:00.994320 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:01.075131 (+ 80811us) raft_consensus.cc:1685] finished
0808 01:52:01.075156 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:01.075353 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":832,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":34146,"thread_start_us":103277,"threads_started":1,"wal-append.queue_time_us":167773}
W20260808 01:52:01.114406  9638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2576) took 1582 ms (client timeout 1731 ms). Trace:
W20260808 01:52:01.123046  9638 rpcz_store.cc:269] 0808 01:51:59.523463 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:00.804673 (+1281210us) service_pool.cc:224] Handling call
0808 01:52:01.105863 (+301190us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:01.044478 15565 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:01.020351 15401 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:01.144013  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:01.146657 15513 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:01.147488 15513 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:01.147949 15600 consensus_queue.cc:237] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:01.150115 15513 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:01.150697  9633 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260808 01:52:01.151294 15436 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:01.151710  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2661) took 2617 ms. Trace:
I20260808 01:52:01.152190 15436 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:01.119889 15383 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:52:01.158978  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 207578us
I20260808 01:52:01.152362  9633 rpcz_store.cc:276] 0808 01:51:58.534660 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.139823 (+2605163us) service_pool.cc:224] Handling call
0808 01:52:01.150399 (+ 10576us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:01.151262 (+   863us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.151599 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:01.154652 15495 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:01.158200 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45803.8us
I20260808 01:52:01.074895  9632 rpcz_store.cc:276] 0808 01:51:57.430612 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:00.556251 (+3125639us) service_pool.cc:224] Handling call
0808 01:52:00.557010 (+   759us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:00.557659 (+   649us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:52:00.784816 (+227157us) log.cc:844] Serialized 14 byte log entry
0808 01:52:01.055061 (+270245us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:01.055134 (+    73us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:52:01.055187 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.055264 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:01.055398 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":59775,"prepare.queue_time_us":110703,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":60716,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":109,"thread_start_us":289670,"threads_started":2,"wal-append.queue_time_us":227436}
I20260808 01:52:01.112140 15509 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:01.164778  9638 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260808 01:52:01.169319  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2660) took 2631 ms. Trace:
I20260808 01:52:01.169732  9638 rpcz_store.cc:276] 0808 01:51:58.534402 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.124501 (+2590099us) service_pool.cc:224] Handling call
0808 01:52:01.139370 (+ 14869us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:01.165617 (+ 26247us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.165819 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:52:01.161114 15406 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.794s	user 0.008s	sys 0.011s
W20260808 01:52:01.190069  9634 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2648) took 2647 ms (client timeout 1737 ms). Trace:
W20260808 01:52:01.190528  9634 rpcz_store.cc:269] 0808 01:51:58.516002 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:59.209211 (+693209us) service_pool.cc:224] Handling call
0808 01:52:01.117619 (+1908408us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b540027ea54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:01.162822 (+ 45203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":267720064}
I20260808 01:52:01.192714 15500 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:01.193727  9636 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
W20260808 01:52:01.195587 15618 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
I20260808 01:52:01.180992 15383 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:01.199069 15383 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:01.199697 15294 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:01.201709 15401 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:01.204345  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:01.213989  9560 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:01.172030 15412 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:01.179526 15406 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:01.210497 15535 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:01.188796 15495 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:01.221099 15565 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:01.222580 15565 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:01.225273  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 715) took 2683 ms. Trace:
I20260808 01:52:01.225759  9633 rpcz_store.cc:276] 0808 01:51:58.541435 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.173955 (+2632520us) service_pool.cc:224] Handling call
0808 01:52:01.224585 (+ 50630us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:01.224911 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:01.224938 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:01.224984 (+    46us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:01.225034 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.225089 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:01.225206 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:01.227103 15436 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:01.227594  9633 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260808 01:52:01.228250  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2670) took 2037 ms. Trace:
I20260808 01:52:01.228560  9633 rpcz_store.cc:276] 0808 01:51:59.191253 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.226816 (+2035563us) service_pool.cc:224] Handling call
0808 01:52:01.227188 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:01.227965 (+   777us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.228159 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:01.230010  9633 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13. Ops: []
I20260808 01:52:01.230237 15292 tablet_bootstrap.cc:492] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:52:01.189299 15509 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:01.231547 15412 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:01.234370 15495 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:01.229251 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:01.239612  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:01.239921  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 721) took 1991 ms. Trace:
W20260808 01:52:01.235452 15304 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.660s	user 0.006s	sys 0.012s
I20260808 01:52:01.243960  9633 rpcz_store.cc:276] 0808 01:51:59.245430 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.229354 (+1983924us) service_pool.cc:224] Handling call
0808 01:52:01.229719 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:01.236321 (+  6602us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.237009 (+   688us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:01.245836  9521 leader_election.cc:400] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12.
I20260808 01:52:01.251775 15304 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:01.257571  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2966) took 1015 ms. Trace:
I20260808 01:52:01.258078  9478 rpcz_store.cc:276] 0808 01:52:00.242384 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:00.242628 (+   244us) service_pool.cc:224] Handling call
0808 01:52:01.257318 (+1014690us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b540021d254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:01.257499 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":276432640}
I20260808 01:52:01.212967 15414 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:01.273970 15347 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:01.280665 15347 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:01.283996 15347 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:01.235354 15294 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:01.286814 15614 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
I20260808 01:52:01.288224 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:01.290298 15594 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:01.258803 15624 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:01.291467  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:01.291491 10363 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260808 01:52:01.293946 15304 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:01.294397 15457 consensus_queue.cc:260] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:01.299978  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2630) took 4213 ms. Trace:
I20260808 01:52:01.300247  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2655) took 2770 ms. Trace:
I20260808 01:52:01.312608  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 714) took 2771 ms. Trace:
I20260808 01:52:01.326231  9638 rpcz_store.cc:276] 0808 01:51:58.541144 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.173938 (+2632794us) service_pool.cc:224] Handling call
0808 01:52:01.236654 (+ 62716us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:01.272360 (+ 35706us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:01.272411 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:01.312250 (+ 39839us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:01.312301 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.312376 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:01.312536 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:01.325412  9629 rpcz_store.cc:276] 0808 01:51:58.532381 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.060961 (+2528580us) service_pool.cc:224] Handling call
0808 01:52:01.108351 (+ 47390us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:01.135522 (+ 27171us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.300047 (+164525us) spinlock_profiling.cc:243] Waited 44.4 ms on lock 0x7b540008d250. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:01.300185 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":102034688}
W20260808 01:52:01.327917  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 609181us
I20260808 01:52:01.325318  9636 rpcz_store.cc:276] 0808 01:51:57.065643 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:00.152977 (+3087334us) service_pool.cc:224] Handling call
0808 01:52:01.162748 (+1009771us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:01.194250 (+ 31502us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.278375 (+ 84125us) spinlock_profiling.cc:243] Waited 986 ms on lock 0x7b540027ea50. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:01.278562 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2266964224}
I20260808 01:52:01.314862 15500 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:01.239768 15535 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:01.316565 15600 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260808 01:52:01.339121  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 164424us
I20260808 01:52:01.360592 15414 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:01.339404  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:01.340442  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:01.362011  9629 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:01.336796 15457 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:52:01.362493  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:01.355201  9633 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260808 01:52:01.361580  9599 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:01.371474  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2674) took 1774 ms. Trace:
I20260808 01:52:01.372529  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:01.372742  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 152841us
I20260808 01:52:01.371912  9633 rpcz_store.cc:276] 0808 01:51:59.596666 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.271525 (+1674859us) service_pool.cc:224] Handling call
0808 01:52:01.333723 (+ 62198us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:01.371121 (+ 37398us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.371398 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:01.372603  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:01.382828  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:01.378468 15411 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:01.347510 15598 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:01.323251 15594 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:01.388108  9601 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:01.383653 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:01.405669  9600 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:01.387271 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:01.402733  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:01.362480 15509 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:01.387823 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:01.383185 15527 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.019s
I20260808 01:52:01.386432 15412 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:01.419580  9560 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 16
I20260808 01:52:01.382030 15603 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:01.405525  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:01.420769  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 91141.4us
I20260808 01:52:01.428920  9550 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:01.450181  9521 leader_election.cc:400] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12.
I20260808 01:52:01.346388 15624 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:01.455536  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 80197.8us
I20260808 01:52:01.340788 15294 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:01.384267 15619 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.066s
I20260808 01:52:01.467525  9633 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14. Ops: []
I20260808 01:52:01.468515  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2581) took 1709 ms. Trace:
I20260808 01:52:01.468995  9633 rpcz_store.cc:276] 0808 01:51:59.759254 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.403412 (+1644158us) service_pool.cc:224] Handling call
0808 01:52:01.467297 (+ 63885us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:01.468130 (+   833us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.468436 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:01.411018 15414 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:01.459339  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 76733.4us
W20260808 01:52:01.471715 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:01.466934  9556 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260808 01:52:01.473057 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:01.474362  9630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2556) took 4093 ms (client timeout 4749 ms). Trace:
W20260808 01:52:01.474905  9630 rpcz_store.cc:269] 0808 01:51:57.381595 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:58.436510 (+1054915us) service_pool.cc:224] Handling call
0808 01:52:01.474116 (+3037606us) spinlock_profiling.cc:243] Waited 2.63 s on lock 0x7b540008cd54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:01.474281 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6050412288}
I20260808 01:52:01.475328  9636 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:01.476025  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 722) took 1528 ms. Trace:
I20260808 01:52:01.455181 15412 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:01.478437  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2657) took 2945 ms. Trace:
I20260808 01:52:01.478262 15412 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:01.479133 15509 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:52:01.479876 14784 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:01.480791 15412 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:01.478855  9637 rpcz_store.cc:276] 0808 01:51:58.533501 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.123348 (+2589847us) service_pool.cc:224] Handling call
0808 01:52:01.124110 (+   762us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:01.163458 (+ 39348us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:01.163510 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:01.189053 (+ 25543us) log.cc:844] Serialized 26 byte log entry
0808 01:52:01.314111 (+125058us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:52:01.314157 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.314235 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:01.478193 (+163958us) raft_consensus.cc:1685] finished
0808 01:52:01.478221 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:01.478374 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19254,"raft.queue_time_us":36760,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":7422,"thread_start_us":70722,"threads_started":1,"wal-append.queue_time_us":207425}
I20260808 01:52:01.481423 15605 raft_consensus.cc:3055] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:01.477670 15624 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:01.481850 15605 raft_consensus.cc:740] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:01.483243  9633 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260808 01:52:01.481588  9638 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260808 01:52:01.485388 15594 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:01.422164 15270 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:52:01.487080  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2662) took 2947 ms. Trace:
I20260808 01:52:01.337932 15347 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.124s
I20260808 01:52:01.476405  9636 rpcz_store.cc:276] 0808 01:51:59.948001 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.450286 (+1502285us) service_pool.cc:224] Handling call
0808 01:52:01.475136 (+ 24850us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:01.475789 (+   653us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.475969 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:01.488008 15270 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:01.507138  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2656) took 2974 ms. Trace:
I20260808 01:52:01.506259  9555 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260808 01:52:01.507637  9631 rpcz_store.cc:276] 0808 01:51:58.533202 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.120950 (+2587748us) service_pool.cc:224] Handling call
0808 01:52:01.121430 (+   480us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:01.170553 (+ 49123us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:01.170590 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:01.206636 (+ 36046us) log.cc:844] Serialized 26 byte log entry
0808 01:52:01.317723 (+111087us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:52:01.333666 (+ 15943us) log.cc:844] Serialized 14 byte log entry
0808 01:52:01.333937 (+   271us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.334055 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:01.506892 (+172837us) raft_consensus.cc:1685] finished
0808 01:52:01.506917 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:01.507075 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41295,"prepare.queue_time_us":15667,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":51561,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":16062,"thread_start_us":82565,"threads_started":1,"wal-append.queue_time_us":235616}
I20260808 01:52:01.488759  9635 rpcz_store.cc:276] 0808 01:51:58.539352 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.157131 (+2617779us) service_pool.cc:224] Handling call
0808 01:52:01.189812 (+ 32681us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:01.218179 (+ 28367us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:52:01.250505 (+ 32326us) log.cc:844] Serialized 14 byte log entry
0808 01:52:01.486598 (+236093us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:01.486724 (+   126us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:52:01.486767 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.486852 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:01.487005 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27806,"prepare.queue_time_us":24394,"prepare.run_cpu_time_us":129,"prepare.run_wall_time_us":17032,"thread_start_us":158506,"threads_started":1}
I20260808 01:52:01.488310 15479 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:01.518633 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:01.511580 15406 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:01.450181 15495 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20260808 01:52:01.522877  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:01.499091 15509 raft_consensus.cc:697] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:01.509413 15270 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:01.524361  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:01.525254  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:01.525632  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:01.526099  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:01.509028 14784 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:01.530233 15388 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.140s
I20260808 01:52:01.533260  9478 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 13. Current term is 14. Ops: []
I20260808 01:52:01.533521  9558 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260808 01:52:01.534555  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:01.588296  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:01.535794 15509 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:01.537282  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:01.537518  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:01.539144 15527 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
W20260808 01:52:01.616919  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 103011us
I20260808 01:52:01.617771  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:01.539495  9480 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:01.618534  9557 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:01.540159  9474 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: [14.8-15.9]
I20260808 01:52:01.540446 15597 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260808 01:52:01.590709  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:01.540832 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:01.503013 15294 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.024s
I20260808 01:52:01.549309  9554 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:01.550565  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:01.590935  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2668) took 2408 ms. Trace:
I20260808 01:52:01.555023 15605 consensus_queue.cc:260] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:52:01.621459 10368 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:52:01.557742  9477 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260808 01:52:01.618626  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:01.563951 15585 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:01.619736  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:01.488174 15414 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:52:01.566470  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:01.575851 15603 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:01.578120  9473 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:01.527016  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:01.590044  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:01.591495  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 181980us
I20260808 01:52:01.508899 15304 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:01.547627 15406 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:01.597256 15630 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.344s	user 0.000s	sys 0.003s
I20260808 01:52:01.602751 15619 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.009s
W20260808 01:52:01.598506  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 109873us
I20260808 01:52:01.598436 15513 raft_consensus.cc:3055] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:01.584992  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:01.598971  9445 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:01.613869  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:01.615186 14784 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:01.620115  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2675) took 2021 ms. Trace:
I20260808 01:52:01.621043  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:01.632570  9523 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:01.633759  9629 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:52:01.623145  9634 rpcz_store.cc:276] 0808 01:51:59.182583 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.191533 (+2008950us) service_pool.cc:224] Handling call
0808 01:52:01.192031 (+   498us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:01.192361 (+   330us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:01.192409 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:01.194523 (+  2114us) log.cc:844] Serialized 26 byte log entry
0808 01:52:01.409468 (+214945us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:52:01.458415 (+ 48947us) log.cc:844] Serialized 14 byte log entry
0808 01:52:01.471352 (+ 12937us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.471461 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:01.585964 (+114503us) raft_consensus.cc:1685] finished
0808 01:52:01.585987 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:01.590870 (+  4883us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":350,"prepare.queue_time_us":50682,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":7361,"raft.queue_time_us":22226,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":96,"thread_start_us":193287,"threads_started":1,"wal-append.queue_time_us":334599}
I20260808 01:52:01.620373 15594 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:01.622706 15294 raft_consensus.cc:3055] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:52:01.670271  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:01.671669 15636 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:01.672319 15636 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:01.673241  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:01.673480 15636 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:01.674959 15624 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:01.675424 15624 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:01.675808 15624 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:01.675886  9476 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:52:01.676004 15304 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:01.677624  9476 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260808 01:52:01.680383  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2795) took 3245 ms. Trace:
I20260808 01:52:01.647655  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:01.652884  9560 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:52:01.654300  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:01.646396  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:01.658957  9638 rpcz_store.cc:276] 0808 01:51:59.599119 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.378886 (+1779767us) service_pool.cc:224] Handling call
0808 01:52:01.481104 (+102218us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:01.485127 (+  4023us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.619888 (+134761us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b540008cd50. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:01.620041 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":233960832}
I20260808 01:52:01.663749 10364 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:01.663760 15594 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:01.667215  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 68543.6us
I20260808 01:52:01.688618  9480 raft_consensus.cc:2393] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:01.667805 15598 raft_consensus.cc:2764] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260808 01:52:01.668510  9629 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14. Ops: [10.8-13.9]
I20260808 01:52:01.668905 15615 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:52:01.668923 15495 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:52:01.669698  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:01.682469 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:01.687808  9521 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:01.711906 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:01.658555 15388 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:01.652081 15513 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:01.722857  9476 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:01.732959  9635 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 12: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:01.680891  9472 rpcz_store.cc:276] 0808 01:51:58.434761 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:59.088321 (+653560us) service_pool.cc:224] Handling call
0808 01:51:59.088738 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:51:59.093236 (+  4498us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:52:00.979753 (+1886517us) spinlock_profiling.cc:243] Waited 34 ms on lock 0x7b18002bab00. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d41889e05 00007f5d483aada7 00007f5d4499dd87 00007f5d449ad49f 00007f5d449ad418
0808 01:52:01.101252 (+121499us) log.cc:844] Serialized 14 byte log entry
0808 01:52:01.521195 (+419943us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:01.563376 (+ 42181us) log.cc:844] Serialized 26 byte log entry
0808 01:52:01.563589 (+   213us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:52:01.563633 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.564534 (+   901us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:01.596512 (+ 31978us) spinlock_profiling.cc:243] Waited 19 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:01.680015 (+ 83503us) raft_consensus.cc:1685] finished
0808 01:52:01.680041 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:01.680199 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18407,"prepare.queue_time_us":323566,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":852,"raft.queue_time_us":27200,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":8824,"spinlock_wait_cycles":121883776,"thread_start_us":490371,"threads_started":3,"wal-append.queue_time_us":151181}
W20260808 01:52:01.736534  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47248.6us
I20260808 01:52:01.733898 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:01.693620 15594 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:01.670275 15294 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:01.744230 15615 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:52:01.645913 15585 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:01.732362 15388 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:01.767678  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:01.690626 15304 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:01.746795  9445 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:01.711061 15495 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:01.747242  9480 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260808 01:52:01.761560  9474 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260808 01:52:01.772300  9477 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 8.
I20260808 01:52:01.775507 15501 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election won for term 16
W20260808 01:52:01.794289  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 143356us
I20260808 01:52:01.797991  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:01.772490 15388 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:01.776109 15557 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:01.799551  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:01.776254 15495 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:52:01.804661  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 81968.7us
I20260808 01:52:01.809391  9445 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:01.813090 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:01.834363  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:01.828783  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 159079us
I20260808 01:52:01.802273 15635 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:01.837875  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:01.801985 15554 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:01.803960 15501 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:01.836169  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:01.808827 15412 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:01.847700 15412 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:01.848592 15509 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:01.849094 15509 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:01.849493 15509 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:01.833949 15615 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260808 01:52:01.853173  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 74773.2us
I20260808 01:52:01.853533 15615 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:01.853921 15535 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:01.854202 15615 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:01.854260  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:01.855207 15655 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:01.855923 15655 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:01.805833 15513 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:01.858367 15513 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:01.858803 15635 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:01.861505 15635 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:01.863389 15650 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260808 01:52:01.859282 15414 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:01.829188 15557 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:01.883582 15649 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
W20260808 01:52:01.917992  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:01.875415 15655 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:01.883922 15414 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:01.927011  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 91141.2us
I20260808 01:52:01.896822 15635 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:01.940763  9550 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:52:01.880538 15624 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:01.924623 15554 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:01.938225 15647 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.158s	user 0.001s	sys 0.001s
I20260808 01:52:01.934376 15412 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:01.931891 15406 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:01.954339  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 74116.7us
I20260808 01:52:01.874503 15294 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:01.980463  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 171144us
I20260808 01:52:01.974498 15139 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:01.947293 15635 raft_consensus.cc:697] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:01.988485  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:01.996176  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:01.997640 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:02.023075 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:02.008141  9554 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:02.041759 15347 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260808 01:52:02.042886  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 55384.9us
W20260808 01:52:02.047487  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 249647us
I20260808 01:52:02.048398 15557 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:02.048795  9521 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:01.975436 15501 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:02.054147 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:02.055181 15661 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:52:02.055734 15661 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:02.056219 15661 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:02.059453 15500 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election won for term 14
W20260808 01:52:02.059834 15654 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
I20260808 01:52:02.060200 15500 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:02.061388 15500 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:02.001072 15139 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:02.065783 15139 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:02.066947 15652 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.233s	user 0.001s	sys 0.001s
I20260808 01:52:02.067142 15294 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:02.068608 15635 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:01.983736 15554 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:01.949079 15414 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:01.989611 15412 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260808 01:52:02.084445 15647 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
W20260808 01:52:02.018214 15642 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.217s	user 0.001s	sys 0.001s
I20260808 01:52:02.105067 15406 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:02.142874 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:02.157351 10364 raft_consensus.cc:2410] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 14 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260808 01:52:02.146412 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:02.122375  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:02.151499  9445 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:02.153842 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:02.066609 15139 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:02.163044  9477 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
W20260808 01:52:02.163594  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 143913us
I20260808 01:52:02.164243 15436 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260808 01:52:02.164901 15270 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.588s	user 0.012s	sys 0.000s
W20260808 01:52:02.165591  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 116972us
I20260808 01:52:02.166088 15660 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.171s
I20260808 01:52:02.166896 15188 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:02.167083 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:02.161855 15492 raft_consensus.cc:2764] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election decision vote started in defunct term 16: won
I20260808 01:52:02.198357 10366 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:52:02.148284  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:02.206756  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:02.208204  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 56901.9us
I20260808 01:52:02.217691  9475 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:02.219461  9477 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260808 01:52:02.211159  9555 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.002s	sys 0.017s
W20260808 01:52:02.221660  9555 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.003s	sys 0.017s
W20260808 01:52:02.221067  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 366826us
I20260808 01:52:02.225895  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:02.226847 15412 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:52:02.228169  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:02.229117 15412 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:52:02.229652 15412 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:02.189409 15406 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:02.192381 15139 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:02.204860 15334 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
I20260808 01:52:02.197139 15270 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:02.252578  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:02.210458 15644 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:02.257737  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:02.237563 15660 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:02.262583  9477 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:52:02.244359 15265 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:02.205081 15188 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:02.274989  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 46889.7us
W20260808 01:52:02.277253  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 77872us
I20260808 01:52:02.279860 15414 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:02.265995 15554 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:02.281641  9477 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13. Ops: [12.12-12.12]
W20260808 01:52:02.283504  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 186134us
I20260808 01:52:02.284555 15406 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:02.285248  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:02.285568 15669 raft_consensus.cc:3055] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:52:02.286021 15669 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:02.271435  9370 catalog_manager.cc:5697] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 8 to 11, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 11 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:52:02.287081 15660 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:52:02.287638 15660 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:02.287273 15669 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:02.288125 15660 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:02.288954 15669 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:52:02.290531 15662 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
I20260808 01:52:02.284566 15270 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:02.294335 15631 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.971s	user 0.004s	sys 0.000s
I20260808 01:52:02.296864 15296 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:52:02.297350 15296 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:02.297433  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 826) took 1692 ms. Trace:
I20260808 01:52:02.298448 15296 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:02.304335 15664 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260808 01:52:02.308856 15296 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:52:02.301851 15656 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.439s	user 0.003s	sys 0.000s
I20260808 01:52:02.310443  9370 catalog_manager.cc:5697] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:52:02.312631  9370 catalog_manager.cc:5697] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 6 to 9, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 9 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:52:02.315711 15668 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:02.317108 15516 consensus_queue.cc:1059] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:02.297910  9551 rpcz_store.cc:276] 0808 01:52:00.605507 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:00.784812 (+179305us) service_pool.cc:224] Handling call
0808 01:52:00.866306 (+ 81494us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:00.960968 (+ 94662us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:00.961004 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:00.984368 (+ 23364us) log.cc:844] Serialized 26 byte log entry
0808 01:52:01.188349 (+203981us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:01.188397 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.188468 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:02.293433 (+1104965us) spinlock_profiling.cc:243] Waited 5.12 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:02.297078 (+  3645us) raft_consensus.cc:1685] finished
0808 01:52:02.297101 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:02.297344 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3525,"raft.queue_time_us":16350,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":23998,"spinlock_wait_cycles":11783040,"thread_start_us":113279,"threads_started":1,"wal-append.queue_time_us":246997}
I20260808 01:52:02.318971 15492 raft_consensus.cc:3055] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:02.319386 15492 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:02.320456 15492 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:02.323122  9370 catalog_manager.cc:5697] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:52:02.323923 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:02.280386  9521 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:02.332284  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 229343us
I20260808 01:52:02.339906  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2664) took 3799 ms. Trace:
I20260808 01:52:02.340567  9632 rpcz_store.cc:276] 0808 01:51:58.540797 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.162402 (+2621605us) service_pool.cc:224] Handling call
0808 01:52:01.168232 (+  5830us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:01.237704 (+ 69472us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:01.237735 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:01.270919 (+ 33184us) log.cc:844] Serialized 26 byte log entry
0808 01:52:01.496024 (+225105us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:01.694320 (+198296us) log.cc:844] Serialized 14 byte log entry
0808 01:52:01.836540 (+142220us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:01.836645 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:02.339548 (+502903us) raft_consensus.cc:1685] finished
0808 01:52:02.339580 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:02.339800 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":18583,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":53507,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":4750,"thread_start_us":489399,"threads_started":3,"wal-append.queue_time_us":382404}
I20260808 01:52:02.358027 10361 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260808 01:52:02.270753 15624 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:02.256597 15436 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.088s
I20260808 01:52:02.372947  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:02.230657 15412 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:02.352021 15296 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260808 01:52:02.374748 15671 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260808 01:52:02.375356 15188 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:02.375730  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:02.377549  9370 catalog_manager.cc:5697] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:52:02.387524 15139 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:02.390125 10360 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 755) took 1574 ms (client timeout 1867 ms). Trace:
W20260808 01:52:02.395040  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 114755us
W20260808 01:52:02.395974 15659 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.543s	user 0.002s	sys 0.000s
I20260808 01:52:02.403147  9601 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:02.409806 15605 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:52:02.413686 15411 consensus_meta.cc:287] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.994s	user 0.015s	sys 0.000s
W20260808 01:52:02.364010 15495 consensus_meta.cc:287] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.520s	user 0.011s	sys 0.002s
W20260808 01:52:02.394425  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:02.423745 15615 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.566s	user 0.007s	sys 0.007s
W20260808 01:52:02.429001  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 193129us
W20260808 01:52:02.426638 10360 rpcz_store.cc:269] 0808 01:52:00.818128 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:00.822591 (+  4463us) service_pool.cc:224] Handling call
0808 01:52:02.389787 (+1567196us) spinlock_profiling.cc:243] Waited 21.9 ms on lock 0x7b5400690fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:02.390012 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50256768}
I20260808 01:52:02.430341  9474 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
W20260808 01:52:02.430907  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:02.418851 15509 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.569s	user 0.012s	sys 0.002s
I20260808 01:52:02.417867 15296 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:02.435267 15495 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:02.450068 15388 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.635s	user 0.013s	sys 0.002s
I20260808 01:52:02.448180 15296 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:02.439239 15585 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:02.445884 15600 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:52:02.429222 15451 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.026s
W20260808 01:52:02.464891  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 89276.4us
W20260808 01:52:02.468222  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 65309.4us
I20260808 01:52:02.454664 15340 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:02.442255 15615 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:02.473762  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:02.478286 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 366223us
I20260808 01:52:02.447494 15509 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:02.488026  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48598.4us
I20260808 01:52:02.494207  9370 catalog_manager.cc:5697] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 2 to 11. New cstate: current_term: 11 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:52:02.495481 15600 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.005s
I20260808 01:52:02.496567  9370 catalog_manager.cc:5697] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:52:02.498397 15586 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:02.498775 15626 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 1.247s	user 0.001s	sys 0.001s
I20260808 01:52:02.499056 15586 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:02.500655  9481 raft_consensus.cc:2410] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
W20260808 01:52:02.500914 15680 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260808 01:52:02.505142 15492 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:02.487134 15270 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20260808 01:52:02.513715 15615 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:02.485208 15292 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.107s	user 0.002s	sys 0.000s
W20260808 01:52:02.519414 15594 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.741s	user 0.011s	sys 0.004s
I20260808 01:52:02.519935 15594 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:02.521489 15347 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:02.521950 15347 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:02.527633 15340 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:02.528578 15685 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:52:02.519986  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 944) took 1656 ms. Trace:
I20260808 01:52:02.530658  9479 rpcz_store.cc:276] 0808 01:52:00.864106 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.804917 (+940811us) service_pool.cc:224] Handling call
0808 01:52:01.839232 (+ 34315us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:01.849755 (+ 10523us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:01.849791 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:01.849846 (+    55us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:52:01.849904 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:02.519713 (+669809us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b54000942d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:02.519775 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:02.519920 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23551360}
I20260808 01:52:02.470060 15388 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:02.533813  9601 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:02.534535 15509 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:02.535002 15513 consensus_meta.cc:287] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.676s	user 0.010s	sys 0.005s
I20260808 01:52:02.535391  9521 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:02.522459 15594 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:02.546142  9370 catalog_manager.cc:5697] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 7 to 10, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 10 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:52:02.547541 15388 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:02.548537  9370 catalog_manager.cc:5697] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 6 to 9, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 9 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:52:02.548681 15347 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:02.550580 15347 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:02.486528 15684 consensus_queue.cc:1059] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
I20260808 01:52:02.556953  9445 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:02.557551 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:02.563742 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:02.564167 10361 raft_consensus.cc:2410] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 14 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260808 01:52:02.563800  9599 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:02.564597 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:02.543963  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:02.595664  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 61941.5us
W20260808 01:52:02.597072  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 55260.1us
I20260808 01:52:02.530458 15661 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:02.563402 14784 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:02.608530 10366 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:02.609153 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:02.587117 15687 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:02.618347  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 64966.2us
W20260808 01:52:02.665844  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:02.666847  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 131545us
W20260808 01:52:02.599049 15636 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.864s	user 0.051s	sys 0.031s
I20260808 01:52:02.669996  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:02.677726  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:02.591578 15513 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:02.598606 15594 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:02.609012 15516 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:52:02.680462 15684 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:02.673933 15636 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.951s	user 0.051s	sys 0.031s
I20260808 01:52:02.691991  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:02.770249  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:02.770757 15661 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:02.599445 15660 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:02.771540  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:02.772794  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:02.773409 15660 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:02.773898  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:02.692233  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:02.692912  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:02.693116  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:02.776122  9555 raft_consensus.cc:2410] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260808 01:52:02.776410  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:02.694437  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:02.695742  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:02.696358 15687 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:02.699438 15479 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:02.702633 15492 raft_consensus.cc:3055] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:52:02.703222 15519 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:02.707353 15684 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:02.683876 15594 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:02.792253 15594 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:52:02.793977 15681 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
W20260808 01:52:02.708917  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:02.795146 15294 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260808 01:52:02.795488  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 839) took 1172 ms. Trace:
I20260808 01:52:02.795361  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:02.712612  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46382.9us
I20260808 01:52:02.787490 15492 raft_consensus.cc:740] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:02.797289  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:02.797834 15492 consensus_queue.cc:260] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:02.727463  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:02.728952  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:02.729708  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:02.815800  9523 leader_election.cc:400] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16.
I20260808 01:52:02.816334  9523 leader_election.cc:403] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:52:02.731146 15565 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:02.736306  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 367721us
I20260808 01:52:02.738114 15411 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:52:02.738886 15406 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:02.739324 15585 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:02.740310  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 183436us
I20260808 01:52:02.741219 15635 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:02.741966 15304 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:02.742422  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2833) took 1200 ms. Trace:
I20260808 01:52:02.743855 15557 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:02.744467 15188 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 01:52:02.747824 15597 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:02.741236  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2835) took 1198 ms. Trace:
I20260808 01:52:02.833520  9599 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:02.753535  9521 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:02.693674  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:02.777673  9554 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:02.781332  9559 raft_consensus.cc:2410] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 16 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260808 01:52:02.791205 15519 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:02.796433  9478 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2832) took 1254 ms (client timeout 1562 ms). Trace:
W20260808 01:52:02.905051  9478 rpcz_store.cc:269] 0808 01:52:01.541728 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.619447 (+ 77719us) service_pool.cc:224] Handling call
0808 01:52:02.787561 (+1168114us) spinlock_profiling.cc:243] Waited 4.08 ms on lock 0x7b5800005498. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:52:02.796353 (+  8792us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9394944}
I20260808 01:52:02.713624 15495 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:02.798485  9601 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:02.800508 15479 raft_consensus.cc:2764] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260808 01:52:02.804819  9524 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:02.819623  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:02.821787  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:02.853178 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:02.795961  9557 rpcz_store.cc:276] 0808 01:52:01.623457 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.753426 (+129969us) service_pool.cc:224] Handling call
0808 01:52:01.776694 (+ 23268us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:01.987732 (+211038us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:01.987768 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:02.020180 (+ 32412us) log.cc:844] Serialized 26 byte log entry
0808 01:52:02.459623 (+439443us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:02.459671 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:02.459747 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:02.793287 (+333540us) spinlock_profiling.cc:243] Waited 1.83 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:02.795126 (+  1839us) raft_consensus.cc:1685] finished
0808 01:52:02.795152 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:02.795414 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":269541,"raft.queue_time_us":230959,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":154,"spinlock_wait_cycles":4198656,"thread_start_us":334945,"threads_started":1,"wal-append.queue_time_us":499852}
W20260808 01:52:02.913192  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 108487us
W20260808 01:52:02.811496 15292 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.053s	user 0.004s	sys 0.000s
I20260808 01:52:02.859090  9551 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260808 01:52:02.854385  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:02.803077 15294 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:02.920289 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:02.909102 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:52:02.886968  9481 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2969) took 2075 ms (client timeout 1805 ms). Trace:
I20260808 01:52:02.913255  9550 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:52:02.852424 15635 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:02.887087  9476 rpcz_store.cc:276] 0808 01:52:01.564197 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.739191 (+174994us) service_pool.cc:224] Handling call
0808 01:52:01.749345 (+ 10154us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:01.778247 (+ 28902us) raft_consensus.cc:1534] Early marking committed up to index 10
0808 01:52:01.876176 (+ 97929us) log.cc:844] Serialized 14 byte log entry
0808 01:52:02.500564 (+624388us) spinlock_profiling.cc:243] Waited 9.58 ms on lock 0x7b18004c1320. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d4453242a 00007f5d4499de0a
0808 01:52:02.740494 (+239930us) log.cc:844] Serialized 14 byte log entry
0808 01:52:02.740709 (+   215us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:02.740767 (+    58us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:02.740890 (+   123us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:02.741005 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:02.741161 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":166969,"prepare.queue_time_us":106165,"prepare.run_cpu_time_us":154,"prepare.run_wall_time_us":5386,"raft.queue_time_us":126334,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":208,"spinlock_wait_cycles":22033280,"thread_start_us":173336,"threads_started":2,"wal-append.queue_time_us":810065}
I20260808 01:52:02.921195  9480 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:02.823097 15668 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:02.924077 15585 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:02.888669 15660 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:02.925505 15644 consensus_queue.cc:1059] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:52:02.925865 15684 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:02.927029 15543 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.126s
I20260808 01:52:02.928186 15687 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:02.978989  9557 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260808 01:52:02.801553 15516 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.124s
I20260808 01:52:02.852818  9473 rpcz_store.cc:276] 0808 01:52:01.542102 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.736345 (+194243us) service_pool.cc:224] Handling call
0808 01:52:01.751985 (+ 15640us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:01.768710 (+ 16725us) raft_consensus.cc:1534] Early marking committed up to index 6
0808 01:52:01.787928 (+ 19218us) log.cc:844] Serialized 14 byte log entry
0808 01:52:02.526791 (+738863us) spinlock_profiling.cc:243] Waited 1.87 ms on lock 0x7b18004b10c0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d4453242a 00007f5d4499de0a
0808 01:52:02.741982 (+215191us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:02.742039 (+    57us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:52:02.742158 (+   119us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:02.742236 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:02.742361 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":197116,"prepare.queue_time_us":37627,"prepare.run_cpu_time_us":135,"prepare.run_wall_time_us":5646,"raft.queue_time_us":141884,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":111,"spinlock_wait_cycles":4290304,"thread_start_us":73561,"threads_started":1,"wal-append.queue_time_us":897449}
I20260808 01:52:02.885679 15406 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:02.988363 15406 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:02.903848 15304 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:02.933315  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 134949us
I20260808 01:52:02.994515  9601 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:02.996680  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:02.935698  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:02.997699 15600 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:52:02.919938 15411 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:02.928326  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 112755us
I20260808 01:52:02.999778 15411 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:03.000969  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:02.907248 15492 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:03.003055  9560 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260808 01:52:02.923655 15557 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:02.937004 15479 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:02.925151  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:02.939606  9481 rpcz_store.cc:269] 0808 01:52:00.811576 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:00.811778 (+   202us) service_pool.cc:224] Handling call
0808 01:52:02.854332 (+2042554us) spinlock_profiling.cc:243] Waited 1.69 s on lock 0x7b540019d2d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:02.886877 (+ 32545us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3880178560}
W20260808 01:52:02.955232  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 121848us
I20260808 01:52:02.788159 15685 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.158s
W20260808 01:52:02.963622  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:02.908043 15334 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16.
I20260808 01:52:03.013876  9554 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260808 01:52:03.014497  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:02.797289 15686 consensus_queue.cc:1059] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.156s
I20260808 01:52:02.937031 15495 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:52:02.921110 14784 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:52:03.016683  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:02.976649  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:03.018246 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 287199us
I20260808 01:52:03.019032  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:03.029661  9476 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:52:03.028318  9472 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:02.923835 15519 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:02.823123 15661 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.135s
I20260808 01:52:02.945410 15294 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 8 FOLLOWER]: Advancing to term 9
W20260808 01:52:03.024519  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:02.951516 15635 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:03.027341  9478 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:02.961261 15668 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:02.853920 15565 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:03.032316  9599 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:03.037305  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:03.027369 15495 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:03.040237  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 286813us
I20260808 01:52:02.928179 15296 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:03.036561 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:03.072726  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 137172us
I20260808 01:52:03.076676  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:03.075587  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 124209us
I20260808 01:52:03.082093  9557 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 11. Current term is 12. Ops: []
I20260808 01:52:03.072523  9473 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:52:03.031183 15600 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:03.095860  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:03.100598 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 64167.2us
I20260808 01:52:03.101852 10334 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:03.039611 15451 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260808 01:52:03.113814  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:03.114995 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:03.114902 10363 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:52:03.117909  9523 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:03.043020 15613 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260808 01:52:03.119261  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 87044us
I20260808 01:52:03.120468  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:03.121731  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52187.4us
I20260808 01:52:03.026122 15660 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:03.073665 15412 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:52:03.089463 15495 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:03.129037  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:03.095036 15436 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:03.136204  9601 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:03.136698 10332 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:03.122304 15500 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:03.113194 15519 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:03.145737  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:03.112352 14784 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:03.119035 15501 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260808 01:52:03.167640  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 49825.5us
W20260808 01:52:03.188661  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59835.2us
I20260808 01:52:03.141235 15479 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260808 01:52:03.189842  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:03.145455 15586 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:03.166401 15500 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:03.166610 15527 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:03.166812 15669 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260808 01:52:03.201768  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 70974.3us
I20260808 01:52:03.183743 15501 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:03.201879  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 120738us
I20260808 01:52:03.188395 15451 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260808 01:52:03.222508  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 102113us
I20260808 01:52:03.224444  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:03.226598  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:03.198613 15644 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260808 01:52:03.204713 15500 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:03.203399 15406 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:52:03.207433 15709 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:52:03.182979 15519 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:03.240123  9521 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:03.240366  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:03.211920 15501 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:52:03.252480  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 106884us
I20260808 01:52:03.252745  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:03.221923 15715 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:03.232031 15411 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:03.254411  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:03.260723  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:03.261210  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:03.244711 15586 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260808 01:52:03.244381 15303 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.087s
I20260808 01:52:03.265424  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:03.259044 15585 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:03.259742 15709 raft_consensus.cc:697] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:03.260896 15406 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:03.263214 15644 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260808 01:52:03.264730 15509 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:03.264885 15669 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:52:03.287070  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:03.290076 15411 consensus_queue.cc:1059] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:52:03.264729 15451 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
W20260808 01:52:03.295323 15711 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260808 01:52:03.296883 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:03.269210 15565 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:03.281539 15668 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:03.232681 15600 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:03.267324 15479 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
W20260808 01:52:03.330121  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:03.271391 15412 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:52:03.364351 10364 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
W20260808 01:52:03.368017  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 81079.7us
I20260808 01:52:03.373027  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:03.261291 15717 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:52:03.415267 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:03.417155  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:03.352085  9524 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:03.356202 15605 consensus_meta.cc:287] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.924s	user 0.015s	sys 0.000s
W20260808 01:52:03.287194 15706 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
I20260808 01:52:03.303188 14784 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:03.376183 15509 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:03.302541 15513 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:03.328506 15585 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:03.458583  9637 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260808 01:52:03.363082 15660 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:03.415122 15717 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:03.303098 15334 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:03.442914 10367 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:03.471158  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:03.450348 14784 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:03.457355 15509 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:03.421773 15451 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Stepping down as leader of term 14
W20260808 01:52:03.467768 15594 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.675s	user 0.013s	sys 0.000s
W20260808 01:52:03.441642  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 89700.9us
I20260808 01:52:03.426571 15411 consensus_queue.cc:1243] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 12.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
W20260808 01:52:03.455598 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:03.440847 15406 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:03.409250 15586 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:03.460800 10366 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
W20260808 01:52:03.448508 15714 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.216s	user 0.002s	sys 0.000s
I20260808 01:52:03.451278 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:03.479602  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62538.8us
I20260808 01:52:03.496469  9524 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:03.470057 15334 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:03.481338  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 216009us
I20260808 01:52:03.470372 10367 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:03.363464 15644 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:03.447024 15721 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:03.490223  9473 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:52:03.501691 15679 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:03.517788  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:03.518604  9632 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:03.521682  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.522748  9630 raft_consensus.cc:2410] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260808 01:52:03.523382  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:03.524135  9634 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:03.524240  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:03.503298 15406 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:03.525204  9636 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:03.525614  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.526103  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:03.526762 15406 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:03.528198  9631 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:52:03.530046 15698 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.600s	user 0.000s	sys 0.004s
I20260808 01:52:03.531417  9633 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 11. Current term is 12. Ops: []
I20260808 01:52:03.523741  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:03.484153 15519 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:03.472357 15717 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:03.524080  9638 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260808 01:52:03.485831 14784 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:52:03.515296  9521 connection.cc:256] RPC call timeout handler was delayed by -1.0108219399990048s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1831s for a response
I20260808 01:52:03.486800 15451 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:03.483001 15412 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:03.541831  9636 raft_consensus.cc:2410] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260808 01:52:03.543433  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.565377  9635 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:52:03.565414  9633 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 11. (term mismatch)
I20260808 01:52:03.567252  9633 pending_rounds.cc:85] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 10
I20260808 01:52:03.567582  9633 pending_rounds.cc:107] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260808 01:52:03.567863  9633 raft_consensus.cc:2889] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:03.513821 15565 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:03.545766  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:03.547940  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:03.569329  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:03.570106  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:03.570719  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:03.553123  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:03.555631  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 76355.3us
I20260808 01:52:03.485462 15585 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:03.579603  9632 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:52:03.613081 14784 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:03.606187  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 67807.2us
I20260808 01:52:03.616992  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:03.510424 15721 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:03.619987  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:03.620436  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:03.622406  9631 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260808 01:52:03.458920 15709 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:03.623595  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.627205  9629 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260808 01:52:03.627612  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.542233 15644 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:03.628443  9632 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 15: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:03.628259 15412 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:03.629383  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.629813 15644 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:03.629987  9632 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
W20260808 01:52:03.626839  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1080633390229195s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1906s for a response
I20260808 01:52:03.618983 14784 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:03.624430  9557 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260808 01:52:03.624739  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:03.585857 10367 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:03.503772 15594 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:03.625385  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:03.587050 15347 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.013s	user 0.015s	sys 0.002s
I20260808 01:52:03.635006  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:03.637059  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:03.638149  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:03.637097  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:03.640713  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.641387  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:03.581615 15340 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:52:03.644639  9473 raft_consensus.cc:2410] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260808 01:52:03.612519 15296 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:03.540345 15457 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.091s
I20260808 01:52:03.584295 15565 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:03.648196  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:03.543008 15679 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:03.551540 15451 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:03.649796  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:03.652010 15594 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:03.611089 15334 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:03.655524  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:03.655823  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:03.657433  9556 raft_consensus.cc:2410] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 15 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260808 01:52:03.658092  9478 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 16: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:03.648772  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 945) took 1457 ms. Trace:
W20260808 01:52:03.643251 15294 consensus_meta.cc:287] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.543s	user 0.012s	sys 0.001s
I20260808 01:52:03.631263 15715 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:03.679194  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:03.662039 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:03.625123 15721 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:03.679617  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.681901  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.666577  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:03.663184  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.733974  9632 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:03.741353  9631 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:03.696409  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.662108 10367 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:03.745013  9630 raft_consensus.cc:2393] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 17: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:52:03.669786 15635 consensus_meta.cc:287] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.555s	user 0.011s	sys 0.001s
W20260808 01:52:03.733561  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 85512.9us
I20260808 01:52:03.761891  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:03.763139  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:03.735239  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:03.734400 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:03.763561 14784 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:03.764230 14784 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:03.668674 15710 raft_consensus.cc:3055] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Stepping down as leader of term 14
W20260808 01:52:03.753556  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 52434.2us
I20260808 01:52:03.771457  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:03.753969  9634 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
W20260808 01:52:03.735713  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 75599.3us
I20260808 01:52:03.772935  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:03.757033 15557 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:03.705619 15495 consensus_meta.cc:287] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.521s	user 0.011s	sys 0.001s
I20260808 01:52:03.733440  9559 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:52:03.675436 15667 raft_consensus.cc:3055] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:52:03.781287 15667 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:03.782390 15667 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:03.784250 15667 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:03.752605 15715 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:03.786391 15495 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:03.745263 15294 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:03.789994 15495 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:03.790717 15294 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:03.791077  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.791661 15347 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:03.792495  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.731119  9480 rpcz_store.cc:276] 0808 01:52:02.189087 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:02.189283 (+   196us) service_pool.cc:224] Handling call
0808 01:52:02.769884 (+580601us) spinlock_profiling.cc:243] Waited 2.59 ms on lock 0x7b5800005498. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:52:02.770251 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:02.946979 (+176728us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:03.645788 (+698809us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54002a6cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:03.646021 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":352583808}
I20260808 01:52:03.767963  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:03.675503 15457 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:52:03.772192  9473 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:52:03.756229  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.773355  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.729764 15565 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:52:03.774971  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:03.775848  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:03.680632 15304 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:03.819532  9634 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:52:03.804791  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:03.821239  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:03.822958  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:03.809423  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:03.667673 15737 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:03.774559  9636 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:03.825088  9632 raft_consensus.cc:2393] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:03.684270 15500 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:03.788257 15586 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:52:03.826627  9630 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 12: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:03.767356  9633 raft_consensus.cc:2410] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 16 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260808 01:52:03.751415 15340 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:03.795924  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:03.800661 14784 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:03.822109  9551 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:03.770794 15710 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:03.779122  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:03.818364 15613 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:03.693439 15501 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:03.828286  9631 raft_consensus.cc:2376] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15. Re-sending same reply.
I20260808 01:52:03.837935  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.831656  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.800213 15605 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:03.839005 14784 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:03.835107  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:03.793823 15451 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:03.840740 14784 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:03.842974  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:03.846184  9559 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:03.847263  9634 raft_consensus.cc:2410] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 16 because replica has last-logged OpId of term: 15 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 6.
I20260808 01:52:03.791517 15294 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260808 01:52:03.849340  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.801234 15457 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:03.849859  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.854561  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:03.835817 15737 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:03.859498  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2677) took 3141 ms. Trace:
I20260808 01:52:03.860631  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:03.863315  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:03.865731  9637 rpcz_store.cc:276] 0808 01:52:00.717988 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:01.481609 (+763621us) service_pool.cc:224] Handling call
0808 01:52:01.481943 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:03.459078 (+1977135us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:03.859206 (+400128us) spinlock_profiling.cc:243] Waited 1.97 s on lock 0x7b5400420ed4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:03.859403 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4521499904}
I20260808 01:52:03.867987  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:03.868870  9632 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:03.870740  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.865144  9633 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:03.872502  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.873725  9634 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 16: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:03.874285  9638 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260808 01:52:03.771492 15635 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:03.843113 15660 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:03.908267  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:03.878187  9557 raft_consensus.cc:2393] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 9: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:03.879328 15735 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:52:03.871155  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:03.919205  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48114.5us
I20260808 01:52:03.919404 10367 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260808 01:52:03.920646  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:03.880192 15636 raft_consensus.cc:3055] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:03.883483  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:03.894927 15501 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:03.878849  9559 raft_consensus.cc:2410] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260808 01:52:03.852324 15406 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:52:03.923774  9560 raft_consensus.cc:2410] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260808 01:52:03.927459  9477 raft_consensus.cc:2368] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15.
I20260808 01:52:03.840505 15720 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.070s
I20260808 01:52:03.930575  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:03.932123  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:03.937361  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.938210  9636 raft_consensus.cc:2393] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 15: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:03.843766 15613 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:03.917403 15679 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:03.944298  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:03.852555 15605 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:52:03.925966 15586 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:03.918356 15735 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:03.926047 15644 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:03.927981 15715 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:03.945920  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:03.923209 15636 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:03.949651  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.960597  9636 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:03.952566  9632 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:03.944706 15406 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:03.965804  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:03.933007 15516 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.022s
I20260808 01:52:03.968755  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:03.932662 15660 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:03.950981 15586 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:03.954249 15735 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:03.952911 15644 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:03.959573 15585 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:03.964463 15406 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:03.980696 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:03.982656  9632 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 14 index: 6. (term mismatch)
I20260808 01:52:03.978854  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:03.920491 15710 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:03.984973  9632 pending_rounds.cc:85] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 5
I20260808 01:52:03.985388  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:03.974850 15516 consensus_queue.cc:1243] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 16.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260808 01:52:03.985415  9632 pending_rounds.cc:107] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 13.6
I20260808 01:52:03.988056  9632 raft_consensus.cc:2889] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:03.977763 15501 consensus_queue.cc:1059] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.013s
I20260808 01:52:03.980053 15586 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
W20260808 01:52:03.994124  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:03.997069  9633 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:52:03.982504 15644 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:04.001220  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:04.004643  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3223) took 1315 ms. Trace:
I20260808 01:52:04.004838  9554 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: []
I20260808 01:52:04.005119  9556 rpcz_store.cc:276] 0808 01:52:02.689673 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:02.692625 (+  2952us) service_pool.cc:224] Handling call
0808 01:52:04.004456 (+1311831us) spinlock_profiling.cc:243] Waited 882 ms on lock 0x7b54003264d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:04.004584 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2027479808}
I20260808 01:52:04.009578  9636 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:04.018162  9560 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:04.019735  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:04.022285  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:03.997866 15724 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:03.982337 15620 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.024s
I20260808 01:52:04.022902  9632 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:52:03.997737 15710 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:52:04.008460 15509 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.510s	user 0.012s	sys 0.000s
I20260808 01:52:04.030923 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:04.031915  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:04.026852 15620 consensus_queue.cc:1243] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 13.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260808 01:52:04.016155 15500 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:04.025722 15729 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:04.019405 15720 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
I20260808 01:52:04.048355  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:04.048903  9635 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:52:04.053251  9636 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260808 01:52:03.998881 15636 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:04.056221 15745 log.cc:927] Time spent T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260808 01:52:04.056466 15636 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
W20260808 01:52:04.057224 15719 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.717s	user 0.000s	sys 0.004s
I20260808 01:52:04.059038  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 957) took 1044 ms. Trace:
I20260808 01:52:04.059930 15660 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:04.060563 15721 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:04.047549 15724 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.014s
I20260808 01:52:04.061807  9550 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:04.059470  9481 rpcz_store.cc:276] 0808 01:52:03.014172 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:03.027586 (+ 13414us) service_pool.cc:224] Handling call
0808 01:52:03.044680 (+ 17094us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:03.071687 (+ 27007us) raft_consensus.cc:1534] Early marking committed up to index 10
0808 01:52:03.122886 (+ 51199us) log.cc:844] Serialized 14 byte log entry
0808 01:52:03.241328 (+118442us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:03.259523 (+ 18195us) log.cc:844] Serialized 26 byte log entry
0808 01:52:03.259703 (+   180us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:03.259747 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:03.259821 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:03.293352 (+ 33531us) spinlock_profiling.cc:243] Waited 4.93 ms on lock 0x7b180031ab00. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:52:04.056507 (+763155us) spinlock_profiling.cc:243] Waited 2.78 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:04.058770 (+  2263us) raft_consensus.cc:1685] finished
0808 01:52:04.058797 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:04.058959 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32298,"prepare.queue_time_us":41793,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":53,"raft.queue_time_us":43030,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":20036,"spinlock_wait_cycles":17731584,"thread_start_us":80173,"threads_started":1,"wal-append.queue_time_us":125840}
I20260808 01:52:04.062866 15660 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:04.063611  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:04.063452 15660 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:04.064234  9635 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:04.065208 15334 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:04.065771 15660 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:04.065958  9472 raft_consensus.cc:2410] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260808 01:52:04.067258  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:04.032624 15294 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.020s
I20260808 01:52:04.034555 15509 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:04.074079  9631 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:04.074661 15721 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:04.040391 15501 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260808 01:52:04.078527 15613 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:04.079748 15509 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:04.040820 15635 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:04.091711  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:04.095949  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:04.043768 15709 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:04.037038 15619 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.055s
I20260808 01:52:04.101678  9560 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260808 01:52:04.102231  9631 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:04.104065  9551 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:04.108304  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:04.098304 15292 tablet_bootstrap.cc:492] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:52:04.110204  9559 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:04.109007  9556 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:04.108696  9599 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:04.112672  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:04.106922  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:04.111109  9635 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:52:04.114921 10332 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:04.102756 15605 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:04.095686 15516 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
I20260808 01:52:04.130645  9635 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:04.122020 15721 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:04.114485 15613 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:04.109555 15709 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:04.149247  9554 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:52:04.142261 15613 raft_consensus.cc:697] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:04.135469 15720 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:04.150445 15709 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:04.124151 15501 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:52:04.152213  9551 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260808 01:52:04.150664 15613 consensus_queue.cc:237] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:04.150836 15720 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:04.159792 15720 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:04.166201  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2845) took 1241 ms. Trace:
I20260808 01:52:04.166498 15635 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:04.167716 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:04.175407  9477 rpcz_store.cc:276] 0808 01:52:02.924708 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:02.924870 (+   162us) service_pool.cc:224] Handling call
0808 01:52:04.165978 (+1241108us) spinlock_profiling.cc:243] Waited 915 ms on lock 0x7b540019c3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:04.166133 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2103906048}
I20260808 01:52:04.175745 15635 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:04.168184 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:04.177284 10363 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:52:04.179283 15747 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.138s	user 0.000s	sys 0.002s
I20260808 01:52:04.184491  9477 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 14 index: 6. (index mismatch)
I20260808 01:52:04.188441 15565 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:04.137570 15713 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.877s	user 0.005s	sys 0.000s
I20260808 01:52:04.191457 15725 raft_consensus.cc:3055] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:52:04.171618 15292 tablet_bootstrap.cc:492] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:52:04.173265 15457 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:04.174902 15501 raft_consensus.cc:697] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:04.169212  9557 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260808 01:52:04.169837  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:04.147346  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 46819.6us
I20260808 01:52:04.198936  9550 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 11. Current term is 12. Ops: []
I20260808 01:52:04.203668  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:04.144716  9631 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:52:04.206975 10363 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:04.210006  9521 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:04.135162 15527 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.857s	user 0.017s	sys 0.012s
W20260808 01:52:04.185237 15717 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.574s	user 0.013s	sys 0.001s
I20260808 01:52:04.191627 15635 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:04.227998 15565 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:04.228616  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:04.229374  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:04.230187  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:04.231316  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:04.231427 15624 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:04.232116  9479 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:04.213907 14784 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:04.233124 15139 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:04.233561 15139 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:04.215284  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:04.192018 15725 raft_consensus.cc:740] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:04.213730 15685 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:52:04.216744 15292 ts_tablet_manager.cc:1403] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 2.987s	user 0.083s	sys 0.014s
I20260808 01:52:04.217005  9478 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260808 01:52:04.217199 15457 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:04.218066  9472 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:04.217440 15501 consensus_queue.cc:237] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:04.194381 15594 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
W20260808 01:52:04.214741 15527 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.947s	user 0.017s	sys 0.012s
I20260808 01:52:04.217384 15334 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:04.224970  9552 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:04.231743  9480 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:04.235221  9550 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:52:04.270546  9559 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260808 01:52:04.228089  9481 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:52:04.212489  9556 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
W20260808 01:52:04.213873  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 44173.9us
I20260808 01:52:04.238094  9521 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:04.234454 15139 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:04.242558 15725 consensus_queue.cc:260] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:04.218917  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3233) took 1512 ms. Trace:
I20260808 01:52:04.256464  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:04.284308  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:04.249676  9555 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:04.268213  9445 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:04.288148  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:04.237977 15720 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:04.294262  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:04.295323  9601 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:04.297415 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:04.191475 15586 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:04.273435  9599 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:04.311589  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:04.311595  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:04.308804  9638 raft_consensus.cc:1217] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Deduplicated request from leader. Original: 9.8->[11.9-16.11]   Dedup: 11.9->[14.10-16.11]
I20260808 01:52:04.303983 15406 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:04.310303  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:04.319919 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:04.320420  9481 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:04.321448  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:04.312739  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:04.322100  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:04.322868  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:04.324131  9601 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:04.314702  9638 pending_rounds.cc:85] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 9
I20260808 01:52:04.315537 14784 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:04.325194  9638 pending_rounds.cc:107] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 13.10
I20260808 01:52:04.325495  9638 raft_consensus.cc:2889] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:04.326982 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:04.327620  9480 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260808 01:52:04.328053 10366 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:52:04.283255  9558 rpcz_store.cc:276] 0808 01:52:02.706256 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:02.919022 (+212766us) service_pool.cc:224] Handling call
0808 01:52:02.919494 (+   472us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:03.009952 (+ 90458us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:03.009983 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:03.010527 (+   544us) log.cc:844] Serialized 26 byte log entry
0808 01:52:03.141326 (+130799us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:52:03.141388 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:03.141490 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:04.110069 (+968579us) spinlock_profiling.cc:243] Waited 2.3 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:04.218576 (+108507us) raft_consensus.cc:1685] finished
0808 01:52:04.218599 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:04.218847 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":62925,"raft.queue_time_us":97318,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":33554,"spinlock_wait_cycles":5300608,"thread_start_us":171485,"threads_started":2,"wal-append.queue_time_us":209533}
I20260808 01:52:04.333815  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:04.337934  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:04.332595 14784 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:04.300231 15594 consensus_queue.cc:1243] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 15.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260808 01:52:04.339933  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:04.342963  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 49894.7us
I20260808 01:52:04.332661  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:04.346982  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:04.347595  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:04.349766  9475 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:04.352624  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:04.237896 15717 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:04.320861 15667 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:52:04.339670 15509 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:04.351747  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:04.371208  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 155819us
I20260808 01:52:04.372112  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:52:04.372732  9521 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:04.374141  9521 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:04.340843 15725 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:04.378985 15139 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:04.293762 15624 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.071s
I20260808 01:52:04.388242 15509 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:04.319500 15679 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260808 01:52:04.389212 15667 raft_consensus.cc:697] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:04.390393 15667 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:04.391111  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:04.391439 15586 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:04.333102 15722 raft_consensus.cc:2764] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260808 01:52:04.347079 15585 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:04.393400 15585 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:04.328109 15600 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:04.393851  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:04.328226 15598 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:04.270717 15636 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.067s
I20260808 01:52:04.310578 15557 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:04.326802 15479 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:04.365552  9523 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:04.333899 15347 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:52:04.339270 15270 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:52:04.247272 15735 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:04.403400  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 57176.2us
I20260808 01:52:04.326375 15613 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:04.394306  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:04.341578 15605 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260808 01:52:04.396116  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:04.449326  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 55108.7us
W20260808 01:52:04.449757  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 53733.5us
I20260808 01:52:04.341792 15304 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:52:04.451079  9521 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:04.451247 15718 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:04.451474 15304 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:52:04.451874 15294 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260808 01:52:04.453245  9524 leader_election.cc:400] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15.
I20260808 01:52:04.402806 15585 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:04.455499  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:04.456866  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:04.457854 15586 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:04.458601 15586 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:04.459133 15644 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:04.460248 15751 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260808 01:52:04.461790 15644 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:04.463042 15717 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:04.464664 15746 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.421s	user 0.000s	sys 0.004s
I20260808 01:52:04.463070 15586 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20260808 01:52:04.466044 15636 raft_consensus.cc:3055] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:52:04.402784 15721 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:04.466601 15636 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:04.467293  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 101836us
I20260808 01:52:04.449663  9476 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 14: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:04.405345  9473 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:04.341725 15733 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:04.418396  9560 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 11. Current term is 12. Ops: []
I20260808 01:52:04.424319  9599 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:52:04.481768  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 57585.7us
I20260808 01:52:04.358106 15412 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:04.435212  9479 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:04.435985 15509 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:04.404072 15600 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:04.440001 15340 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:04.407002 15406 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:04.419634 15557 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:04.447587 15735 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:04.463177 15347 raft_consensus.cc:697] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:04.379035 14784 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:04.473030  9475 raft_consensus.cc:2410] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 15 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260808 01:52:04.473475  9481 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:52:04.480438  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:04.487161 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:04.497239  9480 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260808 01:52:04.499922 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:04.501142  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:04.503633  9556 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260808 01:52:04.506096  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:04.483234 15720 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:04.487907 15334 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260808 01:52:04.506904  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:04.451102 15737 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:04.525969  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:04.523577  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:04.527808  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:04.527691  9474 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:04.519001  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:04.530543  9634 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:04.520859  9633 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260808 01:52:04.488847 15605 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:04.526183  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:04.532680  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:04.528697  9632 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:52:04.507790 15733 consensus_queue.cc:1059] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
I20260808 01:52:04.534955 15292 raft_consensus.cc:359] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:52:04.536732 15516 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:52:04.537953 15495 consensus_queue.cc:1059] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20260808 01:52:04.490823 15406 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:04.499519 15644 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
I20260808 01:52:04.523121 15720 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:04.527279 15411 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.016s
I20260808 01:52:04.540624  9631 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20260808 01:52:04.588876  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:04.589939  9632 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 12: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:04.508149 15721 consensus_queue.cc:1059] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.033s
I20260808 01:52:04.570749 10334 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:04.709381  9634 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:52:04.454653 15294 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.054s
I20260808 01:52:04.493880 15600 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:52:04.547384  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 67064.7us
I20260808 01:52:04.492722 15717 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election won for term 16
W20260808 01:52:04.717010 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 146458us
I20260808 01:52:04.553776  9481 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260808 01:52:04.491508 15340 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:04.557652  9629 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260808 01:52:04.710573 15710 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:04.508406 15729 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260808 01:52:04.547044 15451 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.697s	user 0.006s	sys 0.009s
I20260808 01:52:04.572396  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:04.721544  9633 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:04.499194 15624 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.076s
W20260808 01:52:04.609992 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 303161us
I20260808 01:52:04.578382 15406 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:04.671397 15722 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:04.692891 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:04.714740  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2457) took 1084 ms. Trace:
I20260808 01:52:04.555227 15605 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:04.559515 15685 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260808 01:52:04.559161 15565 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:04.723152  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:04.560127 15292 raft_consensus.cc:740] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:52:04.728415  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:04.670761 15270 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:04.595108 15720 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:04.580176 15495 consensus_queue.cc:1243] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260808 01:52:04.748194  9632 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:04.730412  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:04.584683 15735 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:04.723856 15721 consensus_queue.cc:1243] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 13.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260808 01:52:04.761755  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:04.716082 15598 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:04.574533 15412 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.164s
I20260808 01:52:04.741382 15730 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:04.771306  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:04.746737  9555 rpcz_store.cc:276] 0808 01:52:03.630112 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:03.769197 (+139085us) service_pool.cc:224] Handling call
0808 01:52:04.714502 (+945305us) spinlock_profiling.cc:243] Waited 427 ms on lock 0x7b54003255d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:04.714645 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":982832512}
I20260808 01:52:04.801589  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:04.750052 15605 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:04.840051  9480 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 15: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:52:04.796739 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 79013.6us
I20260808 01:52:04.764815 15270 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:04.850482 15270 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:04.760931 15717 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:04.851346  9523 consensus_peers.cc:597] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:52:04.852142 15717 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:04.764070 15347 consensus_queue.cc:237] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:04.858850 15730 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:04.859632 15730 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:04.803434  9634 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 17. Current term is 18. Ops: [15.9-17.10]
I20260808 01:52:04.797434 15735 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:04.868769 15735 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:04.799896  9481 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:52:04.875322  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:04.877169  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:04.878422  9481 raft_consensus.cc:2410] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 16 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260808 01:52:04.879675  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:04.891252 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 164159us
W20260808 01:52:04.893509  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 122356us
I20260808 01:52:04.873229  9474 raft_consensus.cc:2410] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
W20260808 01:52:04.892086  9636 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.021s	sys 0.036s
W20260808 01:52:04.901458  9636 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.022s	sys 0.036s
I20260808 01:52:04.753144 15722 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:04.897320 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 49536.6us
I20260808 01:52:04.910106 10334 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:04.750619 15709 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.098s
I20260808 01:52:04.756201 15565 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:04.915527  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:04.923525 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:04.935501  9480 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260808 01:52:04.940482  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:04.945299 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:04.947938  9600 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:04.914870 15735 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:04.963109  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2619) took 1139 ms. Trace:
I20260808 01:52:04.997329  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:04.999840  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59508.3us
I20260808 01:52:05.001953  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:04.953359 15457 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.701s	user 0.009s	sys 0.005s
I20260808 01:52:04.945446 15724 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:05.009624 15624 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:05.018993  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 301607us
W20260808 01:52:05.023860  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 76096us
I20260808 01:52:05.023568  9524 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:04.997574  9636 rpcz_store.cc:276] 0808 01:52:03.823582 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:04.103736 (+280154us) service_pool.cc:224] Handling call
0808 01:52:04.115680 (+ 11944us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:04.115972 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:52:04.236040 (+120068us) log.cc:844] Serialized 14 byte log entry
0808 01:52:04.951795 (+715755us) log.cc:844] Serialized 14 byte log entry
0808 01:52:04.962656 (+ 10861us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:04.962739 (+    83us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:52:04.962797 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:04.962882 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:04.963035 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":62494,"prepare.queue_time_us":225771,"prepare.run_cpu_time_us":183,"prepare.run_wall_time_us":180,"raft.queue_time_us":56149,"spinlock_wait_cycles":21607808,"thread_start_us":774123,"threads_started":2}
I20260808 01:52:04.933220 15412 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.085s
I20260808 01:52:04.963302 15411 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260808 01:52:04.997198 15139 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.617s	user 0.007s	sys 0.006s
I20260808 01:52:04.986119 15516 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:04.933490 15500 consensus_queue.cc:1059] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.085s
W20260808 01:52:05.032936 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 101537us
I20260808 01:52:05.046917 15598 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:05.047773 15598 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260808 01:52:05.047958 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 110701us
I20260808 01:52:04.889602 15292 consensus_queue.cc:260] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:52:04.997303 15735 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:05.028018 15754 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.524s	user 0.000s	sys 0.002s
I20260808 01:52:05.031095 15557 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:05.046314 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:05.069829  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:05.025038 15644 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:05.027804 15635 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:05.038427 15685 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:05.032136 15451 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:05.031337 15660 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:05.114187 15660 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:05.032172 15624 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:05.121265  9473 consensus_meta.cc:287] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.638s	user 0.012s	sys 0.000s
I20260808 01:52:05.122246  9473 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:05.123131  9472 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:05.124564  9473 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:05.128229  9473 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:05.051321  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:05.050796 15733 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:05.058840 15139 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.058s
I20260808 01:52:05.063694 15735 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:05.082792 15757 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:05.117427 15685 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:05.098152  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:05.088212 15667 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:05.052929 15516 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:05.134101 10362 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.579s	user 0.012s	sys 0.002s
I20260808 01:52:05.156002  9637 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:05.162542 10362 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:52:05.167611 15667 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.169286 15516 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:05.143456 15733 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.175154 15722 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:05.171620 15292 ts_tablet_manager.cc:1434] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.925s	user 0.011s	sys 0.002s
I20260808 01:52:05.139971 15715 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:05.087913 15771 consensus_queue.cc:1059] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.086s
I20260808 01:52:05.160794 15685 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:05.170356 15667 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:05.203446  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 179061us
I20260808 01:52:05.179114 15733 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:05.178390 15757 raft_consensus.cc:697] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:52:05.183337 15292 ts_tablet_manager.cc:1467] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: Tablet startup took 945ms
I20260808 01:52:05.217162  9637 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:05.226992  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:05.221946  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:05.234480  9629 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:05.234570  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:05.235460  9637 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 14: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:05.236708  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:05.232924 15660 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:05.241466 15733 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 15.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:52:05.201057 15715 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:05.225771 15565 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:05.259343  9630 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
W20260808 01:52:05.245910 15585 consensus_meta.cc:287] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.791s	user 0.009s	sys 0.005s
I20260808 01:52:05.268041  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:05.276005  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:05.247181 15733 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:05.278277  9555 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260808 01:52:05.280987  9630 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:05.219105 15730 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:05.290864  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:05.292136  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:05.280717 15347 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:05.298849  9555 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260808 01:52:05.300617 15788 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:05.301316 15788 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.271373 15667 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:52:05.298005 15347 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:05.305347  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:05.306071  9555 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:05.323504  9637 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
W20260808 01:52:05.310290  9558 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.843s	user 0.002s	sys 0.026s
I20260808 01:52:05.259331 15412 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:05.279206 15733 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:05.333164 15733 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.7, Last appended by leader: 7, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.281368 15757 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:05.326401  9559 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:05.340286 10367 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:05.340797  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:05.342160  9629 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 15: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:05.326792  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:05.343736  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:05.344533  9637 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 18: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:05.253225  9634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:05.345703  9634 raft_consensus.cc:606] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:05.326074  9558 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.859s	user 0.003s	sys 0.027s
I20260808 01:52:05.330638 15715 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:05.331243 15412 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:05.273705 15585 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.303395 15721 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:05.309731 15644 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:05.364106  9637 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260808 01:52:05.366227  9637 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260808 01:52:05.342813 15733 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:05.369339 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:05.376971  9629 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: []
I20260808 01:52:05.379642  9480 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: []
W20260808 01:52:05.389021 15800 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260808 01:52:05.391263 15730 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:05.372049 15624 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:05.394167 15720 consensus_meta.cc:287] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.606s	user 0.013s	sys 0.001s
W20260808 01:52:05.402838  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:05.371162 15644 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.368747 15721 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:05.401484 15479 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.985s	user 0.013s	sys 0.004s
W20260808 01:52:05.415683 15797 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260808 01:52:05.417815 15412 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:05.227078 15292 ts_tablet_manager.cc:1469] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: Trace:
0808 01:51:52.241134 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:52.241506 (+   372us) service_pool.cc:224] Handling call
0808 01:51:52.644266 (+402760us) ts_tablet_manager.cc:934] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
0808 01:51:53.135579 (+491313us) spinlock_profiling.cc:243] Waited 34.6 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4186fa5c 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188965a 00007f5d41889582 00007f5d49938f41
0808 01:51:56.605600 (+3470021us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:51:56.870565 (+264965us) tablet_metadata.cc:665] Metadata flushed
0808 01:51:56.895890 (+ 25325us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:51:57.993926 (+1098036us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:51:57.994385 (+   459us) tablet_replica.cc:206] Creating consensus instance
0808 01:51:57.994852 (+   467us) ts_tablet_manager.cc:989] Replica registered.
0808 01:51:57.995019 (+   167us) inbound_call.cc:177] Queueing success response
0808 01:51:58.483961 (+488942us) spinlock_profiling.cc:243] Waited 4.65 ms on lock 0x7b180039e1a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d41731aa8 00007f5d4188c722 00007f5d4993cb33 00007f5d4993c36d 00007f5d499d81e8 00007f5d49a09d5e 00007f5d499eeded
0808 01:52:01.188998 (+2705037us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0808 01:52:04.246299 (+3057301us) ts_tablet_manager.cc:1435] Starting tablet replica
0808 01:52:04.383400 (+137101us) tablet_replica.cc:261] Starting instrumentation
0808 01:52:04.384442 (+  1042us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":52232,"mutex_wait_us":54974,"raft.queue_time_us":9920,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":14937,"spinlock_wait_cycles":90372608,"tablet-copy.queue_time_us":275036,"tablet-download-pool.queue_time_us":15148,"tablet-download-pool.run_cpu_time_us":2428,"tablet-download-pool.run_wall_time_us":1986068,"thread_start_us":815917,"threads_started":2}
I20260808 01:52:05.419971 15509 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:05.440253 15509 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.433153 15788 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:05.445210 15509 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:05.447839 15585 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:05.447518 15668 raft_consensus.cc:3055] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:05.450407 15668 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:05.451474 15668 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:05.453068 15605 consensus_meta.cc:287] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.547s	user 0.010s	sys 0.001s
I20260808 01:52:05.453349 15668 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:52:05.453955 15770 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.391s	user 0.000s	sys 0.002s
I20260808 01:52:05.453732 15605 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.459769 15347 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:05.437830 15304 consensus_meta.cc:287] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.986s	user 0.013s	sys 0.000s
W20260808 01:52:05.456161 15600 consensus_meta.cc:287] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.718s	user 0.005s	sys 0.008s
W20260808 01:52:05.478495  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:05.482127 15721 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:05.485105 15737 raft_consensus.cc:3055] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:05.485724 15737 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:05.487802  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3253) took 1090 ms. Trace:
I20260808 01:52:05.465699 15479 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:05.476392 15304 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260808 01:52:05.492113 15644 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:05.493706  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:05.494412 15624 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:05.490067 15585 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:05.496023  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:05.496658 15788 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 01:52:05.476938 15412 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:05.497998 15667 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
W20260808 01:52:05.499173 10334 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.598s (SENT) [suppressed 16 similar messages]
I20260808 01:52:05.501045 15635 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:05.507563  9524 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:05.509562  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2981) took 1157 ms. Trace:
I20260808 01:52:05.478219 15600 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:05.512413 15767 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.481s	user 0.003s	sys 0.000s
I20260808 01:52:05.486985 15737 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.408138 15720 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:05.475039 15406 consensus_meta.cc:287] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.713s	user 0.014s	sys 0.001s
I20260808 01:52:05.519328  9481 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:05.523713 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:05.526314  9477 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: []
I20260808 01:52:05.527333  9630 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:05.528388  9474 rpcz_store.cc:276] 0808 01:52:04.351884 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:04.352112 (+   228us) service_pool.cc:224] Handling call
0808 01:52:05.509296 (+1157184us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b540019c8d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:05.509483 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":400636800}
I20260808 01:52:05.535574  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:05.536275  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:05.537810  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:05.453156 15735 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.538789  9556 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:05.538956  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:05.539057  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:05.536598  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:05.545686  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:05.545197  9634 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:52:05.552673  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:05.553689  9474 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:05.557163  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:05.558187  9554 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:52:05.560354  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:05.561347  9474 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:05.566768  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:05.555467  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:05.568444  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:05.569743  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:05.570744  9481 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:05.575202  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:05.579267  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:05.534286 15710 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:05.540593 15644 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:05.581249  9632 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:52:05.522275 15624 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.581478  9636 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:05.523770 15585 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.584888  9473 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:05.529076 15412 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:05.563668 15779 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:05.540830 15771 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.047s
I20260808 01:52:05.563359 15737 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:05.532330 15721 consensus_queue.cc:1059] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
I20260808 01:52:05.577749 15796 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:05.580210 15718 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:05.572080 15685 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.550050 15406 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.595945 15718 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:05.618649 15347 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:05.666890 15347 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:05.668111 15347 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:05.624001  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:05.625254  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3241) took 2051 ms. Trace:
I20260808 01:52:05.596483 15796 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:05.655499 15805 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260808 01:52:05.628628 15412 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:05.629078 15763 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.549s	user 0.001s	sys 0.001s
W20260808 01:52:05.630249 15769 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.571s	user 0.000s	sys 0.002s
I20260808 01:52:05.631345 15735 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:05.633049 15411 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:05.635113 15605 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:05.637269  9481 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
W20260808 01:52:05.637557 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 582895us
W20260808 01:52:05.639395 15774 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.523s	user 0.002s	sys 0.000s
W20260808 01:52:05.641762 15768 log.cc:927] Time spent T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.624s	user 0.004s	sys 0.000s
I20260808 01:52:05.645308 15585 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:05.644925 15479 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:52:05.646459 15644 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:05.648185 15776 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.600s	user 0.000s	sys 0.002s
W20260808 01:52:05.652334 15766 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.541s	user 0.000s	sys 0.001s
I20260808 01:52:05.591663 15710 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:05.594794 15779 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:05.670127  9552 rpcz_store.cc:276] 0808 01:52:03.574213 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:03.594921 (+ 20708us) service_pool.cc:224] Handling call
0808 01:52:03.595372 (+   451us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:04.279172 (+683800us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:05.624820 (+1345648us) spinlock_profiling.cc:243] Waited 629 ms on lock 0x7b5400257e54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:05.625176 (+   356us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1447372032}
I20260808 01:52:05.671805 15796 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:52:05.627776 15764 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.597s	user 0.003s	sys 0.000s
I20260808 01:52:05.668838 15715 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:05.672449 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 173306us
W20260808 01:52:05.674994  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 108334us
I20260808 01:52:05.749450  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:05.656406 15721 consensus_queue.cc:1243] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 15.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260808 01:52:05.679427  9479 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 16: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:05.669346 15718 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:05.565469 15778 consensus_queue.cc:1175] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" responder_term: 16 status { last_received { term: 15 index: 8 } last_committed_idx: 8 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 01:52:05.737943 15406 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:05.740252 15685 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:05.743450 15513 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:05.743975  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3251) took 1497 ms. Trace:
I20260808 01:52:05.713968 15347 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:05.754613  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:05.720470 15644 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:05.746402 15779 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:05.770637 10364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 693) took 1451 ms. Trace:
I20260808 01:52:05.759157 15721 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:05.779737 15685 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:05.772199  9557 rpcz_store.cc:276] 0808 01:52:04.246565 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:04.246766 (+   201us) service_pool.cc:224] Handling call
0808 01:52:04.280865 (+ 34099us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:04.306782 (+ 25917us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:04.306840 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:04.307683 (+   843us) log.cc:844] Serialized 26 byte log entry
0808 01:52:04.592131 (+284448us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:52:04.592193 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:04.592293 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:05.641037 (+1048744us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:05.743530 (+102493us) raft_consensus.cc:1685] finished
0808 01:52:05.743565 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:05.743878 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39819,"raft.queue_time_us":26409,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":19231,"spinlock_wait_cycles":365332864,"thread_start_us":206039,"threads_started":1,"wal-append.queue_time_us":551990}
I20260808 01:52:05.759336 15784 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:05.800633  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2978) took 1498 ms. Trace:
I20260808 01:52:05.783041  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:05.767534 15565 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:05.803378 15565 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.805773 15565 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:05.767189 15811 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:05.807168 15811 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:05.808296  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 750) took 1521 ms. Trace:
I20260808 01:52:05.809581 15811 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:05.801084  9478 rpcz_store.cc:276] 0808 01:52:04.302708 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:04.302889 (+   181us) service_pool.cc:224] Handling call
0808 01:52:04.327246 (+ 24357us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:04.333301 (+  6055us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:04.333333 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:04.355503 (+ 22170us) log.cc:844] Serialized 26 byte log entry
0808 01:52:04.568924 (+213421us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:52:04.729887 (+160963us) log.cc:844] Serialized 14 byte log entry
0808 01:52:04.742660 (+ 12773us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:04.742801 (+   141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:05.628452 (+885651us) spinlock_profiling.cc:243] Waited 97.2 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:05.800303 (+171851us) raft_consensus.cc:1685] finished
0808 01:52:05.800416 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:05.800571 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18455,"prepare.queue_time_us":122188,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":22570,"raft.run_cpu_time_us":409,"raft.run_wall_time_us":20952,"spinlock_wait_cycles":223537408,"thread_start_us":192890,"threads_started":1,"wal-append.queue_time_us":629777}
I20260808 01:52:05.813441  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:05.761636 15759 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.742s	user 0.009s	sys 0.005s
I20260808 01:52:05.782337 10364 rpcz_store.cc:276] 0808 01:52:04.309100 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:04.309262 (+   162us) service_pool.cc:224] Handling call
0808 01:52:04.337872 (+ 28610us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:04.496645 (+158773us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:04.496685 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:04.497180 (+   495us) log.cc:844] Serialized 26 byte log entry
0808 01:52:04.835318 (+338138us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:04.835376 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:04.835479 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:05.629625 (+794146us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:05.754292 (+124667us) raft_consensus.cc:1685] finished
0808 01:52:05.754317 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:05.760597 (+  6280us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30204,"raft.queue_time_us":55961,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":5792,"spinlock_wait_cycles":242054912,"thread_start_us":449041,"threads_started":2,"wal-append.queue_time_us":437185}
W20260808 01:52:05.821166 15759 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.823s	user 0.010s	sys 0.006s
I20260808 01:52:05.787901 15710 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.488376  9558 rpcz_store.cc:276] 0808 01:52:04.397181 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:04.397323 (+   142us) service_pool.cc:224] Handling call
0808 01:52:04.427453 (+ 30130us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:04.428055 (+   602us) raft_consensus.cc:1534] Early marking committed up to index 9
0808 01:52:05.346788 (+918733us) log.cc:844] Serialized 14 byte log entry
0808 01:52:05.487376 (+140588us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:05.487447 (+    71us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:05.487502 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:05.487579 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:05.487734 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20532,"prepare.queue_time_us":599119,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":113,"thread_start_us":1019784,"threads_started":2}
I20260808 01:52:05.825019  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 858) took 1752 ms. Trace:
I20260808 01:52:05.793151 15721 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.825768 15796 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:05.777951 15720 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:05.827154 15755 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.319s	user 0.002s	sys 0.001s
I20260808 01:52:05.776293 15600 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:05.798877 15516 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:05.828249 15516 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:05.828826 15721 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:05.831166  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 748) took 1753 ms. Trace:
I20260808 01:52:05.831696  9635 rpcz_store.cc:276] 0808 01:52:04.077863 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:04.132267 (+ 54404us) service_pool.cc:224] Handling call
0808 01:52:04.151815 (+ 19548us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:04.152185 (+   370us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:04.152205 (+    20us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:04.153702 (+  1497us) log.cc:844] Serialized 26 byte log entry
0808 01:52:04.407469 (+253767us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:04.448846 (+ 41377us) log.cc:844] Serialized 14 byte log entry
0808 01:52:04.449303 (+   457us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:04.449391 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:05.826225 (+1376834us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:05.830760 (+  4535us) raft_consensus.cc:1685] finished
0808 01:52:05.830895 (+   135us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:05.831083 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":41161,"prepare.run_cpu_time_us":148,"prepare.run_wall_time_us":145,"raft.queue_time_us":11988,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":961,"spinlock_wait_cycles":37023616,"thread_start_us":188251,"threads_started":1,"wal-append.queue_time_us":279591}
I20260808 01:52:05.799654 15624 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:05.755790 15715 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:05.786875 15810 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:05.837966 15810 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:05.838397 15810 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:05.840760 15340 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.837426 15715 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:05.821738 15765 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.721s	user 0.003s	sys 0.000s
I20260808 01:52:05.843618 15334 raft_consensus.cc:3055] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:52:05.778808 15513 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.844205 15334 raft_consensus.cc:740] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:05.845391 15334 consensus_queue.cc:260] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.846100 15513 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:05.847886 15516 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:05.848675  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:05.855720 15519 consensus_queue.cc:1059] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:05.856403  9551 raft_consensus.cc:3055] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:05.856875  9551 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:05.811685 15685 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:05.860399 15685 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:05.861622 15340 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:05.825572  9560 rpcz_store.cc:276] 0808 01:52:04.072320 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:04.112110 (+ 39790us) service_pool.cc:224] Handling call
0808 01:52:04.113709 (+  1599us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:04.481417 (+367708us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:05.824698 (+1343281us) spinlock_profiling.cc:243] Waited 304 ms on lock 0x7b54000b40d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:05.824942 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":699771648}
W20260808 01:52:05.863694 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:05.879993  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:05.808832  9638 rpcz_store.cc:276] 0808 01:52:04.287086 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:04.307357 (+ 20271us) service_pool.cc:224] Handling call
0808 01:52:04.308428 (+  1071us) raft_consensus.cc:1471] Updating replica for 3 ops
0808 01:52:04.335150 (+ 26722us) raft_consensus.cc:1534] Early marking committed up to index 9
0808 01:52:04.485233 (+150083us) log.cc:844] Serialized 14 byte log entry
0808 01:52:04.985963 (+500730us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:04.986648 (+   685us) log.cc:844] Serialized 52 byte log entry
0808 01:52:04.986828 (+   180us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:04.987066 (+   238us) log.cc:844] Serialized 14 byte log entry
0808 01:52:04.987340 (+   274us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:04.987446 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:05.498441 (+510995us) spinlock_profiling.cc:243] Waited 9.98 ms on lock 0x7b1800455b00. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4185ac56 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f
0808 01:52:05.499059 (+   618us) spinlock_profiling.cc:243] Waited 3.66 ms on lock 0x7b48002379d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:52:05.647325 (+148266us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:05.807937 (+160612us) raft_consensus.cc:1685] finished
0808 01:52:05.807971 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:05.808205 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37311,"prepare.queue_time_us":44532,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":98,"raft.queue_time_us":65315,"raft.run_cpu_time_us":3709,"raft.run_wall_time_us":523184,"spinlock_wait_cycles":377382016,"thread_start_us":486538,"threads_started":2,"wal-append.queue_time_us":522860}
I20260808 01:52:05.864305  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2990) took 1296 ms. Trace:
I20260808 01:52:05.881451  9479 rpcz_store.cc:276] 0808 01:52:04.567594 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:04.572092 (+  4498us) service_pool.cc:224] Handling call
0808 01:52:05.864071 (+1291979us) spinlock_profiling.cc:243] Waited 957 ms on lock 0x7b540019b9d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:05.864237 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2201276800}
I20260808 01:52:05.864887  9474 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:05.865993 15784 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:05.868778  9475 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
W20260808 01:52:05.869330  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 86382.7us
I20260808 01:52:05.892345  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:05.869992 15667 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:52:05.870816  9551 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:05.896636  9551 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:05.873108  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 2984) took 1453 ms. Trace:
I20260808 01:52:05.897773  9472 rpcz_store.cc:276] 0808 01:52:04.419585 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:04.419989 (+   404us) service_pool.cc:224] Handling call
0808 01:52:05.122952 (+702963us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:05.125773 (+  2821us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:05.872860 (+747087us) spinlock_profiling.cc:243] Waited 665 ms on lock 0x7b54000933d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:05.873036 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1529699712}
I20260808 01:52:05.875031  9524 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:05.906852  9478 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:05.910174  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:05.906970  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:05.912451  9479 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:05.917608  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:05.919530  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:05.921097  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:05.921878  9478 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
W20260808 01:52:05.922971  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:05.923226  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:05.891942 15334 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:52:05.908603 15717 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.021s	user 0.001s	sys 0.022s
W20260808 01:52:05.932459 15717 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.045s	user 0.001s	sys 0.022s
I20260808 01:52:05.915841 15624 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260808 01:52:05.912359 15303 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:05.937749  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:05.938632  9478 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:52:05.913417 15811 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:05.913838 15294 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:05.940307  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:05.927749 15667 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:05.927378 15733 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:05.945278  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:05.943210  9472 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
W20260808 01:52:05.951956  9476 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 967) took 2044 ms (client timeout 1598 ms). Trace:
I20260808 01:52:05.954427  9481 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
W20260808 01:52:05.957927  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:05.958101  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:05.958249  9552 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
W20260808 01:52:05.952498  9476 rpcz_store.cc:269] 0808 01:52:03.907644 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:03.907900 (+   256us) service_pool.cc:224] Handling call
0808 01:52:05.951724 (+2043824us) spinlock_profiling.cc:243] Waited 540 ms on lock 0x7b54002bc0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:05.951880 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1242203392}
I20260808 01:52:05.959574  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:05.961390  9478 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 16: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:05.962477  9554 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260808 01:52:05.963198  9554 pending_rounds.cc:85] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 7
I20260808 01:52:05.921736 15788 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260808 01:52:05.963580  9554 pending_rounds.cc:107] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260808 01:52:05.969369  9554 raft_consensus.cc:2889] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:05.942415 15294 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:05.976086 14784 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:05.976616  9550 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:05.978360  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:05.948015 15796 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:05.981678  9558 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260808 01:52:05.976709 14784 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:05.989913  9521 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:05.991817  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:05.962247 15495 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:05.947134 15303 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:05.963323 15600 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:05.962394 15685 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:05.999153  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 79724us
I20260808 01:52:05.974900 15479 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:06.014071 15685 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:06.004767 15796 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.010s
W20260808 01:52:06.016232 15819 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260808 01:52:06.017630 15495 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:06.018910 15624 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260808 01:52:06.020223 15495 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:06.023700 15624 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:06.027091  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:05.966893 15761 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:06.007261 15516 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:06.030193 15761 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:06.031111 15516 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:05.945943 15811 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:06.034525  9475 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 2987) took 1532 ms (client timeout 1747 ms). Trace:
I20260808 01:52:06.035326 15811 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:06.036579  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 46743.9us
I20260808 01:52:06.006492  9551 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 11. (term mismatch)
I20260808 01:52:06.027560 15788 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:52:05.964015 15784 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:06.004812 15660 consensus_queue.cc:1059] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
I20260808 01:52:06.029232 15303 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:06.006078  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:06.038170  9551 pending_rounds.cc:85] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 10
W20260808 01:52:06.096664  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 56164.2us
I20260808 01:52:06.097119  9551 pending_rounds.cc:107] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260808 01:52:06.097687  9551 raft_consensus.cc:2889] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:52:06.099042  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 72032.6us
I20260808 01:52:05.980855 15294 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:06.099968  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:06.049360  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:06.034293 15516 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:06.014245 15479 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:06.099247 14784 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:06.035109  9475 rpcz_store.cc:269] 0808 01:52:04.502025 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:04.505744 (+  3719us) service_pool.cc:224] Handling call
0808 01:52:05.714616 (+1208872us) spinlock_profiling.cc:243] Waited 5.12 ms on lock 0x7b5400392a28. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7aaa44 00007f5d4998b037 00007f5d4998a8c8 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:06.034330 (+319714us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b5400392bd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:06.034456 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":343098624}
I20260808 01:52:06.099570  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:06.096388  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 104680us
I20260808 01:52:06.001072 15737 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:06.135784  9524 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:06.138255  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:06.143579  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 43723.1us
I20260808 01:52:06.150393  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:06.041658 15761 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:06.165241  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:06.169305  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:06.172066  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:06.172839  9638 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:52:06.175197  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:06.185334  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:06.185694  9637 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:52:06.135455 15771 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
I20260808 01:52:06.044279 15685 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:06.186183  9638 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:52:06.187829  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:06.126056 15715 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:52:06.188562 15821 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:06.209329  9631 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:52:06.188764  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:06.040628 15720 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.137s
I20260808 01:52:06.101653 15513 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.094s
W20260808 01:52:06.189766 10362 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 768) took 1703 ms (client timeout 2076 ms). Trace:
I20260808 01:52:06.095480 15818 raft_consensus.cc:3055] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:06.194199 15824 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:06.195072 15796 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:06.187887  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:06.197669 15820 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.233s	user 0.001s	sys 0.000s
I20260808 01:52:06.199621  9475 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260808 01:52:06.045296 15721 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:52:06.194303  9521 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:06.207792  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:06.210781  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:06.127179 15624 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:06.003316 15717 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:06.214654  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 202366us
I20260808 01:52:06.247017 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:06.127027 15668 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:06.157670 15825 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:52:06.011655 15810 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:06.168502 15811 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:06.222620  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:06.199204 15778 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:06.227901  9630 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
W20260808 01:52:06.228343  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:06.224973 10362 rpcz_store.cc:269] 0808 01:52:04.486605 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:04.486888 (+   283us) service_pool.cc:224] Handling call
0808 01:52:06.189568 (+1702680us) spinlock_profiling.cc:243] Waited 51.8 ms on lock 0x7b54004c3454. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:06.189695 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":119093120}
I20260808 01:52:06.241143  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:06.239106  9636 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
W20260808 01:52:06.302223  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 61211.7us
I20260808 01:52:06.303797  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:06.256824  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62617.4us
I20260808 01:52:06.304967  9521 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:06.257225  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:06.142005 15495 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.079s
I20260808 01:52:06.214113 15715 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:06.259871 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:06.262036  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:06.048311 15600 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:06.269588  9479 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:06.299294  9472 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 15: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:06.250051 15624 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:06.227603 15818 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:06.310742 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48329.9us
I20260808 01:52:06.309223  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:06.256482 15412 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260808 01:52:06.322135 15715 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:06.259455 15812 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.083s
I20260808 01:52:06.316823 15620 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:06.343636 15624 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:06.346414 15836 raft_consensus.cc:3055] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Stepping down as leader of term 8
I20260808 01:52:06.346946 15836 raft_consensus.cc:740] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:06.264430 15721 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:06.344738 15832 raft_consensus.cc:3055] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:06.348160 15832 raft_consensus.cc:740] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:06.349128 15832 consensus_queue.cc:260] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:06.350525 15832 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:06.352257 15836 consensus_queue.cc:260] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:06.354616 10367 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
W20260808 01:52:06.345628 15828 log.cc:927] Time spent T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260808 01:52:06.356134 15836 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 8 FOLLOWER]: Advancing to term 9
I20260808 01:52:06.354609 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:06.357800  9523 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:06.353494 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:06.246511 15796 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:06.367897  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:06.266759 15685 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:06.360873  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:06.361943 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:06.270351 15811 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:06.225136 15821 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:06.394311 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:06.372638 10362 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:06.375447  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:06.260821 15717 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:06.306897 15778 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:06.346818 15412 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:52:06.386757  9479 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 16: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:06.349800 15492 raft_consensus.cc:3055] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:06.308029 15304 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:52:06.263954 15825 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:06.312422 15709 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:52:06.321525 15406 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:52:06.371894  9552 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:52:06.234985 15824 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:06.345919 15620 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:06.368412 15726 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:06.425062 15726 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:06.427527 15726 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:06.405540  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:06.428493 10364 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
W20260808 01:52:06.431213 15816 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260808 01:52:06.417013 15406 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:06.432091 15406 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:06.432220 15796 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:06.432631 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:06.432749 10363 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:06.432979 15796 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:52:06.258821 15644 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:06.386152 15685 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:06.403489 15778 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:06.434183 15685 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:06.436780 15818 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:06.438258 15818 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:06.397030 15811 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:06.322499 15565 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.079s
I20260808 01:52:06.264693 15668 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:06.406299 15492 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:06.413141 15304 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:06.422717 15721 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:06.423854 15717 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:06.424358 15620 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:06.433533 15796 raft_consensus.cc:697] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:06.442530  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:06.401717 15412 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:06.451184  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2872) took 1397 ms. Trace:
I20260808 01:52:06.451314 10362 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEARNER]: Advancing to term 15
W20260808 01:52:06.466118  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:06.464021 15565 consensus_queue.cc:1243] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 16.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260808 01:52:06.478971 15644 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:06.481772 15796 consensus_queue.cc:237] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:06.470643 15304 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:52:06.484865 15492 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:06.417883 15825 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:06.488891 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:06.492749  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:06.494853  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 974) took 1045 ms. Trace:
W20260808 01:52:06.634310  9601 connection.cc:256] RPC call timeout handler was delayed by -1.1644188680220395s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1646s for a response
I20260808 01:52:06.635931  9481 rpcz_store.cc:276] 0808 01:52:05.449468 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:05.717350 (+267882us) service_pool.cc:224] Handling call
0808 01:52:05.954071 (+236721us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:05.954948 (+   877us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:06.494502 (+539554us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b54002bc0d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:06.494744 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":473259264}
I20260808 01:52:06.497015  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2474) took 1992 ms. Trace:
I20260808 01:52:06.637619  9550 rpcz_store.cc:276] 0808 01:52:04.504289 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:04.505637 (+  1348us) service_pool.cc:224] Handling call
0808 01:52:04.506041 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:05.977391 (+1471350us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:06.496623 (+519232us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b5400325ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:06.496924 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3381469952}
I20260808 01:52:06.639161  9554 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:52:06.538847 15412 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:06.615978 14784 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:06.617762 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:06.537039 15479 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:06.503198  9473 rpcz_store.cc:276] 0808 01:52:05.054130 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:05.275242 (+221112us) service_pool.cc:224] Handling call
0808 01:52:05.291624 (+ 16382us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:05.592310 (+300686us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:06.447164 (+854854us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x7b540023aad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:06.451119 (+  3955us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":673804544}
I20260808 01:52:06.612983 15832 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:06.620975 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:52:06.623225  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:06.623121  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:06.611330  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:06.624348 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 222194us
I20260808 01:52:06.625382  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:06.627465  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 306653us
W20260808 01:52:06.622041 15822 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.402s	user 0.005s	sys 0.000s
I20260808 01:52:06.674296  9474 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
W20260808 01:52:06.671578  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:06.686058 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:06.640954 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:06.622391 15779 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.808s	user 0.013s	sys 0.003s
I20260808 01:52:06.445441 15737 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260808 01:52:06.677403  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 319655us
I20260808 01:52:06.766223 10332 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:06.645671  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:06.767092  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:06.645942  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:06.768337  9554 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:06.647300 10366 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:52:06.680958  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:06.641347 14784 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:06.633101 15729 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:06.640203  9601 connection.cc:256] RPC call timeout handler was delayed by -1.0222188680199906s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1804s for a response
W20260808 01:52:06.642256 15334 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.706s	user 0.012s	sys 0.000s
W20260808 01:52:06.642382 15340 consensus_meta.cc:287] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.746s	user 0.011s	sys 0.002s
I20260808 01:52:06.775841  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:06.806353  9637 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:06.802949  9481 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 8. Current term is 9. Ops: []
I20260808 01:52:06.809643  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:06.810371  9637 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:06.644413 15810 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:06.806478  9555 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:52:06.681764 15605 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:06.759260 15492 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:06.896708  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:06.897800  9632 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:52:06.860209 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:06.641002 15644 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:06.861559  9630 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: [11.6-11.6]
W20260808 01:52:06.822295  9629 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.024s	sys 0.041s
I20260808 01:52:06.761710 15824 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:06.905555 15829 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.245s	user 0.005s	sys 0.001s
W20260808 01:52:06.919324  9629 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.027s	sys 0.043s
I20260808 01:52:06.805881 15479 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:06.800005 15837 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:06.931998  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:06.971701  9637 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:06.977538  9632 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:52:06.978546  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:06.862371 15334 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:06.681736 15832 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:06.897280 15605 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:06.988708  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:06.781906 15779 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:06.999652 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:06.929512 15821 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:06.929840 15844 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:06.805253 15729 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:07.012430  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:06.995676  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:06.931399 15668 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:07.008863  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:07.010218  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2876) took 1698 ms. Trace:
I20260808 01:52:06.990183  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:06.985070 15340 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:06.997291 15718 consensus_meta.cc:287] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.227s	user 0.013s	sys 0.012s
I20260808 01:52:06.986145 15837 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:06.983255 15479 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:06.928357 15644 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.013252  9630 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:52:07.016969  9480 rpcz_store.cc:276] 0808 01:52:05.311546 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:05.622521 (+310975us) service_pool.cc:224] Handling call
0808 01:52:05.622978 (+   457us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:05.657423 (+ 34445us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:05.657456 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:05.671189 (+ 13733us) log.cc:844] Serialized 26 byte log entry
0808 01:52:05.945836 (+274647us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:05.945883 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:05.945970 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:07.009966 (+1063996us) raft_consensus.cc:1685] finished
0808 01:52:07.009992 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:07.010155 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33923,"raft.queue_time_us":34143,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":5455,"thread_start_us":226709,"threads_started":1,"wal-append.queue_time_us":330692}
W20260808 01:52:07.152806 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 468478us
I20260808 01:52:07.022665  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:07.148187  9601 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:07.033962 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:07.040740 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:07.082279 15304 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:07.199360  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:07.199854 15304 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:07.195313 15718 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:07.203256 15710 raft_consensus.cc:3055] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:07.200309  9474 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 10. Current term is 11. Ops: []
I20260808 01:52:07.197415 15852 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:07.083362 15827 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.666s	user 0.011s	sys 0.005s
W20260808 01:52:07.073107 15715 consensus_meta.cc:287] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.731s	user 0.004s	sys 0.008s
W20260808 01:52:07.037695 15294 consensus_meta.cc:287] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.819s	user 0.010s	sys 0.000s
W20260808 01:52:07.086933 15624 consensus_meta.cc:287] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.740s	user 0.009s	sys 0.004s
W20260808 01:52:07.136026  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 43706.3us
W20260808 01:52:07.007206 15778 consensus_meta.cc:287] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.573s	user 0.009s	sys 0.003s
I20260808 01:52:07.232197  9445 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:07.085014 15729 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:07.061339  9524 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:07.069643  9638 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:07.184183 10366 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:07.033530 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:07.054833 15779 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:07.053692  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:07.252539  9558 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:52:07.234174 15715 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:07.254068  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:07.254870  9558 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:52:07.065033  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:07.195107  9637 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:52:07.009814  9634 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.008s	sys 0.085s
W20260808 01:52:07.257601  9634 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.774s	user 0.008s	sys 0.085s
W20260808 01:52:07.211992  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 63897.2us
I20260808 01:52:07.203742 15710 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:07.261401  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:07.037369 15685 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.602s	user 0.006s	sys 0.005s
W20260808 01:52:07.111222 15823 log.cc:927] Time spent T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.784s	user 0.000s	sys 0.005s
I20260808 01:52:06.992661 15334 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:07.054571 15843 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.253s	user 0.000s	sys 0.001s
I20260808 01:52:07.156040 15833 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:07.197018 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 430870us
I20260808 01:52:07.227106  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:07.277055  9630 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:52:07.087423 15303 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:07.034663 15811 consensus_meta.cc:287] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.594s	user 0.012s	sys 0.000s
I20260808 01:52:07.052516 15717 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:07.089161 15733 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:07.291312 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:07.053669  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:07.292469  9633 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:07.042572 15832 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:07.008924 15761 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.012s	sys 0.023s
W20260808 01:52:07.237515  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 176266us
I20260808 01:52:07.050572 15851 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:07.239563 15304 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:07.230131 15718 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:07.246794 15778 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:07.347021 15304 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.349313 15304 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:07.349633 15778 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:07.349926 15304 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260808 01:52:07.249141 15406 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.816s	user 0.011s	sys 0.002s
I20260808 01:52:07.285758 15833 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:07.350961 15406 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.351291  9472 raft_consensus.cc:2393] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 16: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:07.079425 15605 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:07.352505 15605 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:07.352993 15406 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:07.299907 15733 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.195114 15340 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:07.031899 15818 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.593s	user 0.007s	sys 0.010s
I20260808 01:52:07.088230 15858 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:52:07.042222 15836 consensus_meta.cc:287] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.668s	user 0.016s	sys 0.000s
I20260808 01:52:07.263731  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:07.149251 15810 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.082s
I20260808 01:52:07.028496  9648 tablet_copy_service.cc:342] P 259bcb3f0d254e8b9effac3277e654e0: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-7c508218361c446898fede7c5682fa9e received from {username='slave'} at 127.0.0.1:45658
I20260808 01:52:07.267172  9632 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 18: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:07.043990 15847 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.184s
I20260808 01:52:07.229698 10334 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:07.377730 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 148196us
I20260808 01:52:07.044179 15636 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.185194 15849 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.119s
I20260808 01:52:07.050892  9477 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:07.302052  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:07.383610 15849 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:07.384099 15849 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:07.385222 15849 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.216581 15848 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:07.314383  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:07.295100 15679 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:07.283335 15519 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:07.387133  9630 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:07.023851 15844 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:07.272780 15685 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:07.318326 15839 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.545s	user 0.000s	sys 0.001s
I20260808 01:52:07.218144 15852 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:07.394044 15836 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:07.393656 15679 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:07.394582 15836 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:07.395066 15836 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:07.396106 15852 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:07.396461 15679 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:07.397419 15519 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:07.297484 15303 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:07.320672  9524 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:07.296070 15811 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:07.301249 15761 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.857s	user 0.013s	sys 0.023s
I20260808 01:52:07.327232 10363 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:57156
I20260808 01:52:07.326843  9637 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:07.410904  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:07.411507  9637 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:07.340083 15479 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:07.342576 15334 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:07.344296 15837 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:07.346105  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 866) took 1661 ms. Trace:
W20260808 01:52:07.355479 10331 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.019s	sys 0.021s
W20260808 01:52:07.438268 10331 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.019s	sys 0.021s
I20260808 01:52:07.438037  9556 rpcz_store.cc:276] 0808 01:52:05.684754 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:05.684924 (+   170us) service_pool.cc:224] Handling call
0808 01:52:05.685334 (+   410us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:05.685749 (+   415us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:05.685782 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:05.717468 (+ 31686us) log.cc:844] Serialized 26 byte log entry
0808 01:52:06.222323 (+504855us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:06.222397 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:06.222563 (+   166us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:07.060288 (+837725us) spinlock_profiling.cc:243] Waited 2.08 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:07.345698 (+285410us) raft_consensus.cc:1685] finished
0808 01:52:07.345734 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:07.346021 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":56242,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":108,"spinlock_wait_cycles":6151680,"thread_start_us":462839,"threads_started":1,"wal-append.queue_time_us":608562}
I20260808 01:52:07.236269 15294 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.441390  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2499) took 1265 ms. Trace:
I20260808 01:52:07.441869  9556 rpcz_store.cc:276] 0808 01:52:06.176489 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:07.440449 (+1263960us) service_pool.cc:224] Handling call
0808 01:52:07.440810 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:07.441022 (+   212us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:07.441056 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:07.441107 (+    51us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:52:07.441141 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:07.441202 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:07.441321 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:07.443563 15294 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:07.444655  9556 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 13. Current term is 14. Ops: []
I20260808 01:52:07.445428  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 871) took 1184 ms. Trace:
I20260808 01:52:07.287528 15717 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:07.445784  9556 rpcz_store.cc:276] 0808 01:52:06.260923 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:07.444088 (+1183165us) service_pool.cc:224] Handling call
0808 01:52:07.444453 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:07.445146 (+   693us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:07.445364 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:07.446951 10368 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:07.446851 15717 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.448724 15717 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:07.451098  9557 raft_consensus.cc:2368] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16.
I20260808 01:52:07.262529 15624 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.452848 14784 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:07.453861 15624 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:07.454102 10364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 971) took 1234 ms. Trace:
I20260808 01:52:07.370061  9558 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:07.454660 10364 rpcz_store.cc:276] 0808 01:52:06.219388 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:06.219547 (+   159us) service_pool.cc:224] Handling call
0808 01:52:07.453850 (+1234303us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b5400620f54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:07.454017 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":415187328}
I20260808 01:52:07.456096  9558 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:07.456938  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 874) took 1195 ms. Trace:
I20260808 01:52:07.457340  9558 rpcz_store.cc:276] 0808 01:52:06.261990 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:07.455408 (+1193418us) service_pool.cc:224] Handling call
0808 01:52:07.455838 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:07.456625 (+   787us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:07.456870 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:07.355741 15851 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:07.458840 15729 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:07.458974 15851 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:07.354667 15832 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:52:07.459537 15729 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.353056 15833 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:07.355020 15778 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:07.462037 15729 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:07.462368 15778 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:07.409847 10363 raft_consensus.cc:606] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:07.367651 15825 consensus_meta.cc:287] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.721s	user 0.012s	sys 0.000s
I20260808 01:52:07.374783 15858 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:07.385409  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:07.476159  9633 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:52:07.264889 15720 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:07.391543  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:07.475345  9552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.008s	sys 0.027s
I20260808 01:52:07.378123 15715 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:07.400179 15644 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:07.426241 15519 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:07.426949  9560 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:07.428453 15685 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:07.436645 15303 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:07.409044 15139 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:07.530553  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:07.443146 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:07.366523  9523 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.004s	sys 0.050s
I20260808 01:52:07.356706 15845 consensus_queue.cc:1059] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.044s
I20260808 01:52:07.465494 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:07.532019  9523 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.715s	user 0.004s	sys 0.050s
I20260808 01:52:07.468499  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 875) took 1206 ms. Trace:
I20260808 01:52:07.532598  9558 rpcz_store.cc:276] 0808 01:52:06.262309 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:07.458336 (+1196027us) service_pool.cc:224] Handling call
0808 01:52:07.467793 (+  9457us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:07.468031 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:07.468069 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:07.468140 (+    71us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:07.468186 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:07.468265 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:07.468423 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:52:07.403844  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 83234.8us
I20260808 01:52:07.536614 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:07.538437 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:07.480407 15720 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:07.547353 15720 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:52:07.527685 15412 consensus_meta.cc:287] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.887s	user 0.006s	sys 0.006s
I20260808 01:52:07.551281 15412 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.552865  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:07.553866 15412 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:07.554215  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:07.478878 15849 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:07.416461  9648 tablet_copy_service.cc:434] P 259bcb3f0d254e8b9effac3277e654e0: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-7c508218361c446898fede7c5682fa9e on tablet 7c508218361c446898fede7c5682fa9e with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:07.555634  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:07.556366  9558 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 15: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:07.556988 15685 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:07.557189 15139 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:07.558166 15139 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.559180 15845 raft_consensus.cc:3055] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:07.559700 15845 raft_consensus.cc:740] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:07.559826 15600 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:07.560037 15811 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:07.560614 15845 consensus_queue.cc:260] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.561368 15733 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:07.559262  9479 raft_consensus.cc:2368] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13.
I20260808 01:52:07.562420 15513 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:07.563426 15858 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:07.479846  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:07.564311  9633 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:52:07.565307  9633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "244e080726394210b5467bf688813cf0"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:07.565881  9633 raft_consensus.cc:606] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:07.566213  9633 raft_consensus.cc:614] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:07.425001 15340 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:07.480643  9475 raft_consensus.cc:2368] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16.
I20260808 01:52:07.478072  9632 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
W20260808 01:52:07.567283  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:07.464569 15778 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:07.459468 15347 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:07.477434 15825 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.568920 15347 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:07.569831 15347 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.570645 15825 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:07.571171 15347 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:07.557587 15685 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.575383  9551 raft_consensus.cc:2368] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13.
I20260808 01:52:07.577060 15685 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:07.577610  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3259) took 1614 ms. Trace:
I20260808 01:52:07.578029  9551 rpcz_store.cc:276] 0808 01:52:05.963161 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:06.168908 (+205747us) service_pool.cc:224] Handling call
0808 01:52:07.577390 (+1408482us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b5400383954. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:07.577543 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3231190912}
W20260808 01:52:07.579052  9475 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 3000) took 1383 ms (client timeout 1623 ms). Trace:
W20260808 01:52:07.579569  9475 rpcz_store.cc:269] 0808 01:52:06.196001 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:06.256777 (+ 60776us) service_pool.cc:224] Handling call
0808 01:52:07.578842 (+1322065us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b540021d754. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:07.578980 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2697991552}
I20260808 01:52:07.580534 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:07.582175  9638 raft_consensus.cc:1217] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Deduplicated request from leader. Original: 11.7->[12.8-14.9]   Dedup: 12.8->[14.9-14.9]
I20260808 01:52:07.582777  9638 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: [12.8-14.9]
W20260808 01:52:07.583434  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 244e080726394210b5467bf688813cf0 (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:07.583732 15710 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.584954 15710 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:07.586846  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:07.524525  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 3001) took 1328 ms. Trace:
I20260808 01:52:07.587608  9551 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:52:07.563485 15513 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:07.513567 15624 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:07.568317 15844 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:07.478735  9552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.009s	sys 0.027s
I20260808 01:52:07.596956  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:07.587548  9472 rpcz_store.cc:276] 0808 01:52:06.196286 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:06.375171 (+178885us) service_pool.cc:224] Handling call
0808 01:52:07.524163 (+1148992us) spinlock_profiling.cc:243] Waited 955 ms on lock 0x7b5400093dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:07.524422 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2196950912}
I20260808 01:52:07.528585  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2500) took 1351 ms. Trace:
I20260808 01:52:07.588987  9637 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
W20260808 01:52:07.522092  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:07.598383  9555 rpcz_store.cc:276] 0808 01:52:06.177687 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:06.622805 (+445118us) service_pool.cc:224] Handling call
0808 01:52:07.528367 (+905562us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b540015bb54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:07.528513 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":308883712}
I20260808 01:52:07.594240  9630 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:07.594940  9631 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:07.589258  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:07.595492  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:07.599895  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:07.589630  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:07.599002  9637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:07.600421  9551 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:52:07.600848  9637 raft_consensus.cc:606] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:07.601094  9558 raft_consensus.cc:2410] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
W20260808 01:52:07.599146  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:07.601306  9637 raft_consensus.cc:614] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:07.605350  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:07.611871  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 872) took 1350 ms. Trace:
I20260808 01:52:07.607606  9475 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:07.605683  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:07.613758  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:07.612376  9556 rpcz_store.cc:276] 0808 01:52:06.261301 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:07.446884 (+1185583us) service_pool.cc:224] Handling call
0808 01:52:07.469029 (+ 22145us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:07.469310 (+   281us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:07.469349 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:07.469410 (+    61us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:07.469455 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:07.469525 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:07.611690 (+142165us) spinlock_profiling.cc:243] Waited 21.8 ms on lock 0x7b540002cbd0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:07.611808 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50185472}
W20260808 01:52:07.601799  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:07.561976 15845 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:07.613796  9481 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:52:07.618777 15729 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:07.619280 15600 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:52:07.616398  9551 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:07.623667  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:07.625087  9473 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:52:07.617326  9556 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
W20260808 01:52:07.626472  9478 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.016s	sys 0.019s
W20260808 01:52:07.626972  9478 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.016s	sys 0.019s
W20260808 01:52:07.623208  9629 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.015s	sys 0.065s
W20260808 01:52:07.627530  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.782s	user 0.004s	sys 0.041s
W20260808 01:52:07.627941  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.782s	user 0.004s	sys 0.041s
W20260808 01:52:07.628593  9636 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.021s	sys 0.021s
W20260808 01:52:07.629107  9636 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.022s	sys 0.022s
I20260808 01:52:07.612937  9630 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 17. Current term is 18. Ops: []
W20260808 01:52:07.627579  9629 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.015s	sys 0.065s
I20260808 01:52:07.626487  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 876) took 1363 ms. Trace:
I20260808 01:52:07.631997  9551 rpcz_store.cc:276] 0808 01:52:06.262583 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:07.615294 (+1352711us) service_pool.cc:224] Handling call
0808 01:52:07.616214 (+   920us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:07.626238 (+ 10024us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:07.626429 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:07.596758 15412 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:07.601037 15513 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:07.621845  9599 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:07.638391  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:07.644733  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 8c49fd321f8b4f89868246239530e78b (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
W20260808 01:52:07.644615  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 383254us
I20260808 01:52:07.678000  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:07.667930  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:07.644238  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:07.686189  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:07.628530 15796 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.018s	sys 0.026s
I20260808 01:52:07.696585  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:07.697934 14784 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:07.698510 14784 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:07.699051 14784 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:07.698354  9599 leader_election.cc:400] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13.
I20260808 01:52:07.700687  9599 leader_election.cc:403] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:52:07.636986 15778 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:07.680655  9635 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260808 01:52:07.671118  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2652) took 1493 ms. Trace:
I20260808 01:52:07.618849 15340 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.062s
I20260808 01:52:07.705560  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:07.606817 15624 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.709571 15761 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260808 01:52:07.711287  9601 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:07.711395  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:07.712811  9599 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:07.713423 15479 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 13
I20260808 01:52:07.675170 15832 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election won for term 15
W20260808 01:52:07.696105 15796 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.018s	sys 0.026s
I20260808 01:52:07.700098 15833 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:52:07.721304  9599 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:52:07.726984  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 49154.3us
I20260808 01:52:07.680598 15733 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:07.636600 15729 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.704576 15837 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260808 01:52:07.704352  9634 rpcz_store.cc:276] 0808 01:52:06.177684 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:06.177813 (+   129us) service_pool.cc:224] Handling call
0808 01:52:06.178365 (+   552us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:06.202032 (+ 23667us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:06.202088 (+    56us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:06.205832 (+  3744us) log.cc:844] Serialized 52 byte log entry
0808 01:52:06.450442 (+244610us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:07.258079 (+807637us) log.cc:844] Serialized 14 byte log entry
0808 01:52:07.274139 (+ 16060us) log.cc:844] Serialized 14 byte log entry
0808 01:52:07.670704 (+396565us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:07.670793 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:07.670837 (+    44us) raft_consensus.cc:1685] finished
0808 01:52:07.670849 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:07.671037 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24858,"prepare.queue_time_us":47833,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":81,"raft.queue_time_us":207479,"raft.run_cpu_time_us":509,"raft.run_wall_time_us":27188,"thread_start_us":1314931,"threads_started":3,"wal-append.queue_time_us":400788}
I20260808 01:52:07.725195 10334 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:07.669106 15412 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:07.715449 15761 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:07.734299 15833 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:07.735507 15733 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:07.740741 15811 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260808 01:52:07.702478  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 470327us
I20260808 01:52:07.790660 15605 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:07.791960 15833 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:07.796517 15412 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:07.796881  9637 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260808 01:52:07.719353 15848 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.078s
I20260808 01:52:07.797330 15636 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:07.798269 15877 raft_consensus.cc:3055] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:07.798782 15877 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:07.753571 15859 consensus_queue.cc:1059] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.059s
I20260808 01:52:07.772251 15832 raft_consensus.cc:697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:07.724252 15836 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:07.711922 15778 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.766134 15619 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:52:07.781870 15837 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:07.753626 15600 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:07.799160  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 983) took 1382 ms. Trace:
I20260808 01:52:07.819482  9474 rpcz_store.cc:276] 0808 01:52:06.416842 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:06.686425 (+269583us) service_pool.cc:224] Handling call
0808 01:52:06.690206 (+  3781us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:07.209741 (+519535us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:07.798899 (+589158us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b540021e154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:07.799094 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1172648192}
I20260808 01:52:07.827998 15729 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:07.800499 15877 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:07.815510  9557 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3258) took 1853 ms (client timeout 1623 ms). Trace:
W20260808 01:52:07.844717  9557 rpcz_store.cc:269] 0808 01:52:05.961779 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:05.991298 (+ 29519us) service_pool.cc:224] Handling call
0808 01:52:07.815226 (+1823928us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54000b4ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:07.815407 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3105278720}
I20260808 01:52:07.810518  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 3002) took 1614 ms. Trace:
I20260808 01:52:07.846462  9479 rpcz_store.cc:276] 0808 01:52:06.196592 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:06.405184 (+208592us) service_pool.cc:224] Handling call
0808 01:52:07.810263 (+1405079us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540021e654. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:07.810432 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2600705280}
I20260808 01:52:07.849807 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:07.851135 10367 raft_consensus.cc:2410] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260808 01:52:07.852551  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2491) took 1681 ms. Trace:
I20260808 01:52:07.743675  9630 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260808 01:52:07.751569 10334 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:07.852985  9552 rpcz_store.cc:276] 0808 01:52:06.171170 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:06.682140 (+510970us) service_pool.cc:224] Handling call
0808 01:52:06.682647 (+   507us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:06.763014 (+ 80367us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:06.763074 (+    60us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:06.763864 (+   790us) log.cc:844] Serialized 52 byte log entry
0808 01:52:07.597488 (+833624us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:52:07.597548 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:07.597659 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:07.852308 (+254649us) raft_consensus.cc:1685] finished
0808 01:52:07.852341 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:07.852488 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6915,"raft.queue_time_us":89702,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":61903,"thread_start_us":668989,"threads_started":1,"wal-append.queue_time_us":842164}
I20260808 01:52:07.752957  9555 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
W20260808 01:52:07.831619 15868 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260808 01:52:07.800608  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2883) took 1894 ms. Trace:
W20260808 01:52:07.859388 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 107911us
I20260808 01:52:07.859555  9477 rpcz_store.cc:276] 0808 01:52:05.905762 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:05.905878 (+   116us) service_pool.cc:224] Handling call
0808 01:52:05.928034 (+ 22156us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:07.383588 (+1455554us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:07.800050 (+416462us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540019c8d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:07.800523 (+   473us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2581697920}
I20260808 01:52:07.729565  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2495) took 1549 ms. Trace:
I20260808 01:52:07.833161 15811 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:07.832814 15867 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260808 01:52:07.834959 15624 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:07.772535 15513 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.838125  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2498) took 1662 ms. Trace:
I20260808 01:52:07.864447  9560 rpcz_store.cc:276] 0808 01:52:06.176090 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:07.088259 (+912169us) service_pool.cc:224] Handling call
0808 01:52:07.088711 (+   452us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:07.498562 (+409851us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:07.837836 (+339274us) spinlock_profiling.cc:243] Waited 338 ms on lock 0x7b540008acd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:07.838053 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":777405696}
I20260808 01:52:07.802162 15709 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:07.839418  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:07.812501 15717 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:07.811004 15600 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:07.805282 15836 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:07.806103 10366 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:07.809264 15619 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:07.744822  9635 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cacb9622c47c4737beff104125e98b77"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:07.872521  9635 raft_consensus.cc:606] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:07.873207  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:07.844296 14784 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:07.858228 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:07.839295  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:07.833451 15733 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:07.861053  9550 rpcz_store.cc:276] 0808 01:52:06.173121 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:06.807853 (+634732us) service_pool.cc:224] Handling call
0808 01:52:06.808263 (+   410us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:06.808545 (+   282us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:52:07.361404 (+552859us) log.cc:844] Serialized 14 byte log entry
0808 01:52:07.722203 (+360799us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:07.722291 (+    88us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:07.722362 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:07.722469 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:07.722650 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12552,"prepare.queue_time_us":267372,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":1170,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":11329,"thread_start_us":825486,"threads_started":2}
I20260808 01:52:07.840922 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:07.868738  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:07.809860 15837 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:07.879926 15837 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:07.881606  9481 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 18: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:07.800908 15859 raft_consensus.cc:3055] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:07.808755 15778 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:07.816243  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:07.884464  9477 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
W20260808 01:52:07.875142  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:07.876022 10367 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 15: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:07.817057 15851 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:07.877420  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2715) took 1075 ms. Trace:
I20260808 01:52:07.908268  9632 rpcz_store.cc:276] 0808 01:52:06.801659 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:07.567672 (+766013us) service_pool.cc:224] Handling call
0808 01:52:07.567974 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:07.568313 (+   339us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:52:07.568685 (+   372us) log.cc:844] Serialized 14 byte log entry
0808 01:52:07.877018 (+308333us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:07.877090 (+    72us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:52:07.877141 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:07.877214 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:07.877355 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8330,"prepare.queue_time_us":6342,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":13245,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":122,"thread_start_us":299529,"threads_started":1}
I20260808 01:52:07.862044 15811 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:07.866334 15835 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:07.870589 15516 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:07.915014 14784 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:07.886148  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2501) took 1070 ms. Trace:
I20260808 01:52:07.935039  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:07.936534  9555 rpcz_store.cc:276] 0808 01:52:06.816156 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:07.602252 (+786096us) service_pool.cc:224] Handling call
0808 01:52:07.602613 (+   361us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:07.855011 (+252398us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:07.885865 (+ 30854us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b540002d0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:07.886084 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":344610048}
I20260808 01:52:07.938108  9600 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:07.749930  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:07.885076  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:07.941635  9474 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:07.877744 15600 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:07.802291 15832 consensus_queue.cc:237] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.913641 15761 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:07.923632 15720 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:08.003260  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:07.876503 15709 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:07.881829 15759 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.050s
I20260808 01:52:07.930704 15847 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:07.932109  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:07.884526 15859 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:07.869225  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2492) took 1697 ms. Trace:
W20260808 01:52:07.911597 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 403616us
I20260808 01:52:07.943194  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:07.944584  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:08.021343  9556 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:08.021915  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2729) took 1156 ms. Trace:
I20260808 01:52:07.945358  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:07.946388  9600 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:07.930281 15811 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:07.947288  9481 raft_consensus.cc:2410] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 17 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
W20260808 01:52:07.949998  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 200166us
I20260808 01:52:07.947224 15849 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:07.996356  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:07.997781  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:08.040246  9477 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 18: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:08.002084  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:08.002524  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:07.925819 15833 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:08.041716  9551 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260808 01:52:08.042380  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:08.017215  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:08.044361  9472 raft_consensus.cc:2410] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 10 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260808 01:52:07.913813 15835 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:08.040925 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:08.024552  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 78448.1us
I20260808 01:52:08.048269  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:07.878435 15619 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:08.025755  9475 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:08.018970  9554 rpcz_store.cc:276] 0808 01:52:06.171622 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:06.802787 (+631165us) service_pool.cc:224] Handling call
0808 01:52:06.845034 (+ 42247us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:06.845271 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 14
0808 01:52:07.402658 (+557387us) log.cc:844] Serialized 14 byte log entry
0808 01:52:07.868714 (+466056us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:07.868795 (+    81us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:52:07.868861 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:07.868961 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:07.869141 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55969,"prepare.queue_time_us":233606,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":30,"raft.queue_time_us":1105,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":7599,"thread_start_us":935134,"threads_started":2}
I20260808 01:52:07.881620 15717 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:08.030949  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:07.951844  9474 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260808 01:52:07.955812 15516 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:08.032567  9550 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 17: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:08.057806  9555 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:07.954979 15737 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260808 01:52:08.038551  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:08.059338  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:08.060453  9472 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 13: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:08.065187  9481 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 16: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:08.066315  9479 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:08.063678 15858 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:08.069836  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:08.070804 15858 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:08.022497  9638 rpcz_store.cc:276] 0808 01:52:06.865935 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:07.585117 (+719182us) service_pool.cc:224] Handling call
0808 01:52:07.585452 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:07.585686 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 10
0808 01:52:07.615994 (+ 30308us) log.cc:844] Serialized 14 byte log entry
0808 01:52:08.020765 (+404771us) log.cc:844] Serialized 14 byte log entry
0808 01:52:08.021224 (+   459us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:08.021319 (+    95us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:08.021442 (+   123us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:08.021647 (+   205us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:08.021836 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49845,"prepare.queue_time_us":30870,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":12847,"raft.queue_time_us":1048,"raft.run_wall_time_us":16223,"spinlock_wait_cycles":9197696,"thread_start_us":383121,"threads_started":1}
I20260808 01:52:08.071831 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:08.071882  9632 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:08.072685  9632 raft_consensus.cc:606] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:08.073062  9632 raft_consensus.cc:614] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:08.075986  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:08.076877  9480 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:52:08.073431 15858 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:08.079334  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:08.080952  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet d0db7b84c3b040c788e0a3f66657d912 (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:08.081614  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260808 01:52:08.082521 10334 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:08.087466  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:08.087919  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:08.088474  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:08.089280  9473 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:08.089334  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:08.090340  9635 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:52:08.090921 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 50102.2us
I20260808 01:52:08.092555 15831 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:52:08.093047 15831 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:08.093465 15831 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:52:08.094883  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:08.095005 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:08.098031  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:08.097577  9377 catalog_manager.cc:5697] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 11 to 15. New cstate: current_term: 15 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:08.098290  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:08.098779  9638 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:08.099329  9632 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:08.099774  9377 catalog_manager.cc:5697] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 9 to 11, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 11 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:08.102928  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:52:08.104332 15597 raft_consensus.cc:3055] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:52:08.104795 15597 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:08.105288  9635 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:52:08.105738 15597 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:08.170156 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:08.174548  9523 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:08.107182  9638 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:08.116376  9634 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 11. Current term is 12. Ops: [10.7-11.8]
I20260808 01:52:08.092849  9472 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260808 01:52:08.101802  9377 catalog_manager.cc:5697] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 12 to 15. New cstate: current_term: 15 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:08.134789 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:08.183368 10363 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:08.117014  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:08.184131  9632 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 16
I20260808 01:52:08.129274 15778 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:08.111783 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:08.141551  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:08.163389  9554 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:08.055678 15835 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:08.014883 15761 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:08.120064 15303 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:08.039467 15848 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:52:08.134189 15513 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:08.114519  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:08.165675 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 83279.3us
I20260808 01:52:08.213449  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:08.215626  9637 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:08.160043  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:08.128890 15851 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:08.129164 15304 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:08.106463  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:08.226925  9630 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:08.166334  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:08.228536  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62334.3us
I20260808 01:52:08.192528  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:08.229353  9479 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:08.189142  9477 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:08.197997  9472 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:08.188212 15737 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:08.123696 15720 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:08.200486  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:08.238284  9634 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:08.200656 10364 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260808 01:52:08.060927 15729 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:08.194651 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:08.239987  9636 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.006s	sys 0.021s
W20260808 01:52:08.240439  9636 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.535s	user 0.007s	sys 0.021s
I20260808 01:52:08.241099  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:08.242441  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:08.250937  9479 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:08.252749  9524 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:08.207721  9556 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:08.193842 15796 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:08.201329 15858 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:52:08.064791 15875 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260808 01:52:08.219568  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 45143.5us
I20260808 01:52:08.131157 15833 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:08.190181 15479 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13.
I20260808 01:52:08.234282  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:08.235515 15597 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:52:08.235127 15710 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:52:08.267927  9632 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:08.263320  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2709) took 1489 ms. Trace:
W20260808 01:52:08.268425  9629 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.009s	sys 0.021s
I20260808 01:52:08.235375 15873 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260808 01:52:08.256060 15139 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.012s	sys 0.019s
I20260808 01:52:08.270885 15667 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:08.235713  9631 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.010s	sys 0.017s
I20260808 01:52:08.275573 15845 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:08.290465 15858 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:08.263927 15788 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.504s	user 0.012s	sys 0.020s
I20260808 01:52:08.202178 15778 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:08.308544  9629 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.009s	sys 0.021s
I20260808 01:52:08.354943  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2654) took 2173 ms. Trace:
I20260808 01:52:08.309314  9523 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:08.321483 15139 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.013s	sys 0.019s
I20260808 01:52:08.301303  9636 rpcz_store.cc:276] 0808 01:52:06.774218 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:06.774367 (+   149us) service_pool.cc:224] Handling call
0808 01:52:06.867470 (+ 93103us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:06.976727 (+109257us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:52:06.992483 (+ 15756us) log.cc:844] Serialized 14 byte log entry
0808 01:52:07.629646 (+637163us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:07.638622 (+  8976us) log.cc:844] Serialized 26 byte log entry
0808 01:52:07.638806 (+   184us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:07.836460 (+197654us) spinlock_profiling.cc:243] Waited 42.4 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:08.240899 (+404439us) log.cc:844] Serialized 14 byte log entry
0808 01:52:08.246460 (+  5561us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:08.246579 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:08.262874 (+ 16295us) raft_consensus.cc:1685] finished
0808 01:52:08.262916 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:08.263217 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":128068,"prepare.queue_time_us":134653,"prepare.run_cpu_time_us":130,"prepare.run_wall_time_us":11831,"raft.queue_time_us":64217,"raft.run_cpu_time_us":498,"raft.run_wall_time_us":30169,"spinlock_wait_cycles":97446784,"thread_start_us":1104376,"threads_started":2,"wal-append.queue_time_us":686502}
I20260808 01:52:08.302757  9524 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:08.364727  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 69434.5us
W20260808 01:52:08.322543  9631 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.011s	sys 0.017s
I20260808 01:52:08.293527  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:08.323129  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 485468us
I20260808 01:52:08.315860 15873 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:08.235957 15304 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:08.307860 15624 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:08.249774 15737 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:08.256274 15721 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.065s
W20260808 01:52:08.358029  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 48835.9us
W20260808 01:52:08.293421  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59251.2us
I20260808 01:52:08.211153 15859 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:08.364220  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2726) took 1499 ms. Trace:
W20260808 01:52:08.323918 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:08.371044 15710 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:08.365917  9524 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:08.373091  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:08.375631 15761 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:08.414960  9633 rpcz_store.cc:276] 0808 01:52:06.865064 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:07.582136 (+717072us) service_pool.cc:224] Handling call
0808 01:52:07.582481 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:07.584715 (+  2234us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:07.584749 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:07.585737 (+   988us) log.cc:844] Serialized 26 byte log entry
0808 01:52:07.872345 (+286608us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:52:07.872410 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:07.872511 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:08.363919 (+491408us) raft_consensus.cc:1685] finished
0808 01:52:08.363947 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:08.364148 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1644,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"thread_start_us":269802,"threads_started":1,"wal-append.queue_time_us":632451}
I20260808 01:52:08.377151 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:08.425980  9599 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.030s	sys 0.071s
W20260808 01:52:08.426271  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:08.426451  9599 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.640s	user 0.030s	sys 0.071s
W20260808 01:52:08.379388 15878 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.244s	user 0.000s	sys 0.001s
W20260808 01:52:08.380219 15874 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
W20260808 01:52:08.381171 15885 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260808 01:52:08.288934 15796 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:08.395454  9635 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:52:08.355625  9629 rpcz_store.cc:276] 0808 01:52:06.181318 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:06.181496 (+   178us) service_pool.cc:224] Handling call
0808 01:52:06.219052 (+ 37556us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:06.929371 (+710319us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:06.929405 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:06.988012 (+ 58607us) log.cc:844] Serialized 26 byte log entry
0808 01:52:07.631126 (+643114us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:08.353970 (+722844us) log.cc:844] Serialized 14 byte log entry
0808 01:52:08.354319 (+   349us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:08.354492 (+   173us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:08.354571 (+    79us) raft_consensus.cc:1685] finished
0808 01:52:08.354594 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:08.354848 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25381,"prepare.queue_time_us":100035,"prepare.run_cpu_time_us":2326,"prepare.run_wall_time_us":8880,"raft.queue_time_us":197359,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":18694,"thread_start_us":1996542,"threads_started":3,"wal-append.queue_time_us":915067}
I20260808 01:52:08.340960 15858 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:52:08.334599 15845 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:08.397424 15875 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:08.306368 15862 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.089s
I20260808 01:52:08.354924 15660 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:08.364040 15860 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260808 01:52:08.421146  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 55306.1us
I20260808 01:52:08.469244  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:08.405030  9523 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:08.470055  9479 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:08.321622 15733 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:08.409368  9478 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.725s	user 0.012s	sys 0.037s
W20260808 01:52:08.476294  9478 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.792s	user 0.013s	sys 0.037s
I20260808 01:52:08.405556  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:08.412629 15624 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 18 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:08.435839 15811 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:08.482812  9377 catalog_manager.cc:5697] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 8 to 16. New cstate: current_term: 16 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:08.377113 15837 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:08.431025  9601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.012s	sys 0.020s
W20260808 01:52:08.487636  9601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.753s	user 0.012s	sys 0.020s
W20260808 01:52:08.488101  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 773101us
I20260808 01:52:08.400607 15831 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:08.489174  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:08.337913 15667 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:08.464282 15845 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:08.450877 15139 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election won for term 16
W20260808 01:52:08.427243  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:08.477665  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:08.506498  9475 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:08.114763  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
W20260808 01:52:08.478367  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:08.480895  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 75432.2us
I20260808 01:52:08.417390 15710 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:08.451185 15720 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:52:08.427630  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 641808us
I20260808 01:52:08.477223 15660 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:08.491812  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:08.455801 15709 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:08.400161 15873 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:08.496259 10362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 774) took 2045 ms. Trace:
I20260808 01:52:08.526069 10362 rpcz_store.cc:276] 0808 01:52:06.450568 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:06.450742 (+   174us) service_pool.cc:224] Handling call
0808 01:52:06.451128 (+   386us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:07.255306 (+804178us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:07.255357 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:07.256027 (+   670us) log.cc:844] Serialized 26 byte log entry
0808 01:52:07.714076 (+458049us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:07.714123 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:07.714198 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:08.495850 (+781652us) raft_consensus.cc:1685] finished
0808 01:52:08.495891 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:08.496161 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":269657,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":117,"thread_start_us":894672,"threads_started":2,"wal-append.queue_time_us":799786}
I20260808 01:52:08.496857 15788 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:08.513074 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:52:08.516997  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:08.397169 15668 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:08.518765 10361 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 13: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:08.519305  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:08.519444  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:08.521522 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:08.528904  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:08.502933  9524 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:08.503515 15479 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:08.503759 15860 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:52:08.533015  9601 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:08.504844 15636 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:08.536087 10364 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 19: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:08.472887 15875 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:08.520195 15139 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:08.543190  9555 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:08.519330 15710 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:08.555401  9524 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:08.556041 15882 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.203s	user 0.001s	sys 0.000s
I20260808 01:52:08.516075 15811 consensus_queue.cc:1059] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.042s
W20260808 01:52:08.542523  9552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.034s	sys 0.038s
W20260808 01:52:08.575616 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 483520us
W20260808 01:52:08.580440  9552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.034s	sys 0.038s
I20260808 01:52:08.577191 14784 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:08.560195  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:08.583416 14784 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260808 01:52:08.586812  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:08.554772 15479 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:08.557617 15844 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:08.557160 15709 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:08.523062 15720 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:08.572547 15721 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:08.557320 15876 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260808 01:52:08.633482 15876 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:08.607156 15479 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:52:08.636020 15893 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260808 01:52:08.637217 15737 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:08.637925 15737 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20260808 01:52:08.639012 15890 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
I20260808 01:52:08.641538 15875 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:08.628916 15709 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:08.643412 15831 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:08.645414 15709 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:08.645926 15831 consensus_queue.cc:1059] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:08.649870  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2894) took 2712 ms. Trace:
I20260808 01:52:08.650395  9478 rpcz_store.cc:276] 0808 01:52:05.937697 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:06.151801 (+214104us) service_pool.cc:224] Handling call
0808 01:52:06.165711 (+ 13910us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:06.167504 (+  1793us) raft_consensus.cc:1534] Early marking committed up to index 5
0808 01:52:07.059394 (+891890us) spinlock_profiling.cc:243] Waited 2.85 ms on lock 0x7b180047ffe0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d483aada7 00007f5d4499dd87 00007f5d449ad49f 00007f5d449ad418
0808 01:52:07.627433 (+568039us) log.cc:844] Serialized 14 byte log entry
0808 01:52:08.480011 (+852578us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:08.484722 (+  4711us) log.cc:844] Serialized 26 byte log entry
0808 01:52:08.484949 (+   227us) raft_consensus.cc:1645] Marking committed up to 5
0808 01:52:08.484999 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:08.485124 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:08.649515 (+164391us) raft_consensus.cc:1685] finished
0808 01:52:08.649643 (+   128us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:08.649802 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23617,"prepare.queue_time_us":852323,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"raft.queue_time_us":4398,"raft.run_cpu_time_us":356,"raft.run_wall_time_us":53097,"spinlock_wait_cycles":6561536,"thread_start_us":1360385,"threads_started":2,"wal-append.queue_time_us":951573}
I20260808 01:52:08.606052 15833 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:08.591660 15761 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:08.579206 15624 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:08.605603 15844 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:08.552788 15668 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:08.658306  9555 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:52:08.670627  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:08.735833  9478 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 13: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:52:08.680585 15600 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.678s	user 0.010s	sys 0.002s
I20260808 01:52:08.683667 15796 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:08.690590 15837 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:08.552675 15788 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
W20260808 01:52:08.654341 15887 log.cc:927] Time spent T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.288s	user 0.001s	sys 0.000s
I20260808 01:52:08.715150  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:08.761675  9556 raft_consensus.cc:2368] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16.
I20260808 01:52:08.634362 15876 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.073s
I20260808 01:52:08.724622 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:08.574944 15849 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.082s
I20260808 01:52:08.626998 15347 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:08.738070  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:08.673739 15778 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:08.741843  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:08.743835 15844 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:08.808869 14784 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 01:52:08.769232 15726 raft_consensus.cc:3055] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:08.717410  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:08.817502  9550 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:08.673049 15721 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:08.768882 15796 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:08.784127 15876 consensus_queue.cc:1243] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 13.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260808 01:52:08.787253 15347 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:08.723364 15833 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:08.830870  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:08.833184  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 428210us
I20260808 01:52:08.760998 15600 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:08.734143 15479 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:52:08.845477  9523 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:08.817173 15849 raft_consensus.cc:3055] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:08.733714 15761 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:08.828850 15347 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:08.839231  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2656) took 1798 ms. Trace:
I20260808 01:52:08.871690  9631 rpcz_store.cc:276] 0808 01:52:07.040411 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:07.622942 (+582531us) service_pool.cc:224] Handling call
0808 01:52:07.651174 (+ 28232us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:07.651487 (+   313us) raft_consensus.cc:1534] Early marking committed up to index 13
0808 01:52:08.366619 (+715132us) log.cc:844] Serialized 14 byte log entry
0808 01:52:08.828213 (+461594us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:08.828306 (+    93us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:08.828382 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:08.838977 (+ 10595us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:08.839146 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56426,"prepare.queue_time_us":106490,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"thread_start_us":1107761,"threads_started":2}
I20260808 01:52:08.806388 15733 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:08.847088  9523 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:08.736773 15139 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:08.849623 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:08.817087 15726 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:08.825896 15778 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:08.875234 15709 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260808 01:52:08.880779 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:08.898293  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:08.836424  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 324800us
I20260808 01:52:08.911123  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:08.756880  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:08.840359 15729 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:08.920014  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:08.925228  9474 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 18: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:08.851097 15849 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:08.928802  9550 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:52:08.865236 15896 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
W20260808 01:52:08.943290 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 362483us
I20260808 01:52:08.834447 15796 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:08.947000  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:08.846459 15858 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:08.948354 15636 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:08.944913 15709 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260808 01:52:08.980003  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 238332us
I20260808 01:52:08.922016 15778 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:08.951186 15896 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:08.983644 10367 raft_consensus.cc:2393] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 10: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:09.030530 15717 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.031922 15717 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:09.032536 15717 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.044535  9377 catalog_manager.cc:5697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 12 to 16. New cstate: current_term: 16 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:09.051318  9377 catalog_manager.cc:5697] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 10 to 12, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 12 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:52:09.007352 15668 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:09.077661  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 462117us
W20260808 01:52:09.068980  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:09.055440 15516 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:08.977859 15726 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:09.057608 15788 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.064950 15720 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.118469  9601 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:09.131707 15600 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:09.114233 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 167516us
I20260808 01:52:08.999315 15849 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.148451  9599 leader_election.cc:400] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16.
I20260808 01:52:09.149319  9599 leader_election.cc:403] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
W20260808 01:52:09.165225 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:09.137995 15709 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:09.155911 15897 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.264s	user 0.000s	sys 0.001s
I20260808 01:52:09.158625 15479 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:09.173604  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:09.147115 15923 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:09.148630 15721 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.174391  9550 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:52:09.184429  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:09.178141  9556 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:09.180439  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:09.185811  9558 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 13: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:09.186434  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:09.186808  9599 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:09.188408 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:09.189677  9556 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:09.166150 15726 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:09.068980 15636 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.186914  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:09.188153  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:09.168044 15717 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.205081  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:09.201650  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:09.203171 10360 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260808 01:52:09.203389  9555 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 9.
I20260808 01:52:09.208586  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:09.209240 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 58758us
I20260808 01:52:09.192711  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:09.232271  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45621.5us
I20260808 01:52:09.208586  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:09.225139  9556 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:09.209286  9555 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:09.211972 10332 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:09.238987  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:09.240052  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2502) took 1297 ms. Trace:
I20260808 01:52:09.172658 15849 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:09.240492  9552 rpcz_store.cc:276] 0808 01:52:07.942455 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:07.942582 (+   127us) service_pool.cc:224] Handling call
0808 01:52:07.942943 (+   361us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:07.947667 (+  4724us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:07.947700 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:07.948477 (+   777us) log.cc:844] Serialized 26 byte log entry
0808 01:52:08.581077 (+632600us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:08.581142 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:08.581223 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:09.239773 (+658550us) raft_consensus.cc:1685] finished
0808 01:52:09.239797 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:09.239985 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4267,"raft.queue_time_us":11813,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":6007,"thread_start_us":584979,"threads_started":1,"wal-append.queue_time_us":872659}
I20260808 01:52:09.175225 15709 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:09.190867 15619 consensus_meta.cc:287] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.056s	user 0.011s	sys 0.006s
I20260808 01:52:09.247036  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:09.195885 15600 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:09.152948 15896 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:09.255944 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:09.256512  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:09.257373  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:09.244004 15904 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.262s	user 0.001s	sys 0.001s
I20260808 01:52:09.254935  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:09.258785  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:09.196004 15796 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.263609  9556 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:09.265901  9555 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:52:09.265782  9600 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:09.267913  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:09.236598 15932 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:09.237488 15858 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:09.246898 15479 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:09.283140  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:09.200718 15919 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.060s
I20260808 01:52:09.258026 15619 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:09.286638  9557 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 13: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:09.292776 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 981) took 1123 ms. Trace:
I20260808 01:52:09.293439 15849 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election won for term 15
W20260808 01:52:09.299194 15886 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.923s	user 0.003s	sys 0.003s
W20260808 01:52:09.302263 15881 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.038s	user 0.002s	sys 0.007s
W20260808 01:52:09.303519 15917 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260808 01:52:09.253589 15761 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:09.303704  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:09.279323 15778 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:09.306921 15911 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.249s	user 0.001s	sys 0.001s
W20260808 01:52:09.263442 15903 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.263s	user 0.002s	sys 0.000s
I20260808 01:52:09.189911 15923 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:09.275352 15902 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.281s	user 0.000s	sys 0.001s
I20260808 01:52:09.313217 10361 rpcz_store.cc:276] 0808 01:52:08.169565 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:08.169804 (+   239us) service_pool.cc:224] Handling call
0808 01:52:09.292545 (+1122741us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b5400620f54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:09.292711 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":795768064}
I20260808 01:52:09.334440 15849 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:09.354449 15849 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:09.205005 15912 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.075s
W20260808 01:52:09.302175 15873 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.755s	user 0.008s	sys 0.004s
W20260808 01:52:09.330488 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 118576us
I20260808 01:52:09.243477 15717 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.042s
I20260808 01:52:09.260393 15929 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16.
I20260808 01:52:09.304523  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2668) took 1192 ms. Trace:
I20260808 01:52:09.261068 15600 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.338090 10364 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12. Ops: []
W20260808 01:52:09.341593  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 223240us
W20260808 01:52:09.343109  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:09.320755 15856 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260808 01:52:09.334905 15907 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.265s	user 0.001s	sys 0.000s
I20260808 01:52:09.334447 15479 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260808 01:52:09.360478 15845 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.847s	user 0.010s	sys 0.005s
I20260808 01:52:09.366238  9635 rpcz_store.cc:276] 0808 01:52:08.112113 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:08.159377 (+ 47264us) service_pool.cc:224] Handling call
0808 01:52:09.304323 (+1144946us) spinlock_profiling.cc:243] Waited 97.3 ms on lock 0x7b54000dfa54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:09.304454 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":223767936}
I20260808 01:52:09.369029  9601 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:09.370993  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1000) took 1147 ms. Trace:
W20260808 01:52:09.364143 15920 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.167s	user 0.000s	sys 0.002s
I20260808 01:52:09.376669 15875 raft_consensus.cc:3055] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:09.378047 15600 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:09.387367  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:09.386541  9473 rpcz_store.cc:276] 0808 01:52:08.223866 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:08.224030 (+   164us) service_pool.cc:224] Handling call
0808 01:52:08.224535 (+   505us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:08.224978 (+   443us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:52:08.225006 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:08.242298 (+ 17292us) log.cc:844] Serialized 26 byte log entry
0808 01:52:08.774050 (+531752us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:08.774116 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:08.774233 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:09.370721 (+596488us) raft_consensus.cc:1685] finished
0808 01:52:09.370749 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:09.370915 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":544,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":8942,"thread_start_us":480711,"threads_started":1,"wal-append.queue_time_us":693197}
I20260808 01:52:09.311659 15932 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:09.385638 15845 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.393121  9551 raft_consensus.cc:2368] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18.
I20260808 01:52:09.394912 15932 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:09.399632 15835 consensus_meta.cc:287] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.188s	user 0.017s	sys 0.000s
W20260808 01:52:09.400334 15710 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.808s	user 0.014s	sys 0.000s
I20260808 01:52:09.400199 15835 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:09.401011 15710 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.389051  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:09.402522  9556 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 18: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:09.370369 15856 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:09.387123 15660 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:09.391935  9601 leader_election.cc:400] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16.
I20260808 01:52:09.401571 10362 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:09.413913  9601 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:09.359660 15139 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:09.415874  9554 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
W20260808 01:52:09.410372 15860 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.832s	user 0.007s	sys 0.011s
I20260808 01:52:09.419297  9601 leader_election.cc:400] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 12. Current term is 13.
I20260808 01:52:09.363235 15873 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.428422  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:09.388425 15875 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:09.433245  9521 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:09.378422 15347 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.411875 15856 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:09.408313 15848 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:52:09.437124  9632 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260808 01:52:09.439873  9633 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: []
W20260808 01:52:09.424489 15860 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.849s	user 0.008s	sys 0.011s
I20260808 01:52:09.463709  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:09.464118  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:09.464893  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:09.424448 15660 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:09.465122  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.463101  9636 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260808 01:52:09.468335  9479 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:09.469031  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.470434  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.490228  9633 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:09.491272  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:09.471078  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:09.471452  9638 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 13: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:09.471122  9630 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:09.471864  9559 raft_consensus.cc:2393] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:52:09.469077 15513 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.240s	user 0.012s	sys 0.006s
I20260808 01:52:09.435467 15919 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:09.470865 15860 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:09.440362 15906 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260808 01:52:09.479995 15844 consensus_meta.cc:287] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.663s	user 0.015s	sys 0.000s
I20260808 01:52:09.431098 15932 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:09.481016  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:09.510244  9629 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:09.438467 15930 raft_consensus.cc:2764] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260808 01:52:09.515518  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.516607  9629 raft_consensus.cc:2410] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260808 01:52:09.484483 15347 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.469827 15833 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:09.426556 15139 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.486157 15710 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.487401  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:09.522034  9631 raft_consensus.cc:2376] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16. Re-sending same reply.
I20260808 01:52:09.522655  9630 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:09.503193  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.523978  9638 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:09.441679 15835 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:09.503221  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.505096  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.507520  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:09.505920 15597 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.202s	user 0.014s	sys 0.003s
I20260808 01:52:09.488061  9634 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 16: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:09.525519  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.532321  9636 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:09.532490  9637 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:09.536768  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:09.537007 15619 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.507843 15717 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:09.538262  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:09.536773  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.539232  9630 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:09.541069  9637 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:09.542541  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.543975  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.543768  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.546368  9638 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 13: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:09.537932 15717 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.543387  9636 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:09.548695  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.548835  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.549685  9634 raft_consensus.cc:2410] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 16 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260808 01:52:09.550732  9636 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260808 01:52:09.551039  9634 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260808 01:52:09.518569 15709 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.558907 15860 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:09.563622  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.569679  9638 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:09.518566 15932 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.573731 15932 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:09.573882  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.519840 15844 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.574842  9634 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:52:09.576052  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:09.577286  9558 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:09.506862 15919 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.580623  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.581312 15919 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:09.582103  9632 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260808 01:52:09.586119  9638 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:52:09.572033 15856 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:09.583493  9633 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:52:09.589834  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:09.594043  9631 raft_consensus.cc:2410] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 16 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260808 01:52:09.592022  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:09.594919  9636 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260808 01:52:09.592087  9637 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:09.600107  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.600229  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.601254  9631 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:09.583679  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.602530  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:09.603294  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:09.603837  9631 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 13: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:09.603950  9630 raft_consensus.cc:2376] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15. Re-sending same reply.
I20260808 01:52:09.601186  9636 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:52:09.610836  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:09.611485  9638 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:09.618398  9634 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:52:09.620585 10364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 983) took 1060 ms. Trace:
I20260808 01:52:09.621120 10364 rpcz_store.cc:276] 0808 01:52:08.560700 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:08.562145 (+  1445us) service_pool.cc:224] Handling call
0808 01:52:08.562519 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:09.370184 (+807665us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:09.620355 (+250171us) spinlock_profiling.cc:243] Waited 773 ms on lock 0x7b5400594054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:09.620525 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1778402176}
I20260808 01:52:09.623610 15873 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.624646 15304 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.630903 15859 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:09.631417 15660 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:09.636698 15856 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:09.615073  9631 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 14. Current term is 15. Ops: []
W20260808 01:52:09.637754  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 706674us
I20260808 01:52:09.645594 15667 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.652827 15733 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:09.653434 15910 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260808 01:52:09.656989 15923 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:09.660741  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:09.664052  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:09.664901  9552 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:52:09.664449  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:09.666193  9558 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:52:09.667786  9599 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:09.653820 15733 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.622453 15757 raft_consensus.cc:3055] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:52:09.673017  9638 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:52:09.687932  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:09.679563 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:09.691784  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:09.694607 14784 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:09.695353 14784 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:09.695904 14784 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:09.687163 15757 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:09.714025 15757 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:09.715620 15757 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:09.716899  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2922) took 1143 ms. Trace:
I20260808 01:52:09.717351  9476 rpcz_store.cc:276] 0808 01:52:08.573814 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:08.574146 (+   332us) service_pool.cc:224] Handling call
0808 01:52:08.574661 (+   515us) raft_consensus.cc:1471] Updating replica for 3 ops
0808 01:52:08.939413 (+364752us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:08.939449 (+    36us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0808 01:52:08.988052 (+ 48603us) log.cc:844] Serialized 78 byte log entry
0808 01:52:09.174454 (+186402us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:52:09.198381 (+ 23927us) log.cc:844] Serialized 14 byte log entry
0808 01:52:09.716506 (+518125us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:09.716582 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:09.716626 (+    44us) raft_consensus.cc:1685] finished
0808 01:52:09.716640 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:09.716815 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":517739,"prepare.queue_time_us":25787,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":69,"raft.queue_time_us":164980,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":22604,"thread_start_us":514830,"threads_started":2,"wal-append.queue_time_us":309913}
I20260808 01:52:09.719767 14784 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.722622 14784 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:09.722805  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:09.723985  9478 raft_consensus.cc:2410] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260808 01:52:09.723776  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:09.724835  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:09.725698  9475 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:09.726231  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:09.727222  9476 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:09.727629  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:09.730350  9473 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:09.730516  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:09.731097  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:09.731853  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:09.733888  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:09.734553  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:09.735442  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:09.735735  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:09.736394  9473 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:09.737253 15733 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.694669 15832 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:09.738962 15832 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:09.739399 15832 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:09.739843  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:09.740746  9475 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 11: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:52:09.738344  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 792262us
I20260808 01:52:09.741622 15668 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:09.742237  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:09.741652 15667 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:09.743294 15667 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:09.743778 15667 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:09.743691  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:09.746227  9474 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:52:09.745328  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:09.747620  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:09.748805  9559 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:09.749783  9472 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:09.751504  9473 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260808 01:52:09.754757  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:09.759874 15837 raft_consensus.cc:3055] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:09.760296 15910 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:09.760666 15837 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:09.761086 15910 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:09.763252  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:09.764950  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:09.767374  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:09.775687  9481 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 8. (term mismatch)
I20260808 01:52:09.779178  9481 pending_rounds.cc:85] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 7
I20260808 01:52:09.780207  9481 pending_rounds.cc:107] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 15.8
I20260808 01:52:09.781193  9481 raft_consensus.cc:2889] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:09.768965  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:09.786180  9601 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:09.767235  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:09.787899 10366 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:09.790292 10364 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:09.791750 10361 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260808 01:52:09.792217  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:09.763726 15636 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:09.764257 15955 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
I20260808 01:52:09.795925  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:09.802179  9472 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:09.794269  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:09.800464  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:09.803391  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:09.796635 14784 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:09.790104 15896 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:09.807927  9524 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:09.813261  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:09.797935 15729 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:09.789691 15914 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:09.818573  9601 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:09.792378 15877 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:09.742266 15668 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.795485 15919 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:09.799727 15932 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:09.803941 15636 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.841176 15636 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.796715 15778 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:52:09.761727 15837 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.843125 15778 raft_consensus.cc:697] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:09.846311 15778 consensus_queue.cc:237] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.795166 15860 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:09.831151 15811 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:09.825588 15914 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:09.854285 15811 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:09.855005 15811 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:09.856290 15914 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:09.819701 15913 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:09.857492 15913 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:09.857890 15913 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:09.819780 15710 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:09.859537 15710 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:09.859928 15710 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:09.811676 15624 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:52:09.825090  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:09.865203  9475 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 16: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:09.865470 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:09.866624 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 501100us
I20260808 01:52:09.834550 15717 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.822988 15605 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:09.810314 15304 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.026s
I20260808 01:52:09.824720 15729 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:09.850121 14784 consensus_queue.cc:1059] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:09.881727  9523 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:09.851126  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:09.842931 15929 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:52:09.816133 15733 consensus_queue.cc:1059] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260808 01:52:09.886034 14784 raft_consensus.cc:3055] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:09.833232  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:09.868767 10332 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:09.798225  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:09.867726  9472 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:09.877782  9474 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
W20260808 01:52:09.884301 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:09.886597 14784 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:09.890581  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:09.890971 14784 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.863453 15624 raft_consensus.cc:697] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:09.892185  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:09.892570 14784 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:09.893440  9550 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 13. Current term is 14. Ops: []
I20260808 01:52:09.893579  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:09.895421  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:09.891340  9474 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:09.895762  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:09.897413  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:09.898206  9558 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
W20260808 01:52:09.888514 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:09.903141  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:09.905912  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 72792.9us
W20260808 01:52:09.916872  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 65868.3us
I20260808 01:52:09.911603  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:09.921255  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:09.891993 15709 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:52:09.917493  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:09.932663 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:09.933248 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:09.934108 10367 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:52:09.936719  9521 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:09.938473  9600 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:09.939950 10334 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:09.941205  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2924) took 1208 ms. Trace:
I20260808 01:52:09.949456  9474 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:52:09.946095  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2917) took 1478 ms. Trace:
I20260808 01:52:09.915535 15733 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:09.955411 10334 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:09.922312 15720 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260808 01:52:09.942287 10366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 702) took 1831 ms (client timeout 2047 ms). Trace:
I20260808 01:52:09.949584  9479 rpcz_store.cc:276] 0808 01:52:08.733229 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:08.737693 (+  4464us) service_pool.cc:224] Handling call
0808 01:52:09.941016 (+1203323us) spinlock_profiling.cc:243] Waited 686 ms on lock 0x7b540038ccd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:09.941139 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1578019712}
I20260808 01:52:09.922035 15896 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:52:09.950392  9477 rpcz_store.cc:276] 0808 01:52:08.467809 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:08.469569 (+  1760us) service_pool.cc:224] Handling call
0808 01:52:08.487376 (+ 17807us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:08.799586 (+312210us) raft_consensus.cc:1534] Early marking committed up to index 10
0808 01:52:08.800004 (+   418us) log.cc:844] Serialized 14 byte log entry
0808 01:52:09.715396 (+915392us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:09.715475 (+    79us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:09.715546 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:09.945843 (+230297us) spinlock_profiling.cc:243] Waited 312 ms on lock 0x7b54001013d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:09.945894 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:09.946030 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":615028,"prepare.queue_time_us":782,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":615593,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":173,"spinlock_wait_cycles":717143808,"thread_start_us":299771,"threads_started":1,"wal-append.queue_time_us":583530}
I20260808 01:52:09.953065  9475 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:09.963183  9521 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:09.891896 15726 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
W20260808 01:52:09.961926 10366 rpcz_store.cc:269] 0808 01:52:08.111215 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:08.111424 (+   209us) service_pool.cc:224] Handling call
0808 01:52:09.942086 (+1830662us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b540037dcd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:09.942218 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3559146240}
I20260808 01:52:09.918109  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:09.904085 15906 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.032s
I20260808 01:52:09.960918  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:09.966107  9445 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:09.935825 15910 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:09.943302 15304 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:09.975483 15906 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:09.976064 15906 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:09.977077 15906 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 14.14, Last appended by leader: 14, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:09.977725 15668 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.921949 15939 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:52:09.933893 15924 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:09.936609 15832 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.980095  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:09.978614 15906 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:09.970114 15929 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:52:09.988545 15139 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.988922 15717 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:52:09.984539 10366 raft_consensus.cc:3055] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:09.970630 15737 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:52:09.989054 15924 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:09.952970 15605 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:52:09.990312 15605 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:09.991055 15605 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:52:09.993124 15924 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.910804 15479 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.048s
I20260808 01:52:09.995325 15479 raft_consensus.cc:3055] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:52:09.962322 15862 consensus_queue.cc:1059] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.028s
I20260808 01:52:09.988686  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:09.996707  9557 raft_consensus.cc:2410] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 19 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260808 01:52:09.990067 15737 raft_consensus.cc:697] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:09.967676 15303 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:52:09.998070 15303 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:09.954070 15733 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:09.978792 15624 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.998982 15733 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:09.999295 15303 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 14.14, Last appended by leader: 14, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:09.995805 15479 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:10.002704 15479 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:10.003746 10360 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:52:10.004478 15479 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:10.004105  9377 catalog_manager.cc:5697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:10.006837  9445 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:10.007848 15636 raft_consensus.cc:3055] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:09.998083 15832 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:10.008437 15668 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:52:10.008980 15668 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:09.965536 15856 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader election won for term 10
I20260808 01:52:10.009400 15668 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:10.010001 15856 raft_consensus.cc:697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:52:09.968603  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50690us
I20260808 01:52:10.009007 15899 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:10.008459 15636 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:10.011263 15899 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:10.012137 15636 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:10.012378 15899 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:10.013412  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3277) took 1495 ms. Trace:
I20260808 01:52:10.014041 15139 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:09.975947 15304 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:10.014617 15844 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:10.008136 15717 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:10.015646 15717 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.015614 15304 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.997992 15737 consensus_queue.cc:237] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:09.965435 15928 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:10.018850  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:10.018493 15928 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:10.021135 15928 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:10.014707 15139 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.013963  9554 rpcz_store.cc:276] 0808 01:52:08.517916 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:08.518143 (+   227us) service_pool.cc:224] Handling call
0808 01:52:08.518648 (+   505us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:09.416363 (+897715us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:10.013130 (+596767us) spinlock_profiling.cc:243] Waited 897 ms on lock 0x7b54001dfe54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:10.013338 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2062566144}
I20260808 01:52:10.023950 15875 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:10.025671 15875 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:10.012503 15845 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:09.989650 10366 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:10.030146 10366 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:10.032781 15933 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:10.033048  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:10.033937  9478 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:10.035610  9599 leader_election.cc:400] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18.
I20260808 01:52:10.037235  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:10.036180 15933 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:10.038344 15933 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.039794 15947 raft_consensus.cc:1064] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:52:10.036100  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:10.040820  9475 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:09.986426  9632 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 19: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:10.042888  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:10.046178 15837 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:10.046805  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:10.048635  9554 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260808 01:52:10.049865  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:10.050719  9557 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 17
I20260808 01:52:10.037779  9599 leader_election.cc:403] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:52:10.052925 15935 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 18
I20260808 01:52:10.053782  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2912) took 1966 ms. Trace:
I20260808 01:52:10.054469  9480 rpcz_store.cc:276] 0808 01:52:08.087031 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:08.087589 (+   558us) service_pool.cc:224] Handling call
0808 01:52:08.087974 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:08.841574 (+753600us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:52:08.861357 (+ 19783us) log.cc:844] Serialized 14 byte log entry
0808 01:52:09.568719 (+707362us) spinlock_profiling.cc:243] Waited 31.8 ms on lock 0x7b18003a41a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d4453242a 00007f5d4499de0a
0808 01:52:09.733258 (+164539us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:09.733317 (+    59us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:52:09.733370 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:10.053463 (+320093us) spinlock_profiling.cc:243] Waited 431 ms on lock 0x7b5400399c54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:10.053523 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:10.053704 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":126858,"prepare.queue_time_us":19451,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":251989,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":213,"spinlock_wait_cycles":1065303296,"thread_start_us":358903,"threads_started":2,"wal-append.queue_time_us":798177}
I20260808 01:52:10.011178 15856 consensus_queue.cc:237] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:10.057247  9635 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
W20260808 01:52:10.058094  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 95038.8us
I20260808 01:52:10.062065  9559 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:52:10.013504 15636 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:10.072036 15710 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.061071  9551 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260808 01:52:10.077515  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:10.006197  9377 catalog_manager.cc:5697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 10 to 14. New cstate: current_term: 14 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:10.078188  9472 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:10.060359  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:10.062911 10366 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:10.064086  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:10.080128  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:10.081002  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:10.079315  9632 raft_consensus.cc:2410] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260808 01:52:10.084196  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:10.085758  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:10.086814  9551 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:10.087534  9558 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:52:10.088246  9472 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260808 01:52:09.955498  9474 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:10.064965  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:10.089376  9633 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:10.065814  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:10.066108 15660 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:10.066650 15729 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:10.068090  9629 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 16. Current term is 17. Ops: [16.8-16.8]
I20260808 01:52:10.069761 15717 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:10.071497 15139 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:10.065829  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47133.1us
I20260808 01:52:09.988698 15709 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:52:10.074473 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:10.075021 15849 consensus_queue.cc:1059] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 01:52:10.075953  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:10.076828  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:10.095254 15908 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:10.096406 15932 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:10.096593 15913 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.092689  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:10.063515  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:10.231928  9556 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260808 01:52:10.227958  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:10.197757  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:10.103312  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:10.192010  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:10.145682  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:10.232427  9559 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:10.235639  9635 raft_consensus.cc:2410] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260808 01:52:10.235251 10360 raft_consensus.cc:2410] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 16 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
W20260808 01:52:10.166769 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:10.102612  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:10.244766  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:10.251385  9638 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:10.211113  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:10.097318 10367 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260808 01:52:10.254015  9558 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:10.102403  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:10.253891  9475 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:10.239117  9557 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:10.264408  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:10.076972 15899 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:10.235765  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:10.207496  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:10.282032  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46366.4us
I20260808 01:52:10.261180  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:10.283469  9479 raft_consensus.cc:2393] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 15: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:10.271950  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:10.284348  9477 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 16: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:10.283488  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:10.261848 10366 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260808 01:52:10.256136  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:10.285386  9559 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:10.285598  9632 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:10.285768  9631 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
W20260808 01:52:10.272043  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 183571us
W20260808 01:52:10.234591  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 172140us
I20260808 01:52:10.266336  9524 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:10.287590  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:10.277801  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:10.276122 14784 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:10.294374 14784 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.187361 15933 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:10.291570  9524 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:10.146093 15944 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:10.211320  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:10.300251  9524 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:10.184751 15935 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18.
I20260808 01:52:10.310153 14784 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:10.304555  9601 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:10.304741  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:10.275645 15479 raft_consensus.cc:3055] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:10.293049 15729 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:52:10.310061  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:10.327791  9629 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:10.311070  9599 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:10.268129 15908 raft_consensus.cc:697] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:10.296367 15914 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:10.312809  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:10.309542 15513 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:10.319114 15933 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:10.292634 15347 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:10.299247 15832 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.023s
I20260808 01:52:10.350996 15933 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.305532 15139 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:10.352569 15139 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:10.353133 15139 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:10.320758 15913 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:10.322896 15710 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:10.313617 15928 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.086961 15877 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.218s
W20260808 01:52:10.325006  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 47346.7us
I20260808 01:52:10.272940 15788 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:10.306952 15836 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
I20260808 01:52:10.333395 15896 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:10.329892  9550 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:52:10.298241 15929 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:10.211254 15811 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:10.331789 14784 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:10.314838 15849 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
I20260808 01:52:10.303122  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:10.262920 15660 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.309340 15844 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:52:10.328328 15876 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260808 01:52:10.438030 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:10.371646  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 67188.9us
I20260808 01:52:10.317565 15479 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:10.411310 15832 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:10.414270  9377 catalog_manager.cc:5697] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:10.316402  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:10.340250 15729 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:10.355748 15513 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:10.465242 15513 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:10.337325  9557 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:10.428500 14784 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.432716  9558 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:10.375909 15912 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:10.472575 15912 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:10.475119 15912 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:10.436635 10332 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.014s	sys 0.019s
W20260808 01:52:10.476037 10332 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.015s	sys 0.019s
W20260808 01:52:10.476368 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 585397us
I20260808 01:52:10.479616  9636 raft_consensus.cc:2410] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 19 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260808 01:52:10.361300 15932 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.075s
W20260808 01:52:10.400043 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:10.457259  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 140943us
W20260808 01:52:10.457515  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:10.494944  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:10.495455  9444 connection.cc:256] RPC call timeout handler was delayed by -1.3886682050069794s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2675s for a response
I20260808 01:52:10.355019 15914 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:10.495814  9444 connection.cc:256] RPC call timeout handler was delayed by -1.227209873002721s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17800000000000002s for a response
I20260808 01:52:10.430114 15605 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:10.461153 15667 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.717s	user 0.013s	sys 0.001s
I20260808 01:52:10.463629 15729 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.317625 15944 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.439716  9633 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:10.481691  9480 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:10.436182 15896 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:10.514792  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:10.504936  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:10.426437 15788 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.519732  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:10.428639 15919 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.071s
I20260808 01:52:10.492640 15844 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:10.520665  9479 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:10.527143  9476 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:52:10.524575  9524 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:10.525164  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:10.529479  9475 raft_consensus.cc:2410] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260808 01:52:10.492221 15876 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:10.533447  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:10.541671  9474 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:10.341975 15908 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:10.519946 15832 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:10.548242  9475 pending_rounds.cc:85] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 7
I20260808 01:52:10.548887  9475 pending_rounds.cc:107] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 11.8
I20260808 01:52:10.549263  9475 raft_consensus.cc:2889] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:10.552060  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:10.552940  9472 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:10.500394 15710 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.042s
I20260808 01:52:10.533411 15844 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:10.537212 15709 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:10.523211 15925 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.034s
I20260808 01:52:10.513512 15667 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.516237 15778 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.032s
I20260808 01:52:10.576622 15605 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:10.577800 15944 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:10.536343 15924 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.033s
I20260808 01:52:10.593724 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:10.567533 15873 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:10.559068 15896 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.612576  9478 raft_consensus.cc:2393] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 16: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:10.538952 15876 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.570818 15844 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:10.557971 15832 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:10.572192 15757 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.046s
I20260808 01:52:10.601233 15709 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:10.634414  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 101214us
W20260808 01:52:10.647456 15965 log.cc:927] Time spent T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260808 01:52:10.648095 15811 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:10.648449 15513 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.649914 15876 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:10.651624 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:10.654186 15709 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.655071  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:10.660000 15733 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.660s	user 0.013s	sys 0.003s
I20260808 01:52:10.707297  9481 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:10.707370 15733 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:10.661100 15875 consensus_meta.cc:287] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.635s	user 0.009s	sys 0.002s
W20260808 01:52:10.661052 15636 consensus_meta.cc:287] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.588s	user 0.012s	sys 0.000s
W20260808 01:52:10.664397  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 373167us
I20260808 01:52:10.666885 15914 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:10.679934  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:10.677480 15928 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:10.727636  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:10.756819 15513 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:10.757774 15733 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:10.758265 15947 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 9, Current term: 15, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:10.737833  9481 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:10.765177  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:10.765883 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:10.766948 10367 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:10.686375  9479 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:10.691006 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:10.721395 15788 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:10.700520 10360 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 11.
I20260808 01:52:10.771232  9600 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:10.670663 15837 consensus_meta.cc:287] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.624s	user 0.007s	sys 0.003s
I20260808 01:52:10.705404  9472 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:10.712873 15860 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20260808 01:52:10.715807 15906 consensus_meta.cc:287] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.694s	user 0.008s	sys 0.005s
I20260808 01:52:10.674261 15660 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:10.720402  9521 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:52:10.782462  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62146.7us
I20260808 01:52:10.784493  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:10.786118  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 55496.8us
I20260808 01:52:10.771827  9524 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:10.790177  9559 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:10.725319 15636 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:10.791345  9637 raft_consensus.cc:2410] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260808 01:52:10.769606 10361 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:10.724735 15914 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:10.774008  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:10.797498  9472 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:10.787618  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:10.736132 15875 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:10.794476  9445 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:10.721839 15923 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260808 01:52:10.724215 15139 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.785840 15513 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:10.792253 15876 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:52:10.801815 15717 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260808 01:52:10.818528 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 295214us
I20260808 01:52:10.825078 10332 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:10.796386 15796 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election won for term 16
W20260808 01:52:10.827168 15668 consensus_meta.cc:287] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.817s	user 0.017s	sys 0.001s
I20260808 01:52:10.827685 15668 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:10.808605 15930 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:10.801676 15906 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:10.832638 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:10.831800  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:10.836133  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:10.836710  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:10.837126  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:10.837389  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:10.837770  9630 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:10.837971  9635 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:10.802328 15928 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.029s
I20260808 01:52:10.841245  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:10.840075  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:10.840147  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:10.839747  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:10.842069  9633 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:10.842180  9629 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 16: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:10.842343  9635 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 18: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:10.842222  9630 raft_consensus.cc:2368] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19.
I20260808 01:52:10.840222 10332 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:10.826574 15347 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:10.846061  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:10.832798 15876 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:10.843443 15925 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:52:10.809017 15914 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.853931 15876 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:10.854528 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:10.855974 15914 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:10.856125 15912 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:10.838099 15906 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:10.842823 15983 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:10.857956 15983 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:10.858425 15983 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:10.859524 15906 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:10.824278 15717 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:10.860261 15717 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:10.858186 15139 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:10.839666 15796 raft_consensus.cc:697] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:10.851747 15668 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:10.824672 15513 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:10.852881 15832 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:10.852377 15347 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:10.853006  9638 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:10.864955 10332 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:10.867213 15796 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.865676 15899 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:10.843276 15981 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:10.870561  9634 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:52:10.875299 15347 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:10.879773 15479 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.880954 15479 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:10.884120 15513 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:10.883603 15981 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.865587 15844 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.901371 15925 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:10.912845 15844 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:10.913199 15717 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:10.921054 15717 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:10.929323 15981 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:10.931160 14784 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:10.933645 15972 raft_consensus.cc:3055] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:10.934352 15972 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:10.939579 15972 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.941650 15972 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:10.947335 15347 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:10.955564 15964 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.512s	user 0.001s	sys 0.005s
I20260808 01:52:10.957031 10362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 791) took 1076 ms. Trace:
W20260808 01:52:10.957422 15982 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.105s	user 0.003s	sys 0.001s
I20260808 01:52:10.957510 10362 rpcz_store.cc:276] 0808 01:52:09.880757 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:09.881081 (+   324us) service_pool.cc:224] Handling call
0808 01:52:09.881473 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:09.924172 (+ 42699us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:09.924213 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:09.924775 (+   562us) log.cc:844] Serialized 26 byte log entry
0808 01:52:10.293979 (+369204us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:10.294034 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:10.294111 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:10.947552 (+653441us) spinlock_profiling.cc:243] Waited 7.06 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:10.956734 (+  9182us) raft_consensus.cc:1685] finished
0808 01:52:10.956759 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:10.956960 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42212,"raft.queue_time_us":43540,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"spinlock_wait_cycles":16248064,"thread_start_us":368860,"threads_started":1,"wal-append.queue_time_us":507089}
I20260808 01:52:10.957082 15347 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:10.976449 15983 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:10.977955 15303 raft_consensus.cc:3055] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:10.978472 15303 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:10.979429 15983 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:10.974083 15987 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.068s	user 0.002s	sys 0.000s
I20260808 01:52:10.979918 15303 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:10.986227 15930 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:10.988673 15303 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:10.987426 15925 raft_consensus.cc:697] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:10.992206 15899 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:10.995767 15930 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:10.996222 15925 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:10.814080  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:11.065104  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 250893us
I20260808 01:52:11.071337  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:11.075639  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 281126us
W20260808 01:52:11.092222  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 285416us
I20260808 01:52:11.094051  9524 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:11.111543  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:11.113473  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:11.115996  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:11.117563  9634 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:11.120193 15479 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:11.122783 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:11.124223 10366 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:11.125720 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:11.128149 15709 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:11.129186 15709 raft_consensus.cc:697] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:11.128830 15636 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:11.131347 15788 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:11.132232 15788 raft_consensus.cc:697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:11.132114 15717 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:11.133453 15717 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:11.135181 15844 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:11.135769 15737 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:11.136448 15737 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:11.137095 15737 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:11.139209  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:11.141182  9477 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:11.140882 15899 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:11.142835 15778 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:11.101984  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:11.144050 15778 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:11.146018  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:11.136216 15844 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:11.101917  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:11.121587 15479 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:11.140798  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:11.151612  9473 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:11.178889  9521 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:11.180780  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 106115us
I20260808 01:52:11.180823 15991 raft_consensus.cc:2764] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260808 01:52:11.183274  9472 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:11.184568 15668 raft_consensus.cc:2764] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260808 01:52:11.186122 15933 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:11.161124  9630 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:11.162591 15479 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:11.150652 15844 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:11.134187 15788 consensus_queue.cc:237] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:11.182371  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:11.217857  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:11.220093 10362 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:52:11.238615  9524 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:11.240618  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:11.243351 15636 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:11.247792 15988 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:11.248903 15988 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:11.203913 15991 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.052s
I20260808 01:52:11.249495 15347 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:11.257558 15347 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:11.252185  9557 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:11.245954 15636 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Stepping down as leader of term 15
W20260808 01:52:11.259862 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:11.260279 15636 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:11.236016  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 53990.1us
I20260808 01:52:11.269528  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:11.252498 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:11.270538  9630 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:11.258283 15347 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:11.277412  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:11.279824  9635 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260808 01:52:11.284327  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:11.285715  9554 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:52:11.287366  9634 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260808 01:52:11.289362  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1014) took 1230 ms. Trace:
I20260808 01:52:11.290215 15709 consensus_queue.cc:237] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:11.291661  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:11.292754  9551 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:11.217962 15933 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:11.252524  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:11.295271 15991 consensus_queue.cc:1243] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 16.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260808 01:52:11.290177  9480 rpcz_store.cc:276] 0808 01:52:10.058397 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:10.063298 (+  4901us) service_pool.cc:224] Handling call
0808 01:52:10.150170 (+ 86872us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:10.509739 (+359569us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:11.288599 (+778860us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b54002c0954. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:11.289003 (+   404us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":705216768}
I20260808 01:52:11.330399  9638 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:11.330734 15933 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:11.331840  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:11.334003 15933 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:11.341241  9599 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:11.363884  9477 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:11.253624 15988 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:11.375784 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:11.380746  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:11.381430  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:11.391647  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:11.401870 15778 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:11.405292 15347 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:11.405915 15636 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:11.408846  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2938) took 1373 ms. Trace:
W20260808 01:52:11.413286 15998 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.154s	user 0.003s	sys 0.000s
I20260808 01:52:11.415833 15914 raft_consensus.cc:2764] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
W20260808 01:52:11.428928  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 133950us
I20260808 01:52:11.423542  9478 rpcz_store.cc:276] 0808 01:52:10.035210 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:10.036882 (+  1672us) service_pool.cc:224] Handling call
0808 01:52:11.408120 (+1371238us) spinlock_profiling.cc:243] Waited 574 ms on lock 0x7b540019b4d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:11.408545 (+   425us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1320041728}
I20260808 01:52:11.439108  9473 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:11.442934  9551 raft_consensus.cc:2410] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 19 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260808 01:52:11.448019  9560 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:11.449707  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:11.450811  9638 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:11.455147 15729 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:11.456108 15729 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:11.459087 15605 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:11.460325  9554 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260808 01:52:11.449988  9630 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
W20260808 01:52:11.444211 15876 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.007s	sys 0.007s
I20260808 01:52:11.462486  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 889) took 1567 ms. Trace:
I20260808 01:52:11.464619  9558 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:11.394469 15737 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:11.462507  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:11.475644 10332 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:11.476794 15933 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:11.445252 15717 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:11.459982 15972 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.517s	user 0.016s	sys 0.012s
I20260808 01:52:11.469784  9550 rpcz_store.cc:276] 0808 01:52:09.894981 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:09.895148 (+   167us) service_pool.cc:224] Handling call
0808 01:52:11.461998 (+1566850us) spinlock_profiling.cc:243] Waited 371 ms on lock 0x7b540029e0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:11.462213 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":853279488}
W20260808 01:52:11.464792 15947 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.009s	sys 0.006s
I20260808 01:52:11.488091 15933 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:11.489337  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2683) took 1408 ms. Trace:
I20260808 01:52:11.489791  9637 rpcz_store.cc:276] 0808 01:52:10.080618 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:10.080742 (+   124us) service_pool.cc:224] Handling call
0808 01:52:11.488854 (+1408112us) spinlock_profiling.cc:243] Waited 709 ms on lock 0x7b540008dc54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:11.489133 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1631373184}
I20260808 01:52:11.484040 15919 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:11.491094  9557 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260808 01:52:11.491348 15919 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:11.492501  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:11.496150 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:11.497185 10367 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:11.498430 15986 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:11.499012 15986 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:11.499390 15986 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:11.503198  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:11.503525  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:11.504328  9554 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 15: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:11.504582  9556 raft_consensus.cc:2410] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 16 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260808 01:52:11.505106  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2681) took 1439 ms. Trace:
I20260808 01:52:11.459990 15605 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:11.494318  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:11.478792  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:11.505575  9636 rpcz_store.cc:276] 0808 01:52:10.065430 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:10.065586 (+   156us) service_pool.cc:224] Handling call
0808 01:52:11.504640 (+1439054us) spinlock_profiling.cc:243] Waited 389 ms on lock 0x7b5400307fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:11.504913 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":893902592}
I20260808 01:52:11.468339 15729 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:11.528800  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:11.543589  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:11.543035  9377 catalog_manager.cc:5697] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:11.496230 15347 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:11.545607 16009 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260808 01:52:11.480424 15479 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:11.551213  9521 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:11.551843  9474 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 2946) took 1361 ms (client timeout 1561 ms). Trace:
I20260808 01:52:11.554550 15930 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:11.179766  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:52:11.561580  9560 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:11.478930 15981 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:11.575979  9445 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:11.576156  9474 rpcz_store.cc:269] 0808 01:52:10.190642 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:10.251831 (+ 61189us) service_pool.cc:224] Handling call
0808 01:52:11.551414 (+1299583us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b540023aad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:11.551652 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":590146816}
I20260808 01:52:11.590884 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:11.564278 15668 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:11.578954 15933 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:11.593665  9555 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:52:11.590447  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:11.596903 10367 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:11.602437 15796 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:11.607906 16001 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.171s	user 0.001s	sys 0.000s
I20260808 01:52:11.618848 15796 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:11.596946  9377 catalog_manager.cc:5697] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 9 to 14, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 14 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:11.622011 15796 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:11.622783 15347 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260808 01:52:11.623989  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 72964.1us
I20260808 01:52:11.624904 15856 consensus_queue.cc:1059] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:11.626073  9521 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:11.628630  9636 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260808 01:52:11.629524  9638 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:11.607278  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:11.617606  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 59386.8us
I20260808 01:52:11.621428  9377 catalog_manager.cc:5697] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 14 to 17, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 17 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:11.635134  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:11.635965  9630 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:11.636747 15667 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:11.636514  9377 catalog_manager.cc:5697] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 6 to 14, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 14 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:11.637894  9634 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260808 01:52:11.638098 15605 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:11.626384 15925 raft_consensus.cc:3055] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:11.639061 15925 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:11.591316 15636 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:11.624145 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:11.595021 15914 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:11.640601  9377 catalog_manager.cc:5697] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:11.658908  9478 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 11: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:52:11.659886 15982 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260808 01:52:11.600462 15729 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:11.660712 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 221782us
I20260808 01:52:11.663252 15992 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:52:11.664312 15899 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:11.664768 15737 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:11.665040 15992 raft_consensus.cc:697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:52:11.606539  9370 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50064 (request call id 238) took 56049 ms (client timeout 15000 ms). Trace:
W20260808 01:52:11.660241 16005 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260808 01:52:11.668627  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:11.669719  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 186599us
I20260808 01:52:11.670029  9472 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 20: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:11.672860  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:11.673319  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:11.673780  9638 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:52:11.674326  9630 raft_consensus.cc:2376] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19. Re-sending same reply.
I20260808 01:52:11.673345  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:11.675683 15844 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:11.676276  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:11.667378  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:11.675180  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 49248.7us
I20260808 01:52:11.664734 15899 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:11.616853 15479 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:11.677472  9479 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260808 01:52:11.666211 15914 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:11.665347  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:11.680392  9477 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 14 index: 14. Preceding OpId from leader: term: 15 index: 15. (index mismatch)
I20260808 01:52:11.641386 15933 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:11.673693 15862 raft_consensus.cc:3055] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:11.702456  9480 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:11.709208  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:11.606879 15930 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:11.712291  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:11.681412 15899 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:11.710023  9474 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:11.617241 15668 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:11.721820  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:11.729813  9630 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 17. Current term is 19. Ops: []
I20260808 01:52:11.707207 15972 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:52:11.748205  9521 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:11.710213 15862 raft_consensus.cc:740] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:11.753495  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:11.732522  9524 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.561s (SENT) [suppressed 25 similar messages]
I20260808 01:52:11.762477  9477 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 18: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:11.648167 15981 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:11.766060  9554 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20260808 01:52:11.769788  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 291175us
I20260808 01:52:11.764204 15856 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.007s
I20260808 01:52:11.775007  9638 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 19. Current term is 20. Ops: [17.13-19.14]
I20260808 01:52:11.668447 15992 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:11.763213 15972 raft_consensus.cc:697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:11.778698  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 112596us
I20260808 01:52:11.778491 15925 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:11.779318 16016 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.040s
I20260808 01:52:11.781418  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:11.782769 14784 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:11.782938 15919 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:11.783260 15933 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:11.776698 15947 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:52:11.787901  9476 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260808 01:52:11.763321  9377 catalog_manager.cc:5697] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 9 to 11, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 11 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 01:52:11.773370 15912 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.784s	user 0.021s	sys 0.043s
I20260808 01:52:11.795912  9550 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260808 01:52:11.806865  9555 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:11.824056  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:11.826766  9600 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:11.834270  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:11.792598 15914 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:11.841995  9445 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:11.790565 15930 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:11.844586 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:11.797833 15303 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.679s	user 0.012s	sys 0.005s
I20260808 01:52:11.807957 15479 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:11.836624 15912 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.980s	user 0.021s	sys 0.044s
I20260808 01:52:11.864735 16020 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:11.865065 15912 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:11.865789 16020 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:11.866811 15912 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:11.869962  9630 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:11.871446  9377 catalog_manager.cc:5697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 6 to 9. New cstate: current_term: 9 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:11.852524 14784 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:11.872819  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 140348us
W20260808 01:52:11.874528  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 56767us
I20260808 01:52:11.883075  9560 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260808 01:52:11.884424 15668 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:11.885445  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:11.890848 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:11.888010  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:11.892788  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:11.903931  9550 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:11.860800 15788 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:11.905457  9555 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:52:11.911270  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 199437us
I20260808 01:52:11.870079 16020 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:11.886427 15933 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:11.893585 15667 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:11.927179  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:11.929924 15981 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:11.931309  9560 raft_consensus.cc:2410] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 14 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260808 01:52:11.931829  9479 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260808 01:52:11.932538  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:11.934094  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:11.934921  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2950) took 1389 ms. Trace:
I20260808 01:52:11.935750  9475 rpcz_store.cc:276] 0808 01:52:10.545595 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:10.547522 (+  1927us) service_pool.cc:224] Handling call
0808 01:52:10.548000 (+   478us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:10.549955 (+  1955us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:10.550004 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:11.601087 (+1051083us) spinlock_profiling.cc:243] Waited 4.29 ms on lock 0x7b2c00134ff8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:52:11.658889 (+ 57802us) log.cc:844] Serialized 26 byte log entry
0808 01:52:11.767115 (+108226us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:52:11.767156 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:11.813487 (+ 46331us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:11.934333 (+120846us) raft_consensus.cc:1685] finished
0808 01:52:11.934356 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:11.934665 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9860992,"thread_start_us":107687,"threads_started":1,"wal-append.queue_time_us":235580}
I20260808 01:52:11.938862 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:52:11.668160  9370 rpcz_store.cc:269] 0808 01:51:15.556648 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:15.557070 (+   422us) service_pool.cc:224] Handling call
0808 01:52:11.605902 (+56048832us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:52:03.596236 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:52:03.596891 (+   655us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:52:03.596956 (+    65us) write_op.cc:435] Acquired schema lock
0808 01:52:03.596996 (+    40us) tablet.cc:662] Decoding operations
0808 01:52:03.598161 (+  1165us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:52:03.598299 (+   138us) write_op.cc:641] Partition lock acquired for write op
0808 01:52:03.598328 (+    29us) tablet.cc:685] Acquiring locks for 9 operations
0808 01:52:03.635866 (+ 37538us) tablet.cc:701] Row locks acquired
0808 01:52:03.647154 (+ 11288us) write_op.cc:260] PREPARE: finished
0808 01:52:03.690962 (+ 43808us) write_op.cc:270] Start()
0808 01:52:03.691220 (+   258us) write_op.cc:276] Timestamp: P: 1786153923661047 usec, L: 0
0808 01:52:03.691276 (+    56us) op_driver.cc:348] REPLICATION: starting
0808 01:52:03.692402 (+  1126us) log.cc:844] Serialized 3646 byte log entry
0808 01:52:04.136158 (+443756us) op_driver.cc:464] REPLICATION: finished
0808 01:52:04.393066 (+256908us) write_op.cc:301] APPLY: starting
0808 01:52:04.393186 (+   120us) tablet.cc:1367] starting BulkCheckPresence
0808 01:52:04.541954 (+148768us) tablet.cc:1370] finished BulkCheckPresence
0808 01:52:04.542107 (+   153us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:52:05.330769 (+788662us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:52:05.331190 (+   421us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=18,key_file_lookups=18,delta_file_lookups=18,mrs_lookups=0
0808 01:52:05.331283 (+    93us) write_op.cc:312] APPLY: finished
0808 01:52:05.331790 (+   507us) log.cc:844] Serialized 89 byte log entry
0808 01:52:05.409723 (+ 77933us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:52:05.453919 (+ 44196us) write_op.cc:454] Released schema lock
0808 01:52:05.454020 (+   101us) write_op.cc:341] FINISH: Updating metrics
0808 01:52:05.458383 (+  4363us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:52:05.458424 (+    41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":400595,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":256557,"apply.run_cpu_time_us":53133,"apply.run_wall_time_us":1065231,"cfile_cache_hit":38,"cfile_cache_hit_bytes":41801,"num_ops":9,"prepare.queue_time_us":374143,"prepare.run_cpu_time_us":114716,"prepare.run_wall_time_us":338421,"raft.queue_time_us":78062,"raft.run_cpu_time_us":13400,"raft.run_wall_time_us":111796,"replication_time_us":444594,"thread_start_us":798937,"threads_started":5,"wal-append.queue_time_us":377588,"wal-append.run_cpu_time_us":30352,"wal-append.run_wall_time_us":2154704}]]}
I20260808 01:52:11.924299  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:11.941026  9638 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260808 01:52:11.940205  9521 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:11.945456 14784 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:11.946173 14784 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:11.946169 15947 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:52:11.940989  9551 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:52:11.949426  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:11.954597  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:11.945667  9521 leader_election.cc:400] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19.
I20260808 01:52:11.959625  9521 leader_election.cc:403] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:52:11.939778  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:11.960747 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:11.962431 10361 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260808 01:52:11.955081  9476 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260808 01:52:11.963086 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:11.961038 15947 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 19
W20260808 01:52:11.963632  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 53715.5us
I20260808 01:52:11.963325 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:11.964598  9524 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:11.964637 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:11.940181  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:11.965678 15844 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:11.966187  9550 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:11.966405 15844 raft_consensus.cc:697] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:11.956609 16028 consensus_queue.cc:1059] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.012s
I20260808 01:52:11.968905 15856 raft_consensus.cc:3055] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:11.969318 15856 raft_consensus.cc:740] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:11.970444 15856 consensus_queue.cc:260] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:11.972111 15856 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:11.965689 16029 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:11.973532  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:11.975252 14784 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:11.963900 15930 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:11.977506 15930 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:11.912688 15919 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:11.978977 15919 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:11.979417 15919 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:11.967867 15844 consensus_queue.cc:237] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:11.960470  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:11.950275  9551 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:11.963842 10362 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:11.989382  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:11.912474 15788 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:11.911439 15914 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260808 01:52:11.998865  9630 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:52:12.004797 14784 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:12.005048  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 223870us
I20260808 01:52:12.006181  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:11.966547 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:12.009344  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:12.014346 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:11.791872 15862 consensus_queue.cc:260] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:12.022364  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 61984.7us
I20260808 01:52:12.028713 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:12.029423 15947 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19.
W20260808 01:52:12.030481 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:12.000944 15667 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:12.035811  9477 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:12.036557  9477 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:12.038899  9480 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
W20260808 01:52:12.046936 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 44716.9us
I20260808 01:52:12.017578 15981 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:12.050352  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:12.051407 15729 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:12.055303 14784 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:12.055810  9599 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:12.056319 14784 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:12.057951 16020 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:12.058439 16020 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:12.058959 16020 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:12.059059 15933 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:12.059497 15933 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:12.059883 15933 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:12.059994 15862 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:12.058158 15899 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:11.979046 15930 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:12.064749 15899 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:12.065734 10367 raft_consensus.cc:2368] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 16. Current term is 17.
I20260808 01:52:12.066752  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:12.067569  9480 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:12.001116 15788 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:12.029587 10362 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:52:12.068612  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:12.076661  9599 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:11.890407 15972 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:12.079648  9599 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:11.991532 16016 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.078s
W20260808 01:52:12.089322  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 213592us
W20260808 01:52:12.090515  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:12.094383 15605 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:12.089426 15981 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:12.097852 15605 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:12.098038 15981 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:12.098347 15605 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:12.098562 15981 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:12.099056 15347 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:12.099627 15347 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:52:12.100617 15986 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.601s	user 0.006s	sys 0.010s
I20260808 01:52:12.101100 15986 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:12.103437  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 89993.8us
I20260808 01:52:12.104770  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:12.105628 15947 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:12.106112  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2954) took 1413 ms. Trace:
I20260808 01:52:12.106079 15947 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:12.107064 15947 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:12.108119 15788 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:12.110960  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 165491us
I20260808 01:52:12.112265  9521 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:12.116314 14784 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:12.113070  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:12.125245  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:12.129392  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:12.132853  9558 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260808 01:52:12.133260  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:12.138450 14784 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:12.141724  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:12.151649  9476 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:52:12.153322  9551 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:52:12.155179 14784 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:12.117702 15788 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:12.106545  9481 rpcz_store.cc:276] 0808 01:52:10.692665 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:10.771414 (+ 78749us) service_pool.cc:224] Handling call
0808 01:52:10.771743 (+   329us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:11.083056 (+311313us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:11.083160 (+   104us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:11.084966 (+  1806us) log.cc:844] Serialized 52 byte log entry
0808 01:52:11.215025 (+130059us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:52:11.301685 (+ 86660us) log.cc:844] Serialized 14 byte log entry
0808 01:52:11.306407 (+  4722us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:12.105017 (+798610us) spinlock_profiling.cc:243] Waited 1.4 ms on lock 0x7b180027b020. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4185ac56 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229
0808 01:52:12.105590 (+   573us) spinlock_profiling.cc:243] Waited 309 ms on lock 0x7b54002c0454. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:12.105642 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:12.105689 (+    47us) raft_consensus.cc:1685] finished
0808 01:52:12.105702 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:12.105918 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":77901,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":9978,"raft.queue_time_us":54820,"raft.run_cpu_time_us":416,"raft.run_wall_time_us":6266,"spinlock_wait_cycles":714976256,"thread_start_us":129197,"threads_started":1,"wal-append.queue_time_us":244710}
I20260808 01:52:12.164623 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:12.161446  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:12.183144  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 70994.2us
I20260808 01:52:12.184564  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:12.185648 15928 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.051s
I20260808 01:52:12.188867 15845 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:12.139025 15636 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:12.201200 10368 raft_consensus.cc:2393] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 16: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:52:12.208043  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 113636us
I20260808 01:52:12.209851  9560 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
W20260808 01:52:12.218968 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 170834us
I20260808 01:52:12.221748 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:12.237368  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:12.248158 14784 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:12.237874 15928 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:12.164942 15919 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:12.268333 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:12.281483  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:12.281664  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:12.283856  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:12.270515 15988 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.006s	sys 0.024s
I20260808 01:52:12.287582  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:12.266737 15986 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:12.290942 15876 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:12.327310 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 86178.2us
I20260808 01:52:12.327486 15876 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:12.332432 16023 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.302s	user 0.004s	sys 0.000s
W20260808 01:52:12.332858 16026 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.277s	user 0.001s	sys 0.000s
W20260808 01:52:12.292126 15988 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.007s	sys 0.024s
I20260808 01:52:12.293576 15636 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:12.339344  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:12.340317  9558 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:12.340520  9473 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:52:12.344671  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:12.330469 15876 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:12.349565  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 165200us
I20260808 01:52:12.337405 15992 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260808 01:52:12.360627  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 102150us
I20260808 01:52:12.376917  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:12.342209 15636 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:12.390405  9552 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 14 index: 14. Preceding OpId from leader: term: 15 index: 15. (index mismatch)
I20260808 01:52:12.391697 16043 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:12.392365 16043 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:12.396396 16043 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:12.398804  9559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2530) took 1990 ms (client timeout 1750 ms). Trace:
W20260808 01:52:12.399494  9559 rpcz_store.cc:269] 0808 01:52:10.304250 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:10.304482 (+   232us) service_pool.cc:224] Handling call
0808 01:52:12.269002 (+1964520us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x7b540029efd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:12.294638 (+ 25636us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1097446272}
I20260808 01:52:12.402125 15667 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:12.411672 15919 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:12.419793  9636 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260808 01:52:12.370416 15992 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:12.421394 15992 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:52:12.422320 16031 log.cc:927] Time spent T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260808 01:52:12.424816 10334 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:12.426093 15996 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:12.426779 15996 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:12.427170  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:12.427973 15925 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:12.428292  9638 raft_consensus.cc:2410] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 17 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260808 01:52:12.428083 15996 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:12.430138 15788 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:52:12.431295 15788 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:12.431720 15788 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:12.433092 16019 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.378s	user 0.002s	sys 0.000s
I20260808 01:52:12.432812  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:12.441401  9635 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:12.432821 15788 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:12.457010  9480 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:52:12.384475 15636 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:12.472396  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3310) took 1015 ms. Trace:
I20260808 01:52:12.470268  9634 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:52:12.476909  9559 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260808 01:52:12.477510  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:12.478437 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:12.479305 10368 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:12.487936 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:12.498265 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:12.475270  9556 rpcz_store.cc:276] 0808 01:52:11.457330 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:11.505361 (+ 48031us) service_pool.cc:224] Handling call
0808 01:52:11.505856 (+   495us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:11.627488 (+121632us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:11.627529 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:11.628033 (+   504us) log.cc:844] Serialized 26 byte log entry
0808 01:52:11.735549 (+107516us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:11.735616 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:11.881227 (+145611us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b540002e254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:11.881294 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:12.432287 (+550993us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:12.472064 (+ 39777us) raft_consensus.cc:1685] finished
0808 01:52:12.472097 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:12.472308 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":932,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":125,"spinlock_wait_cycles":308811904,"thread_start_us":107004,"threads_started":1,"wal-append.queue_time_us":287227}
I20260808 01:52:12.506556  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:12.506947  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:12.509492 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:12.510552 10362 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260808 01:52:12.467187 16020 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:12.520744 16020 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:12.489070 15709 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:12.522151 15709 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:12.523005  9635 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 8. (term mismatch)
I20260808 01:52:12.523440  9635 pending_rounds.cc:85] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 7
I20260808 01:52:12.523689  9635 pending_rounds.cc:107] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 15.8
I20260808 01:52:12.523945  9635 raft_consensus.cc:2889] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:12.524817 15709 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:12.525208 15788 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:12.525977 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:12.526721 10363 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:12.527339  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:12.529629  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 169689us
I20260808 01:52:12.530567  9521 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:12.531376  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:12.532097  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:12.532771  9556 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:12.534292 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:12.534791 16045 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:12.513633  9634 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 13: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:12.532271  9631 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 17: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:12.514204  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:12.536185 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 181665us
I20260808 01:52:12.514119  9636 raft_consensus.cc:2376] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18. Re-sending same reply.
I20260808 01:52:12.539803 10364 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:12.536770 15667 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:12.504179 15729 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:12.552517 15729 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:12.548205 16036 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.195s	user 0.007s	sys 0.000s
I20260808 01:52:12.555295  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:12.556820 15928 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:12.557317 15928 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:12.557718 15928 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
W20260808 01:52:12.564293  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 116737us
I20260808 01:52:12.568473  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:12.569274  9634 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:12.570847 15709 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:12.510383 15947 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:12.571611 15709 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:12.559350 15933 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:12.574056 15933 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:12.576236 15981 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:12.577497  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:12.583321 15729 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:12.535460 16045 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:52:12.585944  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:12.586710  9554 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:12.587033  9632 raft_consensus.cc:2368] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16.
I20260808 01:52:12.587795 15992 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:12.589520 15899 consensus_queue.cc:1059] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:12.565210 15347 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:12.591118 16053 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:12.591861 16053 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:12.593274 15347 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:12.593571 16045 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:12.613190  9558 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:12.627024  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:12.620361 15947 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:12.630509  9629 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:12.631518 10363 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:12.634153  9636 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260808 01:52:12.638363  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:12.639760  9558 raft_consensus.cc:2410] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260808 01:52:12.638648  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:12.611121 16028 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.030s
I20260808 01:52:12.642287  9631 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:12.655009 15947 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:12.655862 15947 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:12.643114 15605 consensus_meta.cc:287] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.544s	user 0.009s	sys 0.006s
I20260808 01:52:12.572875 15709 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:12.637063 16029 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.660s	user 0.011s	sys 0.004s
I20260808 01:52:12.628357 15981 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:12.659713  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:12.658084 15605 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:12.665199  9556 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:12.667248  9634 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:12.671216  9629 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:12.673270  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:12.673758 15636 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:12.676492 15636 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:12.642930 15667 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:12.674248  9558 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 16
I20260808 01:52:12.647481 16053 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:12.658630 15947 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:12.692627  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:12.695737  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:12.666993 15992 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:12.697681  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:12.698640  9634 raft_consensus.cc:2368] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19.
I20260808 01:52:12.667939 16029 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:12.703266  9558 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:12.704304  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:12.707286  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:12.711174  9481 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 16: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:52:12.700702 15972 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.023s	sys 0.020s
I20260808 01:52:12.721241 15605 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:12.721730  9631 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:12.725812  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:12.726006  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:12.728480  9629 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:12.713196 15636 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:12.732283  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:12.733098  9629 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:12.719682 15347 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:12.734405  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:12.724671 15972 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.023s	sys 0.020s
I20260808 01:52:12.754850  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:12.755656  9555 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:12.761030  9630 pending_rounds.cc:85] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 8
I20260808 01:52:12.761658  9630 pending_rounds.cc:107] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 16.9
I20260808 01:52:12.761999  9630 raft_consensus.cc:2889] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:12.763415  9638 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
W20260808 01:52:12.764142 16012 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.087s	user 0.000s	sys 0.006s
I20260808 01:52:12.761082 15925 raft_consensus.cc:3055] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:12.766239 15925 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:12.768004  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:12.771070 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:12.758347 10332 leader_election.cc:336] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.662s (SENT)
I20260808 01:52:12.782718 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:12.784509 16060 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:52:12.784653  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:12.779194  9477 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:12.788192  9477 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:12.773823  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 895) took 1524 ms. Trace:
I20260808 01:52:12.794304  9552 rpcz_store.cc:276] 0808 01:52:11.249601 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:11.249889 (+   288us) service_pool.cc:224] Handling call
0808 01:52:12.292315 (+1042426us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:12.391154 (+ 98839us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:12.773534 (+382380us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540029efd0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:12.773731 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2364964736}
I20260808 01:52:12.793326  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:12.771325 16059 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260808 01:52:12.801079 16060 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:12.801574 15347 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:12.811419 16060 raft_consensus.cc:697] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:52:12.813167 16057 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.123s	user 0.003s	sys 0.000s
I20260808 01:52:12.812875 16060 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:12.813635 15347 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:12.803567  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:12.807219 15908 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260808 01:52:12.772871 16050 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.230s	user 0.002s	sys 0.000s
I20260808 01:52:12.798599  9631 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:12.829110 15928 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:12.829694  9552 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
W20260808 01:52:12.832715 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 74412.2us
I20260808 01:52:12.839211  9477 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260808 01:52:12.833341  9558 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:12.852342  9555 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:12.853197  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:12.819569 15908 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:12.860376  9599 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:52:12.863189  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59783.9us
I20260808 01:52:12.836428 16017 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:12.855255 15972 raft_consensus.cc:2764] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election decision vote started in defunct term 16: won
I20260808 01:52:12.868866 14784 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:12.876645  9599 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:12.879014 15877 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:12.861683 15908 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:12.877620  9599 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:12.884853  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:12.885190  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:12.820379 15925 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:12.870424 16029 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:52:12.889609  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:52:12.976425  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 86870.8us
I20260808 01:52:12.977651  9599 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:52:12.979702  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 98606us
W20260808 01:52:12.976263  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 183105us
I20260808 01:52:12.883992 16053 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:12.981251  9632 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:12.866859 16017 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:12.991771  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:12.995627  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 112250us
W20260808 01:52:12.999109  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:12.998391  9599 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:12.975935 16029 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:13.014098 14784 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:12.890478 16052 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:12.991307 16043 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:13.006973 15981 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:12.981989 15347 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:13.003196 15667 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:12.978766 15910 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.133s
I20260808 01:52:13.017620 14784 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:12.994362 16053 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:13.025915  9632 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260808 01:52:13.000034 15788 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:13.144290 14784 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:13.145843  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1028) took 1202 ms. Trace:
W20260808 01:52:13.145807  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 163960us
I20260808 01:52:13.021710 16029 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:13.071177  9599 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:52:13.157359  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 86499.2us
I20260808 01:52:13.145917 16053 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:13.162626  9632 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:13.159268 15876 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election won for term 17
W20260808 01:52:13.165009 16067 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260808 01:52:13.167332  9601 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:13.154767 15788 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:13.174876 15923 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:13.109192 15605 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:13.175894 15923 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:13.176653 15605 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:13.183411 16063 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260808 01:52:13.184902 15778 raft_consensus.cc:2764] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260808 01:52:13.186411  9636 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260808 01:52:13.068607 15981 raft_consensus.cc:697] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:13.187697 15981 consensus_queue.cc:237] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:13.188163  9636 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260808 01:52:13.189980 15928 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:13.021701 15972 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.128s
I20260808 01:52:13.192660  9633 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:13.195461  9632 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260808 01:52:13.061741 16052 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:13.147440  9479 rpcz_store.cc:276] 0808 01:52:11.942969 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:11.946329 (+  3360us) service_pool.cc:224] Handling call
0808 01:52:11.946911 (+   582us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:12.074770 (+127859us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:12.074839 (+    69us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:12.767849 (+693010us) spinlock_profiling.cc:243] Waited 4.04 ms on lock 0x7b2c00134ff8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:52:12.768358 (+   509us) log.cc:844] Serialized 26 byte log entry
0808 01:52:12.884441 (+116083us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:52:12.884477 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:12.884541 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:13.108928 (+224387us) raft_consensus.cc:1685] finished
0808 01:52:13.108963 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:13.145731 (+ 36768us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56828,"raft.queue_time_us":74143,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":15766,"spinlock_wait_cycles":9337984,"thread_start_us":115655,"threads_started":1,"wal-append.queue_time_us":242789}
I20260808 01:52:13.065411 16043 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:13.159708 15925 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:13.112531 15933 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.091s
I20260808 01:52:13.151065 15667 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:13.209388 16037 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.844s	user 0.005s	sys 0.011s
I20260808 01:52:13.220391  9474 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
W20260808 01:52:13.220698  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 198430us
I20260808 01:52:13.151378 15347 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:13.231225  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:13.235220  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 68018.9us
W20260808 01:52:13.235811 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:13.236217  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:13.238204 15788 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:13.240434 15667 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:13.209128 15876 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:13.242795 15933 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:13.215958 15928 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:13.243572 15876 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:13.216130 16039 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:13.244093 15928 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:13.245702 16017 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:13.246246 15605 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:13.251211 15856 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:13.251884 15347 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:13.251926 15856 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:13.256915 15856 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:13.257746 15928 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:13.258890 16070 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:13.264884 10363 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:13.268254  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:13.275370  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:13.276082 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:13.276918 10367 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:13.275185 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:13.278072 10360 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
W20260808 01:52:13.287322  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:13.260829 15923 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:13.295192 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 142477us
I20260808 01:52:13.309330  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:13.313059  9521 leader_election.cc:336] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.021s (SENT)
I20260808 01:52:13.294412 15667 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:13.323550 16027 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:52:13.323882 15667 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:13.337741  9445 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:13.341215  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:13.342245  9477 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:52:13.295006 15972 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:13.294644 15605 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:13.325309 15928 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:13.352306 16077 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:13.353587  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 777) took 1630 ms. Trace:
I20260808 01:52:13.283290 16070 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:13.329696  9521 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:13.347544 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:13.371096  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 134955us
I20260808 01:52:13.366606 15605 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:13.376226 15788 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:13.376618 15667 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:13.366422 15928 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:13.362082 15972 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:13.307821 16052 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:13.378353  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 65432.1us
I20260808 01:52:13.375461 16076 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:13.359823  9637 rpcz_store.cc:276] 0808 01:52:11.723458 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:11.723667 (+   209us) service_pool.cc:224] Handling call
0808 01:52:11.776400 (+ 52733us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:11.853147 (+ 76747us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:52:11.899524 (+ 46377us) log.cc:844] Serialized 14 byte log entry
0808 01:52:12.255481 (+355957us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:12.300974 (+ 45493us) log.cc:844] Serialized 52 byte log entry
0808 01:52:12.301171 (+   197us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:12.301461 (+   290us) log.cc:844] Serialized 14 byte log entry
0808 01:52:12.455232 (+153771us) log.cc:844] Serialized 14 byte log entry
0808 01:52:12.456561 (+  1329us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:12.456921 (+   360us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:12.592413 (+135492us) spinlock_profiling.cc:243] Waited 8.12 ms on lock 0x7b1800212bc0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:52:13.171587 (+579174us) spinlock_profiling.cc:243] Waited 5.42 ms on lock 0x7b1800212bc0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4185ac56 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4185ac56 00007f5d4185abf0
0808 01:52:13.172783 (+  1196us) spinlock_profiling.cc:243] Waited 6.62 ms on lock 0x7b1800212bc0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4185ac56 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb
0808 01:52:13.174001 (+  1218us) spinlock_profiling.cc:243] Waited 3.9 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:13.353268 (+179267us) raft_consensus.cc:1685] finished
0808 01:52:13.353292 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:13.353472 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57904,"prepare.queue_time_us":28051,"prepare.run_cpu_time_us":282,"prepare.run_wall_time_us":7039,"raft.queue_time_us":357384,"raft.run_cpu_time_us":700,"raft.run_wall_time_us":90538,"spinlock_wait_cycles":55330048,"thread_start_us":335934,"threads_started":3,"wal-append.queue_time_us":277466}
I20260808 01:52:13.384975 15788 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:13.360741 15910 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260808 01:52:13.403381  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 65721.4us
I20260808 01:52:13.382289 15923 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:52:13.408262  9601 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:13.410775  9521 leader_election.cc:336] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.025s (SENT)
I20260808 01:52:13.411902  9521 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:13.408671  9479 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:13.409824  9445 leader_election.cc:400] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 16. Current term is 17.
I20260808 01:52:13.413664  9445 leader_election.cc:403] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260808 01:52:13.338178 16043 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:13.393347 16060 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:13.415962 15844 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:13.408490 16076 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:13.428167 15972 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:13.428514 16070 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:13.429509 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:13.429883 16076 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:13.367398 16077 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:13.408901 15788 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:13.421080 15605 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:13.437270  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 146517us
I20260808 01:52:13.443188 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:13.451514  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:13.448722 15876 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:13.461787  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:13.416847 16060 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:13.467011 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:13.467913 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:13.469110  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 70798.7us
I20260808 01:52:13.456884  9600 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:13.458595 15605 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:13.492059 15605 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:13.467608 16080 log.cc:927] Time spent T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.111s	user 0.000s	sys 0.001s
I20260808 01:52:13.460853 15928 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:13.463035 15908 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:13.495581 16073 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.119s	user 0.003s	sys 0.000s
I20260808 01:52:13.496546 15908 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:13.502526  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:13.503324 10367 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:13.508100 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:13.461407 16069 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.239s	user 0.002s	sys 0.001s
I20260808 01:52:13.511058 14784 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:13.523564  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 66789.8us
I20260808 01:52:13.526355  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:13.546388 14784 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:13.547137  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:13.542230  9600 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:13.551120  9472 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260808 01:52:13.552006  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:13.538404  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 85803.1us
I20260808 01:52:13.554955  9601 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:13.451833 15910 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 16. Current term is 17.
W20260808 01:52:13.525553 16094 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260808 01:52:13.510663 15876 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:13.575136 15876 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:13.559717  9601 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:13.436200 16053 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:13.546110  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:13.577564  9552 raft_consensus.cc:2410] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 18 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
I20260808 01:52:13.499091 15778 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:13.578655 16053 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:13.578807 15778 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:13.581437 15778 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:13.568145  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:13.569665 16045 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.030s
W20260808 01:52:13.550513 15636 consensus_meta.cc:287] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.776s	user 0.015s	sys 0.000s
I20260808 01:52:13.575735 15876 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:13.571514 16070 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:52:13.571555 16083 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:13.582124  9631 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 17: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:13.584611  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:13.505457 16060 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:13.568257  9480 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:13.598146  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:13.617499  9481 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13. Ops: []
I20260808 01:52:13.566123  9600 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:13.618110  9555 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:13.499074 15667 consensus_queue.cc:1059] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.063s
I20260808 01:52:13.617748  9552 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:13.616484  9479 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260808 01:52:13.562006 15788 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:13.616328  9635 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:13.628132  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2789) took 1090 ms. Trace:
W20260808 01:52:13.618883  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 52861.5us
I20260808 01:52:13.569563 15928 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:13.610952  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:13.558396 16052 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:13.599807 15636 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:13.619108 15877 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260808 01:52:13.631033  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:13.628698  9635 rpcz_store.cc:276] 0808 01:52:12.537384 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:13.461468 (+924084us) service_pool.cc:224] Handling call
0808 01:52:13.628049 (+166581us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:52:13.617841 16073 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260808 01:52:13.637111 14784 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260808 01:52:13.642362  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:13.642259  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:13.616922 15910 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:13.600719 16028 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260808 01:52:13.650125  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:13.661237  9480 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:52:13.657033  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:13.624423 16017 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:13.659395  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:13.640224 16088 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.201s	user 0.002s	sys 0.000s
I20260808 01:52:13.665601  9481 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:13.675215  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:13.676879  9474 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: []
W20260808 01:52:13.678983  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 148594us
I20260808 01:52:13.679031  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:13.679394  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:13.679927  9558 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:13.642730 15877 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:13.680166  9560 raft_consensus.cc:2376] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10. Re-sending same reply.
I20260808 01:52:13.682839  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:13.689112  9555 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:13.682500 15877 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:13.653870 16076 raft_consensus.cc:3055] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:13.691100  9599 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:13.692505  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:13.693379  9475 raft_consensus.cc:2410] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 17 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260808 01:52:13.693267  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:13.675607 15923 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:13.675194 15845 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:13.697006 16092 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:13.696897 15845 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:13.698144 16077 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:13.696501 15910 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.003s
I20260808 01:52:13.699923 15845 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:13.660332 16027 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.035s
I20260808 01:52:13.651229 15928 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:13.695605 16076 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:13.703787 16084 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:52:13.705842 15605 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:13.659265 15981 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.034s
I20260808 01:52:13.708331  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:13.709446 15778 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:13.709646 15667 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:13.709990 15778 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:13.710352 15667 raft_consensus.cc:697] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:13.710526 15778 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:13.711598 15667 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:13.696702 15923 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:13.712797  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:13.714720 15923 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:13.709439 15972 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.006s
I20260808 01:52:13.715821 15972 consensus_queue.cc:1243] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 12.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260808 01:52:13.716217  9472 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260808 01:52:13.684268 16078 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
W20260808 01:52:13.718717 16029 consensus_meta.cc:287] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.515s	user 0.011s	sys 0.002s
I20260808 01:52:13.719190 16029 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:13.684470 16017 raft_consensus.cc:697] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:13.720993 16017 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:13.697808 16092 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:13.724648 15636 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:52:13.721590 16030 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election won for term 19
W20260808 01:52:13.743798  9524 leader_election.cc:336] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.561s (SENT)
I20260808 01:52:13.746152  9552 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:13.746712  9475 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260808 01:52:13.750921  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 60846.5us
I20260808 01:52:13.752153  9523 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:13.754021  9523 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:13.756729  9523 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:13.747627  9599 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:52:13.759187  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 73598.8us
I20260808 01:52:13.762558  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:13.763193  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:13.772133  9558 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:52:13.774552  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:13.767874 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:13.779115  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:13.779887  9475 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:13.782110  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:13.782474 16092 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:13.690057 16058 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.084s
I20260808 01:52:13.780521 15910 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:52:13.785320 16030 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:13.786103 15788 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:13.786880 15605 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:13.786582 16030 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:13.783681  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 56691.3us
I20260808 01:52:13.776970 15933 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:13.792950 15933 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:13.797385 15933 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:13.800009 16076 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:13.787497 15605 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:13.803551  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 56056.9us
I20260808 01:52:13.804488  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:13.777913 15709 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:52:13.810794  9634 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260808 01:52:13.788983 15844 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:13.788426 15347 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:13.812219  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:13.817155  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:13.814065  9635 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:13.814484  9632 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260808 01:52:13.815881  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:13.820981  9638 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260808 01:52:13.815904  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:13.801544 16076 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:13.819938  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:13.819516  9630 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260808 01:52:13.821898  9629 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:13.822070  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:13.821307  9478 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:13.820626  9551 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:13.828274  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:13.829186  9634 raft_consensus.cc:2393] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 12: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:13.827203  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:13.775664 15923 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.032s
I20260808 01:52:13.830163  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:13.830519  9476 raft_consensus.cc:2368] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17.
I20260808 01:52:13.830440  9631 raft_consensus.cc:2393] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:13.832495  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:13.829924  9633 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:52:13.841557  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:13.847270 16096 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.157s	user 0.001s	sys 0.000s
I20260808 01:52:13.808295 15986 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:13.848556 15986 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:13.817533 15844 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:13.851403 15844 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:13.853857  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:13.854636  9638 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:13.855264  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:13.855926  9635 raft_consensus.cc:2376] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16. Re-sending same reply.
I20260808 01:52:13.813683 16052 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:13.857154  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:13.857304 16052 raft_consensus.cc:697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:13.858361 16052 consensus_queue.cc:237] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:13.859633  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:13.860179 15605 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:13.862195  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:13.863169  9630 raft_consensus.cc:2376] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18. Re-sending same reply.
I20260808 01:52:13.854336  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:13.858526  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:13.865437  9635 raft_consensus.cc:2376] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17. Re-sending same reply.
I20260808 01:52:13.868290  9558 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260808 01:52:13.869170  9638 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:13.871110  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:13.872264  9638 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 17: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:13.840365  9637 raft_consensus.cc:2376] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16. Re-sending same reply.
I20260808 01:52:13.873488  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:13.875187  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:13.875305  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:13.875825  9637 raft_consensus.cc:2376] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19. Re-sending same reply.
I20260808 01:52:13.876709  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:13.877352 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 45781us
W20260808 01:52:13.879716  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47335.9us
I20260808 01:52:13.881605  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:13.885322 15778 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:13.886948 16103 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.022s
I20260808 01:52:13.886126 16060 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:13.889305 15877 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:13.891774 16060 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:13.891930 15986 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:13.892935  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:13.893770  9630 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:13.895084  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:13.896281  9630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:13.897284  9630 raft_consensus.cc:606] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:13.897686  9630 raft_consensus.cc:614] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:13.887005  9472 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 20: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:13.906334  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:13.907135  9556 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:52:13.908396  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:13.909106  9633 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:13.911434  9634 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260808 01:52:13.912047 15986 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:13.913100  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:13.917088  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:13.916975  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:13.920017  9632 raft_consensus.cc:2376] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18. Re-sending same reply.
I20260808 01:52:13.920125  9635 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:13.925819  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:13.927184  9558 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:13.917393  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:13.928345  9631 raft_consensus.cc:2376] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17. Re-sending same reply.
W20260808 01:52:13.909503  9444 connection.cc:256] RPC call timeout handler was delayed by -2.2575367839890532s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17500000000000002s for a response
I20260808 01:52:13.935206  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:13.935604  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0198287710081786s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1686s for a response
W20260808 01:52:13.938925  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0144287710136268s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16920000000000002s for a response
I20260808 01:52:13.914260 16084 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:13.937516  9476 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:13.933480  9638 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:13.863090 16078 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.070s
I20260808 01:52:13.956595  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:13.957422  9551 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:13.959941 15778 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:13.961520  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:13.962355  9632 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:13.963939  9632 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260808 01:52:13.954047  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:13.965512  9632 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:13.966287  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:13.967419 16020 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:13.969398 16020 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:13.971956  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:13.918376 16060 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:13.922550 15986 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:13.927909 16029 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:13.947779  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:13.976946  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:13.981801  9638 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:13.906131 16027 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.041s
I20260808 01:52:13.943836 16084 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:13.981931  9478 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:13.974365  9632 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260808 01:52:13.983170  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:13.984104  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 44183.3us
I20260808 01:52:13.985666  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:14.003700  9476 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:52:13.999238  9475 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
W20260808 01:52:13.977830  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 217399us
I20260808 01:52:13.971062 15928 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:14.007296  9521 leader_election.cc:400] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19.
I20260808 01:52:14.007977  9521 leader_election.cc:403] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260808 01:52:13.979377 16039 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:14.010963  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:14.012115  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:13.980010 15986 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:14.013399  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:14.014793  9601 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:14.015204  9633 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:14.015726 15729 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:52:14.016482 15729 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:14.016886 15729 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:14.017287 15729 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:14.019071  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:14.023273 16090 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:52:14.008102 16084 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:14.026849 15347 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.015s
I20260808 01:52:13.971797 16016 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.051s
I20260808 01:52:14.031454 10368 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260808 01:52:14.032235  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:13.978214  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 2f740cd3515641a8b2c4add8f809dd82 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:14.033576  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:14.034265  9476 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:14.035213 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:14.037304 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 121420us
I20260808 01:52:14.012145  9445 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:14.039712 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:14.040277  9629 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:52:14.041765  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:14.038626  9445 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:14.046257  9521 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:14.048334  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 280619us
I20260808 01:52:14.049319  9523 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:14.050205  9632 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3759209338e14437aff5f92a2fff36da"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:14.050925  9632 raft_consensus.cc:606] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:14.051373  9632 raft_consensus.cc:614] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:14.052717  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:14.054893  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:14.058867 16090 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:52:14.059731 15788 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:14.060431 15636 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260808 01:52:14.061198  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 50380us
I20260808 01:52:14.062291 15788 raft_consensus.cc:697] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:14.045893 15778 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:52:14.063174  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:14.063452 15778 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:14.064045 15778 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:14.063403 15788 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:14.064658  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:13.979164 16060 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:14.067651  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:14.068099  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:14.069657  9599 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:14.068342  9478 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:14.071676 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:14.047706 16101 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260808 01:52:14.039952 16029 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:14.077641  9599 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:14.077682  9551 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:52:14.079375  9472 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:14.080443  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:14.050420 15928 raft_consensus.cc:3055] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:52:14.046290 15877 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:14.013260 15862 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.060s
I20260808 01:52:14.001770 15876 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:14.083740  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:14.093969  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:14.094596  9631 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:14.092392  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:14.053099 15347 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19.
I20260808 01:52:14.087718  9560 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260808 01:52:14.066552 15910 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:14.084553 16083 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:14.118171  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:14.088673  9599 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:14.121706 14784 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:52:14.122246 14784 raft_consensus.cc:697] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:14.123227 14784 consensus_queue.cc:237] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:14.123901  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:14.087178 15933 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:14.078529 15636 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20260808 01:52:14.120889  9631 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:52:14.085379 15928 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:14.132067  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:14.100135 16078 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:14.134920  9472 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:14.133286 10332 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:14.140671  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:14.099247 16108 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:14.142432 16109 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:14.143961 15908 raft_consensus.cc:2764] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260808 01:52:14.138016 16102 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:14.145735 16104 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:14.081426 15923 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20260808 01:52:14.107704 16090 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:14.153167 10363 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:14.116990 15347 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:52:14.119252 16083 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:14.175700  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 120930us
W20260808 01:52:14.177038  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 127817us
I20260808 01:52:14.180550  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:14.124393 15844 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:14.185191  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:14.165668 16108 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:14.085538 16039 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20260808 01:52:14.129483 15933 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:14.101368 15910 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:14.167096 16078 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:14.134928 16092 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.056s
I20260808 01:52:14.133389 16084 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:14.146996 15972 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260808 01:52:14.205206 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:14.189882 15605 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:14.212001  9551 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260808 01:52:14.232319  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:14.212102  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:14.233785  9556 raft_consensus.cc:2393] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:14.233667  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:14.202261 16108 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:14.235760  9552 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:14.237049  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:14.237812  9555 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:14.238600  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:14.239985  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:14.219923  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:14.198303 16083 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:14.192679 16109 raft_consensus.cc:697] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:14.199463 15347 raft_consensus.cc:697] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:14.224023  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:14.224138  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:14.225153  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 101393us
I20260808 01:52:14.225581 16084 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:14.191381 16104 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:14.241572  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:14.222795  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:14.242456  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:14.222574 10367 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:14.206077 15933 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:14.221839  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:14.267714  9558 raft_consensus.cc:2368] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 16. Current term is 17.
I20260808 01:52:14.253652  9637 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260808 01:52:14.255358  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:14.268885  9638 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:14.258229  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:14.260879 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:14.262600  9555 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:14.222263 16070 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:14.283747  9601 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:14.270444  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:14.285326  9552 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:14.271930  9476 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 17: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:14.272445  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:14.274973 15996 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:52:14.275918 15910 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:14.277803 15862 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:14.278769 16078 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:14.279786 15876 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:14.280254 15347 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:14.188828 15928 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:14.281244 16052 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:52:14.289510  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:14.289686 15844 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:14.291606  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:14.306056  9476 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:14.295213  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:14.301789  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:14.308341  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:14.291628 16104 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:14.266741 16103 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260808 01:52:14.316993  9601 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:14.312137  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:14.351661 14784 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260808 01:52:14.333196  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 286924us
I20260808 01:52:14.311681  9552 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 14. Current term is 15. Ops: []
W20260808 01:52:14.355494  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47270.1us
I20260808 01:52:14.298017 16084 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:14.352713  9524 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:14.356884  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:14.360697  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:14.362103  9475 raft_consensus.cc:2410] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260808 01:52:14.303190 16052 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:14.315423 16107 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.197s	user 0.001s	sys 0.000s
I20260808 01:52:14.263510 16115 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.056s
I20260808 01:52:14.309929 16060 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:14.364440  9524 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:14.315083 15876 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:14.299966 15778 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:14.368280  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:14.378788  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:14.379750  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:14.387706  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:14.371531 16109 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:14.391866 15908 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:14.392369 15908 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:14.392768 15908 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:14.394922 15778 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:14.395713 15778 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:14.396140 15778 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:14.396592 15778 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:14.399351  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:14.400187  9560 raft_consensus.cc:2393] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:14.400326  9559 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260808 01:52:14.401433  9560 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260808 01:52:14.401872  9559 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260808 01:52:14.403326  9557 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260808 01:52:14.404610  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:14.405021 16117 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260808 01:52:14.405360  9551 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:14.297206 16070 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:14.312600 16039 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:14.382465  9524 leader_election.cc:400] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17.
I20260808 01:52:14.413803  9524 leader_election.cc:403] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260808 01:52:14.361572 16077 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:52:14.383976 14784 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:14.316852 16110 consensus_queue.cc:1175] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "259bcb3f0d254e8b9effac3277e654e0" responder_term: 12 status { last_received { term: 12 index: 8 } last_committed_idx: 7 last_received_current_leader { term: 12 index: 8 } }
I20260808 01:52:14.316722 15972 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.066s
I20260808 01:52:14.315783 15862 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.067s
I20260808 01:52:14.372114 16104 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:14.412442  9445 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:14.384487 16090 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:52:14.416307  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:14.439339  9555 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 20: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:14.412379  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:14.440999  9552 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 17: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:14.417228  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:14.442229  9475 raft_consensus.cc:2410] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 18 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
I20260808 01:52:14.376744 16052 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:52:14.420678  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 121824us
I20260808 01:52:14.416726 14784 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:14.424139 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:14.402537  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:14.426606  9476 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:14.445489  9560 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:52:14.429170 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:14.446471  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:14.377835 15928 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:14.433944 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:14.419618 15856 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
I20260808 01:52:14.448451 15605 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:14.448535 16125 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:52:14.449735 14784 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:14.450810  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 68661.4us
I20260808 01:52:14.461787  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:14.462944  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:14.464087  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:14.451645 16102 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17.
I20260808 01:52:14.447098 10368 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:14.452839  9476 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:14.452903  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:14.464835 14784 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:14.464022  9472 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:14.427451 15972 consensus_queue.cc:1243] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260808 01:52:14.472529  9476 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260808 01:52:14.439760 15910 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:14.427742 16104 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:14.471064  9475 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260808 01:52:14.475374 10332 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:14.477404  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:14.501284  9472 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:14.419837 16039 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:14.497198 16017 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.007s	sys 0.012s
I20260808 01:52:14.445086 16108 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:14.472363 15605 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:14.488746 16104 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:14.495786 16060 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:14.519819 15844 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:14.515297 15845 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:14.539856  9556 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260808 01:52:14.528985  9521 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:14.530738  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 141834us
I20260808 01:52:14.534190 16102 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:14.535403 16058 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260808 01:52:14.520124 15729 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:14.520737 16017 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.008s	sys 0.012s
I20260808 01:52:14.503113 15910 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:14.541484  9477 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260808 01:52:14.544734  9521 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:14.553365  9472 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260808 01:52:14.536391 16060 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:14.558142  9630 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:14.549041  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:14.550271 15908 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:14.566785 15908 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:14.566999 15605 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:14.573961 16077 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:14.538311 15862 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
I20260808 01:52:14.575825  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:14.578999 16077 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:14.579622 16077 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:14.580086 16077 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:14.583284 15910 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:14.546736 16115 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:14.601841 15908 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:14.604316 15908 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:14.606737 15908 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:14.610769 16108 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:14.563860 16060 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:14.609465 15933 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:14.615859 15862 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:14.616412 15862 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:14.617317 15862 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:14.618346  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:14.615933  9631 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:14.620079  9557 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:14.619942 15876 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:14.620934  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:14.621910  9631 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 20: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:52:14.623896 16100 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.675s	user 0.003s	sys 0.000s
I20260808 01:52:14.602494 16125 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:14.610769  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:14.625039  9634 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:14.625607  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2967) took 1118 ms. Trace:
I20260808 01:52:14.626021  9474 rpcz_store.cc:276] 0808 01:52:13.506929 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:13.523162 (+ 16233us) service_pool.cc:224] Handling call
0808 01:52:13.560756 (+ 37594us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:13.685718 (+124962us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:14.625345 (+939627us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54001013d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:14.625540 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":266077952}
I20260808 01:52:14.627064  9631 pending_rounds.cc:85] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 7
I20260808 01:52:14.627090  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2972) took 1075 ms. Trace:
I20260808 01:52:14.627698  9631 pending_rounds.cc:107] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 12.8
W20260808 01:52:14.628515  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 83894.8us
I20260808 01:52:14.628098  9631 raft_consensus.cc:2889] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:52:14.629572  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:14.626392  9634 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260808 01:52:14.631834  9521 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:14.601069  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:14.627800  9473 rpcz_store.cc:276] 0808 01:52:13.553146 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:13.561747 (+  8601us) service_pool.cc:224] Handling call
0808 01:52:13.600444 (+ 38697us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:13.615682 (+ 15238us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:13.615726 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:13.640628 (+ 24902us) log.cc:844] Serialized 26 byte log entry
0808 01:52:13.803103 (+162475us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:13.803152 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:13.803352 (+   200us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:14.623199 (+819847us) spinlock_profiling.cc:243] Waited 8.45 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:14.626754 (+  3555us) raft_consensus.cc:1685] finished
0808 01:52:14.626777 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:14.627026 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":341,"raft.queue_time_us":2131,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":3652,"spinlock_wait_cycles":19446016,"thread_start_us":108683,"threads_started":1,"wal-append.queue_time_us":222047}
W20260808 01:52:14.639652  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:14.646628 15729 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:14.647495  9480 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 11. Current term is 12. Ops: [11.11-11.11]
I20260808 01:52:14.547643 16090 consensus_queue.cc:1059] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.085s
I20260808 01:52:14.649847  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:14.650431  9479 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260808 01:52:14.650933  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:14.651597  9557 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:14.651849  9479 raft_consensus.cc:3055] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:14.652292  9479 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:14.657680  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 56718.2us
I20260808 01:52:14.628118 16131 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:14.654663  9479 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:14.663293  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:14.668179  9634 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:14.587589 16076 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.070s
I20260808 01:52:14.635932 15862 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:14.592612 15778 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:14.676396 15908 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:14.676862 10360 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:14.686332  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:14.693945 14784 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:14.630560 16092 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260808 01:52:14.696497  9523 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:14.661268 16115 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:14.670360 16058 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:14.698786 16109 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:52:14.703926  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 72160.5us
I20260808 01:52:14.716315  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:14.703156 15667 consensus_queue.cc:1059] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:14.705816  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:14.679813 16090 consensus_queue.cc:1243] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 18.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260808 01:52:14.678817 16103 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:14.694588 14784 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:14.732122  9521 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:14.706979  9555 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260808 01:52:14.659984 15729 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20260808 01:52:14.740916  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:14.741166 16039 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:14.710374 15933 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:14.741000 16109 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:14.742472 16115 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:14.677816 16131 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:14.667989 16046 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.047s
I20260808 01:52:14.751366 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:14.774693 10367 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:14.772749 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:52:14.775681  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 115871us
I20260808 01:52:14.776101 10362 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 20
I20260808 01:52:14.777261  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:14.778520 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:14.785620  9444 leader_election.cc:400] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 16. Current term is 17.
I20260808 01:52:14.749464 16028 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:14.797521  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:14.701205 16052 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:14.798779 14784 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:14.756208 16108 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:14.778425 15709 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:14.810102  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:14.720350 16129 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:14.804728  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:14.791040 16039 raft_consensus.cc:3055] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:52:14.819393  9559 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260808 01:52:14.819572 14784 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:14.825546 16131 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:14.802980 16108 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:14.827585 15778 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:14.828044 14784 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:14.827594 16108 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:14.829078 16102 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:14.831069 16102 raft_consensus.cc:697] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:14.832106 16102 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:14.833179 16052 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:14.834106  9638 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13. Ops: [12.12-12.12]
I20260808 01:52:14.835000  9474 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 13: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:14.836134  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:14.837476  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:14.828459 15778 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:52:14.839682 16090 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:14.809765 15709 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:14.818641 16039 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:14.791589 16058 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.052s
I20260808 01:52:14.843938 16139 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:14.844646  9474 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260808 01:52:14.845767  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:14.809648 15845 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 14.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260808 01:52:14.865481 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:52:14.866495  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 69106.1us
I20260808 01:52:14.867690  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:14.847088  9558 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:52:14.851119  9557 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:14.851313 14784 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:14.787920 15876 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
I20260808 01:52:14.799621 16028 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:14.800344 15347 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.067s
I20260808 01:52:14.852469 15709 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:14.856407 16076 consensus_queue.cc:1243] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 17.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260808 01:52:14.855644 16058 consensus_queue.cc:1243] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 16.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260808 01:52:14.896288 15347 consensus_queue.cc:1243] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 13.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260808 01:52:14.869103 16139 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:14.867403 16125 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:14.900063 10362 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:14.872340 16141 raft_consensus.cc:2764] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
W20260808 01:52:14.907670 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 124708us
W20260808 01:52:14.911491  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 110756us
I20260808 01:52:14.875609 15933 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.035s
W20260808 01:52:14.966936  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 59112.6us
I20260808 01:52:14.972256  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:14.899575 16151 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:14.987144  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:14.909731 16090 consensus_queue.cc:1059] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.076s
I20260808 01:52:14.995575 16131 consensus_queue.cc:1059] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:14.912859 16145 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.087s
I20260808 01:52:14.967378 16104 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:14.976543 15910 consensus_queue.cc:1059] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.024s
I20260808 01:52:15.039261  9472 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:15.039800  9472 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:15.041182 16144 log.cc:927] Time spent T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Append to log took a long time: real 0.368s	user 0.000s	sys 0.001s
I20260808 01:52:15.048072 16109 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:14.995051 16076 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260808 01:52:15.015028 16142 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260808 01:52:15.015837  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:15.017437  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:15.018980  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:15.023538  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:15.072858  9635 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:15.025576  9560 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260808 01:52:15.028708 16137 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.160s	user 0.000s	sys 0.001s
I20260808 01:52:15.031139 16146 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.122s
W20260808 01:52:15.080099 16148 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260808 01:52:15.080268 15636 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:15.031970  9634 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:15.080888 15636 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.084173 15636 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:15.014869 15986 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.967s	user 0.015s	sys 0.001s
I20260808 01:52:15.088018 15986 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:14.912636 16039 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:15.002323 16060 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:15.012629 16151 consensus_queue.cc:1059] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 15.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.080s
I20260808 01:52:15.062373 15972 raft_consensus.cc:3055] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:15.063692 16104 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:15.066140 16131 raft_consensus.cc:3055] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Stepping down as leader of term 15
W20260808 01:52:15.067534 16083 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.802s	user 0.010s	sys 0.005s
I20260808 01:52:15.068778  9478 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260808 01:52:15.077668  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:15.078114  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:15.078219 16139 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:15.078585  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:15.079598  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:15.096926  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:15.097332  9635 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:15.096505  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:15.100934  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:15.100005  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:15.140020  9559 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:15.098573  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:15.139518  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:15.090451 15986 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:15.060374 16076 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:15.122293  9634 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
W20260808 01:52:15.090655 16077 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.510s	user 0.009s	sys 0.001s
I20260808 01:52:15.150369  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 793) took 1080 ms. Trace:
I20260808 01:52:15.092005 16039 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:15.093451 15636 raft_consensus.cc:3055] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Stepping down as leader of term 15
W20260808 01:52:15.137739 16078 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.016s	sys 0.010s
I20260808 01:52:15.151664  9633 rpcz_store.cc:276] 0808 01:52:14.070341 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:14.070932 (+   591us) service_pool.cc:224] Handling call
0808 01:52:14.071286 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:14.093140 (+ 21854us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:14.093191 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:14.101776 (+  8585us) log.cc:844] Serialized 26 byte log entry
0808 01:52:14.183127 (+ 81351us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:52:14.183192 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:15.142989 (+959797us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b54000a9bd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:15.143048 (+    59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:15.143102 (+    54us) raft_consensus.cc:1685] finished
0808 01:52:15.143117 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:15.150262 (+  7145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5493,"raft.queue_time_us":665,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":8314,"spinlock_wait_cycles":49210240,"thread_start_us":53120,"threads_started":1,"wal-append.queue_time_us":132806}
I20260808 01:52:15.094008 15972 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:15.094159 16028 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:15.163544  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:15.093802 16060 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:15.175473  9633 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:15.095613 16131 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:15.177699 16145 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260808 01:52:15.179105  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 79212.2us
I20260808 01:52:15.095880 16083 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.190073  9636 raft_consensus.cc:2368] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18.
I20260808 01:52:15.191341 16131 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.163508 16077 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.192683 16083 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:15.101214 16108 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:15.165504 16078 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.017s	sys 0.010s
I20260808 01:52:15.195546 10364 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260808 01:52:15.197508 15972 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:15.199388 15972 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:15.151723 16076 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:15.203877  9637 pending_rounds.cc:85] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 5
I20260808 01:52:15.206574  9637 pending_rounds.cc:107] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260808 01:52:15.207145  9637 raft_consensus.cc:2889] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:15.100329 16152 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.076s
W20260808 01:52:15.219754 16162 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
I20260808 01:52:15.222249  9556 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260808 01:52:15.173534 16164 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:15.194463 16077 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:15.224596 16164 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:15.227197  9633 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:15.223899  9556 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:15.165724 15636 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:15.240267 15636 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.7, Last appended by leader: 7, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.242115 15636 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:15.246510  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 795) took 1175 ms. Trace:
I20260808 01:52:15.248121 16164 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:15.247363  9629 rpcz_store.cc:276] 0808 01:52:14.071437 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:14.109320 (+ 37883us) service_pool.cc:224] Handling call
0808 01:52:14.121630 (+ 12310us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:14.121989 (+   359us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:14.122022 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:14.122571 (+   549us) log.cc:844] Serialized 26 byte log entry
0808 01:52:14.393987 (+271416us) spinlock_profiling.cc:243] Waited 6.33 ms on lock 0x7b180048d720. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4453a6ac 00007f5d4186fe3b 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1
0808 01:52:14.394169 (+   182us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:14.394210 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:14.394283 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:14.396799 (+  2516us) spinlock_profiling.cc:243] Waited 2.05 ms on lock 0x7b48002379d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:52:15.212009 (+815210us) spinlock_profiling.cc:243] Waited 7.18 ms on lock 0x7b180048d720. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:52:15.246212 (+ 34203us) raft_consensus.cc:1685] finished
0808 01:52:15.246239 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:15.246433 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1891,"raft.queue_time_us":8704,"raft.run_cpu_time_us":1675,"raft.run_wall_time_us":260632,"spinlock_wait_cycles":35793280,"thread_start_us":17757,"threads_started":1,"wal-append.queue_time_us":18208}
I20260808 01:52:15.252108  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2710) took 1428 ms. Trace:
I20260808 01:52:15.252601  9630 rpcz_store.cc:276] 0808 01:52:13.823772 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:13.916649 (+ 92877us) service_pool.cc:224] Handling call
0808 01:52:13.934696 (+ 18047us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:14.563606 (+628910us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:15.251808 (+688202us) spinlock_profiling.cc:243] Waited 623 ms on lock 0x7b54003faf54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:15.252032 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1433250560}
I20260808 01:52:15.254202  9480 raft_consensus.cc:3055] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:15.254647  9480 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:15.255563  9480 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.263311  9551 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
W20260808 01:52:15.264853  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 76027.6us
I20260808 01:52:15.265751  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:15.277932 15709 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:15.280236  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 3759209338e14437aff5f92a2fff36da (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:15.279635  9559 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:15.283382  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:15.288935  9632 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:15.285094  9635 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:15.289712  9632 raft_consensus.cc:606] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:15.282439 15709 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:15.286265  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 906) took 1605 ms. Trace:
I20260808 01:52:15.280987 10364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 808) took 1071 ms. Trace:
I20260808 01:52:15.292359 10364 rpcz_store.cc:276] 0808 01:52:14.209458 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:14.209645 (+   187us) service_pool.cc:224] Handling call
0808 01:52:14.210006 (+   361us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:15.195956 (+985950us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:15.280660 (+ 84704us) spinlock_profiling.cc:243] Waited 985 ms on lock 0x7b54005550d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:15.280881 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2265822592}
I20260808 01:52:15.291801  9554 rpcz_store.cc:276] 0808 01:52:13.680458 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:13.680656 (+   198us) service_pool.cc:224] Handling call
0808 01:52:13.684636 (+  3980us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:13.812309 (+127673us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:13.812346 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:13.877774 (+ 65428us) log.cc:844] Serialized 52 byte log entry
0808 01:52:14.181941 (+304167us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:14.197416 (+ 15475us) log.cc:844] Serialized 14 byte log entry
0808 01:52:14.203737 (+  6321us) log.cc:844] Serialized 14 byte log entry
0808 01:52:14.203938 (+   201us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:14.204010 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:14.399508 (+195498us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b18001321c0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d498eea98 00007f5d498e6071 00007f5d498e6017 00007f5d499dd098 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318
0808 01:52:15.196822 (+797314us) spinlock_profiling.cc:243] Waited 4.79 ms on lock 0x7b54001598a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490919e 00007f5d44913d10 00007f5d44913cb8 00007f5d44913c38 00007f5d44913bf3 00007f5d44912e7c 00007f5d4a837996 00007f5d4a832148 00007f5d4494fbab 00007f5d449526a0
0808 01:52:15.199400 (+  2578us) raft_consensus.cc:1685] finished
0808 01:52:15.199424 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:15.286076 (+ 86652us) spinlock_profiling.cc:243] Waited 3.58 ms on lock 0x7b540029e0d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:15.286202 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33539,"prepare.queue_time_us":60076,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":85,"raft.queue_time_us":74650,"raft.run_cpu_time_us":2575,"raft.run_wall_time_us":208030,"spinlock_wait_cycles":124650240,"thread_start_us":353760,"threads_started":2,"wal-append.queue_time_us":251583}
I20260808 01:52:15.200208 16108 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:15.314563 15667 consensus_queue.cc:1059] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:15.315874 15667 raft_consensus.cc:3055] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:15.316318 15667 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:15.317327 15667 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:15.319424 15667 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:15.322861 16145 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:15.323460 16145 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:15.323796 16145 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:15.329284 16131 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:15.333122 16070 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:15.334600 15972 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:15.338397  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:15.341857  9631 raft_consensus.cc:2410] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 12 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260808 01:52:15.340427 15972 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:15.345041 15972 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:15.308756 16151 consensus_queue.cc:1059] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260808 01:52:15.355742 16151 consensus_queue.cc:1243] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 14.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
W20260808 01:52:15.358364 16140 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.686s	user 0.006s	sys 0.000s
I20260808 01:52:15.360682  9481 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260808 01:52:15.361838 16109 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260808 01:52:15.366156 16138 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.693s	user 0.000s	sys 0.004s
I20260808 01:52:15.366662  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3072) took 1438 ms. Trace:
I20260808 01:52:15.368271  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:15.367146  9477 rpcz_store.cc:276] 0808 01:52:13.928573 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:14.555809 (+627236us) service_pool.cc:224] Handling call
0808 01:52:14.556204 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:14.560941 (+  4737us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:14.561000 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:14.586919 (+ 25919us) log.cc:844] Serialized 26 byte log entry
0808 01:52:14.662387 (+ 75468us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:52:14.683964 (+ 21577us) log.cc:844] Serialized 14 byte log entry
0808 01:52:14.854072 (+170108us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:14.854163 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:15.320603 (+466440us) spinlock_profiling.cc:243] Waited 9.09 ms on lock 0x7b1800351760. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4185ac56 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb
0808 01:52:15.322218 (+  1615us) spinlock_profiling.cc:243] Waited 33.3 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:15.366419 (+ 44201us) raft_consensus.cc:1685] finished
0808 01:52:15.366442 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:15.366595 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13874,"prepare.queue_time_us":21242,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":6405,"raft.queue_time_us":152214,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":224,"spinlock_wait_cycles":97434880,"thread_start_us":228398,"threads_started":2,"wal-append.queue_time_us":84668}
I20260808 01:52:15.378324  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:15.379297  9632 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:15.380934  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:15.381923  9633 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:15.387869  9472 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.384341  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:15.395363  9472 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:15.399401  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:15.406138  9635 raft_consensus.cc:1217] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Deduplicated request from leader. Original: 13.8->[15.9-17.10]   Dedup: 15.9->[17.10-17.10]
I20260808 01:52:15.406952  9635 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: [15.9-17.10]
I20260808 01:52:15.397392 15856 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:15.414850 16152 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:15.420022 15856 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:15.420161 16152 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.417307  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:15.422293  9550 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
W20260808 01:52:15.422456  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:15.429242  9632 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:15.433305  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:15.435114  9472 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 9. (term mismatch)
I20260808 01:52:15.435770  9472 pending_rounds.cc:85] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 8
I20260808 01:52:15.436097  9472 pending_rounds.cc:107] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 16.9
I20260808 01:52:15.436488  9472 raft_consensus.cc:2889] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:15.446419 10364 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4f04078e3beb42eda250da45f7485537"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:57156
I20260808 01:52:15.447695 10364 raft_consensus.cc:606] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:15.418486 16124 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:15.411787 16168 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260808 01:52:15.455524 16092 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20260808 01:52:15.417116 16146 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:15.460783  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:15.462723  9599 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:15.405645 16076 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.468582  9477 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 17. Current term is 18. Ops: [15.10-17.11]
I20260808 01:52:15.470232  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:15.457705 16168 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:15.476647 16124 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:52:15.501988  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:15.502025 16156 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:15.490489 16083 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:15.467970 15923 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260808 01:52:15.477124 15709 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
I20260808 01:52:15.477020 16114 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20260808 01:52:15.465761 16145 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.485033 16017 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.028s
I20260808 01:52:15.493244 16168 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:15.528209 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:15.469785 16146 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.484375 16043 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.030s
I20260808 01:52:15.529079  9632 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7c508218361c446898fede7c5682fa9e"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:15.487780 16108 consensus_queue.cc:1059] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260808 01:52:15.533653 10368 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260808 01:52:15.534657 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:15.536685  9632 raft_consensus.cc:606] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:15.537971  9632 raft_consensus.cc:614] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:15.540709 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:15.515769 16053 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:15.544641 16152 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:15.545167 15856 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:15.538726 16102 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:15.550020 16102 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:15.550788  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 7c508218361c446898fede7c5682fa9e (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:15.551286 16102 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.552663 16102 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:15.555099 16108 raft_consensus.cc:3055] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:15.555601 16108 raft_consensus.cc:740] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:15.556463 16108 consensus_queue.cc:260] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.557883 16108 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:15.532418 16083 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:15.568032 16146 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:15.569571  9632 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:15.567684 16083 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.571936 16076 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:15.590035  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:15.590647 16145 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:15.591727  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:15.592365  9473 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:15.543238 16020 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Stepping down as leader of term 15
W20260808 01:52:15.613564  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:15.632527  9473 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:15.599892 16020 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:15.570319 16053 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:15.638478  9636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2726) took 1406 ms (client timeout 1770 ms). Trace:
W20260808 01:52:15.659981  9636 rpcz_store.cc:269] 0808 01:52:14.231764 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:14.239679 (+  7915us) service_pool.cc:224] Handling call
0808 01:52:15.636630 (+1396951us) spinlock_profiling.cc:243] Waited 949 ms on lock 0x7b540034fb54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:15.638383 (+  1753us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2183124352}
I20260808 01:52:15.639742  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:15.659176  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:15.662143  9473 raft_consensus.cc:2410] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
W20260808 01:52:15.639264 16169 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
I20260808 01:52:15.708070  9473 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:15.603739 16124 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:15.726052  9550 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:15.729146  9473 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260808 01:52:15.739840  9479 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:15.761444  9479 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260808 01:52:15.762969  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3082) took 1330 ms. Trace:
I20260808 01:52:15.763192  9477 raft_consensus.cc:2410] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260808 01:52:15.763538  9479 rpcz_store.cc:276] 0808 01:52:14.432437 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:14.651422 (+218985us) service_pool.cc:224] Handling call
0808 01:52:14.651717 (+   295us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:15.738993 (+1087276us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b58007c40d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d448ef48a 00007f5d44971827 00007f5d4498a3c7 00007f5d4498bbbe 00007f5d44995f29 00007f5d449967f6 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34
0808 01:52:15.762594 (+ 23601us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:15.762905 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":243590528}
I20260808 01:52:15.769016 16103 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:15.778410 16103 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:15.780429  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:15.782124  9558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2559) took 1512 ms (client timeout 1754 ms). Trace:
W20260808 01:52:15.782702  9558 rpcz_store.cc:269] 0808 01:52:14.270030 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:14.270194 (+   164us) service_pool.cc:224] Handling call
0808 01:52:15.781865 (+1511671us) spinlock_profiling.cc:243] Waited 538 ms on lock 0x7b540002d0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:15.782026 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1238416768}
W20260808 01:52:15.784770  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 351548us
I20260808 01:52:15.785897  9478 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260808 01:52:15.789546 15636 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:15.790602 16156 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:15.791815 16156 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.795909 16168 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.798666 16152 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:15.799825 16152 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.801558  9473 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260808 01:52:15.801335 16131 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
W20260808 01:52:15.798868  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 258893us
I20260808 01:52:15.806746  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3070) took 1916 ms. Trace:
I20260808 01:52:15.807011  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:15.807261  9476 rpcz_store.cc:276] 0808 01:52:13.890307 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:14.487645 (+597338us) service_pool.cc:224] Handling call
0808 01:52:14.488653 (+  1008us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:14.530314 (+ 41661us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:52:14.547211 (+ 16897us) log.cc:844] Serialized 14 byte log entry
0808 01:52:15.351521 (+804310us) spinlock_profiling.cc:243] Waited 996 us on lock 0x7b180043f480. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d4453242a 00007f5d4499de0a
0808 01:52:15.793250 (+441729us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:15.793923 (+   673us) log.cc:844] Serialized 52 byte log entry
0808 01:52:15.794127 (+   204us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:15.794375 (+   248us) log.cc:844] Serialized 14 byte log entry
0808 01:52:15.794577 (+   202us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:15.794694 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:15.806459 (+ 11765us) raft_consensus.cc:1685] finished
0808 01:52:15.806489 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:15.806639 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":322507,"prepare.queue_time_us":22508,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":106,"raft.queue_time_us":339270,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":9865,"spinlock_wait_cycles":2291072,"thread_start_us":114881,"threads_started":1,"wal-append.queue_time_us":952395}
I20260808 01:52:15.810103 16157 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:15.810647 16157 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:15.811829 16053 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:15.811662 16157 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.6, Last appended by leader: 6, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.810047  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:15.828136  9476 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 12: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:15.830838  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:15.836702  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 48349.2us
I20260808 01:52:15.777441 16029 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:15.815056 16162 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260808 01:52:15.866449  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:15.867835  9476 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:15.840775 10360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 733) took 1397 ms. Trace:
I20260808 01:52:15.831022 15709 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260808 01:52:15.825414 16152 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:15.825630 15636 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:15.869896  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:15.886904  9632 raft_consensus.cc:606] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:15.884747  9599 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:52:15.854544 16175 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260808 01:52:15.880059  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:15.889226  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:15.889674  9631 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:52:15.881783  9633 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260808 01:52:15.868865 10360 rpcz_store.cc:276] 0808 01:52:14.443599 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:14.443815 (+   216us) service_pool.cc:224] Handling call
0808 01:52:14.448265 (+  4450us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:14.689600 (+241335us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:15.840507 (+1150907us) spinlock_profiling.cc:243] Waited 228 ms on lock 0x7b5400646c54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:15.840707 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":525120896}
I20260808 01:52:15.880491  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:15.753769 16020 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.883912  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:15.855010 15667 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:15.893160  9637 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260808 01:52:15.894654  9476 raft_consensus.cc:2410] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 17 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 6.
I20260808 01:52:15.883626 16194 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:15.840547 16110 consensus_queue.cc:1059] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.044s
I20260808 01:52:15.869405 16029 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.856714 15862 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.034s
W20260808 01:52:15.936435  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:15.936635 16077 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:15.937143 16077 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:15.940688 16168 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:15.926169 15667 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:15.943193  9634 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2734) took 1278 ms (client timeout 1603 ms). Trace:
W20260808 01:52:15.943686  9634 rpcz_store.cc:269] 0808 01:52:14.664439 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:14.681620 (+ 17181us) service_pool.cc:224] Handling call
0808 01:52:15.943001 (+1261381us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b54003a0054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:15.943127 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":722314496}
I20260808 01:52:15.917758  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:15.948428  9630 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:52:15.931074 16017 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:15.918372  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:15.841639 16114 consensus_queue.cc:1059] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.050s
I20260808 01:52:15.934041  9480 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:15.929494 16194 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.917912  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:15.950086  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:15.951000  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:15.975696  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:15.976605  9476 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 18: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:15.952879  9633 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: [13.11-15.12]
I20260808 01:52:15.954241  9636 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
W20260808 01:52:15.955399 16186 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260808 01:52:15.954998 16017 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:15.955917 16191 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:15.956223  9635 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: []
W20260808 01:52:15.991376  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:15.965853 16184 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.192s	user 0.000s	sys 0.002s
W20260808 01:52:16.090401  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:16.151453  9635 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260808 01:52:15.938166 16077 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:15.887483 16103 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:15.891726 16152 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:52:15.977803  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:16.177603  9633 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:15.980019  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:15.981874  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:15.963919 16174 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260808 01:52:15.984638  9630 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
W20260808 01:52:16.156538  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:16.095919 15923 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:16.096239 16125 raft_consensus.cc:3055] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:16.096511 16164 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:16.174440 16183 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.179s	user 0.001s	sys 0.000s
W20260808 01:52:16.230500  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:16.196267  9629 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 17. Current term is 18. Ops: [15.10-17.11]
I20260808 01:52:16.183257 16109 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:16.228602 16141 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:16.236426  9630 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:16.234808 16164 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:16.244033 16194 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:16.183718 15925 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:16.174052 16110 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:16.173729 16168 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:16.245762  9630 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260808 01:52:16.234038 16077 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:16.262830  9601 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:16.236006 16197 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260808 01:52:16.342783  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:16.343720  9475 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 18
W20260808 01:52:16.343976  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 81324us
I20260808 01:52:16.347050  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:16.353601  9601 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:16.311888  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:16.335574  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:15.959400 15667 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:16.192118 16125 raft_consensus.cc:740] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:16.389083  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:16.406224  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:16.183384 15636 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:16.195045 15923 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:16.349454 16103 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:16.391737 16194 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:16.287158 15925 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:16.256191 16110 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:16.397962 16152 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:16.345031 16168 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:16.419557 15972 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:16.446911  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1056) took 1399 ms. Trace:
I20260808 01:52:16.474357  9481 rpcz_store.cc:276] 0808 01:52:15.047131 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:15.334317 (+287186us) service_pool.cc:224] Handling call
0808 01:52:15.334766 (+   449us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:15.361245 (+ 26479us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:16.441164 (+1079919us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b54002c1354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:16.446814 (+  5650us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28316544}
I20260808 01:52:16.260411 16141 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:16.489472  9476 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:52:16.491060  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:16.476364 16053 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:52:16.487864 16102 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.935s	user 0.021s	sys 0.002s
I20260808 01:52:16.420023 15667 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:16.419622  9630 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:16.250198 16109 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:16.461292  9633 raft_consensus.cc:2410] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260808 01:52:16.511781  9558 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:16.503844  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:16.479828 16125 consensus_queue.cc:260] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:16.490649 16076 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:16.461746  9480 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.500s	user 0.009s	sys 0.001s
I20260808 01:52:16.434967 16103 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:16.478420 16152 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:16.517652  9633 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:16.483264 15972 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:16.522632  9480 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 10. (term mismatch)
I20260808 01:52:16.531502  9480 pending_rounds.cc:85] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 9
I20260808 01:52:16.532300  9480 pending_rounds.cc:107] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20260808 01:52:16.532750  9480 raft_consensus.cc:2889] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:16.533653  9633 raft_consensus.cc:1217] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Deduplicated request from leader. Original: 11.7->[12.8-16.10]   Dedup: 12.8->[14.9-16.10]
I20260808 01:52:16.534543  9633 pending_rounds.cc:85] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 8
I20260808 01:52:16.534987  9633 pending_rounds.cc:107] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260808 01:52:16.513520 16102 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:16.542073  9476 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:52:16.553582 16114 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:16.554383 16207 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:16.554334 16114 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:16.555086 16207 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:16.557268 16207 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:16.514865 16017 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:16.544081  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3079) took 2299 ms. Trace:
I20260808 01:52:16.562105  9480 rpcz_store.cc:276] 0808 01:52:14.244806 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:14.648729 (+403923us) service_pool.cc:224] Handling call
0808 01:52:14.649114 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:15.254088 (+604974us) spinlock_profiling.cc:243] Waited 83.6 ms on lock 0x7b540007e7a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490b1a0 00007f5d449955a0 00007f5d44996611 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c
0808 01:52:15.933191 (+679103us) spinlock_profiling.cc:243] Waited 38 ms on lock 0x7b58007001d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d448ef48a 00007f5d44971827 00007f5d4498a3c7 00007f5d4498bbbe 00007f5d44995f29 00007f5d449967f6 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34
0808 01:52:16.543707 (+610516us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:16.543995 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":279644928}
I20260808 01:52:16.557296 16114 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:16.539974  9475 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:16.538177  9633 raft_consensus.cc:2889] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:16.564330  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:16.565793  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:16.522202 16164 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:16.513958 15667 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:16.536880 16125 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:16.580408  9475 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 15. (term mismatch)
I20260808 01:52:16.580840  9475 pending_rounds.cc:85] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 14
I20260808 01:52:16.581110  9475 pending_rounds.cc:107] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260808 01:52:16.581336  9475 raft_consensus.cc:2889] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:16.582031  9480 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 18: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:16.586980 15856 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:16.587458 16199 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260808 01:52:16.587724 15856 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:16.590082 16198 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
I20260808 01:52:16.590987 15923 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:16.591161 16203 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260808 01:52:16.592514  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3075) took 2640 ms. Trace:
I20260808 01:52:16.592988  9472 rpcz_store.cc:276] 0808 01:52:13.951605 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:14.555884 (+604279us) service_pool.cc:224] Handling call
0808 01:52:14.564709 (+  8825us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:15.039127 (+474418us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b5400193d20. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490b1a0 00007f5d449955a0 00007f5d44996611 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c
0808 01:52:15.387455 (+348328us) spinlock_profiling.cc:243] Waited 42.3 ms on lock 0x7b5400193d20. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4491144f 00007f5d4498a340 00007f5d4498bbbe 00007f5d44995f29 00007f5d449967f6 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2
0808 01:52:15.445261 (+ 57806us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:16.592216 (+1146955us) spinlock_profiling.cc:243] Waited 64.4 ms on lock 0x7b540021d754. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:16.592439 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":879257088}
I20260808 01:52:16.564397 16191 raft_consensus.cc:697] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:16.594929 16103 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:16.595880  9634 pending_rounds.cc:85] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 8
I20260808 01:52:16.596228  9634 pending_rounds.cc:107] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 16.9
I20260808 01:52:16.595641 16191 consensus_queue.cc:237] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:16.596843 16152 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:16.597834 15925 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:16.599057 16168 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:16.519918 16076 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:16.601804 16076 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:16.604185  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:16.521049 16205 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.088s
I20260808 01:52:16.605033 16205 consensus_queue.cc:1243] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 13.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260808 01:52:16.567929 16202 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260808 01:52:16.563652 15788 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 18 current term: 18 is higher.
I20260808 01:52:16.604053 16110 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:16.612699 15667 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:16.614790 16109 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:16.615031 16208 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:16.561354 16102 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:16.615610 16208 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:16.617861 16102 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:16.618173 16208 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:16.590134 15856 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:16.612798 16110 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:16.619920 15972 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:16.619864 16110 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:16.622431 16110 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:16.625114 16108 consensus_meta.cc:287] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.067s	user 0.012s	sys 0.006s
W20260808 01:52:16.625694 16188 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.672s	user 0.001s	sys 0.001s
W20260808 01:52:16.628201 16195 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.670s	user 0.002s	sys 0.003s
I20260808 01:52:16.596501  9634 raft_consensus.cc:2889] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:16.567164 16204 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.072s
I20260808 01:52:16.633883 16204 consensus_queue.cc:1243] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 16.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260808 01:52:16.634210 16053 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:52:16.636437  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3006) took 1161 ms. Trace:
I20260808 01:52:16.636924  9479 rpcz_store.cc:276] 0808 01:52:15.474870 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:15.767299 (+292429us) service_pool.cc:224] Handling call
0808 01:52:15.767749 (+   450us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:15.867196 (+ 99447us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:15.867233 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:15.886976 (+ 19743us) log.cc:844] Serialized 26 byte log entry
0808 01:52:16.014864 (+127888us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:16.014913 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:16.014998 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:16.636183 (+621185us) raft_consensus.cc:1685] finished
0808 01:52:16.636207 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:16.636367 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42752,"raft.queue_time_us":38969,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":5286,"thread_start_us":84157,"threads_started":1,"wal-append.queue_time_us":489082}
I20260808 01:52:16.643882  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3381) took 1063 ms. Trace:
I20260808 01:52:16.644446  9554 rpcz_store.cc:276] 0808 01:52:15.580292 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:15.590800 (+ 10508us) service_pool.cc:224] Handling call
0808 01:52:15.591168 (+   368us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:15.591598 (+   430us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:15.591627 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:15.596723 (+  5096us) log.cc:844] Serialized 52 byte log entry
0808 01:52:15.758259 (+161536us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:15.758326 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:15.758435 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:16.624787 (+866352us) spinlock_profiling.cc:243] Waited 6.5 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:16.643487 (+ 18700us) raft_consensus.cc:1685] finished
0808 01:52:16.643515 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:16.643804 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1211,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":5258,"spinlock_wait_cycles":14950272,"thread_start_us":125604,"threads_started":1,"wal-append.queue_time_us":264634}
I20260808 01:52:16.649446  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 2998) took 1185 ms. Trace:
I20260808 01:52:16.651372  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3004) took 1177 ms. Trace:
I20260808 01:52:16.651813  9473 rpcz_store.cc:276] 0808 01:52:15.474128 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:15.730516 (+256388us) service_pool.cc:224] Handling call
0808 01:52:15.753248 (+ 22732us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:15.802067 (+ 48819us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:16.651070 (+849003us) spinlock_profiling.cc:243] Waited 48 ms on lock 0x7b540019c8d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:16.651306 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":110442880}
I20260808 01:52:16.571650  9481 raft_consensus.cc:2410] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 17 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
W20260808 01:52:16.670135  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:16.650022  9474 rpcz_store.cc:276] 0808 01:52:15.464014 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:15.464181 (+   167us) service_pool.cc:224] Handling call
0808 01:52:15.464554 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:15.569792 (+105238us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:15.569826 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:15.577813 (+  7987us) log.cc:844] Serialized 26 byte log entry
0808 01:52:15.736676 (+158863us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:15.736731 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:16.648878 (+912147us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54004042d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:16.648953 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:16.649021 (+    68us) raft_consensus.cc:1685] finished
0808 01:52:16.649038 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:16.649359 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":241382656,"thread_start_us":124804,"threads_started":1,"wal-append.queue_time_us":254723}
I20260808 01:52:16.680452  9630 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:16.671263 16103 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.024s
I20260808 01:52:16.671525 16152 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:16.696558  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:16.698416  9635 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:16.706717  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:16.709913  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:16.715687  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 326559us
I20260808 01:52:16.717437  9601 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:16.668624 16092 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:16.724506  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 195875us
I20260808 01:52:16.668601 16208 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:52:16.731256  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:16.734293  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:16.735356  9481 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:16.737730  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:16.738663  9481 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:16.743124  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:16.776795  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 92449.6us
I20260808 01:52:16.744096  9474 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
W20260808 01:52:16.781936  9599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.817s	user 0.002s	sys 0.006s
W20260808 01:52:16.782459  9599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.817s	user 0.002s	sys 0.006s
W20260808 01:52:16.782905  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 855582us
I20260808 01:52:16.784071  9481 raft_consensus.cc:3055] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:52:16.784682  9481 raft_consensus.cc:740] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:16.728185 16207 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:16.696573 16152 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:16.798316  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 80975.2us
I20260808 01:52:16.732697 15923 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:16.813179  9521 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:16.792783  9481 consensus_queue.cc:260] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:16.826493  9479 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:16.778712  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:16.832090  9481 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:16.736819 16208 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:16.731701 16092 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:16.835637  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:16.849619  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:16.864748  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:16.867266  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:16.868297  9630 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
W20260808 01:52:16.869091  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:16.892254  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:16.892591  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:16.869191  9601 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:16.896097  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:16.871287 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:16.898516  9630 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:16.901855  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:16.902374 10332 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:16.825687 16210 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:16.912865 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:52:16.913851  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:16.872377 10360 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:16.905709  9558 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:16.906715 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:16.906744  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:16.915756  9550 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:16.912950  9630 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260808 01:52:16.905920  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:16.910104  9556 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 15: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:16.922027 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:16.922797 10367 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEARNER]: Advancing to term 17
I20260808 01:52:16.927982 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:16.938356  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:16.941004  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:16.853540 16108 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:52:16.845526 15876 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:16.821931 16207 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:16.929004 10366 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:52:16.949122  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:16.852255 16194 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:16.893095 16212 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:16.950127 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:16.965628 10368 raft_consensus.cc:2410] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260808 01:52:16.949395  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:16.966473  9555 raft_consensus.cc:2410] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 18 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260808 01:52:16.968763 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:16.971164  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 109584us
I20260808 01:52:16.974647  9556 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 9.
I20260808 01:52:16.976630  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:16.836822 15923 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:17.052702 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:16.906181 15636 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:16.911793 16156 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:16.983996  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:17.072479  9558 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:16.977210 10367 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEARNER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
W20260808 01:52:17.041405  9599 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.194:36095 timed out after 2.130s (SENT) [suppressed 16 similar messages]
I20260808 01:52:17.056792 10366 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260808 01:52:16.894155 16205 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.070s
I20260808 01:52:17.076975 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:17.078084  9523 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:16.953464 16060 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:17.079456  9445 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:17.082261  9523 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:17.079264 15636 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:17.059494 16212 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:17.059177 16209 consensus_queue.cc:1059] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260808 01:52:17.085737 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:16.914130 16204 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.142s
I20260808 01:52:17.086501 15667 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:52:17.087219 16092 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:16.949265 16202 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:17.091077  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:16.915055 16103 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.142s
I20260808 01:52:17.140921  9523 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:17.056394 16220 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:17.074206 15856 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:16.991338 15876 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:17.093573 16060 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:17.106846 16119 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:17.194402  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:17.054286 16194 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:17.137404 16108 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:16.964356 16211 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.202s
I20260808 01:52:17.100939 15925 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 11, Time since last communication: 0.096s
I20260808 01:52:17.202988 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:17.227859  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 392786us
I20260808 01:52:17.095151 16156 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:17.194227 16060 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:16.977912  9550 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:52:17.232049  9630 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260808 01:52:17.230088  9558 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:17.250809  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:17.252184  9558 raft_consensus.cc:2376] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13. Re-sending same reply.
I20260808 01:52:17.198043 16119 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:17.278960  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:17.279402  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 237601us
I20260808 01:52:17.286980  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:17.291657  9481 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260808 01:52:17.292433  9481 pending_rounds.cc:85] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 7
I20260808 01:52:17.292791  9481 pending_rounds.cc:107] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260808 01:52:17.293179  9481 raft_consensus.cc:2889] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:17.107263 16228 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.145s
I20260808 01:52:17.204111 16207 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:17.299726  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:17.232652 15667 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:17.303864  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3089) took 1441 ms. Trace:
W20260808 01:52:17.309512  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 530743us
I20260808 01:52:17.311581  9524 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:17.307806  9481 rpcz_store.cc:276] 0808 01:52:15.862509 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:16.783426 (+920917us) service_pool.cc:224] Handling call
0808 01:52:16.783888 (+   462us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:17.303419 (+519531us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:17.303763 (+   344us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:17.318638  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:17.322997  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:17.325009  9481 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:17.323657  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:17.326129  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:17.167665 16220 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:17.295351 16119 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:17.327284  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:17.345911  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:17.362385  9550 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:17.365150  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:17.325233 16228 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260808 01:52:17.301981 16092 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:17.322266 16110 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.001s	sys 0.016s
I20260808 01:52:17.372897  9481 raft_consensus.cc:2410] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 19 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
W20260808 01:52:17.378135  9475 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.015s	sys 0.023s
I20260808 01:52:17.305181 16131 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:17.387292  9475 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.015s	sys 0.023s
I20260808 01:52:17.392310  9550 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
W20260808 01:52:17.393934  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 113260us
I20260808 01:52:17.395713  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:17.396704  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 253164us
I20260808 01:52:17.365954 16152 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:17.405305  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:17.422197  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:17.371528 16129 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:17.323364 15667 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:17.386240 16110 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.662s	user 0.002s	sys 0.017s
I20260808 01:52:17.373310 16208 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:17.427623  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 109833us
I20260808 01:52:17.360209 16202 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
I20260808 01:52:17.435346  9560 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
W20260808 01:52:17.440388 16235 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20260808 01:52:17.441192 15862 raft_consensus.cc:3055] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:17.428390 15923 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:17.476732  9551 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3359) took 2823 ms (client timeout 2130 ms). Trace:
I20260808 01:52:17.478382  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 796) took 3113 ms. Trace:
W20260808 01:52:17.442159 16231 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
W20260808 01:52:17.443473 16236 log.cc:927] Time spent T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260808 01:52:17.478945  9638 rpcz_store.cc:276] 0808 01:52:14.354112 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:14.627755 (+273643us) service_pool.cc:224] Handling call
0808 01:52:14.628078 (+   323us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:14.834615 (+206537us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:17.467500 (+2632885us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b54004204d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:17.467804 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":473132416}
W20260808 01:52:17.477427  9551 rpcz_store.cc:269] 0808 01:52:14.643844 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:14.643969 (+   125us) service_pool.cc:224] Handling call
0808 01:52:17.467174 (+2823205us) spinlock_profiling.cc:243] Waited 514 ms on lock 0x7b5400325ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:17.467328 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1181162880}
W20260808 01:52:17.447352 16232 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
I20260808 01:52:17.448629 16194 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:17.448966 16213 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.295s	user 0.000s	sys 0.001s
W20260808 01:52:17.447368 16219 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260808 01:52:17.453541 16220 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:17.454872 10366 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260808 01:52:17.456305 15876 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:17.479702 15862 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:17.461028 16156 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:17.423050 16228 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:17.461081  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3010) took 1624 ms. Trace:
I20260808 01:52:17.464712 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:17.464831 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:17.466141 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:17.469018  9600 connection.cc:256] RPC call timeout handler was delayed by -1.38061971400748s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1577s for a response
W20260808 01:52:17.564287  9600 connection.cc:256] RPC call timeout handler was delayed by -1.3401711960032117s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20950000000000002s for a response
I20260808 01:52:17.471218 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:17.390625 16092 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:17.446125 16108 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:17.421576 16131 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:17.473507 16053 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:17.457521 16212 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:17.476814 16083 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:17.485158 16043 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:17.540025 15636 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:17.562527  9475 rpcz_store.cc:276] 0808 01:52:15.836524 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:16.637638 (+801114us) service_pool.cc:224] Handling call
0808 01:52:16.638050 (+   412us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:16.725549 (+ 87499us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:16.725592 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:16.731439 (+  5847us) log.cc:844] Serialized 26 byte log entry
0808 01:52:17.390716 (+659277us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:17.390778 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:17.390883 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:17.460847 (+ 69964us) raft_consensus.cc:1685] finished
0808 01:52:17.460871 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:17.461019 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70067,"raft.queue_time_us":73263,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":7086,"thread_start_us":570967,"threads_started":1,"wal-append.queue_time_us":668733}
I20260808 01:52:17.543989 16200 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:17.588124  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3012) took 1751 ms. Trace:
I20260808 01:52:17.571682  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3093) took 1707 ms. Trace:
I20260808 01:52:17.577909  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:17.588593  9473 rpcz_store.cc:276] 0808 01:52:15.837127 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:16.652861 (+815734us) service_pool.cc:224] Handling call
0808 01:52:16.653321 (+   460us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:16.708694 (+ 55373us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:16.708743 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:16.730113 (+ 21370us) log.cc:844] Serialized 26 byte log entry
0808 01:52:16.942521 (+212408us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:52:16.970845 (+ 28324us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:16.970969 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:17.587876 (+616907us) raft_consensus.cc:1685] finished
0808 01:52:17.587898 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:17.588059 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54903,"raft.queue_time_us":69960,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":4194,"thread_start_us":116124,"threads_started":1,"wal-append.queue_time_us":450973}
I20260808 01:52:17.598239  9638 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:17.564275 16102 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:17.606130  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:17.592635  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 269060us
I20260808 01:52:17.597823  9481 rpcz_store.cc:276] 0808 01:52:15.864040 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:17.398766 (+1534726us) service_pool.cc:224] Handling call
0808 01:52:17.424323 (+ 25557us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:17.563954 (+139631us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:17.564023 (+    69us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:17.571147 (+  7124us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:17.571221 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:17.571342 (+   121us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:17.571571 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10300,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":31197,"thread_start_us":138002,"threads_started":1}
I20260808 01:52:17.609833  9638 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:17.611513 15636 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:17.611708 16103 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260808 01:52:17.612140 15636 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:17.612766 16092 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:17.610848  9524 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:17.571497 16108 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:17.546833 15923 consensus_queue.cc:1059] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260808 01:52:17.616032  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:17.609623  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3092) took 1746 ms. Trace:
I20260808 01:52:17.617111  9481 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
W20260808 01:52:17.609687 10332 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.698s	user 0.025s	sys 0.029s
W20260808 01:52:17.619050 10332 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.708s	user 0.026s	sys 0.029s
W20260808 01:52:17.620587 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 709961us
I20260808 01:52:17.617592  9474 rpcz_store.cc:276] 0808 01:52:15.863645 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:16.785601 (+921956us) service_pool.cc:224] Handling call
0808 01:52:16.786152 (+   551us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:16.802364 (+ 16212us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:16.802399 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:16.803321 (+   922us) log.cc:844] Serialized 52 byte log entry
0808 01:52:17.106780 (+303459us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:17.368162 (+261382us) log.cc:844] Serialized 14 byte log entry
0808 01:52:17.559157 (+190995us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:17.559275 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:17.609317 (+ 50042us) raft_consensus.cc:1685] finished
0808 01:52:17.609346 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:17.609555 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17102,"prepare.queue_time_us":84194,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":45,"raft.queue_time_us":84398,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":14441,"thread_start_us":603389,"threads_started":3,"wal-append.queue_time_us":478014}
I20260808 01:52:17.622859 10332 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:17.547667 16207 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.057s
I20260808 01:52:17.546566 16242 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.065s
I20260808 01:52:17.619509 16131 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:17.629638 16178 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.975s	user 0.010s	sys 0.015s
I20260808 01:52:17.635587  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:17.574486 16209 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.057s
I20260808 01:52:17.626950  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:17.590958 16227 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:17.609645 16102 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:17.639498  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:17.640132  9474 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260808 01:52:17.603546 16249 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:52:17.631048  9600 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:17.637398 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:17.578881 16053 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:17.647841  9638 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:17.622370 15636 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:17.626291 16129 raft_consensus.cc:697] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:17.649854  9475 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:17.586187 16043 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:17.656356  9600 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:17.656935  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:17.657021  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3097) took 1791 ms. Trace:
I20260808 01:52:17.564327 16241 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.209s
I20260808 01:52:17.657677  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3094) took 1793 ms. Trace:
I20260808 01:52:17.650050 16201 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:17.642814 16178 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.054s	user 0.011s	sys 0.015s
I20260808 01:52:17.652890  9601 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:17.589951 16083 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:17.648912 16125 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.041s	user 0.014s	sys 0.002s
W20260808 01:52:17.661703  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 235350us
I20260808 01:52:17.607107 16200 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:17.666981  9481 rpcz_store.cc:276] 0808 01:52:15.865229 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:17.643656 (+1778427us) service_pool.cc:224] Handling call
0808 01:52:17.656345 (+ 12689us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:17.656654 (+   309us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:17.656682 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:17.656726 (+    44us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:52:17.656755 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:17.656829 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:17.656962 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:17.667580  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:17.667580  9600 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:17.675734  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2839) took 1714 ms. Trace:
I20260808 01:52:17.669282  9474 rpcz_store.cc:276] 0808 01:52:15.864333 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:17.639207 (+1774874us) service_pool.cc:224] Handling call
0808 01:52:17.639715 (+   508us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:17.652966 (+ 13251us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:17.657606 (+  4640us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:17.651883 16102 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:17.651983 16233 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:17.684288  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:17.655445 16202 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:17.666612 16228 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:17.686823  9556 raft_consensus.cc:2410] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260808 01:52:17.669934 16209 consensus_queue.cc:1243] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 7201605bb2f54c949a4ebe6e9b3b4b2a log is divergent from this leader: its last log entry 16.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260808 01:52:17.683604  9638 raft_consensus.cc:2393] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 15: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:17.689059  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1066) took 1807 ms. Trace:
I20260808 01:52:17.671350 15981 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:52:17.658818 15876 raft_consensus.cc:2764] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260808 01:52:17.689494 16194 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:52:17.671558 16227 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:17.664119 16251 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:17.694301  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3014) took 1743 ms. Trace:
I20260808 01:52:17.642637 15862 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:17.686053  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:17.691180 16202 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:17.688943 16233 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:17.700862 16200 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:17.702280  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2835) took 1751 ms. Trace:
I20260808 01:52:17.703400 15972 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:17.706871 16228 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:17.706204  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:17.712395  9475 rpcz_store.cc:276] 0808 01:52:15.881839 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:17.680162 (+1798323us) service_pool.cc:224] Handling call
0808 01:52:17.688444 (+  8282us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:17.688661 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:17.688695 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:17.688748 (+    53us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:52:17.688778 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:17.688833 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:17.688997 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:17.718842  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:17.675058 16216 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260808 01:52:17.719703  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66277.3us
I20260808 01:52:17.677089 16249 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:17.685813 15667 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:17.685976 16205 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:17.679200 16250 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:17.745544  9481 rpcz_store.cc:276] 0808 01:52:15.951368 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:17.688282 (+1736914us) service_pool.cc:224] Handling call
0808 01:52:17.693693 (+  5411us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:17.693914 (+   221us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:17.693949 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:17.693999 (+    50us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:52:17.694026 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:17.694093 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:17.694233 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:17.681566 15856 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260808 01:52:17.709949 16260 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:17.680258 16201 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:17.717585 16244 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.248s
I20260808 01:52:17.693511 16129 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:17.824923  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 140697us
W20260808 01:52:17.750331 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 81542.9us
I20260808 01:52:17.683413  9636 rpcz_store.cc:276] 0808 01:52:15.960893 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:15.977678 (+ 16785us) service_pool.cc:224] Handling call
0808 01:52:16.077542 (+ 99864us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:16.442693 (+365151us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:16.459438 (+ 16745us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:16.488539 (+ 29101us) log.cc:844] Serialized 52 byte log entry
0808 01:52:16.669174 (+180635us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:16.704703 (+ 35529us) log.cc:844] Serialized 14 byte log entry
0808 01:52:16.732026 (+ 27323us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:16.732114 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:17.675346 (+943232us) raft_consensus.cc:1685] finished
0808 01:52:17.675384 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:17.675641 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32256,"prepare.queue_time_us":36114,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":192337,"raft.run_cpu_time_us":320,"raft.run_wall_time_us":47666,"thread_start_us":513291,"threads_started":2,"wal-append.queue_time_us":452119}
I20260808 01:52:17.844952  9637 rpcz_store.cc:276] 0808 01:52:15.950591 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:15.950738 (+   147us) service_pool.cc:224] Handling call
0808 01:52:17.701845 (+1751107us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540034fb54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:17.702207 (+   362us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2319194368}
W20260808 01:52:17.766347  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 81561.9us
I20260808 01:52:17.856011  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:17.821995  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34802 (request call id 80) took 2246 ms. Trace:
I20260808 01:52:17.858424  9632 rpcz_store.cc:276] 0808 01:52:15.569020 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:15.569421 (+   401us) service_pool.cc:224] Handling call
0808 01:52:17.815180 (+2245759us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b54003fa054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4498dc33 00007f5d4998d41a 00007f5d42ee1934 00007f5d42ee18c2 00007f5d42ee17f1 00007f5d42ee176c 00007f5d42ee0a22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:17.815315 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":728039168}
I20260808 01:52:17.782032 16223 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:52:17.855595 16205 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:17.861243 16205 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:17.861770  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:17.862725  9630 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:17.864141  9630 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:17.860466 16223 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:17.864957 16223 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:17.856024 10332 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:17.745915 16238 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.204s	user 0.001s	sys 0.001s
I20260808 01:52:17.736404 16251 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:17.879743 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:17.749166 16202 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:17.782299 15862 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:17.859051 16249 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:17.844878 15972 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:17.866482 16029 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:17.855916 16260 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:17.888094  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260808 01:52:17.894171 16265 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:17.865804 16141 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:17.866482 15856 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:17.824844 16263 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:17.913318  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:17.869030 16200 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:17.885169 16251 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:17.920775 16125 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:17.932813 16228 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:17.905835 16211 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:52:17.947957 16270 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:17.912976 16053 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:17.948511 16170 log.cc:927] Time spent T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 2.412s	user 0.008s	sys 0.006s
I20260808 01:52:17.943981  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 913) took 1082 ms. Trace:
I20260808 01:52:17.877334 16233 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:17.746577 16227 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:17.953902  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:17.956233  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 492934us
I20260808 01:52:17.940973 16242 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:17.934319 15856 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:17.984632  9554 rpcz_store.cc:276] 0808 01:52:16.862085 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:16.862281 (+   196us) service_pool.cc:224] Handling call
0808 01:52:16.862769 (+   488us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:16.863138 (+   369us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:16.863166 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:16.863664 (+   498us) log.cc:844] Serialized 26 byte log entry
0808 01:52:17.102831 (+239167us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:17.102871 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:17.102950 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:17.943721 (+840771us) raft_consensus.cc:1685] finished
0808 01:52:17.943749 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:17.943909 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":329210,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":6444,"thread_start_us":157056,"threads_started":1,"wal-append.queue_time_us":522318}
I20260808 01:52:17.867730 16250 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:17.950024 16060 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.654s	user 0.012s	sys 0.003s
I20260808 01:52:18.046487 16060 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:18.047727  9521 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:18.048828 16241 raft_consensus.cc:2764] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260808 01:52:18.048354 16205 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:18.049911 10368 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:18.002349 16270 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:18.057850 16270 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:18.038137 16245 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.253s	user 0.002s	sys 0.000s
I20260808 01:52:18.038584 16266 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:18.039356 16259 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:52:18.042836 16253 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.255s	user 0.001s	sys 0.001s
I20260808 01:52:17.933710 15972 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:18.043779 15856 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:18.059862  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 192933us
I20260808 01:52:18.060307 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:18.070848  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:17.959785 16271 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:18.049172 16060 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:17.984027 16244 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:18.048178 16250 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:17.958505 16083 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:18.073072 15923 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.029s
W20260808 01:52:18.090943 16274 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260808 01:52:18.104846  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3015) took 2153 ms. Trace:
I20260808 01:52:18.105283  9481 rpcz_store.cc:276] 0808 01:52:15.951682 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:17.866549 (+1914867us) service_pool.cc:224] Handling call
0808 01:52:17.866981 (+   432us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:17.868312 (+  1331us) raft_consensus.cc:1534] Early marking committed up to index 10
0808 01:52:17.880359 (+ 12047us) log.cc:844] Serialized 14 byte log entry
0808 01:52:17.957067 (+ 76708us) log.cc:844] Serialized 14 byte log entry
0808 01:52:17.957361 (+   294us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:17.958043 (+   682us) log.cc:844] Serialized 26 byte log entry
0808 01:52:18.001200 (+ 43157us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:18.001247 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:18.001321 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:18.104588 (+103267us) raft_consensus.cc:1685] finished
0808 01:52:18.104614 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:18.104773 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55695,"prepare.queue_time_us":18183,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":1676,"raft.queue_time_us":21063,"raft.run_cpu_time_us":351,"raft.run_wall_time_us":47066,"spinlock_wait_cycles":43885312,"thread_start_us":49432,"threads_started":1,"wal-append.queue_time_us":44681}
I20260808 01:52:18.107836  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:18.108547  9481 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:52:18.109908  9481 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: []
W20260808 01:52:18.112564 16150 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 14 is higher.
I20260808 01:52:18.112905 16227 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:18.113612 16227 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:52:18.114425  9477 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 3002) took 2647 ms (client timeout 2500 ms). Trace:
W20260808 01:52:18.115154  9477 rpcz_store.cc:269] 0808 01:52:15.467270 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:15.469916 (+  2646us) service_pool.cc:224] Handling call
0808 01:52:18.114056 (+2644140us) spinlock_profiling.cc:243] Waited 284 ms on lock 0x7b54002c0e54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:18.114275 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":652996352}
I20260808 01:52:18.119306  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2575) took 3106 ms. Trace:
I20260808 01:52:18.120441 16201 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:18.121031  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:18.122066 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 739) took 1226 ms. Trace:
I20260808 01:52:18.124665 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:17.950857 16200 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:18.132216  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:18.133038 16200 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:18.123736 16271 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:18.137238 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:18.138269 16150 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:18.139421 16260 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:18.139199 16150 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:18.081513 16241 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.063s
W20260808 01:52:18.064337 16210 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.100s	user 0.012s	sys 0.005s
I20260808 01:52:17.953150 16125 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:18.119848  9552 rpcz_store.cc:276] 0808 01:52:15.012946 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:15.177285 (+164339us) service_pool.cc:224] Handling call
0808 01:52:15.177916 (+   631us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:15.178830 (+   914us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:15.178987 (+   157us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:15.179739 (+   752us) log.cc:844] Serialized 26 byte log entry
0808 01:52:15.433560 (+253821us) raft_consensus.cc:1645] Marking committed up to 6
0808 01:52:15.433603 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:15.433675 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:17.933577 (+2499902us) spinlock_profiling.cc:243] Waited 3.44 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:18.118898 (+185321us) raft_consensus.cc:1685] finished
0808 01:52:18.118926 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:18.119207 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1915,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":123,"spinlock_wait_cycles":7908608,"thread_start_us":253252,"threads_started":1,"wal-append.queue_time_us":312927}
I20260808 01:52:18.084798  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3016) took 1899 ms. Trace:
I20260808 01:52:18.117877  9477 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260808 01:52:18.110598  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3017) took 1590 ms. Trace:
I20260808 01:52:18.114172 16227 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:18.139115 15856 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:18.158201  9481 rpcz_store.cc:276] 0808 01:52:16.520145 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.109311 (+1589166us) service_pool.cc:224] Handling call
0808 01:52:18.109699 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:18.110320 (+   621us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:18.110540 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:18.158360  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3102) took 1253 ms. Trace:
I20260808 01:52:18.178434  9477 rpcz_store.cc:276] 0808 01:52:16.904802 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.116772 (+1211970us) service_pool.cc:224] Handling call
0808 01:52:18.117247 (+   475us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:18.157961 (+ 40714us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:18.158290 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:18.180404  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:18.181362  9477 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:18.183238  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:18.184937  9477 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:18.091246 16273 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:18.122502 10361 rpcz_store.cc:276] 0808 01:52:16.895605 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:16.896182 (+   577us) service_pool.cc:224] Handling call
0808 01:52:16.896562 (+   380us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:16.897046 (+   484us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:16.897076 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:16.920785 (+ 23709us) log.cc:844] Serialized 52 byte log entry
0808 01:52:17.398893 (+478108us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:17.398950 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:17.399043 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:18.121820 (+722777us) raft_consensus.cc:1685] finished
0808 01:52:18.121846 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:18.121998 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":148,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":12072,"thread_start_us":314967,"threads_started":1,"wal-append.queue_time_us":614743}
I20260808 01:52:18.100811 16244 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:18.157655  9475 rpcz_store.cc:276] 0808 01:52:16.185706 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.037606 (+1851900us) service_pool.cc:224] Handling call
0808 01:52:18.084027 (+ 46421us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:18.084299 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:18.084341 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:18.084404 (+    63us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:52:18.084453 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:18.084533 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:18.084713 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:18.201682 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:18.059659 16242 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:17.941345 16263 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:18.212337  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:18.213467  9481 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:18.138360 16271 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:18.152340 15972 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:18.187207  9477 pending_rounds.cc:85] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 7
I20260808 01:52:18.156844 16210 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:18.219216  9477 pending_rounds.cc:107] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260808 01:52:18.075218 16259 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:18.223749  9477 raft_consensus.cc:2889] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:18.167523 16205 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:18.134476 16202 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:18.214848 16229 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:18.229912 16229 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:18.232199 16271 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:18.232697 16263 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:18.210949 16249 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:18.239639 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:18.234218  9481 pending_rounds.cc:85] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 10
I20260808 01:52:18.234602 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:18.240933  9481 pending_rounds.cc:107] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 16.11
I20260808 01:52:18.241436  9481 raft_consensus.cc:2889] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:18.241564 10363 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:18.241451  9551 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260808 01:52:18.240834  9475 pending_rounds.cc:85] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 9
I20260808 01:52:18.242695  9475 pending_rounds.cc:107] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 16.10
I20260808 01:52:18.242990  9475 raft_consensus.cc:2889] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:18.237475  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:18.204210 16260 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:52:18.221630 16119 consensus_meta.cc:287] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.851s	user 0.007s	sys 0.009s
I20260808 01:52:18.075416 16266 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:18.257601  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:18.227146 16209 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:18.264500  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 312328us
I20260808 01:52:18.233091 16271 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:18.226177 16265 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:18.238322 16261 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:18.251122 16202 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:18.253579 16150 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:18.280045 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:18.283510 16125 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:18.287304  9523 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:18.287563 16227 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:18.288641 16227 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:52:18.289407  9523 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:18.261025 16119 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:18.291471 16227 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:18.292234 16227 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:18.292919 16227 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:18.293699 16119 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:18.300979 16108 consensus_meta.cc:287] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.671s	user 0.013s	sys 0.003s
I20260808 01:52:18.301671 16108 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:18.257423 16260 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:18.308048  9554 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260808 01:52:18.276026 16261 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:52:18.310112 16242 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:18.312124 16261 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:18.218338 16273 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:18.314929 16273 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:18.271006 16209 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:18.319927 16256 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:18.321065 16256 raft_consensus.cc:3055] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:52:18.321487 16256 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:18.321828  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:18.322229 16209 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:18.323057  9559 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260808 01:52:18.322434 16256 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:18.322584  9480 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:52:18.324267 16256 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:18.324548  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:18.279000 16244 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:18.325263  9480 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:18.325714 16244 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:18.327135 16255 raft_consensus.cc:3055] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:18.327595 16255 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:18.328696 16255 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:18.330072 16255 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:18.331352 16114 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:18.331830 16114 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:18.332715 16114 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:18.333837 16114 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:18.333303 16223 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:18.322420 16227 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:18.334781 10367 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:52:18.335054 15933 raft_consensus.cc:993] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
W20260808 01:52:18.342690  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:18.345147  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:18.346988 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:18.350564 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:18.336405 16102 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.612s	user 0.014s	sys 0.000s
I20260808 01:52:18.356772 16227 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:18.368206  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:18.333709  9480 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:18.344142 16206 raft_consensus.cc:1081] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Signalling peer 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:52:18.312143 16266 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:18.373634  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:18.377596  9480 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:18.387238 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:18.388026 10365 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:18.387714  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:18.389258  9480 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:18.405300  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:18.382972 16157 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:18.371475 16102 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:18.412763  9638 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:18.425706  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:18.426867  9480 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:18.427479 10360 raft_consensus.cc:1217] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Deduplicated request from leader. Original: 12.10->[13.11-15.12]   Dedup: 13.11->[15.12-15.12]
I20260808 01:52:18.428308 10360 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: [13.11-15.12]
W20260808 01:52:18.430752  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 157567us
I20260808 01:52:18.437978  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:18.382781 16265 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:18.439373  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:18.453073  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:18.431977 16273 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 11, Last known committed idx: 10, Time since last communication: 0.015s
I20260808 01:52:18.460954 16273 consensus_queue.cc:1243] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 7201605bb2f54c949a4ebe6e9b3b4b2a log is divergent from this leader: its last log entry 16.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260808 01:52:18.463130 16108 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:18.464879 16102 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:18.468647  9551 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:18.439437 16209 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:18.477178 10365 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:18.477651  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3106) took 1568 ms. Trace:
I20260808 01:52:18.478596 10360 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:18.480197 16043 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:18.446301 15933 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:18.478174  9481 rpcz_store.cc:276] 0808 01:52:16.908897 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.214541 (+1305644us) service_pool.cc:224] Handling call
0808 01:52:18.233996 (+ 19455us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:18.242124 (+  8128us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:18.242160 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:18.242717 (+   557us) log.cc:844] Serialized 26 byte log entry
0808 01:52:18.423204 (+180487us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:18.423233 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:18.423293 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:18.477368 (+ 54075us) raft_consensus.cc:1685] finished
0808 01:52:18.477395 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:18.477582 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":180122,"threads_started":1,"wal-append.queue_time_us":232296}
I20260808 01:52:18.485266  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:18.488214  9523 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:18.500667  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:18.501657  9480 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:18.492386  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:52:18.512271  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:52:18.473294 16284 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:18.517100  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:18.525936  9481 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:18.519170  9480 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
W20260808 01:52:18.494006 16251 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.535s	user 0.010s	sys 0.004s
I20260808 01:52:18.528563  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:18.533335  9480 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:18.487656 16223 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:18.493436 15933 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:18.545213 15933 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:18.496308 16102 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:18.555449  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:18.556859  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:18.558032  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:18.560325  9473 raft_consensus.cc:2410] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260808 01:52:18.546470 16102 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:18.571725 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:18.572540 10362 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 16
I20260808 01:52:18.565526  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:18.566390  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:18.532044 16284 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:18.483124 16209 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:18.584055 16284 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:18.584545 16209 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:18.584631  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:18.547828 16251 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:18.587908  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:18.588927  9472 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:18.589181 10362 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:18.586229  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:18.575083  9480 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:18.577217  9524 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:18.578418 16217 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.130s	user 0.004s	sys 0.000s
I20260808 01:52:18.584864  9636 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:18.596170  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:18.574965  9473 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:52:18.597469 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:18.601876  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:18.599304  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:18.600407  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:18.606142  9445 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:18.610385  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:18.612733  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:18.613622  9632 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 13.
I20260808 01:52:18.614857  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:18.616110  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:18.617208  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3104) took 1709 ms. Trace:
I20260808 01:52:18.617491  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:18.616279  9524 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:18.621719  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1060) took 3115 ms. Trace:
I20260808 01:52:18.622629  9478 rpcz_store.cc:276] 0808 01:52:15.498480 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:15.816085 (+317605us) service_pool.cc:224] Handling call
0808 01:52:15.842848 (+ 26763us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:15.948728 (+105880us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:15.948767 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:15.948814 (+    47us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:15.948860 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:18.613309 (+2664449us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b54002bc5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:18.613367 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:18.613510 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":242918144}
W20260808 01:52:18.627139  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:18.617679  9477 rpcz_store.cc:276] 0808 01:52:16.907580 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.186272 (+1278692us) service_pool.cc:224] Handling call
0808 01:52:18.186886 (+   614us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:18.412070 (+225184us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:18.412393 (+   323us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:18.416504 (+  4111us) log.cc:844] Serialized 52 byte log entry
0808 01:52:18.453583 (+ 37079us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:18.454273 (+   690us) log.cc:844] Serialized 14 byte log entry
0808 01:52:18.495499 (+ 41226us) log.cc:844] Serialized 14 byte log entry
0808 01:52:18.509845 (+ 14346us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:18.509945 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:18.616943 (+106998us) raft_consensus.cc:1685] finished
0808 01:52:18.616965 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:18.617139 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":62306,"prepare.queue_time_us":1851,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":82,"raft.queue_time_us":140703,"raft.run_cpu_time_us":437,"raft.run_wall_time_us":8600,"thread_start_us":162453,"threads_started":2,"wal-append.queue_time_us":20161}
I20260808 01:52:18.628448  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3105) took 1720 ms. Trace:
I20260808 01:52:18.628914  9475 rpcz_store.cc:276] 0808 01:52:16.908199 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.205613 (+1297414us) service_pool.cc:224] Handling call
0808 01:52:18.240580 (+ 34967us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:18.434272 (+193692us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:18.434296 (+    24us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:18.434835 (+   539us) log.cc:844] Serialized 52 byte log entry
0808 01:52:18.489513 (+ 54678us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:18.489557 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:18.489633 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:18.628131 (+138498us) raft_consensus.cc:1685] finished
0808 01:52:18.628158 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:18.628372 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":71056,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":103,"thread_start_us":236134,"threads_started":2,"wal-append.queue_time_us":94465}
I20260808 01:52:18.630371  9524 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:18.631388  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:18.631270  9481 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:18.632756  9524 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:18.638811  9480 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260808 01:52:18.639822  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3109) took 1164 ms. Trace:
I20260808 01:52:18.643515  9480 rpcz_store.cc:276] 0808 01:52:17.475250 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.618230 (+1142980us) service_pool.cc:224] Handling call
0808 01:52:18.638268 (+ 20038us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:18.639416 (+  1148us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:18.639739 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:18.603111 16209 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:52:18.622385 16108 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:18.661828  9480 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:18.629806 16290 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:18.637331 16125 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:18.634127 16266 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:18.642309 16251 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:18.650918 16209 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:18.637534 15923 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:18.688479 15923 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:18.689105 15923 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:18.689921  9555 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
W20260808 01:52:18.668499 16124 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.709s	user 0.030s	sys 0.017s
W20260808 01:52:18.685545 16299 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.077s	user 0.002s	sys 0.000s
I20260808 01:52:18.671876 16194 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:18.666993 16108 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:18.702924  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 70223.4us
W20260808 01:52:18.694468 16124 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.735s	user 0.031s	sys 0.018s
I20260808 01:52:18.686619 16266 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:18.707284  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:18.687409 16114 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:18.686846 16125 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:18.693150 16209 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:18.692667 16263 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:18.711716  9524 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:18.693619 16290 raft_consensus.cc:697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:18.718196 16157 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:18.721032 16125 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:18.709509 16108 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:18.716926 16266 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:18.742882 16294 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
I20260808 01:52:18.746379 10368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 820) took 1607 ms. Trace:
I20260808 01:52:18.748732  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2747) took 1171 ms. Trace:
I20260808 01:52:18.750172  9630 rpcz_store.cc:276] 0808 01:52:17.577225 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:17.870346 (+293121us) service_pool.cc:224] Handling call
0808 01:52:17.870669 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:18.055046 (+184377us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:18.055092 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:18.055145 (+    53us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:52:18.055176 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:18.748309 (+693133us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b540003bbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:18.748391 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:18.748643 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":423310208}
I20260808 01:52:18.752640  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2840) took 2791 ms. Trace:
I20260808 01:52:18.753489  9630 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: [13.11-15.12]
I20260808 01:52:18.747756 10368 rpcz_store.cc:276] 0808 01:52:17.139590 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:17.140422 (+   832us) service_pool.cc:224] Handling call
0808 01:52:17.185543 (+ 45121us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:18.050429 (+864886us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:18.746093 (+695664us) spinlock_profiling.cc:243] Waited 864 ms on lock 0x7b5400785ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:18.746306 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1987181312}
I20260808 01:52:18.714250 16114 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:18.753474  9631 rpcz_store.cc:276] 0808 01:52:15.961220 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:15.987760 (+ 26540us) service_pool.cc:224] Handling call
0808 01:52:15.988059 (+   299us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:16.562148 (+574089us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:16.562182 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:16.569874 (+  7692us) log.cc:844] Serialized 52 byte log entry
0808 01:52:16.828779 (+258905us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:17.281537 (+452758us) log.cc:844] Serialized 14 byte log entry
0808 01:52:17.302341 (+ 20804us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:17.302470 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:18.577365 (+1274895us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:18.747374 (+170009us) raft_consensus.cc:1685] finished
0808 01:52:18.752279 (+  4905us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:18.752543 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":124241,"prepare.queue_time_us":1674839,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":70,"raft.queue_time_us":232384,"raft.run_cpu_time_us":346,"raft.run_wall_time_us":25658,"spinlock_wait_cycles":31118720,"thread_start_us":1108303,"threads_started":3,"wal-append.queue_time_us":795503}
I20260808 01:52:18.759639 16157 raft_consensus.cc:697] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:18.758836  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47227.6us
I20260808 01:52:18.761718 16290 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:18.762887  9524 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:18.762959 16157 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:18.764707  9524 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:18.765568 16273 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:52:18.766379 16273 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:18.766790 16273 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:18.767127 16273 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:52:18.767978  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:18.768253 16304 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:18.773487 16020 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:18.776872  9558 raft_consensus.cc:3055] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:18.777455  9558 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:18.797731  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:18.798462  9551 raft_consensus.cc:2376] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17. Re-sending same reply.
I20260808 01:52:18.696101 16102 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:18.762094  9631 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260808 01:52:18.803561  9636 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:52:18.757102  9637 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:52:18.778481  9558 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:18.813663  9554 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260808 01:52:18.817792 10332 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:18.817924  9637 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260808 01:52:18.825750  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3028) took 1207 ms. Trace:
I20260808 01:52:18.826387  9481 rpcz_store.cc:276] 0808 01:52:17.618333 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.645827 (+1027494us) service_pool.cc:224] Handling call
0808 01:52:18.668473 (+ 22646us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:18.668757 (+   284us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:52:18.682177 (+ 13420us) log.cc:844] Serialized 14 byte log entry
0808 01:52:18.825276 (+143099us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:18.825348 (+    72us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:18.825410 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:18.825503 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:18.825655 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":22001,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"thread_start_us":142556,"threads_started":1}
I20260808 01:52:18.833974  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:18.845181  9599 connection.cc:256] RPC call timeout handler was delayed by -1.3900631199940108s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1736s for a response
I20260808 01:52:18.845808  9558 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:18.787037 16114 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:18.850600  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:18.724552 16263 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:18.860651  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:18.861495  9481 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:18.861608  9560 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260808 01:52:18.856371  9556 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:18.862869  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:18.863677  9524 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:18.864045  9481 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:18.865962  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:18.856302  9637 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:18.800531 16303 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:18.799844 16284 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:18.874092  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:18.874960 16304 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:18.875949  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:18.876271  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:18.905953  9557 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:18.877838  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:18.882093  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:18.843631 15925 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:52:18.905411  9552 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:18.896497  9481 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:18.912176  9481 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260808 01:52:18.915103  9524 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:18.922014  9599 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:18.927412  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2851) took 2738 ms. Trace:
I20260808 01:52:18.823843 16193 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.102s
W20260808 01:52:18.911747 16255 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.581s	user 0.015s	sys 0.003s
I20260808 01:52:18.966689  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:18.968078  9559 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:18.950963 16046 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:18.971278 16046 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:18.971868 16046 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:18.973707 15925 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:18.975093 15925 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:18.927980  9634 rpcz_store.cc:276] 0808 01:52:16.189681 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:16.595213 (+405532us) service_pool.cc:224] Handling call
0808 01:52:16.595628 (+   415us) raft_consensus.cc:1471] Updating replica for 3 ops
0808 01:52:16.629775 (+ 34147us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:16.629806 (+    31us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0808 01:52:16.630497 (+   691us) log.cc:844] Serialized 78 byte log entry
0808 01:52:16.800233 (+169736us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:16.831023 (+ 30790us) log.cc:844] Serialized 14 byte log entry
0808 01:52:18.502017 (+1670994us) spinlock_profiling.cc:243] Waited 6.06 ms on lock 0x7b180037ab60. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d483aada7 00007f5d4499dd87 00007f5d449ad49f 00007f5d449ad418
0808 01:52:18.502366 (+   349us) log.cc:844] Serialized 14 byte log entry
0808 01:52:18.913264 (+410898us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:18.913359 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:18.927104 (+ 13745us) raft_consensus.cc:1685] finished
0808 01:52:18.927136 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:18.927341 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1642051,"prepare.queue_time_us":1763518,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":20111,"raft.queue_time_us":69742,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":253,"spinlock_wait_cycles":13928704,"thread_start_us":531996,"threads_started":2,"wal-append.queue_time_us":356272,"wal-append.run_cpu_time_us":2822,"wal-append.run_wall_time_us":1030701}
I20260808 01:52:18.918337 16242 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:18.923002 16206 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:18.988297  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:18.997898  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:18.902989 15899 consensus_queue.cc:1059] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.077s
I20260808 01:52:18.953300 15636 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:19.036864  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:18.978736 15986 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:18.913679 16284 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:19.039482  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 124506us
I20260808 01:52:18.913874 16303 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:19.102993  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:19.108366  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 919) took 2243 ms. Trace:
I20260808 01:52:19.109172  9600 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:19.001089 16242 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:19.111821  9558 rpcz_store.cc:276] 0808 01:52:16.864594 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:17.317308 (+452714us) service_pool.cc:224] Handling call
0808 01:52:17.317629 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:18.776719 (+1459090us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b54001589a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490b1a0 00007f5d449955a0 00007f5d44996611 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c
0808 01:52:19.107831 (+331112us) raft_consensus.cc:1534] Early marking committed up to index 10
0808 01:52:19.107883 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:19.107946 (+    63us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:19.107992 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:19.108094 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:19.108283 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3760,"raft.queue_time_us":79953,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":27898,"spinlock_wait_cycles":552714112,"thread_start_us":215514,"threads_started":1}
W20260808 01:52:19.127034 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 78976.7us
I20260808 01:52:19.131704  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:19.133275  9558 raft_consensus.cc:2393] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:19.036120 16316 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election won for term 17
W20260808 01:52:19.100256 15856 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.897s	user 0.008s	sys 0.008s
W20260808 01:52:19.146512  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:19.151579 10364 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
W20260808 01:52:19.153749 16305 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.171s	user 0.001s	sys 0.000s
I20260808 01:52:19.154434 16284 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:19.155613 16296 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.231s	user 0.000s	sys 0.001s
W20260808 01:52:19.157960  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:19.036520 16206 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:19.113344 16216 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 11, Time since last communication: 0.051s
W20260808 01:52:19.136159 16110 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 16 current term: 16 is higher.
I20260808 01:52:19.164672  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:19.165511  9632 raft_consensus.cc:2376] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18. Re-sending same reply.
I20260808 01:52:19.113698 16304 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:19.167681  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 806) took 2320 ms. Trace:
I20260808 01:52:19.167920 16316 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:19.168287  9635 rpcz_store.cc:276] 0808 01:52:16.846906 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:16.847399 (+   493us) service_pool.cc:224] Handling call
0808 01:52:16.892228 (+ 44829us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:17.284456 (+392228us) raft_consensus.cc:1534] Early marking committed up to index 14
0808 01:52:18.506857 (+1222401us) log.cc:844] Serialized 14 byte log entry
0808 01:52:18.660476 (+153619us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:18.660547 (+    71us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:52:18.660594 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:19.167389 (+506795us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b54000df054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:19.167451 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:19.167602 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1227752,"prepare.queue_time_us":1223231,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":10447,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":13120,"spinlock_wait_cycles":900135680,"thread_start_us":147193,"threads_started":1,"wal-append.queue_time_us":337471}
I20260808 01:52:19.169931 16316 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.7, Last appended by leader: 7, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:19.038236 16043 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.108s
I20260808 01:52:19.171496  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3113) took 1322 ms. Trace:
W20260808 01:52:19.171654 16306 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260808 01:52:19.171993  9478 rpcz_store.cc:276] 0808 01:52:17.848771 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.649266 (+800495us) service_pool.cc:224] Handling call
0808 01:52:18.675340 (+ 26074us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:18.686223 (+ 10883us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:18.686257 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:18.694633 (+  8376us) log.cc:844] Serialized 26 byte log entry
0808 01:52:18.854362 (+159729us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:52:18.854407 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:18.854480 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:19.171230 (+316750us) raft_consensus.cc:1685] finished
0808 01:52:19.171255 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:19.171424 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5461,"raft.queue_time_us":467,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":4457,"thread_start_us":143869,"threads_started":1,"wal-append.queue_time_us":434848}
I20260808 01:52:19.173967  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:19.174258 16303 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:19.175902  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:19.176771  9632 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 15: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:19.177609  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:19.178290 16263 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:19.174033 16271 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:19.180421  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:19.181511 16260 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:19.107476 16265 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.076s
I20260808 01:52:19.178990  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:19.183874  9635 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
W20260808 01:52:19.184253 16243 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.509s	user 0.005s	sys 0.000s
I20260808 01:52:19.184949  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:19.185635  9635 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:19.130642 16242 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:19.045650 15636 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:19.181981  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:19.187538  9478 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:52:19.188501  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2848) took 3092 ms. Trace:
I20260808 01:52:19.188741  9632 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 20: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:19.186619  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:19.190574  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3107) took 2091 ms. Trace:
I20260808 01:52:19.194471  9473 rpcz_store.cc:276] 0808 01:52:17.099108 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.600079 (+1500971us) service_pool.cc:224] Handling call
0808 01:52:18.600452 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:18.616379 (+ 15927us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:18.616414 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:18.619019 (+  2605us) log.cc:844] Serialized 26 byte log entry
0808 01:52:18.804593 (+185574us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:18.860307 (+ 55714us) log.cc:844] Serialized 14 byte log entry
0808 01:52:18.860610 (+   303us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:18.860703 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:19.190279 (+329576us) raft_consensus.cc:1685] finished
0808 01:52:19.190309 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:19.190504 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":56046,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":47,"raft.queue_time_us":29370,"raft.run_cpu_time_us":451,"raft.run_wall_time_us":41924,"thread_start_us":133462,"threads_started":1,"wal-append.queue_time_us":266863}
I20260808 01:52:19.166323 16206 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:19.208166  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:19.220167  9558 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:19.209753  9635 raft_consensus.cc:2393] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 13: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:19.204972 15788 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:19.217937 15636 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:19.197744  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:19.233419  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:19.223150  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:19.250459  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:19.251346  9632 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:19.251454  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:19.222254  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:19.188930  9633 rpcz_store.cc:276] 0808 01:52:16.096394 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:16.519928 (+423534us) service_pool.cc:224] Handling call
0808 01:52:16.523540 (+  3612us) raft_consensus.cc:1471] Updating replica for 3 ops
0808 01:52:16.673424 (+149884us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:52:16.975587 (+302163us) log.cc:844] Serialized 14 byte log entry
0808 01:52:17.210219 (+234632us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:17.211122 (+   903us) log.cc:844] Serialized 52 byte log entry
0808 01:52:17.211358 (+   236us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:18.505932 (+1294574us) log.cc:844] Serialized 14 byte log entry
0808 01:52:18.506293 (+   361us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:18.506422 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:19.183602 (+677180us) spinlock_profiling.cc:243] Waited 2.83 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:19.188235 (+  4633us) raft_consensus.cc:1685] finished
0808 01:52:19.188258 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:19.188434 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1405303,"prepare.queue_time_us":1377725,"prepare.run_cpu_time_us":147,"prepare.run_wall_time_us":5558,"raft.queue_time_us":6487,"raft.run_cpu_time_us":610,"raft.run_wall_time_us":50712,"spinlock_wait_cycles":6518528,"thread_start_us":347849,"threads_started":2,"wal-append.queue_time_us":565057}
I20260808 01:52:19.253000  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:19.253862  9633 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 18: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:19.254657  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3029) took 1174 ms. Trace:
I20260808 01:52:19.255702  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:19.257738  9480 rpcz_store.cc:276] 0808 01:52:18.080727 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.663963 (+583236us) service_pool.cc:224] Handling call
0808 01:52:18.684662 (+ 20699us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:18.685162 (+   500us) raft_consensus.cc:1534] Early marking committed up to index 10
0808 01:52:19.031195 (+346033us) log.cc:844] Serialized 14 byte log entry
0808 01:52:19.254124 (+222929us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:19.254216 (+    92us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:19.254288 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:19.254393 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:19.254570 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5799,"prepare.queue_time_us":53513,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"thread_start_us":541272,"threads_started":2}
I20260808 01:52:19.192662 15986 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260808 01:52:19.266611  9476 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 3096) took 3401 ms (client timeout 2015 ms). Trace:
I20260808 01:52:19.190549 16290 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:19.267050  9477 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:19.268698  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3026) took 1651 ms. Trace:
W20260808 01:52:19.267793  9476 rpcz_store.cc:269] 0808 01:52:15.864907 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:15.980989 (+116082us) service_pool.cc:224] Handling call
0808 01:52:19.266280 (+3285291us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b54002dbed4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:19.266458 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":712359808}
I20260808 01:52:19.270052  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:19.269187  9475 rpcz_store.cc:276] 0808 01:52:17.616702 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.637204 (+1020502us) service_pool.cc:224] Handling call
0808 01:52:18.666545 (+ 29341us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:18.668961 (+  2416us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:52:18.852114 (+183153us) log.cc:844] Serialized 14 byte log entry
0808 01:52:19.268121 (+416007us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:19.268189 (+    68us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:19.268238 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:19.268308 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:19.268451 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28197,"prepare.queue_time_us":51216,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":62,"raft.queue_time_us":94944,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":9991,"thread_start_us":572733,"threads_started":3}
I20260808 01:52:19.275267  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:19.277213  9476 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:52:19.280608  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:19.282135  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:19.283186  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:19.283349  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:19.294309  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:19.295284  9558 raft_consensus.cc:2410] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
W20260808 01:52:19.284679  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 319670us
I20260808 01:52:19.287974  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:19.297328  9599 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:19.301779  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:19.303926  9634 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 21: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:19.304888  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:19.307166  9473 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 12: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:19.308189 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:19.310238  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:19.311138  9473 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 20: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:19.321283  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:19.321106  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:19.322273  9480 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
W20260808 01:52:19.323653 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 192866us
W20260808 01:52:19.299403 16266 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.558s	user 0.009s	sys 0.008s
I20260808 01:52:19.284649 15986 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:19.325891  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:19.327662  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:19.331827  9558 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
W20260808 01:52:19.305001 16020 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.531s	user 0.016s	sys 0.001s
I20260808 01:52:19.328254  9473 raft_consensus.cc:2410] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 21 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260808 01:52:19.286396 16206 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:19.235307 15788 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:19.312353 16193 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:19.330724  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:19.338719 10334 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:19.340616  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:19.341328  9558 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:19.343218  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3011) took 3506 ms. Trace:
I20260808 01:52:19.343869  9479 rpcz_store.cc:276] 0808 01:52:15.836852 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:16.638481 (+801629us) service_pool.cc:224] Handling call
0808 01:52:16.639345 (+   864us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:16.827100 (+187755us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:19.342860 (+2515760us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b54002bb6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:19.343126 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":429739904}
W20260808 01:52:19.351717  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 158967us
W20260808 01:52:19.352869  9444 connection.cc:256] RPC call timeout handler was delayed by -2.1387563919997774s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19430000000000003s for a response
W20260808 01:52:19.353848  9444 connection.cc:256] RPC call timeout handler was delayed by -1.8768474710232113s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1825s for a response
W20260808 01:52:19.354476  9444 connection.cc:256] RPC call timeout handler was delayed by -1.7229884910047986s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15400000000000003s for a response
W20260808 01:52:19.354956  9444 connection.cc:256] RPC call timeout handler was delayed by -1.7229884910047986s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15400000000000003s for a response
W20260808 01:52:19.355325  9444 connection.cc:256] RPC call timeout handler was delayed by -1.6662884909892455s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1603s for a response
W20260808 01:52:19.356021  9444 connection.cc:256] RPC call timeout handler was delayed by -1.6662884909892455s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1603s for a response
W20260808 01:52:19.325088 16108 consensus_meta.cc:287] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.585s	user 0.010s	sys 0.009s
W20260808 01:52:19.356464  9444 connection.cc:256] RPC call timeout handler was delayed by -1.542947470996296s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21960000000000002s for a response
W20260808 01:52:19.358736  9444 connection.cc:256] RPC call timeout handler was delayed by -1.463765221007634s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1716s for a response
W20260808 01:52:19.359122  9444 connection.cc:256] RPC call timeout handler was delayed by -1.463765221007634s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1716s for a response
W20260808 01:52:19.359506  9444 connection.cc:256] RPC call timeout handler was delayed by -1.3816537199891172s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1754s for a response
I20260808 01:52:19.358076 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:19.290083 16290 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:19.320459 16139 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260808 01:52:19.335883 15986 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:19.360425  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:19.369024  9444 connection.cc:256] RPC call timeout handler was delayed by -1.3637134600139689s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17700000000000002s for a response
I20260808 01:52:19.341487 16191 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:19.374917  9444 connection.cc:256] RPC call timeout handler was delayed by -1.2891884910059161s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2022s for a response
W20260808 01:52:19.375415  9444 connection.cc:256] RPC call timeout handler was delayed by -1.2891884910059161s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2022s for a response
W20260808 01:52:19.376543  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1577884909929708s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21680000000000002s for a response
W20260808 01:52:19.376976  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1577884909929708s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21680000000000002s for a response
I20260808 01:52:19.377714  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:19.325135 16124 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260808 01:52:19.340549 16206 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:19.328809 16284 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.037s
I20260808 01:52:19.357398 16039 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260808 01:52:19.370601 16125 consensus_meta.cc:287] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.634s	user 0.010s	sys 0.005s
W20260808 01:52:19.374284 16313 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.204s	user 0.002s	sys 0.000s
I20260808 01:52:19.369578 16043 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:19.414412 16043 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:19.360339 16223 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:19.337944 16266 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:19.405505  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:19.373790 15856 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:19.417254  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3108) took 2115 ms. Trace:
I20260808 01:52:19.394287 16322 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:19.434393  9472 rpcz_store.cc:276] 0808 01:52:17.301602 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.603551 (+1301949us) service_pool.cc:224] Handling call
0808 01:52:18.615668 (+ 12117us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:18.619517 (+  3849us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:18.619568 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:18.620152 (+   584us) log.cc:844] Serialized 26 byte log entry
0808 01:52:18.960512 (+340360us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:19.010135 (+ 49623us) log.cc:844] Serialized 14 byte log entry
0808 01:52:19.039106 (+ 28971us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:19.039243 (+   137us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:19.169544 (+130301us) spinlock_profiling.cc:243] Waited 7.59 ms on lock 0x7b180032f920. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4185ac56 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121
0808 01:52:19.170248 (+   704us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b180032f920. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:52:19.416988 (+246740us) raft_consensus.cc:1685] finished
0808 01:52:19.417013 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:19.417190 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11693,"prepare.queue_time_us":48140,"prepare.run_wall_time_us":6107,"raft.queue_time_us":16485,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":63436,"spinlock_wait_cycles":111590656,"thread_start_us":307434,"threads_started":1,"wal-append.queue_time_us":477369}
I20260808 01:52:19.438258  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:19.414453  9638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2745) took 1837 ms (client timeout 2022 ms). Trace:
W20260808 01:52:19.439428  9638 rpcz_store.cc:269] 0808 01:52:17.576569 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:17.718527 (+141958us) service_pool.cc:224] Handling call
0808 01:52:19.414236 (+1695709us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b54000db454. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:19.414371 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":917041408}
I20260808 01:52:19.441022  9472 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:19.405452 16039 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:19.395516 16108 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:19.389187 16191 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:19.490264  9638 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 19. Current term is 20. Ops: []
W20260808 01:52:19.491721  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 53528.6us
I20260808 01:52:19.491882 16336 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:19.499297  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:19.416666 16125 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:19.451723 16039 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:19.449174 15636 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:19.451337 16318 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:19.514463  9629 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:19.517709  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:19.519172  9479 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:52:19.528944 16209 consensus_meta.cc:287] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.783s	user 0.010s	sys 0.002s
W20260808 01:52:19.529366 16046 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.557s	user 0.011s	sys 0.001s
I20260808 01:52:19.529623 16209 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:19.529845 16046 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:19.531352  9635 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:19.531481 15788 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:19.532651 15788 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:19.533334 15788 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:19.533910 15788 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:19.538934 16043 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:19.450567 16322 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:19.538918 16046 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:19.535140 16209 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:19.532759 16323 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
I20260808 01:52:19.545075  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 809) took 1107 ms. Trace:
W20260808 01:52:19.545374  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:19.485023 15925 consensus_queue.cc:1059] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.035s
I20260808 01:52:19.552541  9631 rpcz_store.cc:276] 0808 01:52:18.437074 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.822238 (+385164us) service_pool.cc:224] Handling call
0808 01:52:18.822897 (+   659us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:18.930051 (+107154us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:18.930091 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:18.984538 (+ 54447us) log.cc:844] Serialized 52 byte log entry
0808 01:52:19.303637 (+319099us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:19.303684 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:19.303761 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:19.544692 (+240931us) raft_consensus.cc:1685] finished
0808 01:52:19.544725 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:19.544969 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":90982,"raft.queue_time_us":64512,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":32179,"spinlock_wait_cycles":177024,"thread_start_us":166769,"threads_started":1,"wal-append.queue_time_us":364292}
I20260808 01:52:19.557934  9559 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 18: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:19.531713 16290 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:19.545516 16318 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:19.558622 16314 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.152s	user 0.001s	sys 0.000s
I20260808 01:52:19.565961  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 813) took 1127 ms. Trace:
W20260808 01:52:19.600193  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 100987us
I20260808 01:52:19.600363  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:19.606531  9637 rpcz_store.cc:276] 0808 01:52:18.438502 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.871106 (+432604us) service_pool.cc:224] Handling call
0808 01:52:18.871601 (+   495us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:19.135673 (+264072us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:19.135708 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:19.135760 (+    52us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:19.135795 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:19.565599 (+429804us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b54001261d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:19.565663 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:19.565875 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":606717312}
I20260808 01:52:19.542613 16322 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:19.546622 16297 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.210s	user 0.001s	sys 0.000s
W20260808 01:52:19.535336 16327 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260808 01:52:19.544129 16266 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:19.536000 16317 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
I20260808 01:52:19.613104  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:19.617120  9472 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:19.617205  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:19.619855  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3123) took 1338 ms. Trace:
I20260808 01:52:19.598517 16125 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:19.620633  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:19.547161 16139 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:19.613605 15923 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.923s	user 0.011s	sys 0.007s
I20260808 01:52:19.620461  9480 rpcz_store.cc:276] 0808 01:52:18.281451 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:19.323081 (+1041630us) service_pool.cc:224] Handling call
0808 01:52:19.332327 (+  9246us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:19.332545 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 10
0808 01:52:19.332806 (+   261us) log.cc:844] Serialized 14 byte log entry
0808 01:52:19.619411 (+286605us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:19.619490 (+    79us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:19.619558 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:19.619648 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:19.619787 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6093,"prepare.queue_time_us":1423,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":3791,"raft.queue_time_us":45030,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":111,"thread_start_us":277592,"threads_started":2,"wal-append.queue_time_us":114263}
I20260808 01:52:19.623651  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:19.551898 15636 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 18 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:19.625197 16150 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:19.625982  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:19.628456  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:19.633795  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:19.597158 15986 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:19.636050  9479 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:19.612358 16279 consensus_queue.cc:1059] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.020s
I20260808 01:52:19.638093  9473 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:19.638901  9472 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:52:19.626379 16324 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:19.636034 16335 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:19.646922  9472 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260808 01:52:19.648587 16206 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:19.651954 16108 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:19.661070 16259 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:19.660537 15788 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:19.662597 15856 consensus_queue.cc:1059] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:19.663897 16215 raft_consensus.cc:3055] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:19.664327 16215 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:19.664777 16039 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:19.665295 16215 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:19.658917  9473 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 15. (term mismatch)
I20260808 01:52:19.668219  9473 pending_rounds.cc:85] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 14
I20260808 01:52:19.655552  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:19.641702 16114 consensus_meta.cc:287] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.759s	user 0.012s	sys 0.004s
I20260808 01:52:19.668597  9473 pending_rounds.cc:107] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 19.15
I20260808 01:52:19.669260  9473 raft_consensus.cc:2889] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:19.631847 16139 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:19.671924  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:19.658917  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:19.632603 15923 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:19.667393 16039 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:19.688493  9479 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
W20260808 01:52:19.697882  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:19.709146  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 53688.8us
I20260808 01:52:19.710397  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:19.679987 16215 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:19.713627  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:19.719219 16338 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:19.730022 16338 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260808 01:52:19.731170  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:19.687669 16290 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:19.694124 16335 raft_consensus.cc:697] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 18 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:19.714124 16150 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:19.790324 16108 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:19.759111 15788 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:19.698750 15925 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.051s
W20260808 01:52:19.762040 16321 log.cc:927] Time spent T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.435s	user 0.003s	sys 0.000s
I20260808 01:52:19.764132 15923 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:19.764259  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:19.764572 10364 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 1021) took 1640 ms (client timeout 1576 ms). Trace:
I20260808 01:52:19.765812 15986 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:19.767778  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:19.769090  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:19.772250 16139 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:19.687336 16114 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:19.773975  9599 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:52:19.837221  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 63363.3us
I20260808 01:52:19.839708  9599 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:19.776451  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:19.765815  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:19.842103  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:19.843415 16270 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:19.843724  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:19.844501  9556 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:19.845590  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:19.782896 16318 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:19.697392 15636 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:19.787441  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:19.787873 16340 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
W20260808 01:52:19.819795 16242 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.633s	user 0.008s	sys 0.007s
I20260808 01:52:19.849069 16242 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:19.751224 16273 consensus_meta.cc:287] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.984s	user 0.016s	sys 0.001s
I20260808 01:52:19.714459 16288 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:19.851522 16288 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:19.851856 16242 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:19.789032 16259 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:19.852026 16288 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:19.852612 16259 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:19.831702 10364 rpcz_store.cc:269] 0808 01:52:18.124003 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.124294 (+   291us) service_pool.cc:224] Handling call
0808 01:52:19.764320 (+1640026us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400690fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:19.764461 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2359384320}
I20260808 01:52:19.829874 16108 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:19.856360  9629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2852) took 3659 ms (client timeout 1641 ms). Trace:
W20260808 01:52:19.856832  9629 rpcz_store.cc:269] 0808 01:52:16.198410 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:16.232928 (+ 34518us) service_pool.cc:224] Handling call
0808 01:52:19.856114 (+3623186us) spinlock_profiling.cc:243] Waited 3.12 s on lock 0x7b54001b49d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:19.856234 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7170053888}
I20260808 01:52:19.824820 16337 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:19.823664 16290 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:19.845642 16139 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:19.866328  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:19.881717  9473 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:19.863196  9554 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:19.864750  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:19.824503  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 93414.1us
I20260808 01:52:19.863579  9560 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:20.012769  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:19.903187  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:20.013641  9557 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 18: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:20.013497  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:19.862182  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:20.011693  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:20.014770  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3117) took 1739 ms. Trace:
W20260808 01:52:19.968022  9555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2590) took 2916 ms (client timeout 1716 ms). Trace:
I20260808 01:52:20.015239  9550 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:52:19.881538  9556 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:20.018271  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2603) took 1072 ms. Trace:
I20260808 01:52:20.015182  9481 rpcz_store.cc:276] 0808 01:52:18.275661 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.913292 (+637631us) service_pool.cc:224] Handling call
0808 01:52:18.913705 (+   413us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:18.913974 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:18.914006 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:18.947636 (+ 33630us) log.cc:844] Serialized 26 byte log entry
0808 01:52:19.222833 (+275197us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:19.222876 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:19.222948 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:19.761141 (+538193us) spinlock_profiling.cc:243] Waited 1.32 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:20.014395 (+253254us) raft_consensus.cc:1685] finished
0808 01:52:20.014424 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:20.014679 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3030400,"thread_start_us":251141,"threads_started":1,"wal-append.queue_time_us":348956}
W20260808 01:52:20.015508  9555 rpcz_store.cc:269] 0808 01:52:17.051953 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:17.164233 (+112280us) service_pool.cc:224] Handling call
0808 01:52:19.967740 (+2803507us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54002a8354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:19.967925 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3433519488}
I20260808 01:52:20.018829  9556 rpcz_store.cc:276] 0808 01:52:18.946344 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:19.845306 (+898962us) service_pool.cc:224] Handling call
0808 01:52:20.018200 (+172894us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:19.950464  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:20.023149  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:20.027587  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3412) took 1149 ms. Trace:
I20260808 01:52:20.028450  9473 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:20.028892  9552 rpcz_store.cc:276] 0808 01:52:18.877732 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:19.897757 (+1020025us) service_pool.cc:224] Handling call
0808 01:52:20.027512 (+129755us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:52:20.016773  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 113746us
I20260808 01:52:20.027661  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2599) took 1119 ms. Trace:
I20260808 01:52:20.021819 16053 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:20.032651  9556 raft_consensus.cc:2368] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18.
I20260808 01:52:20.034054  9479 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:20.035485 16194 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:19.858065 16335 consensus_queue.cc:237] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:20.036080 16194 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:19.850515 16273 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:20.041482 16273 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:19.857610 16108 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:20.043568  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:19.882889 15986 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:20.043955 15788 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:20.044606 15986 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:20.045161 15986 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:20.045390  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2595) took 1145 ms. Trace:
I20260808 01:52:20.045765 16204 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:20.075884 16204 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:20.077100 16039 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:20.077677 16039 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:20.078282 16204 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:20.079145  9630 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260808 01:52:20.079881 16204 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:20.080982  9601 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:20.075237 16202 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:20.082640 16202 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:20.084051 16206 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:19.862610 16114 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:20.031234  9557 rpcz_store.cc:276] 0808 01:52:18.907947 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:19.898003 (+990056us) service_pool.cc:224] Handling call
0808 01:52:20.027600 (+129597us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:20.086035 16202 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:20.086201  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:20.075232  9636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 814) took 1636 ms (client timeout 1514 ms). Trace:
I20260808 01:52:20.087563  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2600) took 1177 ms. Trace:
I20260808 01:52:20.088223  9524 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:20.088908  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3405) took 1216 ms. Trace:
I20260808 01:52:20.089443 16141 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:20.089532  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:20.090153 16141 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:20.090591 16141 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:20.046629 16324 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:20.091277 16324 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:20.060791  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:20.091619 16227 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:20.092164 16227 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:20.092609 16227 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:20.093847 16268 raft_consensus.cc:3055] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:20.094296 16268 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:20.089300  9554 rpcz_store.cc:276] 0808 01:52:18.873071 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.873687 (+   616us) service_pool.cc:224] Handling call
0808 01:52:20.088718 (+1215031us) spinlock_profiling.cc:243] Waited 977 ms on lock 0x7b540029e5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:20.088846 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2247165312}
I20260808 01:52:20.095293 16268 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:20.096453 16268 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
W20260808 01:52:20.087321  9636 rpcz_store.cc:269] 0808 01:52:18.438801 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.455472 (+ 16671us) service_pool.cc:224] Handling call
0808 01:52:20.075012 (+1619540us) spinlock_profiling.cc:243] Waited 98.4 ms on lock 0x7b5400339354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:20.075154 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":226342784}
I20260808 01:52:19.880688 16139 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:20.098495  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:20.049357  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3025) took 2433 ms. Trace:
I20260808 01:52:20.099454  9477 rpcz_store.cc:276] 0808 01:52:17.615879 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.642411 (+1026532us) service_pool.cc:224] Handling call
0808 01:52:19.266843 (+624432us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:19.285024 (+ 18181us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:20.049070 (+764046us) spinlock_profiling.cc:243] Waited 624 ms on lock 0x7b54002dbed0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:20.049271 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1434841344}
I20260808 01:52:20.061807  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:19.882793 16290 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:20.064356 10364 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260808 01:52:19.950330 16242 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:20.087997  9560 rpcz_store.cc:276] 0808 01:52:18.909814 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:19.763892 (+854078us) service_pool.cc:224] Handling call
0808 01:52:20.087380 (+323488us) spinlock_profiling.cc:243] Waited 31.8 ms on lock 0x7b540029f9d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:20.087500 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":73175808}
I20260808 01:52:20.066057  9600 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:20.068583 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:20.103478 10368 raft_consensus.cc:2410] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 14 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260808 01:52:20.071388 16229 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:20.072263 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:20.104007  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:20.073177 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:19.858559 16256 raft_consensus.cc:3055] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Stepping down as leader of term 19
W20260808 01:52:20.075039 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:20.075402  9559 rpcz_store.cc:276] 0808 01:52:18.900384 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.969107 (+ 68723us) service_pool.cc:224] Handling call
0808 01:52:20.045123 (+1076016us) spinlock_profiling.cc:243] Waited 556 ms on lock 0x7b54000b4ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:20.045310 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1278418176}
I20260808 01:52:20.080662  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2865) took 1646 ms. Trace:
I20260808 01:52:20.106834  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:20.107874  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:20.108016  9630 rpcz_store.cc:276] 0808 01:52:18.434975 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.755414 (+320439us) service_pool.cc:224] Handling call
0808 01:52:20.078818 (+1323404us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:20.079812 (+   994us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:20.080443 (+   631us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b5400339350. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:20.080599 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3042553088}
I20260808 01:52:20.083309 16039 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:19.882515 16337 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:20.072194 16191 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:19.831236 16109 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:19.897399 16259 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:20.092013  9555 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:20.101945  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:20.059377 15986 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:20.107441  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:20.104954 10360 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:52:20.135623  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:20.108587  9477 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:20.142256  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:20.109882 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 273008us
W20260808 01:52:20.114923  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:20.108043 15636 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260808 01:52:20.118340  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 52381us
I20260808 01:52:20.147760  9600 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:20.118355  9524 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:20.112884  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:20.098937 16139 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:20.107558 16256 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:20.121628  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:20.103000 16215 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:20.112094  9599 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:20.159610  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:20.163800  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:20.110576 15981 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.042s
I20260808 01:52:20.117677 16200 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:20.165455 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:20.112284 16242 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:20.117076 16337 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:20.125963 16344 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260808 01:52:20.139398 16338 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260808 01:52:20.123705 16322 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260808 01:52:20.173955  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 61943.1us
I20260808 01:52:20.143702 15788 raft_consensus.cc:2764] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election decision vote started in defunct term 13: won
W20260808 01:52:20.172364  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 54077.4us
I20260808 01:52:20.157441 16204 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:52:20.154805  9555 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:53208
I20260808 01:52:20.208979  9555 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:52:20.209568  9555 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:20.153000 16191 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:20.127754 16109 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:20.211580  9524 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:20.192421 16202 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:52:20.193352 16259 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:52:20.173166 15667 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:52:20.198922 16200 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:20.203343 16322 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:20.234581 16280 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:20.239965 15986 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:20.241947  9555 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:20.242280 16202 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:20.190723 16288 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:20.215742 16204 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:20.233559  9524 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:20.268953  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2609) took 1119 ms. Trace:
I20260808 01:52:20.197968  9477 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:52:20.150627  9601 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:20.196945 16141 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:20.277837 16194 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:20.279234 16200 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:20.348294 16109 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:20.283473 16347 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260808 01:52:20.287712 16139 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:20.299132  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:20.198916 16338 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:20.305205  9555 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:20.173564 16215 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:20.315805  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 82351.2us
I20260808 01:52:20.258464 16346 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:52:20.321681  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:20.304427 16322 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:20.326246  9550 rpcz_store.cc:276] 0808 01:52:19.149815 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:20.110172 (+960357us) service_pool.cc:224] Handling call
0808 01:52:20.268840 (+158668us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:20.305600 16202 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:20.299451 16259 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:20.350937  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 200389us
I20260808 01:52:20.379263 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:20.379817  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:20.317616 16204 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:20.324837 16229 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:20.315479 16280 raft_consensus.cc:697] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:20.245705 16256 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:20.353057 16029 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260808 01:52:20.357267 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 206021us
I20260808 01:52:20.368386  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:53208 (request call id 3410) took 1476 ms. Trace:
I20260808 01:52:20.394749  9555 rpcz_store.cc:276] 0808 01:52:18.876981 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:20.126811 (+1249830us) service_pool.cc:224] Handling call
0808 01:52:20.353469 (+226658us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:20.372057  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:20.376644 16194 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:20.381781  9481 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260808 01:52:20.383332 10368 raft_consensus.cc:2410] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 18 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260808 01:52:20.384135  9601 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:20.354736 16338 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 9 FOLLOWER]: Advancing to term 10
I20260808 01:52:20.396085 16242 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:20.403064  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:20.404677  9601 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:20.389206 15933 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260808 01:52:20.395879 16029 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:20.423184 16029 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:20.428349  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:20.389848  9550 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:20.406425 16017 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:20.429205  9555 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
W20260808 01:52:20.396869  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 254695us
I20260808 01:52:20.430064  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:20.399200 16229 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:20.364779 16346 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:20.434928 16017 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:20.453554 16307 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 1.418s	user 0.002s	sys 0.000s
I20260808 01:52:20.455040 16280 consensus_queue.cc:237] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:20.455219 16229 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:20.455787 16139 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:20.456676 16141 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:20.458880 15981 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:20.458774 15933 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:20.463183  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:20.503492  9477 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 16
I20260808 01:52:20.471131  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:20.506270  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:20.467638  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:20.445923  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3414) took 1567 ms. Trace:
I20260808 01:52:20.508942  9550 rpcz_store.cc:276] 0808 01:52:18.878528 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:20.379328 (+1500800us) service_pool.cc:224] Handling call
0808 01:52:20.389652 (+ 10324us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:20.445577 (+ 55925us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:20.445826 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:20.428745 16124 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:20.485710  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:20.468446 16329 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Stepping down as leader of term 16
W20260808 01:52:20.535723  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50155.8us
I20260808 01:52:20.521639 16356 raft_consensus.cc:2764] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260808 01:52:20.440598 16191 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:52:20.440887 16335 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:20.404560 16108 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.092s
I20260808 01:52:20.531469 16124 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:20.609227  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2593) took 1710 ms. Trace:
I20260808 01:52:20.482745 16017 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:20.514935  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:20.612077  9556 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:20.520072  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:20.539791  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:20.477342  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:20.614392  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:20.557272  9555 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:20.571288  9477 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
W20260808 01:52:20.619112 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 223816us
I20260808 01:52:20.621433  9524 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:20.621753  9521 connection.cc:256] RPC call timeout handler was delayed by -1.7345469020074233s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.42190000000000005s for a response
I20260808 01:52:20.465888 16290 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:20.465950 16227 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:20.513806 15667 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:52:20.624655  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 104722us
I20260808 01:52:20.631603  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:20.625980  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 931) took 1328 ms. Trace:
I20260808 01:52:20.632544  9556 rpcz_store.cc:276] 0808 01:52:19.297529 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:20.625190 (+1327661us) service_pool.cc:224] Handling call
0808 01:52:20.625521 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:20.625696 (+   175us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:20.625717 (+    21us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:20.625757 (+    40us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:52:20.625787 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:20.625834 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:20.625920 (+    86us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:20.634008  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:20.603669 16190 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
W20260808 01:52:20.627584  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 87886.3us
I20260808 01:52:20.536155 16329 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:20.538389 16288 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:20.652912  9636 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13. Ops: []
I20260808 01:52:20.669706  9475 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:20.654481  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:20.670619  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:20.609851  9550 rpcz_store.cc:276] 0808 01:52:18.899322 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:20.526471 (+1627149us) service_pool.cc:224] Handling call
0808 01:52:20.608424 (+ 81953us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:20.608813 (+   389us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:20.608854 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:20.608909 (+    55us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:20.608949 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:20.609019 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:20.609164 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10236}
I20260808 01:52:20.618970 16124 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:20.670604  9556 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 14: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:20.677416  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:20.678182  9632 raft_consensus.cc:2376] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16. Re-sending same reply.
I20260808 01:52:20.679853 16346 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:20.681118 16215 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:20.630301 16284 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:52:20.682614  9560 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 18: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:20.466097 16259 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:20.628969 16335 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:20.717308  9630 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:20.669217 16356 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:20.778579  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:20.668761 16141 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election won for term 16
W20260808 01:52:20.696916 16354 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260808 01:52:20.730019 16346 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:20.918203  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:20.768682 16194 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:20.922412  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 299467us
I20260808 01:52:20.951840  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:20.970973 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 342447us
W20260808 01:52:21.017118  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 65436us
I20260808 01:52:21.018409  9521 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:20.914052 16284 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:21.020910  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:21.038321  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:21.039577  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:21.040174  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:21.041105  9474 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:20.937471 16141 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:21.026078 16268 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.929s	user 0.013s	sys 0.004s
I20260808 01:52:21.057986  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 815) took 2617 ms. Trace:
I20260808 01:52:21.019354 16125 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:21.017707 16150 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:21.029690 16288 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:21.030086 16335 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:21.058611  9634 rpcz_store.cc:276] 0808 01:52:18.442453 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:19.352793 (+910340us) service_pool.cc:224] Handling call
0808 01:52:19.367269 (+ 14476us) raft_consensus.cc:1471] Updating replica for 3 ops
0808 01:52:19.375720 (+  8451us) raft_consensus.cc:1534] Early marking committed up to index 12
0808 01:52:19.410733 (+ 35013us) log.cc:844] Serialized 14 byte log entry
0808 01:52:20.053191 (+642458us) spinlock_profiling.cc:243] Waited 2.98 ms on lock 0x7b18004375c0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d4453242a 00007f5d4499de0a
0808 01:52:20.317117 (+263926us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0808 01:52:20.325951 (+  8834us) log.cc:844] Serialized 78 byte log entry
0808 01:52:20.326130 (+   179us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:52:20.603241 (+277111us) log.cc:844] Serialized 14 byte log entry
0808 01:52:21.036710 (+433469us) log.cc:844] Serialized 14 byte log entry
0808 01:52:21.046889 (+ 10179us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:21.057538 (+ 10649us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:21.057626 (+    88us) raft_consensus.cc:1685] finished
0808 01:52:21.057646 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:21.057887 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":215300,"prepare.queue_time_us":501457,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":25240,"raft.queue_time_us":340462,"raft.run_cpu_time_us":880,"raft.run_wall_time_us":62429,"spinlock_wait_cycles":31888000,"thread_start_us":726871,"threads_started":3,"wal-append.queue_time_us":902268}
W20260808 01:52:21.068011  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 446641us
W20260808 01:52:21.036502 16338 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.629s	user 0.012s	sys 0.000s
I20260808 01:52:20.978587 16346 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:21.072971  9521 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:21.074358  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:21.005082 16194 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:21.049141 15788 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:21.063185 16029 consensus_meta.cc:287] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.639s	user 0.012s	sys 0.000s
I20260808 01:52:21.081401  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:21.092700  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:21.035976 16284 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:21.102165 16249 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:21.019719 16329 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:21.107120 16249 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260808 01:52:21.102398  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 470840us
I20260808 01:52:21.108480  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:21.111954  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:21.114028  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:52:21.118224  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 338523us
I20260808 01:52:21.126925  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:21.097455 16202 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.711s	user 0.011s	sys 0.001s
I20260808 01:52:21.130270  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:21.077569 16338 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:21.133849  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:21.075827 16150 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:21.116389 16125 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:21.131601  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:21.082867 16288 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:52:21.134958  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:21.143541  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:21.164211  9637 raft_consensus.cc:2376] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16. Re-sending same reply.
I20260808 01:52:21.082504 16335 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:21.169566  9478 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 9. Current term is 10. Ops: []
W20260808 01:52:21.169809 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 166433us
W20260808 01:52:21.171166 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:21.171674  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:21.096849 15788 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:21.133210 16290 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:21.143187 16356 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:21.147181 16369 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Stepping down as leader of term 20
W20260808 01:52:21.178063 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:21.115710 16029 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:21.181417  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:21.122714 16364 consensus_queue.cc:1175] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "259bcb3f0d254e8b9effac3277e654e0" responder_term: 16 status { last_received { term: 16 index: 13 } last_committed_idx: 13 last_received_current_leader { term: 16 index: 13 } }
I20260808 01:52:21.192353  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:21.193790  9635 raft_consensus.cc:2410] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 21 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260808 01:52:21.209529  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:21.213505  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 106678us
I20260808 01:52:21.143697 16202 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:21.215058  9476 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 7. Preceding OpId from leader: term: 17 index: 8. (index mismatch)
I20260808 01:52:21.218050  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:21.219506  9635 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:21.128242 16141 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.6, Last appended by leader: 6, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:21.222652  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 112326us
I20260808 01:52:21.189632 16356 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:21.227751  9631 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:21.228842  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:21.190157 16369 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:21.211936 16290 raft_consensus.cc:2764] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260808 01:52:21.228917  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:21.240772  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 126831us
W20260808 01:52:21.231007 16337 consensus_meta.cc:287] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.025s	user 0.015s	sys 0.000s
I20260808 01:52:21.247263  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:21.245754  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:21.270423  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:21.273500 16335 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:21.244379 16233 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:21.275902 16338 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:21.280783 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:21.285328 16346 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:21.286144 16346 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:21.286787 16346 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:21.289908 16346 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:21.291272 16369 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:21.292897 16369 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:21.296073 16288 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:21.297869 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:21.299345 16194 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:21.299662  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 54034.5us
I20260808 01:52:21.301542  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:21.302598 16337 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:21.303231  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:21.304054  9472 raft_consensus.cc:2393] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:21.305294 16337 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:21.302513 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:21.314349 10360 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:21.266743  9637 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:21.318760  9480 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:21.321457 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:21.240397 16356 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
W20260808 01:52:21.245460 16362 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.222s	user 0.000s	sys 0.001s
I20260808 01:52:21.332792 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:21.331756  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:21.297721 16150 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:21.290490 16346 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:21.354171 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:21.306090 16337 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:21.353665  9637 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:21.356097 10362 raft_consensus.cc:2410] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260808 01:52:21.335371  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:21.453001  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1082) took 1177 ms. Trace:
I20260808 01:52:21.453501  9478 rpcz_store.cc:276] 0808 01:52:20.275930 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:20.276157 (+   227us) service_pool.cc:224] Handling call
0808 01:52:20.316082 (+ 39925us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:21.170313 (+854231us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:21.452684 (+282371us) spinlock_profiling.cc:243] Waited 853 ms on lock 0x7b54000942d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:21.452922 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1962391936}
I20260808 01:52:21.459089 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:21.364464 15788 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:21.366600 16029 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:21.376423 16324 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.285s	user 0.010s	sys 0.005s
W20260808 01:52:21.379515 16017 consensus_meta.cc:287] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.748s	user 0.011s	sys 0.002s
I20260808 01:52:21.385911 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:21.404528  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:21.419075 10365 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 16: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:21.445223 16178 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:21.339350 16288 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:21.553551 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:21.620522  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:21.322006 16290 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
W20260808 01:52:21.380522 16322 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.998s	user 0.013s	sys 0.000s
W20260808 01:52:21.576864 10334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.001s	sys 0.009s
I20260808 01:52:21.446542 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: true dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:21.591759 16017 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:21.592301  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:21.592569 16124 consensus_meta.cc:287] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.824s	user 0.009s	sys 0.004s
I20260808 01:52:21.592780 15788 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:21.593246 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:21.465798 16346 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:21.623073 10363 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260808 01:52:21.381624 16383 consensus_queue.cc:1059] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.254s
I20260808 01:52:21.448339 16337 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:21.629998  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:21.632972 10364 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:21.668427  9637 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:52:21.633198  9557 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:21.670287  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:21.671746  9637 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:21.659564 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:52:21.636804  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:21.633582 10334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.001s	sys 0.010s
I20260808 01:52:21.636389 16017 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:21.638183 16029 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:21.635694 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:21.686597 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 50988us
I20260808 01:52:21.687773 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:21.630759 16204 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.234s	user 0.009s	sys 0.005s
I20260808 01:52:21.689242  9637 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
W20260808 01:52:21.630762 15667 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.987s	user 0.006s	sys 0.008s
I20260808 01:52:21.694196  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:21.630424 16376 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:21.699258  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:21.696226 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:21.697319  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:21.705737  9637 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:52:21.685420 16190 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.012s	user 0.013s	sys 0.000s
I20260808 01:52:21.680146 16337 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:21.611851 16324 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:21.665123 16378 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:21.629935 16288 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:21.652469 16124 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:52:21.694427 16029 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:21.722271 16397 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:21.744661 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:21.630170 16178 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:21.633944 16322 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:21.713325 15667 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:21.754567 16378 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:21.730396 16190 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:21.694562 16204 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:21.714918 16376 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:21.806406 16204 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:21.821871 16378 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:21.850943 16392 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
W20260808 01:52:21.852989 16394 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
W20260808 01:52:21.585626  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 181239us
I20260808 01:52:21.899144  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:21.902455  9473 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:21.632405  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:21.630460  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:21.905246  9472 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:52:21.907020  9474 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.834s	user 0.001s	sys 0.009s
W20260808 01:52:21.907845  9474 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.835s	user 0.001s	sys 0.009s
I20260808 01:52:21.907836  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:21.912339  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:21.917375  9481 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
W20260808 01:52:21.919957  9550 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.025s	sys 0.031s
W20260808 01:52:21.920248  9555 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.893s	user 0.000s	sys 0.027s
W20260808 01:52:21.920668  9550 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.025s	sys 0.031s
W20260808 01:52:21.920899  9555 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.894s	user 0.000s	sys 0.027s
I20260808 01:52:21.922622  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3417) took 2142 ms. Trace:
I20260808 01:52:21.923655  9550 rpcz_store.cc:276] 0808 01:52:19.779644 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:21.027327 (+1247683us) service_pool.cc:224] Handling call
0808 01:52:21.052954 (+ 25627us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:21.053237 (+   283us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:52:21.215853 (+162616us) log.cc:844] Serialized 14 byte log entry
0808 01:52:21.315562 (+ 99709us) log.cc:844] Serialized 14 byte log entry
0808 01:52:21.921513 (+605951us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:21.921712 (+   199us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:21.921872 (+   160us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:21.922033 (+   161us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:21.922375 (+   342us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":190018,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":97,"raft.queue_time_us":777134,"raft.run_cpu_time_us":446,"raft.run_wall_time_us":21950,"thread_start_us":850948,"threads_started":3,"wal-append.queue_time_us":535765}
I20260808 01:52:21.922775  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3416) took 2143 ms. Trace:
I20260808 01:52:21.927495  9555 rpcz_store.cc:276] 0808 01:52:19.779313 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:20.615580 (+836267us) service_pool.cc:224] Handling call
0808 01:52:20.616096 (+   516us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:21.023712 (+407616us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:21.023745 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:21.024272 (+   527us) log.cc:844] Serialized 26 byte log entry
0808 01:52:21.921927 (+897655us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:21.922041 (+   114us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:21.922208 (+   167us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:21.922294 (+    86us) raft_consensus.cc:1685] finished
0808 01:52:21.922311 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:21.922628 (+   317us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21255,"raft.queue_time_us":172622,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":121712,"thread_start_us":1279915,"threads_started":2,"wal-append.queue_time_us":790717}
I20260808 01:52:21.938235  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:21.939503  9559 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:21.943269  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.004s	sys 0.016s
W20260808 01:52:21.944056  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.004s	sys 0.016s
I20260808 01:52:21.947716  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:21.948603  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:21.949338  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:21.949468  9633 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:52:21.949997  9550 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:21.950153  9633 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:21.964620  9633 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:21.971741 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:21.972863 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:21.951639  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:21.952497  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:21.953438  9636 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 12. Current term is 13. Ops: []
W20260808 01:52:21.959465  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:21.994639 16367 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:21.997174 10364 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:52:21.995713  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:22.000950  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:22.002575 16017 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:22.003954 16017 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:21.818787 16233 consensus_meta.cc:287] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.544s	user 0.012s	sys 0.002s
I20260808 01:52:22.002408  9550 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:22.005154 16324 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:22.019845 16029 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:22.021703 16411 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:22.022905 16411 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:22.024096 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 336293us
I20260808 01:52:22.025261 16124 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:22.027199 16411 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:22.028215 16288 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:22.028303 16412 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:22.038070 16322 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:22.038885 15856 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:52:22.039685 16190 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:22.043843 16389 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:22.044529  9557 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:22.043835  9559 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:22.047803 16376 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:22.098328  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:22.098718 16376 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:52:22.099627  9638 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:22.100478 16376 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:22.101003  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:22.103551 10368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 1036) took 1955 ms (client timeout 2000 ms). Trace:
I20260808 01:52:22.104323  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2598) took 3185 ms. Trace:
W20260808 01:52:22.105067  9599 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.194:36095 timed out after 1.593s (SENT) [suppressed 14 similar messages]
W20260808 01:52:22.104162 10368 rpcz_store.cc:269] 0808 01:52:20.148001 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:20.148165 (+   164us) service_pool.cc:224] Handling call
0808 01:52:22.102954 (+1954789us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b540041f0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:22.103278 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":443367808}
I20260808 01:52:22.048100 16017 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:22.049226 16396 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.197s	user 0.001s	sys 0.000s
I20260808 01:52:22.043305 16178 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:22.058439 15636 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:22.103760  9554 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260808 01:52:22.059196 16375 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:22.113199 16375 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:22.062729 16386 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:22.066440 15667 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:22.115935 15667 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:22.117156 16053 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:22.117727 16053 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:22.067967 16338 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:22.070748  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3071) took 1602 ms. Trace:
I20260808 01:52:22.119658  9473 rpcz_store.cc:276] 0808 01:52:20.468504 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:20.468685 (+   181us) service_pool.cc:224] Handling call
0808 01:52:20.469009 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:21.903592 (+1434583us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.070339 (+166747us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b540021d254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:22.070637 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3294721280}
I20260808 01:52:22.119876 16338 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:22.071066  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:22.072304  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3070) took 1606 ms. Trace:
I20260808 01:52:22.072419  9633 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:22.079321 16412 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:22.080791  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:22.082002 16223 raft_consensus.cc:3055] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Stepping down as leader of term 12
W20260808 01:52:22.083410  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 185048us
I20260808 01:52:22.125295  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3439) took 1063 ms. Trace:
I20260808 01:52:22.125804  9557 rpcz_store.cc:276] 0808 01:52:21.062342 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:21.449320 (+386978us) service_pool.cc:224] Handling call
0808 01:52:22.125159 (+675839us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:22.127977  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:22.083577  9556 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:52:22.129315  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3418) took 2349 ms. Trace:
I20260808 01:52:22.129680  9552 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260808 01:52:22.129750  9556 rpcz_store.cc:276] 0808 01:52:19.779957 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.082341 (+2302384us) service_pool.cc:224] Handling call
0808 01:52:22.083070 (+   729us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:22.128970 (+ 45900us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.129222 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:22.087294 16346 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:22.133682 16386 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:22.122267  9475 rpcz_store.cc:276] 0808 01:52:20.465917 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:20.466197 (+   280us) service_pool.cc:224] Handling call
0808 01:52:20.466560 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:20.670271 (+203711us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.071686 (+1401415us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b54002c0454. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:22.072062 (+   376us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":466596608}
W20260808 01:52:22.090010  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 61266.7us
I20260808 01:52:22.136263  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:22.136512  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:22.091424 16266 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:22.137519  9475 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:52:22.137789 16266 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:22.139027 16389 raft_consensus.cc:697] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:22.140782 16389 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:22.130615  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3423) took 1723 ms. Trace:
I20260808 01:52:22.092157  9635 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:22.055869  9634 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:22.095299 16190 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:22.104902  9560 rpcz_store.cc:276] 0808 01:52:18.901394 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:20.126534 (+1225140us) service_pool.cc:224] Handling call
0808 01:52:22.086106 (+1959572us) spinlock_profiling.cc:243] Waited 518 ms on lock 0x7b5400325ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:22.086408 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1192020224}
I20260808 01:52:22.146443  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3426) took 1737 ms. Trace:
I20260808 01:52:22.148495  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3427) took 1739 ms. Trace:
I20260808 01:52:22.101056 16376 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:22.150969  9558 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:22.152127 16376 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:22.153452  9558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2606) took 3047 ms (client timeout 1867 ms). Trace:
I20260808 01:52:22.147063  9557 rpcz_store.cc:276] 0808 01:52:20.408701 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.144939 (+1736238us) service_pool.cc:224] Handling call
0808 01:52:22.145621 (+   682us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:22.146038 (+   417us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:22.146067 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:22.146142 (+    75us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:52:22.146180 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.146252 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:22.146374 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:22.146288  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:22.146579  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3438) took 1095 ms. Trace:
I20260808 01:52:22.105695  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:22.143587  9552 rpcz_store.cc:276] 0808 01:52:20.407215 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.128698 (+1721483us) service_pool.cc:224] Handling call
0808 01:52:22.129341 (+   643us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:22.130289 (+   948us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.130517 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:22.149161  9560 rpcz_store.cc:276] 0808 01:52:20.409045 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.146980 (+1737935us) service_pool.cc:224] Handling call
0808 01:52:22.147543 (+   563us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:22.148053 (+   510us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:22.148089 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:22.148167 (+    78us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:52:22.148220 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.148288 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:22.148423 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:22.155756  9559 rpcz_store.cc:276] 0808 01:52:21.028414 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:21.667170 (+638756us) service_pool.cc:224] Handling call
0808 01:52:22.111840 (+444670us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:52:22.125545 16414 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
W20260808 01:52:22.159484  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:22.154059 16376 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
W20260808 01:52:22.154197  9558 rpcz_store.cc:269] 0808 01:52:19.106293 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:19.342039 (+235746us) service_pool.cc:224] Handling call
0808 01:52:22.153083 (+2811044us) spinlock_profiling.cc:243] Waited 2.77 s on lock 0x7b54001acf54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:22.153315 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6366144384}
I20260808 01:52:22.154287 15667 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:22.125396 16223 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:22.164330 16017 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:22.165390 16017 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:22.166057 15788 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.513s	user 0.012s	sys 0.004s
I20260808 01:52:22.166616 15788 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:22.167527  9478 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:22.146385 16412 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:22.167392  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:22.165483 16223 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:22.153357 15981 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:22.148523 16375 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:22.176836 16190 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:22.181352 16029 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:22.183120  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 77820.2us
I20260808 01:52:22.183883  9560 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:22.185562  9552 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21. Ops: [20.12-20.12]
W20260808 01:52:22.153861 16356 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.798s	user 0.012s	sys 0.010s
W20260808 01:52:22.194392  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 58088.3us
I20260808 01:52:22.191526  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3432) took 1693 ms. Trace:
I20260808 01:52:22.193881 16415 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:22.224246 16338 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:22.194392  9638 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:22.200641 10334 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:22.228178  9637 consensus_meta.cc:287] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.522s	user 0.012s	sys 0.009s
I20260808 01:52:22.228765  9637 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:22.229292 16344 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:52:22.229782 16344 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:22.230183 16344 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:22.230089  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:22.230582 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:22.230997 16029 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:22.226298  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:22.232630  9638 raft_consensus.cc:2410] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 22 because replica has last-logged OpId of term: 21 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260808 01:52:22.233661 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:22.234004  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:22.201355 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:22.187546  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3141) took 1129 ms. Trace:
I20260808 01:52:22.201582 16346 consensus_queue.cc:1059] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:22.236043 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:22.236953  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3434) took 1633 ms. Trace:
I20260808 01:52:22.237267 10369 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:52:22.191999 16114 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:22.201576  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 73810.3us
I20260808 01:52:22.238430 16114 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:22.239383  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:22.240036  9521 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:22.242408 16114 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:22.187109 16124 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:52:22.189646 16284 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:22.210157 16369 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.917s	user 0.017s	sys 0.012s
I20260808 01:52:22.218138  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3430) took 1778 ms. Trace:
I20260808 01:52:22.320118  9552 rpcz_store.cc:276] 0808 01:52:20.440231 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.175334 (+1735103us) service_pool.cc:224] Handling call
0808 01:52:22.185015 (+  9681us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:22.217558 (+ 32543us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.218052 (+   494us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:22.217960  9550 rpcz_store.cc:276] 0808 01:52:20.498458 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.189932 (+1691474us) service_pool.cc:224] Handling call
0808 01:52:22.190524 (+   592us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:22.191044 (+   520us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:22.191069 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:22.191118 (+    49us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:22.191153 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.191222 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:22.191391 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:22.215464 16316 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.007s
I20260808 01:52:22.223057  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:22.223783 16017 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:22.224102  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3431) took 1744 ms. Trace:
I20260808 01:52:22.327284  9560 rpcz_store.cc:276] 0808 01:52:20.479538 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.175188 (+1695650us) service_pool.cc:224] Handling call
0808 01:52:22.183479 (+  8291us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:22.217007 (+ 33528us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.223870 (+  6863us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:22.232596 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:22.237334 16346 raft_consensus.cc:3055] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Stepping down as leader of term 9
I20260808 01:52:22.238122 16266 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:22.186604 16256 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:22.207100 15981 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:22.301222 16259 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:22.302277 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 55452us
I20260808 01:52:22.303149 16338 raft_consensus.cc:2764] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260808 01:52:22.263764  9558 rpcz_store.cc:276] 0808 01:52:20.604051 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.234833 (+1630782us) service_pool.cc:224] Handling call
0808 01:52:22.235798 (+   965us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:22.236415 (+   617us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:22.236471 (+    56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:22.236539 (+    68us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:22.236666 (+   127us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.236747 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:22.236887 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:22.303422 16129 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:22.303702 16290 consensus_meta.cc:287] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.673s	user 0.012s	sys 0.016s
I20260808 01:52:22.300524 16114 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:22.354349 16114 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:22.355619  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2879) took 1188 ms. Trace:
I20260808 01:52:22.356076  9631 rpcz_store.cc:276] 0808 01:52:21.167777 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:21.171366 (+  3589us) service_pool.cc:224] Handling call
0808 01:52:22.355154 (+1183788us) spinlock_profiling.cc:243] Waited 52.2 ms on lock 0x7b54003a0054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:22.355413 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":119948160}
I20260808 01:52:22.357012 16114 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:22.233701 16029 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:22.359606 16290 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:22.360013  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:22.358443 16259 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:22.361475  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:22.362277  9632 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:52:22.363004 16259 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:22.301457  9638 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 18: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:22.364856  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:22.365068  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:22.366235  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:22.371011  9638 raft_consensus.cc:2393] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 17: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:22.371785 15636 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:22.372572 15636 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:22.256032 16053 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:22.235423  9474 rpcz_store.cc:276] 0808 01:52:21.058223 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:21.058360 (+   137us) service_pool.cc:224] Handling call
0808 01:52:21.058800 (+   440us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:21.909138 (+850338us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:21.909234 (+    96us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:21.910432 (+  1198us) log.cc:844] Serialized 26 byte log entry
0808 01:52:22.010957 (+100525us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:22.011004 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.030449 (+ 19445us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:22.172977 (+142528us) raft_consensus.cc:1685] finished
0808 01:52:22.173002 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:22.187450 (+ 14448us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12091,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":5240,"thread_start_us":934519,"threads_started":2,"wal-append.queue_time_us":190795}
I20260808 01:52:22.263674  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:22.309854 16395 raft_consensus.cc:3055] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:22.310405 16249 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260808 01:52:22.312752  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:22.313534  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2873) took 1187 ms. Trace:
I20260808 01:52:22.316012  9637 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:22.252611 16415 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:22.316335 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:22.317657 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:22.327994  9633 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:52:22.331640 10361 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:22.380147  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:22.350917  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 111078us
I20260808 01:52:22.353641  9558 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:22.245407  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:22.367290 16129 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:22.366263  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:22.372483  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:22.389914  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3440) took 1306 ms. Trace:
I20260808 01:52:22.249974 16356 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:22.375118  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 152244us
I20260808 01:52:22.349391  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:22.373834 16053 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:22.329905 16346 raft_consensus.cc:740] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:22.357539  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:22.387617  9634 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:22.390331  9558 rpcz_store.cc:276] 0808 01:52:21.083866 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.348519 (+1264653us) service_pool.cc:224] Handling call
0808 01:52:22.353363 (+  4844us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:22.385550 (+ 32187us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.389845 (+  4295us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:22.382151  9636 rpcz_store.cc:276] 0808 01:52:21.125949 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:21.126100 (+   151us) service_pool.cc:224] Handling call
0808 01:52:21.146730 (+ 20630us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:21.993562 (+846832us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.313151 (+319589us) spinlock_profiling.cc:243] Waited 806 ms on lock 0x7b540008c854. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:22.313450 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1853412608}
I20260808 01:52:22.342610 16124 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:22.394464  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:22.381438 16395 raft_consensus.cc:740] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:22.406581  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2890) took 1088 ms. Trace:
I20260808 01:52:22.418324  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:22.419862  9560 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:22.421103  9631 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:22.423751  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3073) took 1310 ms. Trace:
I20260808 01:52:22.421125  9473 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:22.415993  9634 rpcz_store.cc:276] 0808 01:52:21.313591 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:21.950822 (+637231us) service_pool.cc:224] Handling call
0808 01:52:22.401735 (+450913us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:22.435137  9558 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:52:22.435292 16419 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.056s	user 0.000s	sys 0.002s
I20260808 01:52:22.435541  9474 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:22.437156  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:22.438282  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:22.439260  9551 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:22.437328  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:22.433269  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:22.439374  9475 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 10.
I20260808 01:52:22.490481  9475 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:22.502926  9524 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:22.493232 16356 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:22.503917  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 64207.1us
I20260808 01:52:22.505119  9524 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:22.505416  9632 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 17: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:22.505774  9475 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260808 01:52:22.506398 16284 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:22.491685  9472 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:22.507710 16284 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:22.496874 15981 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:22.515666 16227 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:22.518013  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2885) took 1295 ms. Trace:
W20260808 01:52:22.499316 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:22.524796 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:22.520059  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:22.519789  9632 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260808 01:52:22.518527  9635 rpcz_store.cc:276] 0808 01:52:21.222617 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:21.228663 (+  6046us) service_pool.cc:224] Handling call
0808 01:52:22.517665 (+1289002us) spinlock_profiling.cc:243] Waited 712 ms on lock 0x7b54001b44d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:22.517932 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1638273152}
W20260808 01:52:22.474900 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 55272us
I20260808 01:52:22.446182  9633 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:22.535202  9559 raft_consensus.cc:2393] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 17: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:22.465265  9473 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:22.461431  9521 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:22.528827  9636 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:22.564559  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:22.564878  9630 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 7. Preceding OpId from leader: term: 17 index: 8. (index mismatch)
I20260808 01:52:22.575203  9635 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260808 01:52:22.567747  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2899) took 1093 ms. Trace:
W20260808 01:52:22.536868  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 101662us
I20260808 01:52:22.536413  9552 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:22.567498  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:22.579749  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3149) took 1014 ms. Trace:
I20260808 01:52:22.583482  9473 rpcz_store.cc:276] 0808 01:52:21.552880 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.464208 (+911328us) service_pool.cc:224] Handling call
0808 01:52:22.464858 (+   650us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:22.566905 (+102047us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.567194 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:22.580744  9633 rpcz_store.cc:276] 0808 01:52:21.443455 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.405850 (+962395us) service_pool.cc:224] Handling call
0808 01:52:22.426830 (+ 20980us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:22.535332 (+108502us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.536147 (+   815us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:52:22.587553  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:22.590512  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 124752us
I20260808 01:52:22.590862  9636 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:22.591718 16129 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:22.595647  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3429) took 2158 ms. Trace:
I20260808 01:52:22.453351 16029 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:22.533917 16249 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:22.596205  9557 rpcz_store.cc:276] 0808 01:52:20.429344 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.175574 (+1746230us) service_pool.cc:224] Handling call
0808 01:52:22.313193 (+137619us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:22.314359 (+  1166us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:22.314397 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:22.315211 (+   814us) log.cc:844] Serialized 52 byte log entry
0808 01:52:22.408721 (+ 93510us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:52:22.408765 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.408841 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:22.587418 (+178577us) raft_consensus.cc:1685] finished
0808 01:52:22.587443 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:22.587622 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":155,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":160,"thread_start_us":92838,"threads_started":1,"wal-append.queue_time_us":184861}
I20260808 01:52:22.533967 16415 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:22.592185 16227 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:52:22.537173 16290 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:22.433282  9479 rpcz_store.cc:276] 0808 01:52:21.084768 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:21.084937 (+   169us) service_pool.cc:224] Handling call
0808 01:52:21.085561 (+   624us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:21.204183 (+118622us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:21.204234 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:21.229769 (+ 25535us) log.cc:844] Serialized 590 byte log entry
0808 01:52:21.349891 (+120122us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:21.349958 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.318887 (+968929us) spinlock_profiling.cc:243] Waited 93.7 ms on lock 0x7b54001013d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:22.318962 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:22.319048 (+    86us) raft_consensus.cc:1685] finished
0808 01:52:22.319063 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:22.395047 (+ 75984us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":94311,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"spinlock_wait_cycles":215540608,"thread_start_us":115314,"threads_started":1,"wal-append.queue_time_us":410230}
I20260808 01:52:22.625219  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:22.611270 16227 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:22.627444 16227 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:52:22.628013  9600 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:22.619242 16338 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:52:22.632680 16337 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.881s	user 0.007s	sys 0.014s
I20260808 01:52:22.633013 16129 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:22.633127 16338 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:22.633358 16337 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:22.636008  9557 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:22.633769 16129 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:22.629863  9600 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:22.635416  9632 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:22.610657 16249 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:22.609234 16415 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:22.611596 16356 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:22.649376 16337 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:22.657860  9472 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:22.658721 16129 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:22.659510 16290 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:22.661985 16290 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:22.499531 16346 consensus_queue.cc:260] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:22.644696 16017 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:22.652053  9631 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:22.596055 16124 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:22.676307  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:22.682315  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:22.685817 16338 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:22.689651 10363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 752) took 1109 ms. Trace:
W20260808 01:52:22.691689 16421 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260808 01:52:22.697003 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:22.697260 16356 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:22.700975 16356 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:22.702394  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3419) took 2922 ms. Trace:
I20260808 01:52:22.703023  9554 rpcz_store.cc:276] 0808 01:52:19.780278 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.084578 (+2304300us) service_pool.cc:224] Handling call
0808 01:52:22.103063 (+ 18485us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:22.122148 (+ 19085us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.702094 (+579946us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b5400325ad0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:22.702318 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24359296}
I20260808 01:52:22.704517  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:22.708179 16227 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:22.651046 16344 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:22.712397 15933 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:22.713470 15933 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:22.719236 16227 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:22.665902 16017 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:22.500087 16395 consensus_queue.cc:260] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:22.733641 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 150484us
I20260808 01:52:22.681502  9552 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260808 01:52:22.690227 10363 rpcz_store.cc:276] 0808 01:52:21.579976 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:21.582298 (+  2322us) service_pool.cc:224] Handling call
0808 01:52:21.590935 (+  8637us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:21.647856 (+ 56921us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.689144 (+1041288us) spinlock_profiling.cc:243] Waited 8.23 ms on lock 0x7b5400681d50. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:22.689402 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18923392}
I20260808 01:52:22.665532 16249 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:22.754551  9473 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:22.754985  9558 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260808 01:52:22.741930 16284 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:22.762470 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:22.651997 16415 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:22.713459 16344 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:22.733999 16017 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:22.746438 16395 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:22.780633  9637 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:22.784600  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:22.785745  9479 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:52:22.788194  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 370187us
I20260808 01:52:22.790961 16338 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:22.798511 10368 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260808 01:52:22.798349  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:22.784507 16356 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:22.810648  9472 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 6. Preceding OpId from leader: term: 16 index: 7. (index mismatch)
I20260808 01:52:22.829818 16290 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:22.830794 16290 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:22.831362 16290 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:22.774438 15933 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:22.843142  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 44594us
I20260808 01:52:22.866426  9521 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:22.875139  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:22.811480 16356 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:22.817411 15636 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:22.893705 16378 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:22.894412 16378 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:22.855608 16338 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:22.913486  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3424) took 2505 ms. Trace:
I20260808 01:52:22.925707 16202 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:22.925875 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:22.927290 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:22.929231 10367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 832) took 1636 ms. Trace:
I20260808 01:52:22.931432  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3074) took 1699 ms. Trace:
I20260808 01:52:22.931871  9478 rpcz_store.cc:276] 0808 01:52:21.231895 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:21.915236 (+683341us) service_pool.cc:224] Handling call
0808 01:52:21.916334 (+  1098us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:22.167975 (+251641us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.931127 (+763152us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b540023aad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:22.931332 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":577035520}
I20260808 01:52:22.933749  9478 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260808 01:52:22.900328 16337 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:22.939257  9472 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:22.952692 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:22.929685 10367 rpcz_store.cc:276] 0808 01:52:21.292353 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:21.293743 (+  1390us) service_pool.cc:224] Handling call
0808 01:52:22.103281 (+809538us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:22.103985 (+   704us) raft_consensus.cc:1534] Early marking committed up to index 14
0808 01:52:22.327322 (+223337us) log.cc:844] Serialized 14 byte log entry
0808 01:52:22.432815 (+105493us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:22.432882 (+    67us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:52:22.432932 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.453809 (+ 20877us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:22.928845 (+475036us) spinlock_profiling.cc:243] Waited 809 ms on lock 0x7b540041f0d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:22.929039 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":315250,"prepare.run_wall_time_us":31464,"raft.queue_time_us":7457,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":32695,"spinlock_wait_cycles":1908173056,"thread_start_us":252411,"threads_started":2,"wal-append.queue_time_us":540072}
W20260808 01:52:22.973165  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:22.943545 16344 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:22.914034  9556 rpcz_store.cc:276] 0808 01:52:20.407751 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.131101 (+1723350us) service_pool.cc:224] Handling call
0808 01:52:22.131802 (+   701us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:22.192483 (+ 60681us) raft_consensus.cc:1534] Early marking committed up to index 9
0808 01:52:22.398772 (+206289us) log.cc:844] Serialized 14 byte log entry
0808 01:52:22.584464 (+185692us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:22.648360 (+ 63896us) log.cc:844] Serialized 26 byte log entry
0808 01:52:22.648525 (+   165us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:22.648569 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:22.648640 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:22.910708 (+262068us) raft_consensus.cc:1685] finished
0808 01:52:22.910735 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:22.913403 (+  2668us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":68031,"prepare.queue_time_us":390665,"prepare.run_cpu_time_us":134,"prepare.run_wall_time_us":5920,"raft.queue_time_us":65667,"raft.run_cpu_time_us":440,"raft.run_wall_time_us":10536,"thread_start_us":355772,"threads_started":2,"wal-append.queue_time_us":367501}
W20260808 01:52:22.979467  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 126668us
I20260808 01:52:22.984747  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:22.984844  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:22.985837  9554 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:22.993705 16229 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260808 01:52:22.999715  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 147457us
I20260808 01:52:22.985824 16029 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:23.014583  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:23.017076 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:23.014464  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:23.020464 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:23.025094  9554 raft_consensus.cc:2410] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 20 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260808 01:52:23.020154  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:23.004926 16029 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:23.038456  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:23.052212  9473 raft_consensus.cc:2410] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 8.
I20260808 01:52:23.005069 16344 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:23.057542  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:23.058565  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:23.060877 16150 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:52:23.049741 16338 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:23.063318  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:23.062783 15636 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:23.082230 16398 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.415s	user 0.001s	sys 0.003s
W20260808 01:52:23.085798  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:23.091548 16329 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:23.094229  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2904) took 1544 ms. Trace:
I20260808 01:52:23.100723  9634 rpcz_store.cc:276] 0808 01:52:21.549969 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.434189 (+884220us) service_pool.cc:224] Handling call
0808 01:52:22.483522 (+ 49333us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:22.619942 (+136420us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:52:22.649748 (+ 29806us) log.cc:844] Serialized 14 byte log entry
0808 01:52:22.981894 (+332146us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:22.981955 (+    61us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:22.985630 (+  3675us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:23.093730 (+108100us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b540027e054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:23.093839 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:23.094111 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":32961,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":52,"raft.queue_time_us":141,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":23819,"spinlock_wait_cycles":279291776,"thread_start_us":198329,"threads_started":1,"wal-append.queue_time_us":322890}
I20260808 01:52:23.097113  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:23.104560  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:23.105635  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:23.100132  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:23.107326  9554 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:23.113834  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:23.115384  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:23.051724 16290 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:23.123346  9479 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:52:23.123832  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:23.114600  9558 raft_consensus.cc:2376] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17. Re-sending same reply.
I20260808 01:52:23.108590  9557 raft_consensus.cc:2376] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17. Re-sending same reply.
W20260808 01:52:23.128193 16432 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260808 01:52:23.133814  9521 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:23.096150 16356 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:23.137841  9556 raft_consensus.cc:2410] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260808 01:52:23.141516  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:23.117406 15925 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:23.146701  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:23.148249  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:23.109939 15933 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:23.153339  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:23.135685 16338 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:23.155426 16114 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:23.159125 16335 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260808 01:52:23.157078  9554 raft_consensus.cc:2376] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19. Re-sending same reply.
I20260808 01:52:23.139952 15636 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.036s
I20260808 01:52:23.157742 16338 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:23.152014 16415 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:23.191864 16290 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:23.182855 16338 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:23.160804 15925 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:23.198966 15925 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:23.200776  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:23.202191 16202 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:23.202721 16202 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:23.203132 16202 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:23.210121  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:23.185029 16356 consensus_queue.cc:1059] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
I20260808 01:52:23.215359 16290 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:52:23.176057 15933 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:23.223704 15933 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:23.231706 16415 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:23.239528  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 105892us
I20260808 01:52:23.248848 16150 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:23.250779  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:23.251771  9555 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:52:23.252275  9555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3442) took 1895 ms (client timeout 1997 ms). Trace:
W20260808 01:52:23.252668  9555 rpcz_store.cc:269] 0808 01:52:21.357312 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:23.250346 (+1893034us) service_pool.cc:224] Handling call
0808 01:52:23.252205 (+  1859us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:23.255867  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3400) took 4444 ms. Trace:
I20260808 01:52:23.256461  9551 rpcz_store.cc:276] 0808 01:52:18.811667 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:18.811889 (+   222us) service_pool.cc:224] Handling call
0808 01:52:18.812474 (+   585us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:22.571291 (+3758817us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:23.255374 (+684083us) spinlock_profiling.cc:243] Waited 3.59 s on lock 0x7b540002c6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:23.255757 (+   383us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8251465472}
W20260808 01:52:23.309913 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:23.354744 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:23.254396 16202 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:23.463317  9444 leader_election.cc:336] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Timed out: RequestConsensusVote RPC to 127.7.165.194:36095 timed out after 1.510s (SENT)
W20260808 01:52:23.465086  9444 leader_election.cc:336] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Timed out: RequestConsensusVote RPC to 127.7.165.194:36095 timed out after 1.535s (SENT)
I20260808 01:52:23.465613  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:23.467146 10365 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:23.470944 16215 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:23.471628 16215 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:23.483389  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 282849us
I20260808 01:52:23.484743  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:23.486147 15933 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260808 01:52:23.520797 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:23.578963  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 90868.2us
I20260808 01:52:23.611904 16442 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260808 01:52:23.608989 16335 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:23.614353 16335 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:23.618175 16335 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:23.548554  9521 connection.cc:256] RPC call timeout handler was delayed by -1.2946423280227464s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.168s for a response
W20260808 01:52:23.680873  9521 connection.cc:256] RPC call timeout handler was delayed by -1.2748423280136194s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17020000000000002s for a response
W20260808 01:52:23.681381  9521 connection.cc:256] RPC call timeout handler was delayed by -1.2109423280053306s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1773s for a response
W20260808 01:52:23.681792  9521 connection.cc:256] RPC call timeout handler was delayed by -1.2082423280226067s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1776s for a response
W20260808 01:52:23.682188  9521 connection.cc:256] RPC call timeout handler was delayed by -1.1695423280179966s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1819s for a response
I20260808 01:52:23.685653 16150 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:23.686610  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2614) took 1380 ms. Trace:
I20260808 01:52:23.687333  9560 rpcz_store.cc:276] 0808 01:52:22.306151 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.332443 (+ 26292us) service_pool.cc:224] Handling call
0808 01:52:23.686346 (+1353903us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b54001ac554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:23.686521 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":38505600}
W20260808 01:52:23.682550  9521 connection.cc:256] RPC call timeout handler was delayed by -1.163242328009801s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1826s for a response
W20260808 01:52:23.689194  9521 connection.cc:256] RPC call timeout handler was delayed by -1.1164423279988114s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1878s for a response
I20260808 01:52:23.691524 16053 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:23.692312 16053 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:23.692165 16150 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:23.691569 16202 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:23.718187 16202 consensus_queue.cc:1059] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:52:23.705909 16053 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:23.720019 16053 raft_consensus.cc:3055] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:23.720534 16053 raft_consensus.cc:740] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:23.721791 16053 consensus_queue.cc:260] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:23.735559  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1096) took 1366 ms. Trace:
W20260808 01:52:23.733580 16029 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.680s	user 0.015s	sys 0.000s
W20260808 01:52:23.690891 16017 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.901s	user 0.015s	sys 0.002s
I20260808 01:52:23.759395  9474 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17. Ops: []
W20260808 01:52:23.760771  9480 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 3144) took 2636 ms (client timeout 2076 ms). Trace:
W20260808 01:52:23.761314  9480 rpcz_store.cc:269] 0808 01:52:21.122710 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:21.134577 (+ 11867us) service_pool.cc:224] Handling call
0808 01:52:23.758085 (+2623508us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b54002a6cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:23.758403 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":338719616}
W20260808 01:52:23.766170  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:23.771798  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 306751us
I20260808 01:52:23.775570  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:23.777860  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:23.780122  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:23.781702 16290 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:23.782510 16290 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:23.784679 16290 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:23.781690 15933 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.004s
I20260808 01:52:23.787081  9634 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 13. Current term is 14. Ops: []
I20260808 01:52:23.766719 16017 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:23.790203 16017 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:23.792119 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:23.799949 16338 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.562s	user 0.012s	sys 0.000s
I20260808 01:52:23.801586 16337 raft_consensus.cc:3055] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:23.801841  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3072) took 2722 ms. Trace:
I20260808 01:52:23.800595 16338 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:23.748068 16029 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:23.802202 16337 raft_consensus.cc:740] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:23.810042 16443 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:23.814782 16443 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:23.815599 16337 consensus_queue.cc:260] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:23.817432 16443 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:23.817760 16337 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:23.820209 16443 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:23.802374  9476 rpcz_store.cc:276] 0808 01:52:21.079769 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:21.079960 (+   191us) service_pool.cc:224] Handling call
0808 01:52:21.080304 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:21.215503 (+135199us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:23.801250 (+2585747us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b54004047d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:23.801615 (+   365us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":309084032}
I20260808 01:52:23.820793 16344 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:23.821580 10365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 753) took 2231 ms (client timeout 1730 ms). Trace:
W20260808 01:52:23.838809  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 129097us
W20260808 01:52:23.846050 10365 rpcz_store.cc:269] 0808 01:52:21.590178 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:21.590435 (+   257us) service_pool.cc:224] Handling call
0808 01:52:23.807252 (+2216817us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x7b540046c154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:23.821465 (+ 14213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3709730688}
I20260808 01:52:23.821837 15899 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:23.822450 16367 raft_consensus.cc:3055] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:23.824049 16346 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:52:23.821846 16249 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.033s	user 0.011s	sys 0.005s
I20260808 01:52:23.860340 16249 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:23.862016 15899 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:23.863341  9559 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 16: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:23.828145 16338 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:23.736294  9478 rpcz_store.cc:276] 0808 01:52:22.369136 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.934585 (+565449us) service_pool.cc:224] Handling call
0808 01:52:22.951931 (+ 17346us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:23.041927 (+ 89996us) raft_consensus.cc:1534] Early marking committed up to index 15
0808 01:52:23.167472 (+125545us) log.cc:844] Serialized 14 byte log entry
0808 01:52:23.734069 (+566597us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:23.734138 (+    69us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:52:23.735238 (+  1100us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:23.735320 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:23.735482 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":182860,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":96,"raft.queue_time_us":40136,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"spinlock_wait_cycles":2082560,"thread_start_us":645900,"threads_started":3,"wal-append.queue_time_us":389706}
I20260808 01:52:23.811008  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2903) took 2261 ms. Trace:
I20260808 01:52:23.847610  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:23.865870  9473 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 19
I20260808 01:52:23.858772 16229 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:23.867561 16229 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:23.838419 16443 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:23.875631  9480 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:23.878403 16338 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260808 01:52:23.875720 16425 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.910s	user 0.007s	sys 0.040s
I20260808 01:52:23.883108 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:23.883911 10368 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:23.885252 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:23.886066 10364 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:23.887334 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:23.889981  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:23.891122  9635 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
W20260808 01:52:23.892742  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:23.892614  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:23.893819  9635 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 22: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:23.888190 16249 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:23.894569  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:23.888797 16215 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:23.865641  9630 rpcz_store.cc:276] 0808 01:52:21.549466 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.422616 (+873150us) service_pool.cc:224] Handling call
0808 01:52:22.528879 (+106263us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:22.577466 (+ 48587us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:23.810698 (+1233232us) spinlock_profiling.cc:243] Waited 52.7 ms on lock 0x7b54001b44d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:23.810928 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":121272192}
I20260808 01:52:23.897045 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:23.897629  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:23.897651  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:23.898382  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2792) took 1213 ms. Trace:
I20260808 01:52:23.898524 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:23.898837  9631 rpcz_store.cc:276] 0808 01:52:22.687715 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:23.897342 (+1209627us) service_pool.cc:224] Handling call
0808 01:52:23.898301 (+   959us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:23.899724  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:23.865124 15899 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:23.869798 16229 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:23.855329 16367 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:23.895481  9632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 825) took 1230 ms (client timeout 1610 ms). Trace:
I20260808 01:52:23.901587  9629 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:23.899893  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:23.904794  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:23.849277  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:23.908789  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1089) took 1828 ms. Trace:
I20260808 01:52:23.901870 16444 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:52:23.905362  9632 rpcz_store.cc:269] 0808 01:52:22.675570 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:23.894222 (+1218652us) service_pool.cc:224] Handling call
0808 01:52:23.895422 (+  1200us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:23.914314  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:23.914171  9474 rpcz_store.cc:276] 0808 01:52:22.080767 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.620900 (+540133us) service_pool.cc:224] Handling call
0808 01:52:23.759079 (+1138179us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:23.760211 (+  1132us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:23.908412 (+148201us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54002a6cd0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:23.908705 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2492566400}
I20260808 01:52:23.907295 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:23.878481  9473 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:23.906833  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:23.917302 10362 raft_consensus.cc:2410] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260808 01:52:23.905107 16367 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:23.920397 16367 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:23.921366 16369 raft_consensus.cc:3055] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Stepping down as leader of term 16
W20260808 01:52:23.921521  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 62856.7us
I20260808 01:52:23.922127 16369 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:23.923341  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:23.924407  9479 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260808 01:52:23.923568  9481 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 3137) took 2886 ms (client timeout 1782 ms). Trace:
I20260808 01:52:23.918411  9630 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 22: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:23.925439  9629 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
W20260808 01:52:23.926841  9481 rpcz_store.cc:269] 0808 01:52:21.037639 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:21.037964 (+   325us) service_pool.cc:224] Handling call
0808 01:52:23.923266 (+2885302us) spinlock_profiling.cc:243] Waited 878 ms on lock 0x7b54002dc3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:23.923483 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2018384256}
I20260808 01:52:23.930639 16444 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:23.933525 16444 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:23.935350  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:23.933064 16029 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:23.945250 16017 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:23.947870 16447 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:23.936693 16369 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:23.937779  9445 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:23.938794  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:23.942605 16371 consensus_queue.cc:1059] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:23.065018  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:52:23.951253  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:23.956152  9637 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 13: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:52:23.964891 16440 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.389s	user 0.005s	sys 0.005s
I20260808 01:52:23.967500  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:23.942178  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:23.968807  9633 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:23.969297  9550 raft_consensus.cc:2393] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 11: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:52:23.969774  9550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2615) took 1663 ms (client timeout 1535 ms). Trace:
W20260808 01:52:23.970161  9550 rpcz_store.cc:269] 0808 01:52:22.306529 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:22.327529 (+ 21000us) service_pool.cc:224] Handling call
0808 01:52:23.941937 (+1614408us) spinlock_profiling.cc:243] Waited 2.12 ms on lock 0x7b5800003c98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:52:23.969701 (+ 27764us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4880256}
I20260808 01:52:23.970793  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:23.973369 16378 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:23.975618  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:23.976352  9479 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:23.976922  9475 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 10. Current term is 11. Ops: []
I20260808 01:52:23.979113  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:23.979913  9476 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:52:23.980140 16223 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:23.984553  9599 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:23.986013 16202 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:23.986814 16053 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:23.988246 16356 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:23.953239  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:23.989614 16371 raft_consensus.cc:3055] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:23.992170  9631 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:52:24.025975  9480 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:52:23.993929  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 81166.2us
I20260808 01:52:23.994086 16290 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:23.997789  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:23.998821  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:24.030304  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:23.999814  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:24.000806 15636 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:24.004290 16369 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:23.951582 16447 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:23.937280  9632 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260808 01:52:24.035693 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 359657us
I20260808 01:52:24.020210  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:24.028788  9445 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:24.031126  9635 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:24.030351  9633 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:24.034375 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:24.037616 10367 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:24.031023  9636 raft_consensus.cc:2376] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17. Re-sending same reply.
I20260808 01:52:24.039145  9638 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:52:24.045739  9479 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:52:24.012538 16444 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:24.019608 16356 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:24.009778 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:24.025986 16371 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:24.049615 16316 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:24.034229 16290 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:24.059705  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 75196.4us
I20260808 01:52:24.036640 16227 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:24.053450  9445 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:24.100035  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 46718.1us
I20260808 01:52:24.053793  9474 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
W20260808 01:52:24.122395  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:24.122947  9552 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:24.124037  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2622) took 1362 ms. Trace:
I20260808 01:52:24.124361  9552 rpcz_store.cc:276] 0808 01:52:22.765649 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:24.115767 (+1350118us) service_pool.cc:224] Handling call
0808 01:52:24.116176 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:24.123761 (+  7585us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:24.123965 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:24.120893  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:24.089365 16346 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:24.079214 15636 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:24.129168  9473 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:24.129019  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:24.130388  9474 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260808 01:52:24.103048 16029 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:24.135165  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:24.138460  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:24.140674  9473 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:24.088964 16316 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:24.138046  9476 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 12. Current term is 13. Ops: []
I20260808 01:52:24.118350 16338 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:24.138731  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.146947  9552 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:24.141058  9478 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:24.078796 16444 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:24.052531 16406 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.080s
I20260808 01:52:24.152204  9481 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260808 01:52:24.152614  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:24.132681 15636 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:24.155439 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:24.159705  9552 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 9 FOLLOWER]: Advancing to term 10
W20260808 01:52:24.164265  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 113116us
I20260808 01:52:24.146711 16338 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:24.171348 16338 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:24.165935 10334 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:24.175137  9635 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:24.175997  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:24.151742 16389 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:24.132778 16227 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:24.135221 16346 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:24.181740  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.155805 16457 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:52:24.185367 10334 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:24.161914 15636 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:24.195019  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.195951  9559 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 15: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:52:24.205051  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 87261us
I20260808 01:52:24.199849  9524 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:24.211009  9473 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:24.181757 16395 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:52:24.184264 16345 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:52:24.215418 16279 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:24.215611 16304 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:24.216499 16400 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:24.216771  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.217253 16229 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:24.217615 16249 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 01:52:24.218907  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.219646  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.223121  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:24.214359  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 190761us
I20260808 01:52:24.177078 16378 consensus_queue.cc:1059] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.030s
I20260808 01:52:24.230854  9599 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:24.214309  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:24.264246  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:24.267309  9556 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:24.264626  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:24.268304  9558 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
W20260808 01:52:24.270409  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 44593.6us
W20260808 01:52:24.272364  9377 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55058 (request call id 130) took 31288 ms (client timeout 15000 ms). Trace:
I20260808 01:52:24.272614  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:24.274869 16455 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
W20260808 01:52:24.278329 16454 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260808 01:52:24.280180  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.161353 16356 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:24.270506  9636 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 18. Current term is 19. Ops: []
W20260808 01:52:24.289906  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 59160.1us
I20260808 01:52:24.236068  9551 raft_consensus.cc:2410] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 19 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260808 01:52:24.290942  9599 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:24.294904  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 80728.2us
I20260808 01:52:24.301483  9559 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:24.302793  9552 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10.
I20260808 01:52:24.304890  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.293527 16017 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:24.184725 16451 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.115s
I20260808 01:52:24.261505 16397 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:24.325026 16397 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:24.325556 16397 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:24.327916 16150 consensus_queue.cc:1059] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:52:24.235678 16395 raft_consensus.cc:697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:24.314007  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.185002 16389 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:24.351333  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:24.351637  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:24.353361 16389 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:24.354180 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 54549us
I20260808 01:52:24.317199 16125 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:24.362993 16125 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:24.363513 16125 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:24.185390 16371 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:24.367051 16462 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:24.367800 16462 raft_consensus.cc:697] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:24.369187 16462 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:24.370666 16227 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:24.371477  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.372407  9555 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:24.373309  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.380941 16444 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:24.382632 16316 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:24.383530 16290 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:24.298466 16345 raft_consensus.cc:697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:24.388494 16345 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:24.389629 16125 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:24.294338 16229 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:24.395656 16229 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:24.397341  9560 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:24.399415  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.400260  9555 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:24.401216  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.401916  9555 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:24.402827  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.404572 16406 consensus_queue.cc:1059] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260808 01:52:24.405831 16406 consensus_queue.cc:1059] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.000s
I20260808 01:52:24.408286 16242 raft_consensus.cc:3055] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:52:24.408717 16242 raft_consensus.cc:740] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:24.409739 16242 consensus_queue.cc:260] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:24.303632 16437 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.108s
I20260808 01:52:24.413234 16194 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:24.314994 16017 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:24.413861 16194 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:24.414028 16017 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:24.415302 16415 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:24.415750 16415 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:24.416687 16415 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:24.419292 16395 consensus_queue.cc:237] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:24.421767 16346 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:24.322791 16443 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:24.322984 16383 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.099s
I20260808 01:52:24.337574  9473 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:24.312322  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:24.306165 16356 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:24.457777  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.229313 16279 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:24.343189 10367 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:52:24.346832  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:24.339648  9554 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:24.336918 16386 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:24.367800 16371 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:24.486289  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:24.446120  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:24.487877  9524 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:24.488253  9551 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
W20260808 01:52:24.451439  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:24.321266  9523 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:24.456516  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:24.398618  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:24.459154  9555 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 15: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:24.425592 16329 raft_consensus.cc:3055] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Stepping down as leader of term 10
I20260808 01:52:24.451112 16415 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:24.478235  9559 raft_consensus.cc:2393] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:52:24.481334  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 168669us
I20260808 01:52:24.481940  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:24.492425  9601 connection.cc:256] RPC call timeout handler was delayed by -1.6448671399848536s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17820000000000003s for a response
W20260808 01:52:24.499115  9601 connection.cc:256] RPC call timeout handler was delayed by -1.3802671399898827s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2076s for a response
I20260808 01:52:24.496685  9554 raft_consensus.cc:2393] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 17: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:24.473614  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:24.494136  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.495553  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:24.500783  9555 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
W20260808 01:52:24.503145 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52477.8us
I20260808 01:52:24.511546  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:24.514732  9475 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:24.515467  9474 raft_consensus.cc:2393] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:24.486594  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:24.480475 16029 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:24.517953  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:24.523763  9473 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 16: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:24.519182  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:24.519627  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:24.525867  9559 raft_consensus.cc:2376] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 10. Re-sending same reply.
I20260808 01:52:24.479116 16279 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:24.469429 16290 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:24.536633  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.476426 16444 raft_consensus.cc:3055] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:52:24.527328  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:24.543627  9479 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:24.543830  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:24.522244  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.530228  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.531203  9478 raft_consensus.cc:2393] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:24.538017  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:24.539425  9558 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:24.539570  9472 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260808 01:52:24.548525  9633 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: []
I20260808 01:52:24.543071 16444 raft_consensus.cc:740] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:24.549628  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:24.551594  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:24.502796 16316 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:24.552533  9479 raft_consensus.cc:2410] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 22 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260808 01:52:24.497377 16329 raft_consensus.cc:740] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:24.554071  9559 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 18: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:24.474601 16367 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.070s
I20260808 01:52:24.557861  9636 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260808 01:52:24.578364  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:24.573696 16290 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:24.574998 16437 raft_consensus.cc:3055] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Stepping down as leader of term 17
W20260808 01:52:24.570403 16461 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.278s	user 0.000s	sys 0.001s
W20260808 01:52:24.272908  9377 rpcz_store.cc:269] 0808 01:51:52.984363 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:52.984613 (+   250us) service_pool.cc:224] Handling call
0808 01:52:24.272019 (+31287406us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:52:12.712872 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:52:12.747660 (+ 34788us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:52:12.747729 (+    69us) write_op.cc:435] Acquired schema lock
0808 01:52:12.747748 (+    19us) tablet.cc:662] Decoding operations
0808 01:52:12.749261 (+  1513us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:52:12.749382 (+   121us) write_op.cc:641] Partition lock acquired for write op
0808 01:52:12.749418 (+    36us) tablet.cc:685] Acquiring locks for 16 operations
0808 01:52:12.756018 (+  6600us) tablet.cc:701] Row locks acquired
0808 01:52:12.756056 (+    38us) write_op.cc:260] PREPARE: finished
0808 01:52:12.756254 (+   198us) write_op.cc:270] Start()
0808 01:52:12.756443 (+   189us) write_op.cc:276] Timestamp: P: 1786153932756181 usec, L: 0
0808 01:52:12.756486 (+    43us) op_driver.cc:348] REPLICATION: starting
0808 01:52:12.757333 (+   847us) log.cc:844] Serialized 6278 byte log entry
0808 01:52:13.323963 (+566630us) op_driver.cc:464] REPLICATION: finished
0808 01:52:13.407090 (+ 83127us) write_op.cc:301] APPLY: starting
0808 01:52:13.416360 (+  9270us) tablet.cc:1367] starting BulkCheckPresence
0808 01:52:13.717781 (+301421us) tablet.cc:1370] finished BulkCheckPresence
0808 01:52:13.717920 (+   139us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:52:14.647838 (+929918us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:52:14.648455 (+   617us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=32,key_file_lookups=32,delta_file_lookups=32,mrs_lookups=0
0808 01:52:14.648526 (+    71us) write_op.cc:312] APPLY: finished
0808 01:52:14.649052 (+   526us) log.cc:844] Serialized 148 byte log entry
0808 01:52:14.699740 (+ 50688us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:52:14.740487 (+ 40747us) write_op.cc:454] Released schema lock
0808 01:52:14.740632 (+   145us) write_op.cc:341] FINISH: Updating metrics
0808 01:52:14.741200 (+   568us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:52:14.741238 (+    38us) write_op.cc:454] Released schema lock
0808 01:52:15.040244 (+299006us) spinlock_profiling.cc:243] Waited 41.4 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
Metrics: {"mutex_wait_us":3431286,"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":67451,"apply.run_cpu_time_us":37370,"apply.run_wall_time_us":1346009,"cfile_cache_hit":66,"cfile_cache_hit_bytes":71103,"num_ops":16,"prepare.queue_time_us":275448,"prepare.run_cpu_time_us":32556,"prepare.run_wall_time_us":287195,"raft.queue_time_us":303495,"raft.run_cpu_time_us":6636,"raft.run_wall_time_us":67829,"replication_time_us":567236,"spinlock_wait_cycles":95184768,"thread_start_us":520463,"threads_started":5,"wal-append.queue_time_us":272897,"wal-append.run_cpu_time_us":81342,"wal-append.run_wall_time_us":2616216}]]}
I20260808 01:52:24.577610  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:24.580561 16406 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:24.537245 16304 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260808 01:52:24.582643  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 261556us
I20260808 01:52:24.578096 16456 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260808 01:52:24.590377  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 283110us
I20260808 01:52:24.325340 16447 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:24.575327 16242 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:52:24.605370 16337 consensus_meta.cc:287] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.759s	user 0.009s	sys 0.004s
I20260808 01:52:24.603876  9638 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:24.494174 16386 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:24.627862  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:24.548652 16125 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:24.609012  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:24.649147  9474 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:52:24.638584  9475 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 20: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:24.642433  9636 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:24.597055 16316 raft_consensus.cc:697] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:24.550310 16444 consensus_queue.cc:260] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:24.654765  9630 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:24.601423 16437 raft_consensus.cc:740] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:24.661935 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:24.669317  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:24.682804  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:24.699276  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:24.700356  9638 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:24.704873 15636 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:24.501997 16194 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:24.639750 16406 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:24.712708 10332 leader_election.cc:336] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Timed out: RequestConsensusVote RPC to 127.7.165.194:36095 timed out after 1.535s (SENT)
W20260808 01:52:24.719174 10332 leader_election.cc:336] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.535s (SENT)
I20260808 01:52:24.719969 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:24.730504  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:24.738616  9636 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260808 01:52:24.657078 16337 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:24.637720 16304 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:52:24.744443  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 233006us
W20260808 01:52:24.752398  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:24.722750  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:24.759250  9472 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:24.762322  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:24.764317  9479 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
I20260808 01:52:24.698458  9551 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:24.758075  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:24.689242 16443 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260808 01:52:24.776831  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:24.780797 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 61738.1us
W20260808 01:52:24.781805 10332 leader_election.cc:336] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.610s (SENT)
W20260808 01:52:24.782593 10332 leader_election.cc:336] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Timed out: RequestConsensusVote RPC to 127.7.165.194:36095 timed out after 1.610s (SENT)
I20260808 01:52:24.783145 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:24.755550 16461 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260808 01:52:24.785020  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:24.647536 16397 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:24.758111 16444 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:52:24.790465  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 162739us
W20260808 01:52:24.793624  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 145517us
I20260808 01:52:24.797256  9636 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260808 01:52:24.795284  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:24.771462 16447 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:24.775586 16194 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
W20260808 01:52:24.803262  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:24.805442  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 47516.3us
I20260808 01:52:24.788223 16346 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:24.823117  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:24.829479 16290 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:24.834811 16443 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:24.835402 16443 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:24.835824 16443 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:24.836808 10334 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:24.837669 16467 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260808 01:52:24.837827 15899 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:24.837961 10334 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:24.838409 15899 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:24.838982 15899 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:24.841681  9475 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:24.756001 16329 consensus_queue.cc:260] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:24.844952 16329 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:52:24.818868 16316 consensus_queue.cc:237] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:24.769563 16337 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:24.851501  9475 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 10. Current term is 11. Ops: []
I20260808 01:52:24.852576 16337 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:24.855317  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46114.8us
I20260808 01:52:24.856189  9599 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:24.765483 16406 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:24.856977  9630 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:24.862015  9638 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 6. Preceding OpId from leader: term: 16 index: 7. (index mismatch)
W20260808 01:52:24.873356  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:24.801988 16202 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:24.805501 16417 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260808 01:52:24.925653  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 414785us
I20260808 01:52:24.875057 16346 consensus_queue.cc:1059] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.051s
I20260808 01:52:24.927445  9600 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:24.927650  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:24.927606  9636 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260808 01:52:24.931145 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:24.927352 15636 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:24.934900  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 139690us
W20260808 01:52:24.863900 16053 consensus_meta.cc:287] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.845s	user 0.014s	sys 0.000s
I20260808 01:52:24.774073 16304 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:24.944222  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 88106us
W20260808 01:52:24.878456 16223 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.898s	user 0.008s	sys 0.007s
I20260808 01:52:24.874832 15856 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260808 01:52:24.944980  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:24.969746  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2802) took 1118 ms. Trace:
I20260808 01:52:24.947149  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:24.960407 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:24.875862 16029 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:24.980365 16029 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:24.981570 16304 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:24.984069 16406 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:24.984733  9373 catalog_manager.cc:5697] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 6 to 11, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 11 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:52:24.991593 16337 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:24.992136 16337 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:24.992491 16337 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:24.993657  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:24.994410  9475 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:24.995743  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:24.927168 16367 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:24.967291  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:24.997051  9636 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:24.971271  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:24.998438  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:24.971956  9630 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:24.996357  9475 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
W20260808 01:52:25.000507  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:24.861119 16465 consensus_queue.cc:1059] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.103s
I20260808 01:52:24.943058 16462 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:24.971598  9634 rpcz_store.cc:276] 0808 01:52:23.851263 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:23.858554 (+  7291us) service_pool.cc:224] Handling call
0808 01:52:23.859366 (+   812us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:23.918634 (+ 59268us) raft_consensus.cc:1534] Early marking committed up to index 7
0808 01:52:24.086248 (+167614us) log.cc:844] Serialized 14 byte log entry
0808 01:52:24.294787 (+208539us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:24.313527 (+ 18740us) log.cc:844] Serialized 52 byte log entry
0808 01:52:24.313686 (+   159us) raft_consensus.cc:1645] Marking committed up to 7
0808 01:52:24.313723 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:24.313794 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:24.969501 (+655707us) raft_consensus.cc:1685] finished
0808 01:52:24.969526 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:24.969680 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":137127,"prepare.queue_time_us":213616,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"raft.queue_time_us":141306,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":10034,"thread_start_us":227690,"threads_started":2,"wal-append.queue_time_us":78374}
I20260808 01:52:25.005273  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:25.000892  9524 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:25.016360  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 193215us
I20260808 01:52:25.012694  9373 catalog_manager.cc:5697] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:52:25.024533  9373 catalog_manager.cc:5697] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 7 to 8, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 8 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:52:25.026341  9523 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:24.963161 16227 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:52:24.954059 16470 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:52:25.026490  9373 catalog_manager.cc:5697] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:52:24.974063 15856 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:25.029079  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:25.014034 16150 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:25.035756 16265 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:25.044318 16290 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:25.049779 16265 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:25.049870  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 84914.1us
W20260808 01:52:25.025480 16369 consensus_meta.cc:287] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.986s	user 0.015s	sys 0.000s
I20260808 01:52:25.055420  9475 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260808 01:52:25.054889  9373 catalog_manager.cc:5697] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 10 to 11, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 11 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:52:25.061190  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:25.070844  9634 raft_consensus.cc:2376] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18. Re-sending same reply.
I20260808 01:52:25.062268  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:25.072180  9638 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:25.034268 16316 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:24.941344 16437 consensus_queue.cc:260] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:25.073451 15636 raft_consensus.cc:3055] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:25.080065 15636 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:25.016461 16367 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:25.081339 15636 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:25.044582 15856 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 10 FOLLOWER]: Advancing to term 11
W20260808 01:52:25.066150  9556 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.760s	user 0.009s	sys 0.003s
I20260808 01:52:25.044787 16457 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:25.095672  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:25.078060  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:25.096581  9634 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:25.091137  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:25.093680  9556 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
W20260808 01:52:25.093397 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 132114us
I20260808 01:52:25.074945 16316 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:25.060194 16386 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:25.051461 16462 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:25.053740 16150 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:25.102020  9630 raft_consensus.cc:2393] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 17: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:25.056247 16249 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:25.048751 16477 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:25.103808 15636 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:25.097463  9373 catalog_manager.cc:5697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 7 to 11, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 11 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:52:25.217751  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:25.108068  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:25.227258  9634 raft_consensus.cc:2376] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16. Re-sending same reply.
I20260808 01:52:25.108922  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:25.226433  9472 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:52:25.140223  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:25.071256 16470 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:52:25.141919  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 66365us
W20260808 01:52:25.144205 16469 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.207s	user 0.001s	sys 0.000s
I20260808 01:52:25.145000 15899 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:25.146296 16397 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:25.141166 16367 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:25.147418 16480 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260808 01:52:25.158474  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:25.165423 16017 consensus_meta.cc:287] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.751s	user 0.007s	sys 0.004s
W20260808 01:52:25.168428  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 142242us
I20260808 01:52:25.106014 16316 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:25.140452 16150 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:25.275508 16462 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:25.172037 16265 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:25.172195 16338 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.000s	user 0.012s	sys 0.000s
I20260808 01:52:25.279140  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:25.279395 16338 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:25.104729 16457 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:25.184922 16444 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:25.186771 16356 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.699s	user 0.005s	sys 0.007s
I20260808 01:52:25.188975 16437 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:25.168298  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:25.210915 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:25.222725  9636 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:25.218438  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:25.222580  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 109702us
I20260808 01:52:25.229674  9629 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:25.305297  9637 raft_consensus.cc:2410] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260808 01:52:25.207371  9560 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:25.230914  9638 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
W20260808 01:52:25.154547 16371 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.660s	user 0.000s	sys 0.009s
W20260808 01:52:25.227308 10332 leader_election.cc:336] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.009s (SENT)
W20260808 01:52:25.307294 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 113009us
W20260808 01:52:25.307943 10332 leader_election.cc:336] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Timed out: RequestConsensusVote RPC to 127.7.165.194:36095 timed out after 2.009s (SENT)
I20260808 01:52:25.308318 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:25.234760  9373 catalog_manager.cc:5697] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:52:25.309125  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:25.247148 16477 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:25.243621  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:25.190608 16415 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.680s	user 0.009s	sys 0.000s
I20260808 01:52:25.252137  9556 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:25.255250  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:25.255102  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:25.257378  9555 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: []
W20260808 01:52:25.232646 16485 log.cc:927] Time spent T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.058s	user 0.001s	sys 0.006s
I20260808 01:52:25.102824 16443 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:25.232386 16466 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.307s	user 0.003s	sys 0.003s
W20260808 01:52:25.232208 16472 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.126s	user 0.000s	sys 0.004s
I20260808 01:52:25.260844  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:25.347426 16194 consensus_meta.cc:287] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.528s	user 0.013s	sys 0.000s
W20260808 01:52:25.347942 16483 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260808 01:52:25.348433  9635 raft_consensus.cc:2410] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 8.
I20260808 01:52:25.349434  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2643) took 1033 ms. Trace:
I20260808 01:52:25.349907  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2638) took 1129 ms. Trace:
I20260808 01:52:25.349838  9560 rpcz_store.cc:276] 0808 01:52:24.316363 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:24.398334 (+ 81971us) service_pool.cc:224] Handling call
0808 01:52:25.349220 (+950886us) spinlock_profiling.cc:243] Waited 586 ms on lock 0x7b5400382f54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:25.349363 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1346750336}
I20260808 01:52:25.350425  9552 rpcz_store.cc:276] 0808 01:52:24.220593 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:24.308641 (+ 88048us) service_pool.cc:224] Handling call
0808 01:52:24.309304 (+   663us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:24.343716 (+ 34412us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:24.343786 (+    70us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:24.530624 (+186838us) log.cc:844] Serialized 26 byte log entry
0808 01:52:24.815737 (+285113us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:24.815777 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:24.815852 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:25.349682 (+533830us) raft_consensus.cc:1685] finished
0808 01:52:25.349706 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:25.349844 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":245550,"threads_started":1,"wal-append.queue_time_us":477841}
I20260808 01:52:25.352087 16223 raft_consensus.cc:3055] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:25.262874  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:25.353822  9559 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 22: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:25.267484  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:25.353976  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:25.268347  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:25.354700  9474 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:25.354867  9552 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260808 01:52:25.272431  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:25.356057  9473 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:25.273164 16017 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:25.293026  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 209379us
I20260808 01:52:25.300539  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:25.363919  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:25.364681  9634 raft_consensus.cc:2376] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19. Re-sending same reply.
I20260808 01:52:25.365655  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:25.366616  9634 raft_consensus.cc:2410] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260808 01:52:25.296321  9632 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:25.302395 16290 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:25.316220  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 148052us
W20260808 01:52:25.319808  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:25.323081  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:25.323160 16337 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:25.324059 16367 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:25.323995  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:25.325459 16443 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:25.326002 16457 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:25.331578 16478 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.217s	user 0.001s	sys 0.000s
I20260808 01:52:25.332094  9373 catalog_manager.cc:5697] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 1 to 12. New cstate: current_term: 12 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:52:25.333340  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:25.416752  9637 raft_consensus.cc:2376] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14. Re-sending same reply.
W20260808 01:52:25.336536  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:25.417666  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:25.337833 16477 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:25.339334 15899 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:25.340348  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:25.316943 16444 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:25.322227  9523 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:25.359001  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:25.427614  9635 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:25.313763  9480 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 21: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:25.367726  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:25.368999  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:25.370914  9557 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:25.314566 16346 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:25.377410 16017 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:25.380096 16479 log.cc:927] Time spent T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260808 01:52:25.380895 16279 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.826s	user 0.012s	sys 0.000s
I20260808 01:52:25.385035 16194 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:25.385424  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:25.387064  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:25.389424 15856 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:25.392410 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:25.398605 16290 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:25.399658  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:25.374320  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:25.403795 10367 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 13 index: 12. (term mismatch)
I20260808 01:52:25.457757 10367 pending_rounds.cc:85] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting all ops after (but not including) 11
I20260808 01:52:25.400642  9629 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:25.404393  9550 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:25.411055  9474 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:52:25.361217 16223 raft_consensus.cc:740] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:25.435523  9636 raft_consensus.cc:2376] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14. Re-sending same reply.
I20260808 01:52:25.436765  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:25.437914  9444 leader_election.cc:336] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Timed out: RequestConsensusVote RPC to 127.7.165.194:36095 timed out after 1.867s (SENT)
I20260808 01:52:25.459928  9637 raft_consensus.cc:2376] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17. Re-sending same reply.
I20260808 01:52:25.459998  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:25.438230  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:25.440207  9634 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:25.410084 16367 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:25.415947 16202 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.011s
I20260808 01:52:25.445029  9373 catalog_manager.cc:5697] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 9 to 10, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 10 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:52:25.427008 16386 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:25.449545  9472 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:52:25.464725  9632 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 17. Current term is 18. Ops: [15.11-17.12]
W20260808 01:52:25.439680 16242 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.798s	user 0.011s	sys 0.003s
I20260808 01:52:25.410073 16415 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.035s
W20260808 01:52:25.461529  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 87302.2us
I20260808 01:52:25.468585  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:25.469195  9558 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:25.440426 16369 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:25.467130  9635 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
W20260808 01:52:25.476328  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 154256us
I20260808 01:52:25.478102  9629 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:25.452543 16279 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:25.466305  9631 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:25.463639 10367 pending_rounds.cc:107] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting uncommitted NO_OP operation due to leader change: 12.12
I20260808 01:52:25.488130 10367 raft_consensus.cc:2889] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:25.467870  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:25.472173  9524 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:25.466360 16367 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:25.491204  9557 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
W20260808 01:52:25.509986  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 71848.6us
W20260808 01:52:25.505239  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:25.507359  9476 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:25.464916 16400 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:52:25.472795 16464 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:25.523176  9601 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:25.503494 16194 raft_consensus.cc:3055] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:25.529565  9599 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:25.513451 16242 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:25.540696  9558 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:25.543246  9599 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:25.449600 16470 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:25.522794 16029 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260808 01:52:25.559592 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:25.473199 16316 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:25.527797 16400 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:25.548197  9373 catalog_manager.cc:5697] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 9 to 10, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 10 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:52:25.566510  9523 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:25.567494  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 129598us
I20260808 01:52:25.529223 16194 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:25.542196 16242 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:25.569370  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:25.509878 16386 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:25.555361 16053 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:25.558776 16462 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election won for term 17
W20260808 01:52:25.577325  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 54245.5us
I20260808 01:52:25.583818  9373 catalog_manager.cc:5697] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 10 to 11, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 11 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:52:25.585927  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3098) took 1053 ms. Trace:
W20260808 01:52:25.587613 16504 log.cc:927] Time spent T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
I20260808 01:52:25.555676 16202 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260808 01:52:25.541968 16371 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.029s
I20260808 01:52:25.607520  9477 rpcz_store.cc:276] 0808 01:52:24.494482 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:24.494738 (+   256us) service_pool.cc:224] Handling call
0808 01:52:24.524930 (+ 30192us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:24.553083 (+ 28153us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:24.553120 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:24.637559 (+ 84439us) log.cc:844] Serialized 52 byte log entry
0808 01:52:24.955954 (+318395us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:52:24.956008 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:24.956089 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:25.547297 (+591208us) raft_consensus.cc:1685] finished
0808 01:52:25.547320 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:25.547497 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":29217,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":148,"thread_start_us":190773,"threads_started":1,"wal-append.queue_time_us":539177}
I20260808 01:52:25.576747 16242 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:25.607177 16053 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:25.630947  9601 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:25.641834  9480 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 1101) took 1543 ms (client timeout 1561 ms). Trace:
W20260808 01:52:25.642467  9480 rpcz_store.cc:269] 0808 01:52:24.091863 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:24.093328 (+  1465us) service_pool.cc:224] Handling call
0808 01:52:25.634399 (+1541071us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540038d1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:25.634622 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2720020736}
I20260808 01:52:25.613926 16406 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260808 01:52:25.636839  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 90392.3us
I20260808 01:52:25.621038 16462 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:25.638232 16053 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:25.577911 16223 consensus_queue.cc:260] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:25.654582 15933 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:25.671136 16316 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:25.694164  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:25.637502 16444 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:25.697340  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66521.5us
W20260808 01:52:25.705384 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:25.710597  9601 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:25.712352  9523 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:25.663061 15856 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:25.725582 16367 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:25.736022 16316 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:52:25.672827 16194 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:25.747617 16029 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260808 01:52:25.763262  9551 raft_consensus.cc:2368] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21.
I20260808 01:52:25.656037 16400 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:25.774775  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:25.776225  9474 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:25.779323  9558 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:25.779793 16386 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:25.664513 16462 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:25.782465  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:25.788968  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:25.789157  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:25.799850  9556 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
W20260808 01:52:25.816771 16150 consensus_meta.cc:287] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.541s	user 0.013s	sys 0.000s
I20260808 01:52:25.817451 16150 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:25.833041  9637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "244e080726394210b5467bf688813cf0"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:25.834173  9637 raft_consensus.cc:606] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:25.836338  9637 raft_consensus.cc:614] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:25.829965  9474 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:25.839636  9475 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:25.843626  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:25.827150 16316 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:25.851735 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:25.855370 15856 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:25.871109 15856 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:25.878614 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:25.886328 15856 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:25.887140  9556 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260808 01:52:25.887252 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:25.888178 10363 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:25.889878 16279 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:25.890913 16383 consensus_queue.cc:1059] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20260808 01:52:25.893491  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:25.900450  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:25.857765 16338 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:25.938896 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:25.961702 10332 leader_election.cc:336] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.568s (SENT)
W20260808 01:52:25.966809  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 76050.2us
W20260808 01:52:25.973055  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 244e080726394210b5467bf688813cf0 (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:25.973702  9556 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:25.975212  9557 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260808 01:52:25.976891 16367 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:25.988306  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:25.989451  9558 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
W20260808 01:52:25.993415  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 201400us
I20260808 01:52:25.972270  9524 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:26.004940  9634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3354328ed023401e83a2361925b02e81"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:26.005715  9634 raft_consensus.cc:606] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:26.012264  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.013245  9558 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
W20260808 01:52:25.996918 16249 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.519s	user 0.003s	sys 0.006s
W20260808 01:52:26.043035  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 72757.7us
W20260808 01:52:26.075181  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 58351.2us
I20260808 01:52:26.075748  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.080159  9558 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:52:26.077437 16202 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.005s
W20260808 01:52:26.082700 16519 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.120s	user 0.000s	sys 0.001s
I20260808 01:52:26.083685  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:26.084462 16512 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.203s	user 0.001s	sys 0.000s
I20260808 01:52:26.086174  9554 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 11. (term mismatch)
I20260808 01:52:26.086660  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.087659  9555 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:52:26.086615  9554 pending_rounds.cc:85] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 10
I20260808 01:52:26.089191  9554 pending_rounds.cc:107] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 15.11
I20260808 01:52:26.088932  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.089536  9554 raft_consensus.cc:2889] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:26.011488 16223 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:26.095942  9556 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 9. Current term is 10. Ops: []
I20260808 01:52:26.097329  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.098269  9556 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:52:26.100818  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.101887  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.104485  9555 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 10.
W20260808 01:52:26.140606  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 145945us
I20260808 01:52:26.107743  9556 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 10. Current term is 11. Ops: []
I20260808 01:52:26.077813 16415 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election won for term 17
W20260808 01:52:26.162312 16526 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
W20260808 01:52:26.165318  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:26.075150 16447 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.079s
I20260808 01:52:26.173007  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:26.166659  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:26.165197  9521 leader_election.cc:336] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.680s (SENT)
I20260808 01:52:26.166225  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.174489  9521 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:26.175441  9556 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:52:26.174851  9554 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:52:26.175789  9552 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:52:26.184248  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.017098 16316 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:26.196988 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:26.182129  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:26.185588  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 415912us
I20260808 01:52:26.188740 16337 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:26.178982 16150 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:26.147156 16053 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:26.176546 16202 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.019s
I20260808 01:52:26.195881 16447 raft_consensus.cc:3055] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:26.198989  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:26.200927  9552 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:52:26.207540  9634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "439040d168824496acc076b1c280bde3"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:26.227788  9634 raft_consensus.cc:606] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:26.228281  9634 raft_consensus.cc:614] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:26.158056 16242 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:26.209563  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:26.232507  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:26.240275 16053 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:26.220425 16447 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:26.243752 16447 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:26.245056 16437 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260808 01:52:26.246500 16371 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:26.247138 16371 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:26.249042  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 439040d168824496acc076b1c280bde3 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:26.253954  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:26.257388  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 688063us
I20260808 01:52:26.257539 16356 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260808 01:52:26.261013  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 95850.3us
W20260808 01:52:26.264744  9521 leader_election.cc:336] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.702s (SENT)
W20260808 01:52:26.266073  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 309679us
I20260808 01:52:26.266556  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:26.234010 16249 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:26.270346  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:26.211418 16468 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.053s
I20260808 01:52:26.267820  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:26.263597 16356 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:26.263377 16415 raft_consensus.cc:697] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:26.274698 16356 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
W20260808 01:52:26.282023  9521 leader_election.cc:336] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.738s (SENT)
W20260808 01:52:26.283278  9521 leader_election.cc:336] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.773s (SENT)
I20260808 01:52:26.283594  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:26.283703  9521 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:26.280669  9554 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:26.287030 10367 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:57156
I20260808 01:52:26.287770 10367 raft_consensus.cc:606] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:26.288210 10367 raft_consensus.cc:614] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:26.284783 16386 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:26.279409 16371 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:26.289558 16386 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
W20260808 01:52:26.290393 16520 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.157s	user 0.000s	sys 0.002s
I20260808 01:52:26.282701 16406 raft_consensus.cc:3055] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:26.291847 16406 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:26.278725 16415 consensus_queue.cc:237] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:26.293001  9521 leader_election.cc:336] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.776s (SENT)
I20260808 01:52:26.279421 16338 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:26.293506  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:26.293901 16338 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:26.285545  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:26.295501 16514 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:26.288702 16337 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:26.296941 16337 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:26.297222 16338 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:26.297423 16337 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:26.297889 16338 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:26.298341 16338 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:26.298749 16338 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:26.288141 16383 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260808 01:52:26.296491 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:26.304316 10362 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: []
I20260808 01:52:26.304836 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:26.305279 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:26.292887 16406 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:26.304225 10365 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEARNER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260808 01:52:26.304073  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.306365 16406 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:26.305963 10364 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:52:26.306897  9554 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:26.307396 16242 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:26.304901 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:26.308216 16249 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:26.309397  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:26.305589 10368 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:26.288388  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3468) took 1034 ms. Trace:
I20260808 01:52:26.314045  9551 rpcz_store.cc:276] 0808 01:52:25.253557 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:25.253698 (+   141us) service_pool.cc:224] Handling call
0808 01:52:26.288141 (+1034443us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b540016a8d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:26.288299 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":938315008}
I20260808 01:52:26.315364  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:26.316185  9475 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:26.316561  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.315474  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.319376  9557 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:26.320082  9477 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260808 01:52:26.320860  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 1116) took 1092 ms. Trace:
W20260808 01:52:26.320916  9550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2651) took 2002 ms (client timeout 1954 ms). Trace:
W20260808 01:52:26.321535  9550 rpcz_store.cc:269] 0808 01:52:24.318832 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:24.463731 (+144899us) service_pool.cc:224] Handling call
0808 01:52:26.320738 (+1857007us) spinlock_profiling.cc:243] Waited 857 ms on lock 0x7b540029f4d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:26.320842 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1970914176}
W20260808 01:52:26.323105  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 8cbead77343f44dd82199bebcfb2def7 (from 7201605bb2f54c949a4ebe6e9b3b4b2a to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:26.324760 10334 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:26.326088 16397 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:26.326714 16397 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:52:26.327658 16499 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.975s	user 0.007s	sys 0.005s
I20260808 01:52:26.327991 16397 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:26.321476  9472 rpcz_store.cc:276] 0808 01:52:25.228006 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:25.248815 (+ 20809us) service_pool.cc:224] Handling call
0808 01:52:26.320705 (+1071890us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b540021e154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:26.320806 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":350020864}
I20260808 01:52:26.331212 10367 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:26.334189 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:26.273878  9552 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:26.308255  9554 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
W20260808 01:52:26.339056  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:26.347200 10332 leader_election.cc:336] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.991s (SENT)
I20260808 01:52:26.347905 10332 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:26.347098  9521 leader_election.cc:336] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.819s (SENT)
I20260808 01:52:26.351737  9556 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:52:26.361928  9475 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:26.351361 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:26.370517  9636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:26.371553  9636 raft_consensus.cc:606] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:26.391333  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 82025.1us
W20260808 01:52:26.391554 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:26.393002  9599 leader_election.cc:400] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21.
I20260808 01:52:26.393419  9599 leader_election.cc:403] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
W20260808 01:52:26.395201  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:26.361783 16223 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:26.396212 16223 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:26.398954 16482 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
W20260808 01:52:26.400801  9633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2914) took 1712 ms (client timeout 2208 ms). Trace:
W20260808 01:52:26.401309  9633 rpcz_store.cc:269] 0808 01:52:24.689292 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:24.716954 (+ 27662us) service_pool.cc:224] Handling call
0808 01:52:26.400547 (+1683593us) spinlock_profiling.cc:243] Waited 693 ms on lock 0x7b540025b7d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:26.400703 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1594525568}
I20260808 01:52:26.401696  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:26.402391  9556 raft_consensus.cc:2376] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17. Re-sending same reply.
I20260808 01:52:26.399192  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.387351  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:26.403910  9480 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 10 FOLLOWER]: Advancing to term 11
I20260808 01:52:26.398313  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:26.333588 16249 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:26.347922 16386 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:26.402990  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:26.417768  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:26.406232 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59092.2us
I20260808 01:52:26.404987 10334 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:26.407297  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.419423  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:26.408726  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:26.419873  9477 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 21: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:26.420571  9554 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:26.398494 16223 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:26.421921  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.422520  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:26.435776  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 88728.6us
I20260808 01:52:26.436050  9550 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:26.436461  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.437137  9554 raft_consensus.cc:2376] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19. Re-sending same reply.
I20260808 01:52:26.437814  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.439138  9557 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 21: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:26.389475 15856 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:26.443102  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:26.443224  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:26.444682 10332 leader_election.cc:336] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Timed out: RequestConsensusVote RPC to 127.7.165.194:36095 timed out after 2.009s (SENT)
W20260808 01:52:26.445744 10332 leader_election.cc:336] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.009s (SENT)
I20260808 01:52:26.446535 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:26.447443  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 49223us
I20260808 01:52:26.447775  9554 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
W20260808 01:52:26.448298  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:26.448402  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:52:26.443974  9521 leader_election.cc:336] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.826s (SENT)
I20260808 01:52:26.452337  9552 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:26.455988  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.418824 16530 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:26.398506 16477 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.044s
I20260808 01:52:26.421090 16444 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:26.456188  9599 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:26.417753 16053 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:26.477283  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:26.478009  9552 raft_consensus.cc:2376] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17. Re-sending same reply.
I20260808 01:52:26.454340 16476 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:52:26.458142 16493 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:26.441625 16532 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.035s
I20260808 01:52:26.419421 16371 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.048s
I20260808 01:52:26.422866 16386 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:26.507048  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 58746us
I20260808 01:52:26.510047  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:26.511953  9552 raft_consensus.cc:2376] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17. Re-sending same reply.
I20260808 01:52:26.512082  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:26.461247 16530 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:26.514142  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 92782.8us
I20260808 01:52:26.485103 15856 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:52:26.523780  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:26.524034 15856 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:26.524503 15856 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:26.524860  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.525506  9560 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:26.525662  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.526567  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.527220  9560 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
W20260808 01:52:26.520665  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 76728.5us
I20260808 01:52:26.527828  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:26.520816 16524 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.261s	user 0.000s	sys 0.001s
W20260808 01:52:26.528275  9521 leader_election.cc:336] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.878s (SENT)
I20260808 01:52:26.524503  9559 raft_consensus.cc:2376] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20. Re-sending same reply.
I20260808 01:52:26.513722 16493 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:26.530117  9521 leader_election.cc:336] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.015s (SENT)
I20260808 01:52:26.531648 16493 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:26.518596 16530 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:26.521492  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:26.522290  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:26.533943  9632 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 18: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:26.523046  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.535398  9557 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:26.527104  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.536586  9554 raft_consensus.cc:2393] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:26.536388 16338 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:26.538533  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:26.539122 16290 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:26.539613 16290 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:26.540073 16290 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:26.540817  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:26.541553  9632 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:26.519254 16482 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:26.530622  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:26.543254 16482 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:26.543792 16482 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:26.486428 16470 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:26.527623 16476 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:26.527030  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:26.550833  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.553221  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:26.586956  9634 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:26.588554 10368 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:52:26.592473  9632 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:26.600032  9523 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:26.601085 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:26.602312  9523 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:26.602415  9601 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:26.614286 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 209374us
W20260808 01:52:26.654512  9521 leader_election.cc:336] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.168s (SENT)
I20260808 01:52:26.659709 10369 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260808 01:52:26.663887  9633 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
W20260808 01:52:26.667443  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 63369us
I20260808 01:52:26.668448  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:26.669229  9638 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
W20260808 01:52:26.589051 16479 log.cc:927] Time spent T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
W20260808 01:52:26.678869  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:26.569701 16451 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:26.590430 16397 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:26.596722 16017 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Stepping down as leader of term 16
W20260808 01:52:26.700789  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 98454.5us
W20260808 01:52:26.740389 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 224343us
I20260808 01:52:26.740795  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:26.589378 16338 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:26.586267 16476 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:26.602945 16316 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:26.744210  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.661885 16470 raft_consensus.cc:697] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:26.667207 16249 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:26.702179 16447 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:26.602104 16415 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.148s
W20260808 01:52:26.774739  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 120307us
I20260808 01:52:26.748966 16456 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260808 01:52:26.778141  9521 leader_election.cc:336] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.200s (SENT)
I20260808 01:52:26.780676  9521 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:26.749868 16017 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:26.681715 16242 consensus_queue.cc:1059] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.089s
I20260808 01:52:26.789286  9634 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:26.782059 16150 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:26.774363 16249 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:26.793303  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 281289us
I20260808 01:52:26.800078  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:26.796221  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:26.794924  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 54263.9us
I20260808 01:52:26.782127 16456 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:26.802788  9601 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:26.784122 16371 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:26.819600  9476 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
W20260808 01:52:26.821467 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 205539us
I20260808 01:52:26.822970  9477 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:26.778136 16397 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.026s
I20260808 01:52:26.810309  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:52:26.814285  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:26.827929  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:26.828325  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:26.829155  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:26.824654 16532 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:26.830415 16532 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:26.837025  9476 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:26.826332  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:26.838374  9477 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:26.777514 16542 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 15.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.040s
I20260808 01:52:26.801288 16249 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:26.807902 16456 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:26.852851  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:26.812471 16477 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
W20260808 01:52:26.861819  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:26.849615  9480 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 11.
I20260808 01:52:26.853711  9475 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:26.794297 16386 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:26.800745 16150 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:26.855619 16541 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260808 01:52:26.802386 16476 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:26.856273 16356 consensus_meta.cc:287] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.575s	user 0.011s	sys 0.000s
I20260808 01:52:26.793406 16337 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:26.856803  9638 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:26.899152  9638 raft_consensus.cc:606] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:26.899820  9638 raft_consensus.cc:614] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:26.901072 16150 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:26.857026 16462 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260808 01:52:26.858479 16532 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:26.863533  9555 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260808 01:52:26.903400  9475 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:26.864355  9552 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []
W20260808 01:52:26.856573  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 53923.6us
I20260808 01:52:26.905777  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:26.907433  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet d0db7b84c3b040c788e0a3f66657d912 (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:26.907586  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:26.908231  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 01:52:26.910542  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:52:26.824868 16316 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260808 01:52:26.884339  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 56124.5us
I20260808 01:52:26.841114 16464 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:26.832789 16470 consensus_queue.cc:237] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:26.889060  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:26.879705 10334 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:26.923841 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:26.894102  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 115998us
W20260808 01:52:26.925833  9521 leader_election.cc:336] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.201s (SENT)
I20260808 01:52:26.896976 16337 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:26.897768  9473 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:26.892549 16356 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:26.937618 16371 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:26.938174  9478 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:26.939656 16356 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:26.915525  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:26.940428 16367 raft_consensus.cc:3055] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Stepping down as leader of term 21
W20260808 01:52:26.926381 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:26.922538  9558 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 13.
I20260808 01:52:26.941280 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:26.943586 16477 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:26.940887 16367 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:26.948477 16542 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:26.949263 16542 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:26.919500  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:26.935705 16406 consensus_meta.cc:287] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.629s	user 0.012s	sys 0.001s
I20260808 01:52:26.936259 16223 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader election won for term 11
I20260808 01:52:26.926407  9521 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:26.884409 16548 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260808 01:52:26.949605 16536 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.475s	user 0.006s	sys 0.003s
I20260808 01:52:26.833086 16017 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:26.920447 16532 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:26.952071 16476 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:26.924937 16464 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:26.970631  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:26.977412  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 58043.2us
I20260808 01:52:26.961658 16444 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:26.995779  9601 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:26.984992  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:26.971215 16223 raft_consensus.cc:697] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:27.008342  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:26.968591 16316 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:52:27.014868 16223 consensus_queue.cc:237] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:27.018249 16451 raft_consensus.cc:3055] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:27.018849 16451 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:27.019811 16451 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:27.021185 16451 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:27.022974 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:27.023692 10365 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 12.
I20260808 01:52:26.976877 16053 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21.
I20260808 01:52:26.971521 16477 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:26.983139 16464 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:26.925419 16544 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.068s
I20260808 01:52:26.972954 16356 raft_consensus.cc:2764] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader election decision vote started in defunct term 10: won
I20260808 01:52:26.977280 16552 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:27.030067  9600 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:27.065099  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:27.031813  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 106047us
W20260808 01:52:27.067955  9521 leader_election.cc:336] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.206s (SENT)
I20260808 01:52:27.068589  9521 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:26.977351 16371 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:27.071995 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:27.002208  9556 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:27.034385  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:26.991451 16017 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:27.010478 15933 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:27.075941 16468 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:52:27.079720 16444 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:27.080206 16544 consensus_queue.cc:1243] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 15.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260808 01:52:27.025067 16548 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:27.032428 16532 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:27.066386 16462 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:27.003221 15856 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:27.106902 16406 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:26.960081 16542 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 15.15, Last appended by leader: 15, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:27.115221 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 373804us
W20260808 01:52:27.078090  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 43840.4us
W20260808 01:52:27.116068  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 107828us
I20260808 01:52:27.116530  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:27.117349  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:27.096335 16053 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:27.093750 16468 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:27.062184 16530 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:27.064585 16477 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:27.112267 16532 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:27.164284  9601 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:27.166740  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:27.126332 10364 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:52:27.100682 16371 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:27.131110 16468 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:27.175809  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:27.176307  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:27.173574  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 329296us
I20260808 01:52:27.218896 16406 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 01:52:27.219673  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:27.163041 16356 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:27.221257 16444 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:27.221815 16444 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:27.222257 16444 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:27.221920 16356 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:27.222816 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 49246.6us
I20260808 01:52:27.225025 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:27.225343  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:27.225508 16356 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:27.227185  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:27.227306 10332 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:27.203601 16290 consensus_meta.cc:287] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.663s	user 0.017s	sys 0.000s
I20260808 01:52:27.167500 16338 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:27.170282 16548 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:27.194388 16150 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:27.205523 16194 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
W20260808 01:52:27.280824  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 53696.8us
W20260808 01:52:27.259980  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 83742.8us
I20260808 01:52:27.283157  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:27.207538 16542 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:27.208410 16456 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:27.209208 16482 consensus_meta.cc:287] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.665s	user 0.018s	sys 0.000s
I20260808 01:52:27.162101 16535 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:27.076710 16552 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:27.216957  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:27.229414  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 161498us
I20260808 01:52:27.256242 16530 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:27.257510 16554 log.cc:927] Time spent T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Append to log took a long time: real 0.080s	user 0.001s	sys 0.001s
I20260808 01:52:27.214665 16053 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:27.258633 15856 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:27.291591 16338 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:27.226459 16493 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:27.250368 16356 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:27.305518  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:27.280339 16029 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:27.226056 16544 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.023s
I20260808 01:52:27.323411  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:27.329332 16279 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:27.247992 16415 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
I20260808 01:52:27.290758 16476 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:27.367408 16279 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:27.336824 16552 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:27.338647 16456 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:27.345254 16558 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:52:27.251097 16290 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:27.301191 16482 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:27.310763 16386 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:27.374500  9477 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260808 01:52:27.258880 16548 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:27.369967  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 64723.4us
W20260808 01:52:27.380461  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 160930us
I20260808 01:52:27.386827  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:27.387683  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:27.387949  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:27.388495  9480 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:52:27.296897 16550 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.090s
I20260808 01:52:27.372097 16547 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:27.372269 16532 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:27.394382  9476 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:27.391999  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:27.379927 16279 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:27.397794  9473 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:27.409085  9524 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:27.382114 16558 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:27.414017 16367 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:27.415640 16367 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:27.417178 16415 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:27.417999 16415 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:27.419415  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2662) took 1011 ms. Trace:
I20260808 01:52:27.419929  9556 rpcz_store.cc:276] 0808 01:52:26.408242 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:26.408478 (+   236us) service_pool.cc:224] Handling call
0808 01:52:27.419200 (+1010722us) spinlock_profiling.cc:243] Waited 550 ms on lock 0x7b54003147d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:27.419343 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1265852288}
I20260808 01:52:27.420820 16415 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:27.376876 16476 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:27.422658  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3122) took 1142 ms. Trace:
I20260808 01:52:27.423259  9478 rpcz_store.cc:276] 0808 01:52:26.280461 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:26.312606 (+ 32145us) service_pool.cc:224] Handling call
0808 01:52:26.312945 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:26.938703 (+625758us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:27.422376 (+483673us) spinlock_profiling.cc:243] Waited 625 ms on lock 0x7b54002bbbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:27.422589 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1437412608}
I20260808 01:52:27.424978 16476 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:27.428733 16444 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:27.430653  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:27.431548  9480 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:27.432346 16560 raft_consensus.cc:3055] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:27.432775 16560 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:27.433637 16560 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:27.392863 16356 raft_consensus.cc:2764] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260808 01:52:27.435456  9472 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:27.436794  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:27.437448  9472 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:27.437780 16570 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:27.439103  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:27.439492 16570 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:27.439949 16570 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:27.440294 16570 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:27.440828  9474 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260808 01:52:27.442397  9481 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21. Ops: [20.16-20.16]
I20260808 01:52:27.393774 16386 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:27.443818  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:27.443975 16386 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:27.444639  9479 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
W20260808 01:52:27.456131  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:27.467059  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:27.468086  9552 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:27.477058  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:27.498340  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.499667  9634 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:52:27.501398  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.502455  9636 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:52:27.404644 16493 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:27.507815  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:27.509377  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:27.509994 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:27.511477  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.515426  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.387694 16242 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:27.374579 16338 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:27.517437  9635 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:27.517813  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:27.553437  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.518191  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:27.554514  9636 raft_consensus.cc:2376] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16. Re-sending same reply.
W20260808 01:52:27.554901  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:27.555572  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:27.518883  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 60508.2us
I20260808 01:52:27.556835  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.523463  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.557551  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.518966  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:27.508124  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:27.559058  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:27.560011  9630 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
W20260808 01:52:27.560310  9521 leader_election.cc:336] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.502s (SENT)
W20260808 01:52:27.561728  9521 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.568s (SENT) [suppressed 74 similar messages]
I20260808 01:52:27.569024  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.580940  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:27.581651  9630 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:27.479244 16560 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:27.457209 16558 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:27.522927 16476 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:27.556759 16447 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:27.494428 16356 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:27.572021 16482 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:27.572902 16532 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:27.576205 16548 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:27.579223  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.609601  9632 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:52:27.609707  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:27.580264  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.610467  9633 raft_consensus.cc:2368] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19.
I20260808 01:52:27.569868  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:27.612093  9559 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:27.612229  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.613358  9629 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:27.551586  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:27.587555 16477 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:27.478399 16568 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:27.594561 16371 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:27.479082 16550 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:27.611454  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:27.638652  9636 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:27.623641  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.644622  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:52:27.649099  9472 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:27.646049  9599 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:27.652952  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:27.656128  9559 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:27.595969 16457 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.039s
I20260808 01:52:27.597956 16356 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:27.654229 16464 consensus_meta.cc:287] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.552s	user 0.010s	sys 0.002s
I20260808 01:52:27.622936 16415 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:27.640357  9633 raft_consensus.cc:2376] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22. Re-sending same reply.
I20260808 01:52:27.599771  9552 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:27.614351 16316 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:27.617265 16569 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:27.663045  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.631695  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:27.753422  9634 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:27.670419  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:27.597550 16476 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:27.597745 16447 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:27.614053  9479 raft_consensus.cc:2393] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:27.633702 16279 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:27.744926  9480 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:27.706127  9635 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260808 01:52:27.792804  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:27.793557  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:27.794441  9634 raft_consensus.cc:2376] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16. Re-sending same reply.
I20260808 01:52:27.754885  9472 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:27.757791  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.758857  9636 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:52:27.678854 16457 consensus_queue.cc:1243] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 15.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260808 01:52:27.761314  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.612210 16532 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:27.748632  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 339651us
I20260808 01:52:27.661041 16444 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:27.759392  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 73192.3us
I20260808 01:52:27.683223 16356 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:27.815729  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.816533  9633 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:27.663405 16029 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260808 01:52:27.801136  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50222.7us
I20260808 01:52:27.826130  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:27.805035  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.806180  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.811715  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.820837  9638 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:52:27.652168 16550 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:27.829252  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.834960  9633 raft_consensus.cc:2410] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260808 01:52:27.837266  9633 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260808 01:52:27.839274  9633 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:27.831529  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:27.824090  9524 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:27.841077  9636 raft_consensus.cc:2410] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
W20260808 01:52:27.801656 16150 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.511s	user 0.010s	sys 0.001s
W20260808 01:52:27.817477 16534 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260808 01:52:27.790583 16290 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:27.831003 16547 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:27.849992  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:27.850812  9636 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260808 01:52:27.846339 16017 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.006s
I20260808 01:52:27.847167  9635 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:27.850836 16547 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:27.855362 16567 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.195s	user 0.000s	sys 0.000s
I20260808 01:52:27.762708 16415 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:27.869115  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:27.869059 16406 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:27.870018 16406 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:27.871480 16415 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:27.872506 16415 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260808 01:52:27.871176 16406 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:27.873592 16279 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:27.905527  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:27.880398 16532 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:27.882318 16476 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:27.891441 16468 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.643s	user 0.011s	sys 0.000s
I20260808 01:52:27.893630 16550 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:27.909426  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 347708us
I20260808 01:52:27.650914 16568 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:27.921913  9521 leader_election.cc:336] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.775s (SENT)
I20260808 01:52:27.928452  9521 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:27.912941  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.014s	sys 0.015s
W20260808 01:52:27.931264  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.015s	sys 0.015s
W20260808 01:52:27.931795  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 539856us
I20260808 01:52:27.798380 16194 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.098s
I20260808 01:52:27.843015 16053 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:52:27.889956 16570 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:27.896596 16451 consensus_meta.cc:287] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.865s	user 0.009s	sys 0.004s
W20260808 01:52:27.918519 16249 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.047s	user 0.014s	sys 0.001s
W20260808 01:52:27.917243  9443 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.006s	sys 0.024s
W20260808 01:52:27.955439  9443 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.006s	sys 0.025s
I20260808 01:52:27.933034  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:27.895437 16150 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260808 01:52:27.944554 10332 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.716s	user 0.020s	sys 0.024s
I20260808 01:52:27.825026 16029 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:27.909090 16242 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:27.959811 10332 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.020s	sys 0.024s
I20260808 01:52:27.766819 16569 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:27.750201 16464 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:27.967213  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:27.988592  9472 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
W20260808 01:52:27.936645 16574 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.079s	user 0.001s	sys 0.001s
I20260808 01:52:27.993230 10364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 1062) took 1403 ms. Trace:
I20260808 01:52:27.975194  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:27.833153 16571 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.112s
I20260808 01:52:27.993361  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:27.997900 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:27.759124 16316 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:28.001511 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:28.015297  9476 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:27.997243 10364 rpcz_store.cc:276] 0808 01:52:26.589979 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:26.590160 (+   181us) service_pool.cc:224] Handling call
0808 01:52:26.667777 (+ 77617us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:27.175153 (+507376us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:27.992923 (+817770us) spinlock_profiling.cc:243] Waited 458 ms on lock 0x7b5400449b54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:27.993157 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1053825408}
W20260808 01:52:28.013826 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 786659us
I20260808 01:52:27.915264 16017 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.058s
I20260808 01:52:28.007505  9479 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:52:28.018167  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 169263us
I20260808 01:52:28.018909 10332 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:27.971416 16029 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:28.022745 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:28.026607  9600 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:28.028260  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:27.923075 16468 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:27.995034 16194 consensus_queue.cc:1243] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 17.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260808 01:52:28.027604  9477 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:28.029963 10332 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:28.024966 10369 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
W20260808 01:52:28.047816  9445 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.046s	sys 0.079s
I20260808 01:52:27.972934 16249 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:28.048352  9445 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.640s	user 0.046s	sys 0.079s
I20260808 01:52:28.017220 16577 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election won for term 20
W20260808 01:52:28.048348  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 73293.8us
I20260808 01:52:28.027263 16223 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260808 01:52:28.051597 10332 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:28.053304  9630 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:52:28.053390 10334 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:28.056560  9631 raft_consensus.cc:2368] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 13. Current term is 14.
I20260808 01:52:28.057633 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:28.033723 16457 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:28.055325 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:28.062239 10365 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 11.
I20260808 01:52:28.056550  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:28.050401 16367 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.634s	user 0.014s	sys 0.000s
I20260808 01:52:28.073908  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:28.080278 10332 leader_election.cc:336] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.212s (SENT)
W20260808 01:52:28.053901  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 128420us
I20260808 01:52:28.087167  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:52:28.087671 10332 connection.cc:256] RPC call timeout handler was delayed by -1.4377305319940206s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1625s for a response
W20260808 01:52:28.076237 16462 consensus_meta.cc:287] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.955s	user 0.013s	sys 0.004s
I20260808 01:52:28.089385  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:28.089622 16462 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:28.082728  9479 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:52:28.091813  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:28.091609  9472 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260808 01:52:28.054121 16550 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:28.094053 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:28.052354 16386 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:28.059333 16279 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:28.094089 10334 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:28.047991 16371 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260808 01:52:28.100025  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:28.071257 16544 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:28.050658 16194 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.045s
I20260808 01:52:28.071727 16577 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:28.079713 16552 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:52:28.062484 16542 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.044s
I20260808 01:52:28.103204 16386 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:28.142217  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 220357us
I20260808 01:52:28.097922 16444 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:28.096614  9445 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:28.131618  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:28.094448 16150 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:28.101899 16415 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:28.147864 16150 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:28.148885 16476 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:28.149739 16476 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:28.150450 16150 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:28.151526 16415 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.003s
I20260808 01:52:28.154870 16585 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:28.155375 16585 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:28.156417 16585 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:28.157883 16585 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
W20260808 01:52:28.143064  9521 leader_election.cc:336] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.868s (SENT)
I20260808 01:52:28.100520 16279 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:28.123509 16578 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:28.107681 16544 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:28.180711 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:28.105020 16447 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:28.123656 16575 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:28.188309 10369 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:28.196228  9554 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260808 01:52:28.197808  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:28.199414  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 180089us
I20260808 01:52:28.136572 16367 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:28.200392  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:28.126607 16338 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:28.206324  9556 raft_consensus.cc:1217] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Deduplicated request from leader. Original: 12.10->[13.11-16.12]   Dedup: 13.11->[16.12-16.12]
I20260808 01:52:28.208590  9556 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17. Ops: [13.11-16.12]
W20260808 01:52:28.212051  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 115523us
I20260808 01:52:28.208627  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:28.214365  9558 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:28.180660 16279 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:28.230329 16338 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:28.230420  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:28.231314  9554 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:28.231546 16338 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:28.232568  9601 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:28.097433 16242 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:28.235363  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:28.236124  9550 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260808 01:52:28.236155  9559 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:28.198999 16544 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:52:28.242511  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 207051us
I20260808 01:52:28.205504 16570 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:28.256541  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:28.241210  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 153043us
I20260808 01:52:28.210917 16552 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:52:28.207310 16367 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:28.265812  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:28.266831  9637 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:28.252882  9476 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:28.187875 16229 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 16.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.056s
I20260808 01:52:28.271175  9554 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260808 01:52:28.254106  9477 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260808 01:52:28.263111  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:28.272670  9630 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:28.229369 16530 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:28.240312 16447 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:28.280539  9631 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260808 01:52:28.282795 16552 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:28.282990  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:28.277345 16464 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:28.282999 16468 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:28.283198 16249 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:28.283979 16552 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:28.284547 16530 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:28.282868  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:28.288329  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:28.288579 16386 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:28.289887 16584 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
I20260808 01:52:28.290069 16530 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:28.259740 16568 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:28.291440  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:28.292404  9631 raft_consensus.cc:2393] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 17: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:28.211066 16575 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:28.295068 16549 raft_consensus.cc:3055] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:28.295626 16549 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:28.295974  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:28.296680  9630 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:28.297012 16575 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:28.297154 16582 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.195s
W20260808 01:52:28.310058  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:28.298065 16590 consensus_queue.cc:1059] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.119s
I20260808 01:52:28.298583 16367 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:28.299500 16447 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:28.294550  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:28.306088  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:28.259388 16592 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:28.307463  9475 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:28.308125  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:28.309160  9630 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:28.311146 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:28.314441  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:28.322819  9559 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:28.329933  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:28.345000  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:28.349996  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:28.364607  9637 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:52:28.364801  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:28.365756  9630 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:52:28.352077  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:28.352065 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:28.357674  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:28.358345  9474 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:28.242265 16476 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:28.310022 15933 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:52:28.368927  9559 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
W20260808 01:52:28.353049  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 96625us
W20260808 01:52:28.374406  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 169301us
I20260808 01:52:28.308444 16530 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:28.381889  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:28.387610  9554 raft_consensus.cc:2410] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260808 01:52:28.389154  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:28.389411  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:28.350507 16447 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:28.394804  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:28.367882 16464 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:52:28.368923 16415 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:28.362313 16569 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:28.344135 16575 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.053s
I20260808 01:52:28.402570  9600 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:28.403695 10332 connection.cc:256] RPC call timeout handler was delayed by -1.1977566059795208s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18630000000000002s for a response
I20260808 01:52:28.393770 16444 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260808 01:52:28.404559  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:28.399672 16595 raft_consensus.cc:3055] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:28.401300  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:28.403776 16573 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:52:28.419934  9559 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:52:28.438266  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:28.319111 16468 consensus_queue.cc:1059] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.079s
W20260808 01:52:28.423827 10332 connection.cc:256] RPC call timeout handler was delayed by -1.1394579649786465s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3143s for a response
I20260808 01:52:28.280838 16577 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:28.398516 16367 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:28.431171  9630 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:28.382967  9558 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:52:28.349754 16593 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.104s
I20260808 01:52:28.397728 16597 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.039s
I20260808 01:52:28.418570  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:28.296768 16549 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:28.444149 16590 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:28.444764 16582 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.016s
I20260808 01:52:28.403514 16530 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:28.433650 16595 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:28.447180  9556 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 16. Current term is 17. Ops: [14.12-16.13]
I20260808 01:52:28.453348 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:28.435956  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:28.367746 16493 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:28.482620 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:28.482959  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47123.1us
I20260808 01:52:28.410889 16415 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:28.372602 16592 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:28.480738 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:28.469295 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:28.468886 10369 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260808 01:52:28.488647  9555 raft_consensus.cc:1217] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Deduplicated request from leader. Original: 12.10->[13.11-17.13]   Dedup: 13.11->[16.12-17.13]
I20260808 01:52:28.494176 10367 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:28.432530 16464 raft_consensus.cc:3055] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:28.490664 10365 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:28.494084  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:28.495335  9555 pending_rounds.cc:85] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 11
W20260808 01:52:28.460963 16558 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.852s	user 0.010s	sys 0.004s
I20260808 01:52:28.499818  9555 pending_rounds.cc:107] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 15.12
I20260808 01:52:28.504546  9555 raft_consensus.cc:2889] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:28.500546 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:28.450909 16468 raft_consensus.cc:3055] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Stepping down as leader of term 17
W20260808 01:52:28.480515 16356 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.648s	user 0.011s	sys 0.002s
I20260808 01:52:28.457827 16290 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:28.442978 16575 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:52:28.460752 16590 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:28.529583  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 87023.5us
I20260808 01:52:28.526510  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:28.508491 10369 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
W20260808 01:52:28.458563 16542 consensus_queue.cc:1175] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "259bcb3f0d254e8b9effac3277e654e0" responder_term: 16 status { last_received { term: 16 index: 12 } last_committed_idx: 12 last_received_current_leader { term: 16 index: 12 } }
I20260808 01:52:28.464427 16530 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:28.458503 16549 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:28.493592 16586 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:28.455559 16597 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260808 01:52:28.472590 16550 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:28.503623 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:28.535882  9478 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.012s	sys 0.013s
W20260808 01:52:28.545703  9478 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.013s	sys 0.014s
I20260808 01:52:28.498091 16464 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:28.460261 16029 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:28.446883 16591 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.083s
I20260808 01:52:28.529224 16585 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:28.453783 16150 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.076s
I20260808 01:52:28.506358 16565 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:28.526079 16468 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:28.551381  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:28.589290  9521 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:28.553118 16601 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:28.492925 16493 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:28.532320 16590 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:28.599844  9521 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:28.532131 16356 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:28.653613  9550 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:28.662411  9638 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:52:28.677428  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 77671.5us
W20260808 01:52:28.525650  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:28.526604 16558 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:28.537315 16290 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:28.475744 16595 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:28.680644  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:28.588616 16573 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:28.540697 16530 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.047s
I20260808 01:52:28.598204 16406 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260808 01:52:28.724077  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 197648us
I20260808 01:52:28.728226 10368 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:28.725900 16573 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:28.592217 16585 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:28.759258  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:28.673516 16547 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:28.661895 16550 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.126s
I20260808 01:52:28.739840  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:28.673143 16279 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:28.662894 16601 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:28.760860  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:28.762221  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:28.806905  9635 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:28.808358  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:28.762954  9636 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260808 01:52:28.811214  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:28.812805  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:28.813548  9636 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:28.766548 10368 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
W20260808 01:52:28.784963  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 45234.5us
I20260808 01:52:28.597213 16565 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:28.819993  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:28.822628 16565 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:28.751395  9630 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260808 01:52:28.707762 16229 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.073s
I20260808 01:52:28.801698  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:28.833690  9633 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:28.801996  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:28.834893  9631 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:52:28.726895 16451 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:28.803463  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:28.836149  9637 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:28.710006 16570 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:28.809608  9635 raft_consensus.cc:2410] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 8.
I20260808 01:52:28.837456 16570 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:28.766390  9472 pending_rounds.cc:85] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 9
I20260808 01:52:28.826444  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:28.837921 16570 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:28.767146 16249 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:28.838161  9472 pending_rounds.cc:107] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 16.10
I20260808 01:52:28.693156 16569 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
I20260808 01:52:28.838429  9481 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:28.838565  9472 raft_consensus.cc:2889] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:28.733661 16580 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:28.840507  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:28.838560 16249 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:28.769768 16560 consensus_meta.cc:287] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.174s	user 0.012s	sys 0.004s
I20260808 01:52:28.852761 10364 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:28.854331  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:28.849905  9475 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: [11.11-13.12]
I20260808 01:52:28.858362  9636 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
W20260808 01:52:28.863010 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:28.692603 16464 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:28.786017 16573 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:28.848268 16597 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:52:28.894390 16602 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.343s
W20260808 01:52:28.897948 16615 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
W20260808 01:52:28.899840 16610 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.208s	user 0.001s	sys 0.000s
I20260808 01:52:28.893985 16565 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:28.853322 16582 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:28.901419 16565 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:28.865507 16603 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
W20260808 01:52:28.916869 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 434296us
I20260808 01:52:28.896629 16596 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:28.904335 16371 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:28.861186 16451 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:28.804849 16468 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:28.917951 16560 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:28.938871 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:28.925057 16476 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:28.889837 16614 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.074s
I20260808 01:52:28.839728 16580 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:28.930714 16617 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:28.930866 16597 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:28.963117 16605 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.236s	user 0.000s	sys 0.002s
I20260808 01:52:28.963496 16617 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:28.938304 16029 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:28.975782 10369 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:28.978374  9523 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:28.944468 16371 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:28.988610 16249 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:28.989809 16371 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:28.990269 16587 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.741s	user 0.004s	sys 0.002s
W20260808 01:52:28.993896 16611 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.213s	user 0.001s	sys 0.000s
W20260808 01:52:28.980670 16607 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.292s	user 0.002s	sys 0.000s
I20260808 01:52:29.003124 16592 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:29.004760 16617 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:29.005771 16585 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:28.948556 16621 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.030s
I20260808 01:52:29.006913 16451 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:28.985385 16279 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:29.009552  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:29.011018  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:29.011169  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:28.980721 16356 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:29.013125 16356 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:29.013638 16356 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:29.014746 16604 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.212s	user 0.001s	sys 0.000s
I20260808 01:52:29.014585 16356 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 16.7, Last appended by leader: 7, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.015772 16569 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:28.951766 16560 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.016309 16569 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.016566  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:29.017804  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:29.018303 16560 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:29.018435 16451 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:29.018854 16591 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:28.940373 16596 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.019070 16451 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.019474 16591 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.018435 16569 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:29.021025 16596 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:28.983913 16290 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:29.021790 16591 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:29.022047 16290 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:29.022529 16290 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:28.982820 16406 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:28.935813 16601 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:29.023471 16290 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:29.024191 16406 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:29.024744 16601 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:29.025207 16406 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:29.025774 16451 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:29.028398 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 91363.4us
I20260808 01:52:28.953213 16476 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.029297 10332 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:29.004905 16616 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260808 01:52:29.032157 10332 leader_election.cc:400] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 13. Current term is 14.
I20260808 01:52:29.012180  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:29.012132  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:29.033392  9559 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.009s	sys 0.014s
I20260808 01:52:29.036026 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:29.040616  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:29.048455  9552 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:29.048969 10369 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
W20260808 01:52:29.049062  9559 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.572s	user 0.009s	sys 0.014s
I20260808 01:52:29.049592  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:29.073928 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:29.091470 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:29.092376 10369 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:29.094878  9523 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:29.097019  9523 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:29.108146  9478 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.000s	sys 0.015s
W20260808 01:52:29.110548  9478 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.000s	sys 0.015s
I20260808 01:52:29.114589  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:29.012111 16580 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:29.114238 16625 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:29.115509  9555 raft_consensus.cc:2368] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 13. Current term is 14.
I20260808 01:52:29.028443 16634 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260808 01:52:29.071226 16367 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.596s	user 0.007s	sys 0.005s
I20260808 01:52:29.023608 16565 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:28.985215 16575 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:29.163431  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:29.119756 16592 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:52:29.130021  9473 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
W20260808 01:52:29.175028  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 48148.3us
I20260808 01:52:29.176509  9473 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:29.129144  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:29.018880 16560 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:29.043430 16585 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:29.091862 16558 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:28.977851 16597 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:29.152356 16609 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:29.194530 16609 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.196979 16609 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:29.187604 16535 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.012s
I20260808 01:52:29.152894 16631 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:29.197768 16631 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:29.198189 16631 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:29.153331 16639 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:52:29.169057 16625 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:29.198974 16639 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:29.199448 16639 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:29.199760 16625 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:29.119401 16578 consensus_meta.cc:287] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.903s	user 0.006s	sys 0.009s
I20260808 01:52:29.202972 16578 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:29.203799  9476 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:29.159165  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:29.207072  9555 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:29.206745 16573 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.207996  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:29.211439 16597 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:29.213142  9632 raft_consensus.cc:3055] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:29.213671  9632 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:52:29.215792 16589 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.924s	user 0.002s	sys 0.003s
I20260808 01:52:29.214535  9632 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.120878 16577 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:29.230230 10368 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260808 01:52:29.169667  9480 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
W20260808 01:52:29.137625 16447 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.716s	user 0.013s	sys 0.000s
I20260808 01:52:29.169661  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 963) took 1011 ms. Trace:
I20260808 01:52:29.127135 16593 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:29.225478 16386 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:29.239732  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:29.239897 16386 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:29.236294  9557 rpcz_store.cc:276] 0808 01:52:28.158960 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:28.159714 (+   754us) service_pool.cc:224] Handling call
0808 01:52:28.160260 (+   546us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:28.160789 (+   529us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:28.160831 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:28.192443 (+ 31612us) log.cc:844] Serialized 26 byte log entry
0808 01:52:28.488493 (+296050us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:52:28.827208 (+338715us) log.cc:844] Serialized 14 byte log entry
0808 01:52:28.892035 (+ 64827us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:28.892123 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:29.169410 (+277287us) raft_consensus.cc:1685] finished
0808 01:52:29.169433 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:29.169578 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23158,"prepare.queue_time_us":187987,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":5136,"raft.queue_time_us":40342,"raft.run_cpu_time_us":466,"raft.run_wall_time_us":20832,"thread_start_us":578470,"threads_started":3,"wal-append.queue_time_us":375422}
I20260808 01:52:29.241304  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:29.179737  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:29.148092 16634 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:29.246075 16634 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:29.245692 16415 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.7, Last appended by leader: 7, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.230317 16577 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:29.247253 16577 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:29.185765  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:29.173962 16549 consensus_meta.cc:287] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.631s	user 0.014s	sys 0.000s
I20260808 01:52:29.147398 16580 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:29.187484 16565 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.005s
I20260808 01:52:29.249830 10365 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260808 01:52:29.250236 16565 raft_consensus.cc:3055] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:29.192304 16585 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.249418 16580 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.187875 16560 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:29.193321 16623 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:29.193500 16279 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:52:29.252840 16623 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:29.253410 16623 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:29.221072  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3183) took 1796 ms. Trace:
I20260808 01:52:29.221797  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:29.255116 16535 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:29.255654 16535 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:29.256054 16535 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
W20260808 01:52:29.126894  9445 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.007s	sys 0.004s
I20260808 01:52:29.224148  9600 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:29.257807  9445 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.007s	sys 0.005s
I20260808 01:52:29.178200 16592 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:29.227785  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:29.174088 16456 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:29.260293  9555 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:29.242141  9631 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
W20260808 01:52:29.242851  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 63190.5us
I20260808 01:52:29.244355 16447 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.254597  9478 rpcz_store.cc:276] 0808 01:52:27.424292 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:27.463091 (+ 38799us) service_pool.cc:224] Handling call
0808 01:52:27.520284 (+ 57193us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:27.589822 (+ 69538us) raft_consensus.cc:1534] Early marking committed up to index 10
0808 01:52:27.922907 (+333085us) log.cc:844] Serialized 14 byte log entry
0808 01:52:29.111916 (+1189009us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:29.111982 (+    66us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:29.112052 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:29.220829 (+108777us) spinlock_profiling.cc:243] Waited 69.2 ms on lock 0x7b5400392bd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:29.220877 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:29.221011 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16115,"prepare.queue_time_us":423975,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":56,"raft.queue_time_us":211788,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":44993,"spinlock_wait_cycles":159142784,"thread_start_us":1489896,"threads_started":3,"wal-append.queue_time_us":773535}
I20260808 01:52:29.224367  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:29.264240  9637 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
W20260808 01:52:29.258188  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 752834us
I20260808 01:52:29.249387  9559 raft_consensus.cc:2410] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260808 01:52:29.189617 16603 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:29.233485 10368 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: []
I20260808 01:52:29.270393  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:29.270933  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 46875.9us
I20260808 01:52:29.271690  9557 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:29.261497  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:29.217666 16573 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:29.277622  9555 raft_consensus.cc:2410] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260808 01:52:29.278751 16609 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:52:29.186988 16617 consensus_queue.cc:1059] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.060s
W20260808 01:52:29.259876 16544 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.996s	user 0.014s	sys 0.000s
I20260808 01:52:29.250769 16565 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:29.263721 16592 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:29.291152  9521 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:29.263406 16457 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:29.286489  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:29.286154 16447 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:29.313884  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:29.315534  9475 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:29.315259 16565 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.316025 16386 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:29.316694  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:29.338158  9478 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:29.269280 16456 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.317432 16544 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:29.274233 16415 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:29.341666  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:29.357558  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 66549.6us
I20260808 01:52:29.355042  9474 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:29.358979  9559 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260808 01:52:29.367075  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:29.376310 16570 consensus_meta.cc:287] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.524s	user 0.015s	sys 0.003s
I20260808 01:52:29.376926 16570 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:29.383633 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:29.384690 10368 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
W20260808 01:52:29.384964 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:29.382050 16570 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:29.294165 16596 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.059s
I20260808 01:52:29.397010  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:29.404121  9559 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
W20260808 01:52:29.405031  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:29.353542 16573 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:29.406507 10363 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:29.408458  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:29.409353  9478 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 20: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:29.410629  9481 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:29.388327 16249 raft_consensus.cc:2764] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
W20260808 01:52:29.421389  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 54434.5us
I20260808 01:52:29.428539  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:29.425765 10368 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:29.430249  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:29.372453 16447 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260808 01:52:29.290448 16575 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.339910 16457 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:52:29.432137  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:29.438241  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:29.441944  9481 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:29.442107  9477 raft_consensus.cc:2368] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15.
I20260808 01:52:29.443771  9521 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:29.440276  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:29.435662 16612 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.017s
W20260808 01:52:29.453305 16629 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260808 01:52:29.453600 16612 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election won for term 19
W20260808 01:52:29.454859 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:29.455271 16641 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260808 01:52:29.456562 16544 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:29.454288 16612 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:29.465629 16612 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:29.467784 16649 log.cc:927] Time spent T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260808 01:52:29.388216 16591 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.045s
I20260808 01:52:29.483896  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:29.439944  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:29.490226  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:29.510627  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:29.430135 16609 raft_consensus.cc:697] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:52:29.523221 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:29.485028 16249 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:29.420413 16573 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:29.489671 16530 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Stepping down as leader of term 17
W20260808 01:52:29.538977  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:29.489410 16367 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election won for term 17
W20260808 01:52:29.544054  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 255934us
I20260808 01:52:29.512455 16578 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260808 01:52:29.546581  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 58077.7us
I20260808 01:52:29.546976  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:29.547174  9637 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: [16.7-16.7]
I20260808 01:52:29.549723  9477 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260808 01:52:29.555308  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:29.555899  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:29.556648  9481 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:29.557972  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:29.560997 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:29.569998 10367 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 19: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:29.577374  9474 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 11.
I20260808 01:52:29.578274  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:29.578734  9631 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:29.581072  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:29.539927 16249 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:29.582458  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:29.589037  9631 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:29.543484 16530 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:29.583791  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:29.590080 16249 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:29.590050  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:29.580646 16356 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:52:29.581501 16639 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.596505  9637 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:29.716439  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:29.717134  9637 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:29.717145 16639 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:29.718189  9631 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:29.718155  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:29.719321  9637 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:29.580827 16646 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:52:29.723933 16530 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:29.727300  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2684) took 1452 ms. Trace:
I20260808 01:52:29.727926  9550 rpcz_store.cc:276] 0808 01:52:28.274708 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:28.287980 (+ 13272us) service_pool.cc:224] Handling call
0808 01:52:29.627258 (+1339278us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b5400409554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:29.727221 (+ 99963us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":482192256}
I20260808 01:52:29.731328 16457 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:29.581625 16623 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.707434  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
W20260808 01:52:29.707931  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 156136us
W20260808 01:52:29.703150 16655 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.157s	user 0.004s	sys 0.000s
I20260808 01:52:29.749362  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2681) took 1558 ms. Trace:
I20260808 01:52:29.749883  9556 rpcz_store.cc:276] 0808 01:52:28.191332 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:28.209775 (+ 18443us) service_pool.cc:224] Handling call
0808 01:52:28.446880 (+237105us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:28.477082 (+ 30202us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:29.749106 (+1272024us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b540029f9d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:29.749283 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":475010432}
I20260808 01:52:29.713356  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:29.639271 16609 consensus_queue.cc:237] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:29.676329 10363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 779) took 1621 ms (client timeout 1829 ms). Trace:
I20260808 01:52:29.704430 16367 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:29.741709  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:29.760504 10363 rpcz_store.cc:269] 0808 01:52:28.054776 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:28.061659 (+  6883us) service_pool.cc:224] Handling call
0808 01:52:29.602144 (+1540485us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b540056fed4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:29.676242 (+ 74098us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3010402176}
W20260808 01:52:29.762079  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 48952.8us
I20260808 01:52:29.827555 16671 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260808 01:52:29.827924 16666 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
I20260808 01:52:29.829962 16671 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:29.764560 16646 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:29.831140 16669 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:29.831441 16367 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 15.15, Last appended by leader: 15, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:29.831884 16669 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:29.832298 16573 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:29.832442 16415 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:29.833104 16415 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:29.833701 16664 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:29.834599 16664 consensus_queue.cc:1243] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 7201605bb2f54c949a4ebe6e9b3b4b2a log is divergent from this leader: its last log entry 14.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260808 01:52:29.830669 16249 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:29.836695 16639 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260808 01:52:29.835273 16664 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:52:29.836241 16530 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:29.837615 16670 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:29.838194 16670 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.839020  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2677) took 1652 ms. Trace:
I20260808 01:52:29.840734 16670 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:29.839433  9560 rpcz_store.cc:276] 0808 01:52:28.186877 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:28.187016 (+   139us) service_pool.cc:224] Handling call
0808 01:52:28.187374 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:28.198815 (+ 11441us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:52:28.489912 (+291097us) log.cc:844] Serialized 14 byte log entry
0808 01:52:29.411642 (+921730us) spinlock_profiling.cc:243] Waited 4.58 ms on lock 0x7b18004316e0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d4453242a 00007f5d4499de0a
0808 01:52:29.540515 (+128873us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:29.542015 (+  1500us) log.cc:844] Serialized 26 byte log entry
0808 01:52:29.542168 (+   153us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:29.542778 (+   610us) log.cc:844] Serialized 14 byte log entry
0808 01:52:29.712743 (+169965us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:29.712950 (+   207us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:29.838767 (+125817us) raft_consensus.cc:1685] finished
0808 01:52:29.838789 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:29.838952 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":68716,"prepare.queue_time_us":293877,"prepare.run_cpu_time_us":129,"prepare.run_wall_time_us":10775,"raft.queue_time_us":647495,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":31951,"spinlock_wait_cycles":10524416,"thread_start_us":500982,"threads_started":3,"wal-append.queue_time_us":1105445}
I20260808 01:52:29.841096 16623 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:29.842317 16415 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:29.766819 16535 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.510s	user 0.016s	sys 0.006s
I20260808 01:52:29.843302 16535 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.845008 16249 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:29.846405 16535 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:29.850896 16665 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.091s	user 0.001s	sys 0.003s
I20260808 01:52:29.862236 16657 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:29.861931 16356 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260808 01:52:29.881012  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 437310us
I20260808 01:52:29.882213  9521 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:29.889864  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 338552us
I20260808 01:52:29.834357 16669 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:29.880003  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 327694us
I20260808 01:52:29.862919 16657 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:29.892290  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:29.868927 16646 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:29.895223  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:29.896267  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 317783us
I20260808 01:52:29.894325  9599 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:29.907529 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:29.897604  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:29.919404  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:29.925858  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:29.926033  9599 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:29.935413  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:29.936378  9560 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 15
I20260808 01:52:29.940227  9524 leader_election.cc:400] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15.
I20260808 01:52:29.940848  9524 leader_election.cc:403] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:52:29.937944  9521 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:29.945448  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:29.891898 16637 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:29.957005 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:29.951143  9559 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260808 01:52:29.931061 16669 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:29.960039  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:29.960225 16672 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:29.960829  9551 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:29.961440  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:29.961094 16672 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:29.898370 16386 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:29.963559 16576 raft_consensus.cc:3055] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:29.964064 16576 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:29.965179 16576 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:29.968389 16679 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:29.969231 16679 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.970180  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:29.893708 16573 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:29.972108 16673 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:29.972821 16673 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:29.934409 16578 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:29.973402 16673 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:29.975356 16464 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:29.976387  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:29.972481 16573 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.981264  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:29.936897 16530 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:29.973438  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:29.989035  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:29.996136  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:29.997028  9477 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:29.948712 16415 consensus_queue.cc:1059] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 17.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.051s
I20260808 01:52:29.948367 16650 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:30.000376 16650 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:30.001748  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:30.005318  9477 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 15: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:52:30.005913 16592 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.667s	user 0.011s	sys 0.002s
I20260808 01:52:30.027460 16592 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:30.028887  9557 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 22. Current term is 23. Ops: []
I20260808 01:52:30.029255 16579 raft_consensus.cc:3055] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:30.029701 16579 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:30.036136 16569 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:30.036880 16569 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:30.030658 16579 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:30.037930 16579 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:30.040570 16592 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:30.012452 16657 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:30.012563 16573 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:30.013656 16590 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.322s	user 0.010s	sys 0.006s
I20260808 01:52:29.985333 16637 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:29.943511 16646 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:30.023792  9559 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260808 01:52:30.025624  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:30.009493 16672 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:30.062304  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:30.061687 16590 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:30.065260  9473 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:30.066097  9629 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:30.066435  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:30.067376 16590 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:30.068451 16444 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:30.069073 16444 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:52:30.069293  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 123863us
I20260808 01:52:29.966614 16576 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:30.042315 16569 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:30.032840 16415 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260808 01:52:30.043131  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:30.043771 16650 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:30.046231  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:30.047210 16560 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:30.048115 16591 raft_consensus.cc:3055] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:30.049695 16585 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:30.051467  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:30.084352  9474 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
W20260808 01:52:30.040756 16631 consensus_meta.cc:287] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.842s	user 0.013s	sys 0.000s
I20260808 01:52:30.031878  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3518) took 1484 ms. Trace:
I20260808 01:52:30.073689  9478 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: [11.11-13.12]
I20260808 01:52:30.089016  9557 rpcz_store.cc:276] 0808 01:52:28.547629 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:29.323280 (+775651us) service_pool.cc:224] Handling call
0808 01:52:29.340595 (+ 17315us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:30.029332 (+688737us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:30.031529 (+  2197us) spinlock_profiling.cc:243] Waited 688 ms on lock 0x7b540002cbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:30.031792 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1582371072}
W20260808 01:52:30.085974  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 160045us
I20260808 01:52:30.071346  9560 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:30.079751  9477 raft_consensus.cc:2410] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260808 01:52:30.022771 16669 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:30.086526  9601 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:30.070619  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:30.126339  9474 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:52:30.128099  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:30.130302  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:30.100447  9475 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260808 01:52:30.131845  9551 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:30.095139 16249 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15.
I20260808 01:52:30.069597 16673 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:30.249274  9555 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:30.062387 16646 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:30.088836 16577 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.841s	user 0.001s	sys 0.014s
I20260808 01:52:30.084206 16415 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:30.253583  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 124088us
I20260808 01:52:30.083827 16591 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:30.088258 16679 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:30.258576  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 128344us
I20260808 01:52:30.287724  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:30.132483  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62038.4us
W20260808 01:52:30.292877  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 409535us
I20260808 01:52:30.294016  9599 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:30.294308  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:30.088097 16672 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.176s
I20260808 01:52:30.254663 16670 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:30.262271 16575 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:30.099977 16631 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:30.255503 16530 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:30.248828 16464 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:30.289650 16594 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:30.134163 16674 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:30.290766 16457 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:30.314157 16415 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:30.359305 16547 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:30.360216 16662 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260808 01:52:30.289944 16535 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.075s
I20260808 01:52:30.290272 16577 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:30.330540 16575 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:30.371292 16591 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.8, Last appended by leader: 8, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:30.375151 16673 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:30.378194  9552 raft_consensus.cc:2368] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20.
I20260808 01:52:30.332144 16670 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:30.365185  9635 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260808 01:52:30.342402 16530 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:30.383610 16547 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:30.312767 16542 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.053s
I20260808 01:52:30.360889 16594 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:30.377957 16680 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260808 01:52:30.397296  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:30.366071 16462 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:30.399952 16575 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:52:30.484153  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.021s	sys 0.057s
I20260808 01:52:30.414790  9472 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 16: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:52:30.490903  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.022s	sys 0.057s
I20260808 01:52:30.402640 16623 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260808 01:52:30.490473  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 93303.4us
W20260808 01:52:30.486138 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 398482us
I20260808 01:52:30.382150 16457 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:30.406808 16670 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
W20260808 01:52:30.491390  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 545998us
I20260808 01:52:30.409654 16530 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:30.414243 16631 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:30.368286 16464 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:30.411347 16591 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
W20260808 01:52:30.491624 16634 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.245s	user 0.008s	sys 0.009s
I20260808 01:52:30.492700 16444 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:30.514602 16444 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:30.516145 16493 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:30.496654  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:30.524549  9521 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:30.413108 16577 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:30.507893 16570 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:30.531137 16571 raft_consensus.cc:2764] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260808 01:52:30.531769 16637 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:30.499720 16462 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:30.496573 16150 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:30.532791 16150 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:30.533030 16462 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:30.533870 16150 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:30.501449  9523 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.020s	sys 0.025s
I20260808 01:52:30.506326  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2687) took 2101 ms. Trace:
I20260808 01:52:30.484328 16594 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:30.526947 16242 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:30.407318 16657 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:30.541466  9523 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.021s	sys 0.025s
I20260808 01:52:30.542657  9633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "17e220e6a25747939bdf18f26b732407"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
W20260808 01:52:30.543357  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 249440us
W20260808 01:52:30.545254  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 598413us
I20260808 01:52:30.549975  9633 raft_consensus.cc:606] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:30.550457  9633 raft_consensus.cc:614] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:30.395576 16669 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:30.552332  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:30.555850  9599 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:30.551342  9524 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:30.544397  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 256754us
I20260808 01:52:30.547392 16242 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:30.564926  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:30.566834  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:30.559860  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:30.572630  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:30.558133 16586 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.965s	user 0.011s	sys 0.021s
W20260808 01:52:30.596939  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 17e220e6a25747939bdf18f26b732407 (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:30.597148  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:30.557160 16631 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:30.512020 16634 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:30.560508 16053 raft_consensus.cc:3055] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Stepping down as leader of term 13
W20260808 01:52:30.567909 16612 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.098s	user 0.016s	sys 0.010s
I20260808 01:52:30.560156 16571 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:52:30.602427  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:30.570842 16242 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:30.560310 16451 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:30.568682 16367 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260808 01:52:30.613922  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:30.557492 16657 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:30.602816 16678 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:30.580317 16577 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:30.605674 16682 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:30.576828 16698 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:30.573908 16679 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.054s
I20260808 01:52:30.578915 16650 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:52:30.561564 16639 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.045s
I20260808 01:52:30.630505  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:30.633255  9637 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260808 01:52:30.634100 16571 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:30.611698 16612 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.141s	user 0.017s	sys 0.011s
I20260808 01:52:30.597045 16569 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:30.564610 16637 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
I20260808 01:52:30.608330  9599 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:30.598775 16586 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.014s	user 0.012s	sys 0.022s
I20260808 01:52:30.645802  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:30.607425 16053 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:30.573215 16677 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:30.667302  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:30.634912 16457 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:30.672358 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:30.650002 16579 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.612s	user 0.015s	sys 0.001s
W20260808 01:52:30.677484  9476 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 3144) took 2370 ms (client timeout 1834 ms). Trace:
I20260808 01:52:30.613831 16631 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:30.658811 16593 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260808 01:52:30.693136  9476 rpcz_store.cc:269] 0808 01:52:28.307141 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:28.307301 (+   160us) service_pool.cc:224] Handling call
0808 01:52:30.677188 (+2369887us) spinlock_profiling.cc:243] Waited 847 ms on lock 0x7b5400399754. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:30.677385 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1947685248}
I20260808 01:52:30.694908 16542 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:30.662999 16701 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:30.696844 16701 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:30.699065 16476 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:30.704034 16468 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:30.699198 16701 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:30.707067  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:30.695016 16316 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:30.707964  9637 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:30.709164  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:30.711880 16673 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:30.717550  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:30.710421  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:30.721071  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:30.683470 16586 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:30.722612  9475 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260808 01:52:30.636168 16451 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:30.722263  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 114071us
I20260808 01:52:30.733771  9599 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:30.725982 10332 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:30.666985 16577 raft_consensus.cc:697] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:30.750494  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:30.817829 16451 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:30.819895  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 86292.7us
I20260808 01:52:30.821812  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:30.823079  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:30.824224  9560 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:30.827555  9630 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:30.827696  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:30.721480 16593 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:30.683096 16569 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:30.735461 16464 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:30.856382  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:30.664462 16682 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:30.728520 16542 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:30.809814 16700 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
W20260808 01:52:30.835253 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 109404us
W20260808 01:52:30.871311  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 274248us
W20260808 01:52:30.873000  9524 connection.cc:256] RPC call timeout handler was delayed by -1.0756842400296591s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1834s for a response
I20260808 01:52:30.829874 16029 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260808 01:52:30.878549  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 51698.2us
I20260808 01:52:30.830318 15933 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:30.692457 16677 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:30.941479 15933 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:30.944252 16476 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:30.939709  9560 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:30.752326 16650 raft_consensus.cc:697] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:30.851833 16593 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:30.670219 16698 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:30.948518  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 138294us
I20260808 01:52:30.843597 16462 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:30.752074 16547 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:30.959553 16694 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.356s	user 0.002s	sys 0.000s
I20260808 01:52:30.951960 16468 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:30.950245 16678 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:30.948800 16367 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:30.877146 16692 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:30.844007 16053 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:30.987907 16692 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:30.987972 16367 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:30.949101 16579 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:30.964112 16577 consensus_queue.cc:237] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:30.998118 16695 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.354s	user 0.001s	sys 0.001s
W20260808 01:52:30.978816 16415 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.585s	user 0.004s	sys 0.009s
I20260808 01:52:31.007205  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3208) took 1006 ms. Trace:
I20260808 01:52:30.878552 16542 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:31.007941 16650 consensus_queue.cc:237] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:31.021669 16547 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.023485 16542 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:30.867138 16705 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.143s
W20260808 01:52:30.996472 16703 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
I20260808 01:52:31.027743  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.028884  9478 raft_consensus.cc:2393] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 18: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:31.029345 16705 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:30.945711 16696 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:31.009965 16713 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:30.994313 16579 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:31.017668  9479 rpcz_store.cc:276] 0808 01:52:30.000816 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:30.000984 (+   168us) service_pool.cc:224] Handling call
0808 01:52:30.001388 (+   404us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:30.001827 (+   439us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:30.001855 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:30.002381 (+   526us) log.cc:844] Serialized 26 byte log entry
0808 01:52:30.494394 (+492013us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:52:30.494469 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:30.494622 (+   153us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:31.006900 (+512278us) raft_consensus.cc:1685] finished
0808 01:52:31.006927 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:31.007130 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1147,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":94,"spinlock_wait_cycles":123648,"thread_start_us":491569,"threads_started":1,"wal-append.queue_time_us":600949}
I20260808 01:52:31.030344 16367 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:31.030951 16415 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:30.943608 16586 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:31.032155 16715 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260808 01:52:30.989872 16678 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:30.955628 16476 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:30.969192 16717 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:30.999552 16634 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:31.039912 16591 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.505s	user 0.007s	sys 0.010s
W20260808 01:52:30.993187  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:31.058174  9559 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260808 01:52:31.059262 16682 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.043352 16692 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:31.060084 16697 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.404s	user 0.000s	sys 0.001s
I20260808 01:52:31.032356 16696 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:31.032850 16713 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:31.075275  9634 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
W20260808 01:52:31.056664 16644 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.597s	user 0.001s	sys 0.013s
I20260808 01:52:31.082471  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1138) took 1826 ms. Trace:
I20260808 01:52:31.083158  9480 rpcz_store.cc:276] 0808 01:52:29.256063 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:29.256220 (+   157us) service_pool.cc:224] Handling call
0808 01:52:29.256603 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:29.256996 (+   393us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:29.257027 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:29.257542 (+   515us) log.cc:844] Serialized 26 byte log entry
0808 01:52:29.985071 (+727529us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b1800186ca0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:52:30.397517 (+412446us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:52:30.397618 (+   101us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:30.397766 (+   148us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:31.077379 (+679613us) raft_consensus.cc:1685] finished
0808 01:52:31.077402 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:31.082368 (+  4966us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9595,"raft.queue_time_us":24331,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":25563,"spinlock_wait_cycles":39769856,"thread_start_us":408956,"threads_started":1,"wal-append.queue_time_us":1267220}
I20260808 01:52:31.072081 16462 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:31.005667 16468 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:31.039017 16705 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:31.093184  9550 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: []
W20260808 01:52:31.110599 16617 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.747s	user 0.011s	sys 0.018s
W20260808 01:52:31.140888 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 270381us
I20260808 01:52:31.072033 16717 raft_consensus.cc:697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:31.143858 10332 leader_election.cc:400] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 13. Current term is 14.
I20260808 01:52:31.146749 10332 leader_election.cc:403] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260808 01:52:31.087255 16657 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:31.088433 16701 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:31.142589 16617 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.785s	user 0.012s	sys 0.018s
I20260808 01:52:31.160750 16705 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.136039 16571 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.167338 16701 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:31.176242 16701 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:31.105679 16462 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:31.184513  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:31.153767 15933 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:31.191416  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:31.163924 16569 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.202996 16579 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:31.186750 16634 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:31.213866  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3517) took 2669 ms. Trace:
I20260808 01:52:31.218266  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2701) took 1279 ms. Trace:
W20260808 01:52:31.198405 16594 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.646s	user 0.008s	sys 0.008s
W20260808 01:52:31.218123  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:31.222443  9555 rpcz_store.cc:276] 0808 01:52:28.544135 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:29.301187 (+757052us) service_pool.cc:224] Handling call
0808 01:52:29.310599 (+  9412us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:30.249810 (+939211us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:31.213610 (+963800us) spinlock_profiling.cc:243] Waited 938 ms on lock 0x7b540002c1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:31.213803 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2158155648}
I20260808 01:52:31.223203  9550 rpcz_store.cc:276] 0808 01:52:29.939332 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:29.939696 (+   364us) service_pool.cc:224] Handling call
0808 01:52:29.940162 (+   466us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:31.137652 (+1197490us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:31.217985 (+ 80333us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540008acd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:31.218197 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2651029376}
I20260808 01:52:31.189850 16677 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.234354  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:31.240069  9555 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:31.243593  9555 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 7. Preceding OpId from leader: term: 17 index: 8. (index mismatch)
W20260808 01:52:31.213529 16530 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.703s	user 0.012s	sys 0.004s
W20260808 01:52:31.221405 16242 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.584s	user 0.007s	sys 0.007s
W20260808 01:52:31.217483 16654 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 1.754s	user 0.004s	sys 0.005s
I20260808 01:52:31.206601 16726 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 13. Current term is 14.
I20260808 01:52:31.213178 16456 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:31.208333 16462 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:31.216105 16698 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:31.245572 16576 consensus_meta.cc:287] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.170s	user 0.017s	sys 0.006s
I20260808 01:52:31.194994 16650 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:31.246281 16575 consensus_meta.cc:287] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.747s	user 0.014s	sys 0.003s
I20260808 01:52:31.216295 16476 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:31.215538 16696 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.271085 16677 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:31.258248 16415 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.243225 16594 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:31.293481 16677 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:31.311882  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:31.253187 16530 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:31.314532  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:31.312809 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 170787us
I20260808 01:52:31.254065 16242 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:31.239163 16634 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:31.316155 10332 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:31.317838  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:31.319913  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:31.325995  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:31.313054 16670 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.804s	user 0.016s	sys 0.000s
I20260808 01:52:31.333688  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:31.334558  9636 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:31.335621  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:31.311983 16713 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.312005 16290 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:31.338084  9637 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 20: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:31.342095  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:31.335886  9521 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:31.336591 16670 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:31.340694 16726 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:31.215777 16717 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:31.350000  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:31.354799  9557 raft_consensus.cc:2410] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 18 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260808 01:52:31.268937 16575 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:31.362226 16575 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:31.284785 16650 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:31.391229 10332 connection.cc:256] RPC call timeout handler was delayed by -1.3525027689756826s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2134s for a response
W20260808 01:52:31.391980 10332 connection.cc:256] RPC call timeout handler was delayed by -1.2661027689755429s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.223s for a response
I20260808 01:52:31.291728 16593 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:31.356077  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:31.357306 16476 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:31.329510 16663 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:31.399139 16663 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:31.400399 16593 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:31.396924 16669 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:31.401589 16663 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:31.327611 16700 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:31.371328  9633 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 18. Ops: []
I20260808 01:52:31.402194 16728 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:52:31.405395  9551 raft_consensus.cc:2393] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 12: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:31.406667 16476 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:31.409086 16476 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:31.372767 16670 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:31.387697 16242 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.350822  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:31.394397  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:31.413980  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:31.416206  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:31.354007 16249 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:31.414628  9557 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:52:31.420418  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:31.425184 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:31.427085 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:31.433220  9557 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:31.434198  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:31.437494  9636 raft_consensus.cc:2410] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
W20260808 01:52:31.423841 16727 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260808 01:52:31.410305 16730 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:31.424036 16249 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:52:31.449716  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:31.432305 16728 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:31.441583 16634 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:31.450846 16249 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:31.452291 16728 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:31.455247 16677 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.455616 16580 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.448243 16730 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:31.465276 16634 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:30.540948  9558 rpcz_store.cc:276] 0808 01:52:28.405322 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:28.448432 (+ 43110us) service_pool.cc:224] Handling call
0808 01:52:28.497750 (+ 49318us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:28.497991 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:52:28.789400 (+291409us) log.cc:844] Serialized 14 byte log entry
0808 01:52:30.055534 (+1266134us) spinlock_profiling.cc:243] Waited 18.1 ms on lock 0x7b1800475180. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d4453242a 00007f5d4499de0a
0808 01:52:30.501214 (+445680us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:30.501298 (+    84us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:30.501361 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:30.501479 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:30.506219 (+  4740us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30696,"prepare.queue_time_us":233810,"prepare.run_cpu_time_us":2720,"prepare.run_wall_time_us":18778,"raft.queue_time_us":132,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":2405,"spinlock_wait_cycles":41696000,"thread_start_us":680277,"threads_started":2}
I20260808 01:52:31.470207 16576 consensus_queue.cc:1059] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.010s
I20260808 01:52:31.479142 16634 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:31.479697 16634 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:31.480185 16689 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.079s	user 0.000s	sys 0.002s
I20260808 01:52:31.485944 16634 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.486614 16634 raft_consensus.cc:3055] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:31.487035 16634 raft_consensus.cc:740] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:31.500162  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:31.500255 16415 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:31.501248  9558 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 15: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:31.501148 16415 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:31.503188 16650 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.504308 16730 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:31.474004 16530 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:31.505751 16580 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:31.504421  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:31.472368 16594 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:31.510062 16710 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.460s	user 0.002s	sys 0.000s
I20260808 01:52:31.469624 16728 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:31.501416 16565 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:31.510795 16580 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:31.513696 16595 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:31.514546 16580 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.517834  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:31.520578  9638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2832) took 3258 ms (client timeout 2706 ms). Trace:
W20260808 01:52:31.521063  9638 rpcz_store.cc:269] 0808 01:52:28.262295 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:28.268225 (+  5930us) service_pool.cc:224] Handling call
0808 01:52:31.520318 (+3252093us) spinlock_profiling.cc:243] Waited 316 ms on lock 0x7b54004213d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:31.520450 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":726067968}
I20260808 01:52:31.522322  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:31.487898 16634 consensus_queue.cc:260] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:31.525873  9638 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:52:31.526487  9557 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:31.528630 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:31.528981  9631 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:31.530058 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:31.523407 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:31.530824 10363 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:31.530958 10367 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:31.524183 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:31.532011 10362 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:31.532498 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:31.533233 10360 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:31.524883 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:31.535069 10369 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:31.536041 16601 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:31.536491  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:31.537057 16601 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:31.539727 10363 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:31.539894 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:31.541667 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:31.542527  9632 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:31.505630 16677 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:31.543979 16677 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:31.547449  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:31.526080 16631 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.553555 16623 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:31.554509 16623 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:31.560127 10364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 783) took 3346 ms. Trace:
I20260808 01:52:31.561070 16601 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:31.560645 10364 rpcz_store.cc:276] 0808 01:52:28.213933 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:28.214226 (+   293us) service_pool.cc:224] Handling call
0808 01:52:28.241258 (+ 27032us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:28.861568 (+620310us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:31.559718 (+2698150us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b54006108d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:31.560009 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1396723072}
W20260808 01:52:31.565639 16711 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.611s	user 0.001s	sys 0.000s
I20260808 01:52:31.569139 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:31.570117  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2706) took 1628 ms. Trace:
I20260808 01:52:31.570605  9559 rpcz_store.cc:276] 0808 01:52:29.941963 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:30.064593 (+122630us) service_pool.cc:224] Handling call
0808 01:52:30.072392 (+  7799us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:31.069524 (+997132us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:31.569811 (+500287us) spinlock_profiling.cc:243] Waited 985 ms on lock 0x7b540029f4d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:31.570051 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2266243456}
I20260808 01:52:31.570119 10361 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 14.
I20260808 01:52:31.579162  9560 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260808 01:52:31.580437  9632 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260808 01:52:31.581670  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:31.583473  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:31.584380  9560 raft_consensus.cc:2410] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260808 01:52:31.588507  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:31.589545  9557 raft_consensus.cc:2410] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260808 01:52:31.589774  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:31.588510  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:31.589942  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:31.589579  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:31.591954  9638 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 16
I20260808 01:52:31.593734 10369 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260808 01:52:31.596547 16470 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20260808 01:52:31.597519 16596 raft_consensus.cc:3055] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:31.598032 16596 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:31.596741  9556 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:31.602352  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:31.603411  9550 raft_consensus.cc:2410] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 18 because replica has last-logged OpId of term: 16 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 6.
I20260808 01:52:31.604732  9477 raft_consensus.cc:1217] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Deduplicated request from leader. Original: 10.10->[11.11-13.12]   Dedup: 13.12->[]
I20260808 01:52:31.606897  9477 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 13. Current term is 14. Ops: [11.11-13.12]
I20260808 01:52:31.607131  9480 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:31.607438  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2966) took 2823 ms. Trace:
I20260808 01:52:31.607848  9632 rpcz_store.cc:276] 0808 01:52:28.759566 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:28.759699 (+   133us) service_pool.cc:224] Handling call
0808 01:52:28.802532 (+ 42833us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:29.213031 (+410499us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b54003596a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490b1a0 00007f5d449955a0 00007f5d44996611 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c
0808 01:52:30.358570 (+1145539us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b58006cbcd0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d448ef48a 00007f5d44971827 00007f5d4498a3c7 00007f5d4498bbbe 00007f5d44995f29 00007f5d449967f6 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34
0808 01:52:31.542123 (+1183553us) spinlock_profiling.cc:243] Waited 3.8 ms on lock 0x7b54003596a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44908dfd 00007f5d448ef584 00007f5d44971827 00007f5d4498a3c7 00007f5d4498bbbe 00007f5d44995f29 00007f5d449967f6 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229
0808 01:52:31.581007 (+ 38884us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:31.583023 (+  2016us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2606313344}
I20260808 01:52:31.605963 16692 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:31.611311  9637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "244e080726394210b5467bf688813cf0"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:31.612010  9637 raft_consensus.cc:606] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:31.611573 16692 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:31.612435  9637 raft_consensus.cc:614] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:31.612969  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.610363  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.616308  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:31.617203  9559 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:31.617712  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.623565  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.624466  9480 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:31.625751  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.626655  9476 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:52:31.629315  9476 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260808 01:52:31.620970  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.630575  9475 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
W20260808 01:52:31.630620 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 189590us
I20260808 01:52:31.633005  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:31.633325  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:31.633781  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:31.634020  9559 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:31.604624 16590 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:31.604064 16645 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:31.662843 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:31.614390 16677 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.663327  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:31.666694  9472 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 1139) took 2209 ms (client timeout 2237 ms). Trace:
W20260808 01:52:31.669723  9472 rpcz_store.cc:269] 0808 01:52:29.457576 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:29.489871 (+ 32295us) service_pool.cc:224] Handling call
0808 01:52:31.666376 (+2176505us) spinlock_profiling.cc:243] Waited 917 ms on lock 0x7b54002bc0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:31.666580 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2108689664}
I20260808 01:52:31.670872  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.672372  9478 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:31.667752  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.665259  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3001) took 1013 ms. Trace:
I20260808 01:52:31.675860  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:31.676391  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 244e080726394210b5467bf688813cf0 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:31.676214  9636 rpcz_store.cc:276] 0808 01:52:30.652320 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:31.607675 (+955355us) service_pool.cc:224] Handling call
0808 01:52:31.664473 (+ 56798us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:31.664761 (+   288us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:31.664799 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:31.664864 (+    65us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:31.664927 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:31.665015 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:31.665173 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:31.680918 10362 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:52:31.622547 16692 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.681249  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:31.686013 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:31.686290  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:31.695186  9638 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:52:31.685220 16645 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:31.691121 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:31.691582  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:31.706422 10367 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:31.692173 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:31.707674 10363 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:31.691938 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:31.703833  9478 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:31.695431  9523 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:31.705188 10360 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:31.695017  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:31.711386  9557 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:52:31.712903  9636 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 12. (term mismatch)
I20260808 01:52:31.713320  9636 pending_rounds.cc:85] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 11
I20260808 01:52:31.713568  9636 pending_rounds.cc:107] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 17.12
I20260808 01:52:31.713831  9636 raft_consensus.cc:2889] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:31.677099 16590 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:31.693679  9473 raft_consensus.cc:2410] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 19 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
W20260808 01:52:31.708021  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:31.716734 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:31.719971 10367 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:31.719987  9523 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:31.693224 16678 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:31.723310  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.745321  9477 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 22: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:31.724309  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:31.747298  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.747820  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.748685  9474 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 20: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:31.725091 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:31.731390 16249 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:31.732545 16565 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:31.732661 16596 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:31.746044  9556 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:31.734927 16631 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:31.710212 16728 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:31.736020 16675 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:31.765856  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.711458 16719 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:31.743393  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.766911  9477 raft_consensus.cc:2393] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 21: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:31.767148  9479 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 14.
I20260808 01:52:31.743562  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:31.768210  9475 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:31.768605  9638 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3759209338e14437aff5f92a2fff36da"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:31.744529  9472 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:31.769310  9638 raft_consensus.cc:606] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:31.751443  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:31.755311 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:31.757694  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.770643 10366 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:31.758481 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:31.760543  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:31.771930  9523 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:31.761806  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:31.761436 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:31.772075  9480 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:31.772535 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:31.699124 16570 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:31.772735  9636 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:31.772871  9474 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:31.774495 10362 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:31.775204 16736 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:31.778704 16736 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:31.779158 16736 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:31.780884 10369 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 16: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:31.752759  9478 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:31.775611 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:31.782451  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.784587  9474 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:31.784451 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:31.768951 16677 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:31.763763 16631 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:31.787715 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:31.780297 16728 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.789479  9476 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260808 01:52:31.786922 10362 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:31.766981 16229 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.026s
I20260808 01:52:31.792848  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2998) took 1153 ms. Trace:
I20260808 01:52:31.794162  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2971) took 3031 ms. Trace:
I20260808 01:52:31.795413 16692 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:31.759298 16290 consensus_queue.cc:1059] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.025s
I20260808 01:52:31.796136 16535 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260808 01:52:31.773599 16625 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:31.774950 10332 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:31.799949 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:31.798594 16632 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.337s	user 0.012s	sys 0.030s
W20260808 01:52:31.800022  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:31.800254  9478 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:31.774355 16719 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:31.801590  9523 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:31.777942  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.777487 16570 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:31.768486 16580 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.031s
I20260808 01:52:31.797662 16596 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:31.797473  9601 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:31.784518 16645 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:31.814939  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.797394 16677 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:31.815423 10365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 1075) took 3280 ms. Trace:
I20260808 01:52:31.844198  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:31.844602 10365 rpcz_store.cc:276] 0808 01:52:28.535400 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:28.535589 (+   189us) service_pool.cc:224] Handling call
0808 01:52:28.535972 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:29.250297 (+714325us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:31.815126 (+2564829us) spinlock_profiling.cc:243] Waited 713 ms on lock 0x7b54005550d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:31.815349 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1640747776}
I20260808 01:52:31.806937  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:31.846166  9638 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:31.847442  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:31.849086  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:31.818841 10363 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 12.
I20260808 01:52:31.819634  9475 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:31.852327  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.853420  9475 raft_consensus.cc:2393] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 12: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:52:31.816202  9552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2704) took 1875 ms (client timeout 1793 ms). Trace:
I20260808 01:52:31.820338  9480 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:31.809952  9631 rpcz_store.cc:276] 0808 01:52:30.639079 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:31.191108 (+552029us) service_pool.cc:224] Handling call
0808 01:52:31.792612 (+601504us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x7b5400307fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:31.792778 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":774771072}
I20260808 01:52:31.808404  9479 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: []
W20260808 01:52:31.854154  9552 rpcz_store.cc:269] 0808 01:52:29.940927 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:29.976021 (+ 35094us) service_pool.cc:224] Handling call
0808 01:52:31.815992 (+1839971us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b54003269d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:31.816126 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":921660672}
I20260808 01:52:31.854849  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.821442  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:31.796762 16249 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:31.857863  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 60557.2us
I20260808 01:52:31.810088  9629 rpcz_store.cc:276] 0808 01:52:28.762291 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:28.762405 (+   114us) service_pool.cc:224] Handling call
0808 01:52:28.805087 (+ 42682us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:30.066491 (+1261404us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:31.793767 (+1727276us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54001b3ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:31.793997 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2899502976}
W20260808 01:52:31.858798 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:31.859323  9601 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:31.800930  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:31.863389  9636 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:31.867812  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:31.830502 16646 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:31.836850  9477 raft_consensus.cc:2393] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 18: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:31.819372 16692 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:31.807219 16631 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:31.829046 16650 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:31.831857 16614 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:31.870393  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:31.835124 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 60279.6us
I20260808 01:52:31.897156 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:31.882623  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:31.883926  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:31.901170  9601 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:31.885628  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:31.835765 16717 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:31.889930  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:31.828641 16671 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
W20260808 01:52:31.845491 16632 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.372s	user 0.012s	sys 0.031s
I20260808 01:52:31.874023  9474 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:52:31.903415  9636 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 18: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:31.867934 16677 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:31.912163  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:31.913084  9629 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:31.907079  9631 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4f04078e3beb42eda250da45f7485537"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:31.914237  9631 raft_consensus.cc:606] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:31.900506 10332 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:31.881559 16580 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:31.909622  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:31.920238  9477 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 20
I20260808 01:52:31.888049 16594 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:31.892702 16645 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:31.975111  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:31.989372  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 88303.2us
I20260808 01:52:31.887542 16692 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:31.952409  9631 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260808 01:52:31.886998 16242 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:52:31.886340 16535 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:32.004256  9477 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:31.909194 16625 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260808 01:52:31.908169 16415 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:31.923566 16580 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:31.922842 16728 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:32.003783 16530 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:31.992844 16594 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:32.034356 10363 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:57156
I20260808 01:52:32.041867 10363 raft_consensus.cc:606] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:32.042410 10363 raft_consensus.cc:614] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:31.888795 16646 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:31.911150 16614 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:31.906306 16650 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:32.010138 16194 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.036s
I20260808 01:52:32.014341 16645 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:32.080412  9476 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260808 01:52:32.048434  9477 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 15. Preceding OpId from leader: term: 17 index: 16. (index mismatch)
I20260808 01:52:32.072065  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:32.064572  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 8cbead77343f44dd82199bebcfb2def7 (from 7201605bb2f54c949a4ebe6e9b3b4b2a to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:32.027971 16242 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:32.088517 16595 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.574s	user 0.012s	sys 0.000s
W20260808 01:52:32.078045 16623 consensus_meta.cc:287] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.522s	user 0.007s	sys 0.007s
I20260808 01:52:32.117411 16053 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:32.118261  9475 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:52:32.121083  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2990) took 1492 ms. Trace:
I20260808 01:52:32.135473  9633 rpcz_store.cc:276] 0808 01:52:30.628477 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:30.630807 (+  2330us) service_pool.cc:224] Handling call
0808 01:52:30.649740 (+ 18933us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:31.402873 (+753133us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:32.120771 (+717898us) spinlock_profiling.cc:243] Waited 721 ms on lock 0x7b540027e054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:32.121007 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1658919296}
W20260808 01:52:32.136090 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 440779us
I20260808 01:52:32.124389 16645 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:32.139201  9480 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
W20260808 01:52:32.137946  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 347603us
I20260808 01:52:32.048846 16728 raft_consensus.cc:697] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:32.141605  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:32.044903 16594 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:32.162729  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:32.162992  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 306181us
I20260808 01:52:32.044983 16661 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:32.171700 10334 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:32.165992  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:32.167662  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2846) took 2944 ms. Trace:
I20260808 01:52:32.154058  9629 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:32.173408  9635 rpcz_store.cc:276] 0808 01:52:29.223604 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:29.241098 (+ 17494us) service_pool.cc:224] Handling call
0808 01:52:29.269859 (+ 28761us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:30.403473 (+1133614us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:32.167238 (+1763765us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540027ea54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:32.167572 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2509062016}
I20260808 01:52:32.045961 16530 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:32.171816  9633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:32.177062  9633 raft_consensus.cc:606] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:32.177561  9633 raft_consensus.cc:614] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:32.162169 16595 raft_consensus.cc:993] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 6e7856bed1be43acaa821290d506ee97 to start an election
I20260808 01:52:32.158664 16660 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:32.184620  9633 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260808 01:52:32.196069 16646 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:32.126187 16736 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:32.201521  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:32.220361  9478 raft_consensus.cc:2410] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 20 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 8.
W20260808 01:52:32.202165  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 9949664c54584858ae8fc74c9dc07a4a (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:32.089494 16580 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:32.222617  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 01:52:32.146970  9523 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:32.209693  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:32.232761  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 85903.4us
I20260808 01:52:32.197088 16468 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:32.217783  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:32.184856 16586 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:32.181000 16530 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:32.157780 16623 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:32.240828 16586 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:32.241838 16614 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:32.242274  9476 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:32.214802 16242 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:32.241954 16623 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:32.189325  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:32.239224  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:32.235958  9523 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:32.226716 16194 raft_consensus.cc:1081] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Signalling peer 6e7856bed1be43acaa821290d506ee97 to start an election
I20260808 01:52:32.243633 16586 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:32.251542  9478 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
W20260808 01:52:32.289232  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:32.261760 10369 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260808 01:52:32.262566 10363 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 22. Current term is 23. Ops: []
W20260808 01:52:32.290910  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 101651us
I20260808 01:52:32.266188 10362 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:32.292340  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:32.293638 16623 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:32.294180 16623 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:32.294595 16623 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:32.239470 16628 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:32.253909 16661 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:32.299803 16661 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:32.300077 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:32.300841 10364 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:32.299285 16628 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:32.302102 16628 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:32.256356 16719 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:32.305903  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3003) took 1590 ms. Trace:
I20260808 01:52:32.306362  9632 rpcz_store.cc:276] 0808 01:52:30.715566 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:31.610106 (+894540us) service_pool.cc:224] Handling call
0808 01:52:31.629011 (+ 18905us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:31.663206 (+ 34195us) raft_consensus.cc:1534] Early marking committed up to index 10
0808 01:52:31.663261 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:31.686101 (+ 22840us) log.cc:844] Serialized 26 byte log entry
0808 01:52:32.035510 (+349409us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:32.036080 (+   570us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:32.036203 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:32.305629 (+269426us) raft_consensus.cc:1685] finished
0808 01:52:32.305655 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:32.305834 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":595607,"raft.queue_time_us":5912,"raft.run_cpu_time_us":324,"raft.run_wall_time_us":600153,"thread_start_us":348908,"threads_started":1,"wal-append.queue_time_us":525473}
I20260808 01:52:32.289937  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:32.311154  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:32.311434  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:32.311931  9552 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:32.312130 16602 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:32.312345  9556 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 18: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:32.312773 16602 raft_consensus.cc:697] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:32.313294  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:32.314669  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:32.314401 16602 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:32.315500  9557 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 20: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:52:32.316064  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 80176.7us
I20260808 01:52:32.316473  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:32.317138  9523 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:32.318157 16749 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:52:32.320117 10332 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:32.322572  9552 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:32.325032  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:32.263305 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:32.327708  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:32.267042  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 2992) took 1637 ms. Trace:
I20260808 01:52:32.329291  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:32.331821  9630 rpcz_store.cc:276] 0808 01:52:30.629406 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:30.632052 (+  2646us) service_pool.cc:224] Handling call
0808 01:52:30.632468 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:30.828367 (+195899us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:32.250629 (+1422262us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b540003bbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:32.266947 (+ 16318us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":448039296}
I20260808 01:52:32.338114  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:32.490406  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:32.491856  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:32.338551  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:32.364377  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:32.570802  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 873) took 1160 ms. Trace:
I20260808 01:52:32.579046  9629 rpcz_store.cc:276] 0808 01:52:31.409703 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:32.174598 (+764895us) service_pool.cc:224] Handling call
0808 01:52:32.174984 (+   386us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:32.175821 (+   837us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:52:32.224047 (+ 48226us) log.cc:844] Serialized 14 byte log entry
0808 01:52:32.568374 (+344327us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:32.568654 (+   280us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:32.568987 (+   333us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:32.569288 (+   301us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:32.570089 (+   801us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":43869,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"raft.queue_time_us":670,"raft.run_cpu_time_us":412,"raft.run_wall_time_us":412,"thread_start_us":341683,"threads_started":1}
I20260808 01:52:32.359496  9554 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260808 01:52:32.357473  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:32.584631  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:32.585428  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:32.586071  9554 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 18: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:32.588050  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:32.360922  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:32.589003  9554 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260808 01:52:32.338425  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:32.590792  9560 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:32.591579  9554 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260808 01:52:32.352584  9599 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:32.592823 16645 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:32.594125 16645 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:32.339704  9633 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260808 01:52:32.596307  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3004) took 1645 ms. Trace:
I20260808 01:52:32.357738 16447 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:32.594075  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:32.600955 16447 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:32.601881  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:32.340657 10332 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:32.382097  9559 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:32.354962 16194 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260808 01:52:32.330785 16565 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.012s	sys 0.024s
W20260808 01:52:32.611254 16565 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.816s	user 0.013s	sys 0.025s
W20260808 01:52:32.380967  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 385604us
I20260808 01:52:32.349262  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:32.617144 16242 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:32.617571  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:32.618247 16242 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:32.359870  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:32.359020 16586 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:32.641183  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:32.656807 16586 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:32.657609 16586 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:32.658053  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:32.354898 16600 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:32.660591 16600 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:32.429332 16631 consensus_meta.cc:287] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.523s	user 0.013s	sys 0.000s
I20260808 01:52:32.660821  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:32.662601  9551 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2709) took 2017 ms (client timeout 1500 ms). Trace:
I20260808 01:52:32.662739 16631 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:32.663264  9551 rpcz_store.cc:269] 0808 01:52:30.645171 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:30.645382 (+   211us) service_pool.cc:224] Handling call
0808 01:52:32.662091 (+2016709us) spinlock_profiling.cc:243] Waited 725 ms on lock 0x7b54000ff854. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:32.662364 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1666592640}
I20260808 01:52:32.664377  9557 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:32.343952 16650 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:32.432982 16661 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:32.634177  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:32.667852  9635 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:32.358207 16668 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260808 01:52:32.387802 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:32.654327  9473 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 3214) took 2609 ms (client timeout 2711 ms). Trace:
I20260808 01:52:32.670964 16668 consensus_queue.cc:1243] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 21.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260808 01:52:32.671227  9555 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260808 01:52:32.342739 10365 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260808 01:52:32.343698 16290 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:52:32.350039 16470 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:32.335342 16573 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.012s
I20260808 01:52:32.235841 16590 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:32.671627  9473 rpcz_store.cc:269] 0808 01:52:30.047644 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:30.066091 (+ 18447us) service_pool.cc:224] Handling call
0808 01:52:32.653649 (+2587558us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54002bc5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:32.654019 (+   370us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3740084096}
I20260808 01:52:32.332657 16614 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:32.360532 16628 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:32.357060 16677 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:32.571158  9477 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:32.678643 16470 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:32.687872  9634 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 2968) took 3927 ms (client timeout 2435 ms). Trace:
I20260808 01:52:32.689061 16470 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:32.211650 16660 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:32.349110 16623 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:32.443578 16717 consensus_meta.cc:287] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.534s	user 0.014s	sys 0.000s
I20260808 01:52:32.321412 16720 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:32.270592 16728 consensus_queue.cc:237] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:32.351999 16736 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:32.744572 16568 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:32.229171 16575 raft_consensus.cc:3055] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Stepping down as leader of term 17
W20260808 01:52:32.745901 16570 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.901s	user 0.003s	sys 0.027s
I20260808 01:52:32.747020 16573 consensus_queue.cc:1243] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 21.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260808 01:52:32.544886 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:32.566507 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:32.751488 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:32.580696  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:32.344012  9552 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
W20260808 01:52:32.578313 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 406522us
I20260808 01:52:32.756070 16575 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:32.578236  9630 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:32.759262 16579 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:32.747093 16570 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:32.760267 16579 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:32.605917 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 265114us
W20260808 01:52:32.595649  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 243010us
I20260808 01:52:32.762318  9599 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:32.763717 16596 consensus_meta.cc:287] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.920s	user 0.009s	sys 0.019s
I20260808 01:52:32.599334  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:32.631330  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:32.641148 16763 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260808 01:52:32.766593  9478 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 21. Current term is 22. Ops: []
I20260808 01:52:32.745784 16568 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:32.767345  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:32.768368  9474 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 16: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:32.715556 16717 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:32.769528 16568 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:32.769786  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 358463us
I20260808 01:52:32.770473  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:32.757968 16575 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:32.771875  9477 raft_consensus.cc:2410] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 21 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260808 01:52:32.772130 16575 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:32.772680 16753 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:32.643042  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:32.643481 16447 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:32.655342  9479 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
I20260808 01:52:32.356626 16595 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:52:32.776516  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 159526us
I20260808 01:52:32.777129 16595 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:32.778620 16242 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:32.779124 16757 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:32.779639 16757 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:32.779740 16595 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:32.658068  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:32.659554  9481 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 3203) took 2689 ms (client timeout 1520 ms). Trace:
I20260808 01:52:32.669847  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:32.781409  9481 rpcz_store.cc:269] 0808 01:52:29.970067 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:29.973110 (+  3043us) service_pool.cc:224] Handling call
0808 01:52:32.658816 (+2685706us) spinlock_profiling.cc:243] Waited 1.9 s on lock 0x7b54002bb1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:32.659270 (+   454us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4381189888}
I20260808 01:52:32.343492 16719 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:32.782843  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:32.783231 16719 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:32.783794 16719 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:32.785598 16595 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election won for term 18
W20260808 01:52:32.786242  9523 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.196:33455 timed out after 1.674s (SENT) [suppressed 7 similar messages]
I20260808 01:52:32.696455  9473 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:52:32.703012  9524 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:32.704312  9638 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:52:32.705078  9558 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:32.707419 16600 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:32.709551 16677 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:32.678617 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:32.710297  9632 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:32.710919  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 972) took 1367 ms. Trace:
I20260808 01:52:32.715247  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:32.669914 16661 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:32.706318 16628 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:32.672123 16668 raft_consensus.cc:3055] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:52:32.715543 16720 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:32.680054 16614 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:32.704398 16623 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:32.741104 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:32.764523 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:32.764775 16596 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:32.780134 16757 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:32.781754  9635 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
W20260808 01:52:32.688422  9634 rpcz_store.cc:269] 0808 01:52:28.760467 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:28.760584 (+   117us) service_pool.cc:224] Handling call
0808 01:52:32.687294 (+3926710us) spinlock_profiling.cc:243] Waited 2.27 s on lock 0x7b540003c5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:32.687584 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5217264384}
I20260808 01:52:32.776520  9479 pending_rounds.cc:85] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 13
I20260808 01:52:32.787104 16671 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:32.787235 10334 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:32.597057  9637 rpcz_store.cc:276] 0808 01:52:30.950550 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:31.636632 (+686082us) service_pool.cc:224] Handling call
0808 01:52:31.675413 (+ 38781us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:31.675655 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:52:31.687817 (+ 12162us) log.cc:844] Serialized 14 byte log entry
0808 01:52:32.595104 (+907287us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:32.595302 (+   198us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:52:32.595447 (+   145us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:32.595676 (+   229us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:32.595969 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":106724,"prepare.queue_time_us":13688,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":83783,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":72376,"thread_start_us":884774,"threads_started":2,"wal-append.queue_time_us":571029}
I20260808 01:52:32.788267 16576 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:32.789984 16595 raft_consensus.cc:697] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:32.769433 10369 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 11. Current term is 12. Ops: []
I20260808 01:52:32.784471  9601 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:32.791291  9599 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:32.776674  9444 leader_election.cc:400] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20.
I20260808 01:52:32.799552  9555 rpcz_store.cc:276] 0808 01:52:31.343413 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:31.343560 (+   147us) service_pool.cc:224] Handling call
0808 01:52:32.670432 (+1326872us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:32.671841 (+  1409us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:32.690804 (+ 18963us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54000ff850. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:32.710787 (+ 19983us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3050275200}
I20260808 01:52:32.807567  9479 pending_rounds.cc:107] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 17.14
I20260808 01:52:32.810835  9629 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 16: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:32.816172  9479 raft_consensus.cc:2889] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:32.816246  9444 leader_election.cc:403] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:52:32.779423 16573 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.026s
I20260808 01:52:32.824591  9560 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:52:32.801796 16668 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:32.809172 16720 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:32.810585 16614 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
W20260808 01:52:32.837636 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50717.7us
I20260808 01:52:32.823082 16596 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:32.835757 10365 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:32.841107  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:32.838464  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:32.846282 10366 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 15: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:32.831897 16631 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:32.844447  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:32.841063 16053 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.639s	user 0.003s	sys 0.024s
I20260808 01:52:32.851459 16736 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.011s
I20260808 01:52:32.839924 16242 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:32.859063 16544 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:32.859853 16544 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:32.860404 16544 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:52:32.847922  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 63621.2us
I20260808 01:52:32.861629 16447 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:32.872639  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:32.876600 16570 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:32.877197 16692 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.852s	user 0.013s	sys 0.014s
I20260808 01:52:32.879875 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:32.880795 10366 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEARNER]: Advancing to term 19
I20260808 01:52:32.887920  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:32.884601 10361 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:32.894938 16595 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:32.897089 16579 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:32.890300 16590 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:32.902318  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:32.903055  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 249671us
I20260808 01:52:32.904606  9634 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260808 01:52:32.883482 16651 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
I20260808 01:52:32.907534  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1148) took 1071 ms. Trace:
I20260808 01:52:32.907994  9479 rpcz_store.cc:276] 0808 01:52:31.836071 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:31.855163 (+ 19092us) service_pool.cc:224] Handling call
0808 01:52:32.654085 (+798922us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:32.852035 (+197950us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:32.907268 (+ 55233us) spinlock_profiling.cc:243] Waited 760 ms on lock 0x7b54002bc5d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:32.907431 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1746911488}
I20260808 01:52:32.909839  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:52:32.922565  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:52:32.924719 16668 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:32.915319  9636 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260808 01:52:32.916301 16596 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 17.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:32.963191 16761 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.259s	user 0.000s	sys 0.003s
W20260808 01:52:32.917840  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 126765us
I20260808 01:52:32.914322 16660 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:32.928506  9475 raft_consensus.cc:3055] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:32.915059 16575 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20.
I20260808 01:52:32.908089 16651 raft_consensus.cc:3055] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:32.906535  9634 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 6. Preceding OpId from leader: term: 17 index: 8. (index mismatch)
I20260808 01:52:32.933458 16634 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:32.935798 16661 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election won for term 20
W20260808 01:52:32.942559 16769 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.237s	user 0.000s	sys 0.004s
I20260808 01:52:32.954162 16645 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:32.954741  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 178338us
I20260808 01:52:32.959546  9637 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:32.877897 16692 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:32.974294 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:32.973032  9475 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:32.977320 10366 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEARNER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:32.978947  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:32.975880 16651 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:32.987160  9635 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:32.996277  9630 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:32.869122 16671 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:33.002988  9629 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:33.012977  9632 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:32.989512 16661 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:33.025380  9635 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:33.022254  9634 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260808 01:52:32.999559 16719 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:33.044876  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 874) took 1323 ms. Trace:
I20260808 01:52:33.045423  9638 rpcz_store.cc:276] 0808 01:52:31.721981 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:31.885342 (+163361us) service_pool.cc:224] Handling call
0808 01:52:33.044580 (+1159238us) spinlock_profiling.cc:243] Waited 793 ms on lock 0x7b54000a9bd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:33.044766 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1824565632}
I20260808 01:52:33.020514 16586 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:33.051084 16651 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:33.051693 16751 consensus_queue.cc:1059] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.017s
I20260808 01:52:33.052819 16651 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:33.053191 16586 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:33.055187 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:33.056982 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:33.057348 16771 log.cc:927] Time spent T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
I20260808 01:52:33.058290  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:33.059607 16781 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260808 01:52:33.060173 16753 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:33.062896  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 864) took 1717 ms. Trace:
I20260808 01:52:33.063627  9636 rpcz_store.cc:276] 0808 01:52:31.352656 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:31.910104 (+557448us) service_pool.cc:224] Handling call
0808 01:52:31.982934 (+ 72830us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:32.972500 (+989566us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:33.062459 (+ 89959us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b540003c0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:33.062800 (+   341us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2022578560}
I20260808 01:52:33.069203 16573 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:33.070827 16573 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:33.071974 16668 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:33.069132 16614 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:33.072952 16576 consensus_queue.cc:1059] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:33.049027  9475 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:33.071318  9599 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:33.074426 16576 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:33.074618 16614 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:33.071296 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 206303us
I20260808 01:52:33.075690 16728 raft_consensus.cc:3055] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:52:33.076164 16728 raft_consensus.cc:740] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:33.076557 16762 consensus_queue.cc:1059] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:52:33.078125 16762 raft_consensus.cc:3055] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:52:33.077538 16728 consensus_queue.cc:260] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:33.078547 16762 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:33.078908 16754 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260808 01:52:33.079982 16768 log.cc:927] Time spent T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.375s	user 0.002s	sys 0.004s
I20260808 01:52:33.074743  9475 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:33.075083 16736 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.006s
I20260808 01:52:33.080061 16762 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:33.082424 16762 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:52:33.083582 16594 consensus_meta.cc:287] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.903s	user 0.016s	sys 0.010s
I20260808 01:52:33.083300 16661 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:33.089391 16669 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:33.090261 16669 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:33.092724 16669 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:33.078891 16753 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:33.097178 16671 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:33.052902 16751 raft_consensus.cc:3055] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:33.075256 16576 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:33.103770  9629 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:52:33.106235 10366 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:33.110852 10365 raft_consensus.cc:2368] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22.
I20260808 01:52:33.084146 16594 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:33.110571 16645 raft_consensus.cc:3055] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:33.119782 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:33.118127  9523 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:33.101536 16751 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:33.123386  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 131911us
I20260808 01:52:33.127374 10363 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:33.127717  9638 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260808 01:52:33.136427  9635 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260808 01:52:33.113161 16586 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.010s
W20260808 01:52:33.139822  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 68649.5us
W20260808 01:52:33.142367 16530 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.898s	user 0.017s	sys 0.007s
I20260808 01:52:33.142566 10360 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260808 01:52:33.143474 16576 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:33.143474 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:33.144131 16592 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:33.144804 16592 raft_consensus.cc:697] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:33.145200 16753 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:33.120112 16645 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:33.146958  9600 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:33.098098 16742 consensus_queue.cc:1059] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.037s
I20260808 01:52:33.147747 16645 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:33.148521 16742 raft_consensus.cc:3055] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Stepping down as leader of term 11
I20260808 01:52:33.149034 16742 raft_consensus.cc:740] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:33.149374 16645 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:33.149858 16470 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:52:33.150934 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 74555.3us
I20260808 01:52:33.151909 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:33.152047 16692 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:33.153733 16576 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:33.155260 16717 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260808 01:52:33.155462 16544 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:33.157522 16717 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:52:33.157961 16544 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:33.158607 16762 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:33.171327 16762 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:33.172983 16762 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:33.174270 16762 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:33.159013  9524 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:33.143018 16530 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:33.159677 16579 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260808 01:52:33.207178  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:33.208494  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:33.175004 16675 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:33.175951 16728 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 11 FOLLOWER]: Advancing to term 12
I20260808 01:52:33.179199 16692 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:33.145956 16592 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:33.182503  9599 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:33.182902  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 62154.3us
I20260808 01:52:33.185187 10367 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 22: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:33.183195 16764 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.008s
I20260808 01:52:33.193866  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:33.195576 16751 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:33.197049  9632 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 21. Current term is 22. Ops: []
I20260808 01:52:33.209183 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:33.217860 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:33.215277 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:33.217926  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:33.224174  9630 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:33.218145  9523 leader_election.cc:400] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22.
I20260808 01:52:33.219172  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:33.210767 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:33.224925  9523 leader_election.cc:403] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260808 01:52:33.218266  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:33.225925  9636 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
W20260808 01:52:33.227515 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 91828.3us
I20260808 01:52:33.230450 16576 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:33.231221 16576 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:33.231886  9474 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3759209338e14437aff5f92a2fff36da"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:50140
I20260808 01:52:33.232319 16229 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:33.234494 10363 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:33.238845  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:33.239344  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:33.243367 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:33.239694  9472 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:52:33.251503  9633 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:52:33.252668 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:33.260103  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 113322us
I20260808 01:52:33.262588  9475 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 11. (term mismatch)
I20260808 01:52:33.263123  9475 pending_rounds.cc:85] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 10
I20260808 01:52:33.263382  9475 pending_rounds.cc:107] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 20.11
I20260808 01:52:33.263643  9475 raft_consensus.cc:2889] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:33.162416 16669 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.081s
I20260808 01:52:33.268159  9629 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:33.232522 16576 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:33.272831  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 58470.9us
I20260808 01:52:33.275293 10365 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:33.276302 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:33.150343 16742 consensus_queue.cc:260] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:33.234715 16580 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:33.283972  9600 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:33.290491  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 108161us
I20260808 01:52:33.291543  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:33.274602 16753 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:33.292390  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:33.279755 16570 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:52:33.298992 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:33.311151  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:33.319077  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:33.320688 10363 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:33.285715 16651 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:33.324450  9523 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:33.326301  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 167441us
I20260808 01:52:33.329831  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:33.331370 16290 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 21 current term: 21 is higher.
W20260808 01:52:33.332026  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 88755.4us
I20260808 01:52:33.332547 16713 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:33.333258 16675 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:52:33.333839 16675 raft_consensus.cc:697] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:33.334955  9557 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:52:33.335059 16675 consensus_queue.cc:237] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:33.336828 16757 consensus_meta.cc:287] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.500s	user 0.013s	sys 0.001s
I20260808 01:52:33.294929 16692 consensus_queue.cc:1059] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.043s
I20260808 01:52:33.337320 16757 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:33.338294 16229 raft_consensus.cc:3055] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:33.338771 16229 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:33.338912  9635 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:33.339892 16229 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:33.341387 16229 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
W20260808 01:52:33.347218  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 68357us
I20260808 01:52:33.348198  9600 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:33.332175  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 464441us
I20260808 01:52:33.311621 16751 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:33.365008  9629 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:52:33.368453  9475 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:33.373315  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:33.374114  9629 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:33.375507 16651 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:33.363790 16719 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:33.363025 16594 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:33.379838 16785 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:33.380757 16785 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:33.381204  9632 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 17. Current term is 18. Ops: []
W20260808 01:52:33.383132  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 48444.3us
I20260808 01:52:33.387907  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:33.389405  9599 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:33.352152 16660 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260808 01:52:33.384253  9445 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:33.405755  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:33.368053 16742 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:52:33.414296  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:33.414522  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:33.415935  9629 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
W20260808 01:52:33.410828  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 59798.8us
I20260808 01:52:33.427268  9445 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:33.411633 16470 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260808 01:52:33.472615 16720 consensus_meta.cc:287] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.600s	user 0.014s	sys 0.000s
I20260808 01:52:33.473254 16720 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:33.474893 10362 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:33.425945 16757 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:33.483850  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 118269us
W20260808 01:52:33.492177  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:33.496582  9521 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:33.497889 16669 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:33.498327 16669 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:33.498710 16669 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:33.506851 16530 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:33.508872  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:33.510880 16764 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260808 01:52:33.511238  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 126140us
I20260808 01:52:33.511453 16764 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:33.511871 16764 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:33.430552 16594 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:33.514689 16594 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:33.515177 16594 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:33.520802 16720 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:33.523602 16570 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22.
I20260808 01:52:33.525435 10366 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:33.525538  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:33.541450 16669 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:33.527318  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 106367us
I20260808 01:52:33.527819 10368 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:33.528275  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:33.537942 16650 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:33.428736 16194 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260808 01:52:33.558633  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:33.559468  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:33.426901 16651 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:33.571934  9475 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:52:33.575557  9638 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:52:33.429039 16229 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:52:33.612696  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:33.617462  9632 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:33.622124 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:33.564379 16650 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:33.625485  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:33.626579  9472 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:33.566762 16194 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:33.625523  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:33.564430 16736 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:33.620738 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:33.640136  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:33.643280 16736 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:33.646083 16646 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:33.577143 16692 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.071s
W20260808 01:52:33.660045  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 347233us
I20260808 01:52:33.643144  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:33.639324 16650 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:33.649211 16194 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:33.677832  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:33.679131  9634 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:33.653538 16229 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:33.692778  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:33.696543 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:33.704620  9445 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:33.721920  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:33.703890 16596 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election won for term 22
W20260808 01:52:33.733513 10368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 861) took 2197 ms (client timeout 1590 ms). Trace:
I20260808 01:52:33.738808 10365 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
W20260808 01:52:33.738507  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 104744us
I20260808 01:52:33.707607 16692 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:33.751142  9634 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
W20260808 01:52:33.751353  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 225919us
I20260808 01:52:33.718375 16675 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.034s
I20260808 01:52:33.705914 16229 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:33.738885 10368 rpcz_store.cc:269] 0808 01:52:31.523167 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:31.528282 (+  5115us) service_pool.cc:224] Handling call
0808 01:52:33.707249 (+2178967us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b5400785ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:33.720176 (+ 12927us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4254872448}
I20260808 01:52:33.736421 16753 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:33.758989  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:33.761720 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:52:33.762435  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 72926.5us
W20260808 01:52:33.763329  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 70726.4us
W20260808 01:52:33.766350  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:33.732873 16645 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader election won for term 12
I20260808 01:52:33.791659 16645 raft_consensus.cc:697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:33.792793 16645 consensus_queue.cc:237] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:33.770040  9557 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:33.771620 16573 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:33.772588 16249 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:33.794806 16249 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:33.795012  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:33.795719  9634 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:33.795814 16249 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:33.796861  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:33.797314 16249 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:33.785662  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:33.799067  9476 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:33.750247 16596 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:33.762967 16530 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:33.803612  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 224296us
I20260808 01:52:33.754015 16692 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:33.790778 16579 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:33.790931 16669 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:33.800818  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:33.770007  9632 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 22.
I20260808 01:52:33.786841  9521 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:33.755916 16470 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:33.809110  9472 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 22.
W20260808 01:52:33.794495 16728 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.579s	user 0.010s	sys 0.000s
I20260808 01:52:33.766568 16757 consensus_queue.cc:1059] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.042s
W20260808 01:52:33.803357 16580 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.502s	user 0.012s	sys 0.000s
I20260808 01:52:33.817829 16229 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:33.818753 16770 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.174s	user 0.006s	sys 0.000s
I20260808 01:52:33.820258  9481 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:33.800551 16573 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:33.823715  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2871) took 1235 ms. Trace:
W20260808 01:52:33.853075  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 83186.3us
I20260808 01:52:33.809832 16579 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:33.849483 16596 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:33.858475  9478 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:33.851982  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:33.847074  9476 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:52:33.853147  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:33.809608 16692 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:33.877923 16470 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:33.887857 16470 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:33.853698  9637 rpcz_store.cc:276] 0808 01:52:32.588674 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:32.846199 (+257525us) service_pool.cc:224] Handling call
0808 01:52:33.823487 (+977288us) spinlock_profiling.cc:243] Waited 59.9 ms on lock 0x7b5400420ed4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:33.823639 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":137838848}
I20260808 01:52:33.889504  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:33.825515  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:33.854784  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 68759.8us
I20260808 01:52:33.892364  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:33.893402 16580 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:33.811496 16801 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.056s
I20260808 01:52:33.808770 16530 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:33.871177  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 872) took 2462 ms. Trace:
I20260808 01:52:33.903304  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:33.861325 16573 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:33.864291  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:33.866863  9632 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260808 01:52:33.873641  9474 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:52:33.907859  9474 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:33.868974  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:33.908062  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:33.895568 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:33.909746 10365 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:52:33.872330 16780 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.758s	user 0.000s	sys 0.018s
W20260808 01:52:33.912910  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 87494.8us
I20260808 01:52:33.912700 16809 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:33.920667  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:33.896063  9444 connection.cc:256] RPC call timeout handler was delayed by -1.6375211000267882s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1759s for a response
I20260808 01:52:33.871219 16669 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:33.930567  9637 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:52:33.933463  9557 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:33.903753  9631 rpcz_store.cc:276] 0808 01:52:31.408653 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:32.007342 (+598689us) service_pool.cc:224] Handling call
0808 01:52:32.007793 (+   451us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:32.383029 (+375236us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:32.383077 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:32.501791 (+118714us) log.cc:844] Serialized 26 byte log entry
0808 01:52:32.567255 (+ 65464us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:52:32.699914 (+132659us) log.cc:844] Serialized 14 byte log entry
0808 01:52:32.722879 (+ 22965us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:32.725655 (+  2776us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:33.092349 (+366694us) spinlock_profiling.cc:243] Waited 2.53 ms on lock 0x7b18001aae20. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:52:33.817951 (+725602us) spinlock_profiling.cc:243] Waited 52.8 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:33.870712 (+ 52761us) raft_consensus.cc:1685] finished
0808 01:52:33.870763 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:33.870951 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":191238,"prepare.queue_time_us":19422,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":81,"raft.queue_time_us":123801,"raft.run_cpu_time_us":372,"raft.run_wall_time_us":148074,"spinlock_wait_cycles":127310976,"thread_start_us":436429,"threads_started":2,"wal-append.queue_time_us":141646}
I20260808 01:52:33.936017  9472 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:33.909495 16661 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:33.868438 16764 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:33.925057 16669 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:33.923743 16595 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.752s	user 0.004s	sys 0.043s
I20260808 01:52:33.940231  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:33.941932  9481 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 20: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:33.953785  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:33.954074  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:33.955541  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:33.955888 16757 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:33.956133  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 129164us
I20260808 01:52:33.956660 16757 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:33.959041 16757 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:33.964003  9476 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:52:33.941761  9599 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:33.944113  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:33.932046 16573 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:33.932854 16580 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:33.974982  9636 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
W20260808 01:52:33.952395 16595 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.781s	user 0.004s	sys 0.044s
I20260808 01:52:33.968101  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:33.968659  9472 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:52:33.972972 16530 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:33.939030 16623 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Stepping down as leader of term 19
W20260808 01:52:33.974181  9633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2866) took 1467 ms (client timeout 1808 ms). Trace:
I20260808 01:52:33.945034 16800 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:33.984498  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:33.983124  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:33.985249  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:33.988101  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:33.988526  9632 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:33.988857  9476 raft_consensus.cc:2368] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22.
I20260808 01:52:33.985585  9481 raft_consensus.cc:2368] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19.
I20260808 01:52:33.966377 16669 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
W20260808 01:52:33.984416  9633 rpcz_store.cc:269] 0808 01:52:32.506501 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:32.598549 (+ 92048us) service_pool.cc:224] Handling call
0808 01:52:33.973904 (+1375355us) spinlock_profiling.cc:243] Waited 479 ms on lock 0x7b54004218d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:33.974098 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1101298048}
I20260808 01:52:33.937141 16594 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:33.997184  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 46534.2us
I20260808 01:52:33.996363  9559 raft_consensus.cc:1217] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Deduplicated request from leader. Original: 17.9->[19.10-20.11]   Dedup: 19.10->[20.11-20.11]
I20260808 01:52:33.999581  9557 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:33.999890  9559 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: [19.10-20.11]
I20260808 01:52:34.000976  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:33.999119 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:34.004598  9472 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:34.005656  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:34.006368  9524 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:34.001942  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:34.010385  9633 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 22: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:33.965793 16661 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:33.984151 16623 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:34.012568  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:34.017623  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:33.996610 16803 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:33.987682 16800 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:34.031301  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 106812us
I20260808 01:52:34.013260 16530 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:34.036468 16580 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election won for term 20
W20260808 01:52:34.036492  9444 connection.cc:256] RPC call timeout handler was delayed by -1.4350211000128184s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19840000000000002s for a response
W20260808 01:52:34.044675  9444 connection.cc:256] RPC call timeout handler was delayed by -1.3945211000100244s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2029s for a response
I20260808 01:52:34.038149  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:34.039274  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:34.039978  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:33.970111 16579 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:34.045003  9444 connection.cc:256] RPC call timeout handler was delayed by -1.3709030420286581s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18080000000000002s for a response
I20260808 01:52:34.052304  9554 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:34.054057  9632 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260808 01:52:34.056380  9476 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:34.057324  9552 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:34.045348 16742 consensus_meta.cc:287] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.622s	user 0.010s	sys 0.004s
I20260808 01:52:34.083168  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:34.091228  9559 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:34.089244  9472 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:34.094633  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:34.089215  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 77933.3us
W20260808 01:52:34.098719  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 44622.6us
I20260808 01:52:34.100812 16764 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:34.101408 16594 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:34.102366  9444 connection.cc:256] RPC call timeout handler was delayed by -1.239721100020688s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22010000000000002s for a response
W20260808 01:52:34.102900  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1060063769982662s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2116s for a response
W20260808 01:52:34.103227  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0199030420044437s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2198s for a response
I20260808 01:52:34.103964 16661 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:34.106108 16661 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:34.090485 16596 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:34.107323 16596 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:34.110133 16596 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:34.111402 16650 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:34.027045 16803 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:34.114183 16650 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:34.115330 16803 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:34.117745 16773 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 01:52:34.120488  9474 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:34.121783 16800 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.16, Last appended by leader: 16, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:34.115511 16623 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:34.124186 16623 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:34.124466 16661 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:34.125440 16194 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:34.126853  9634 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260808 01:52:34.127786 16800 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
W20260808 01:52:34.129781 16807 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260808 01:52:34.132164  9558 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260808 01:52:34.136217  9478 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:34.137523  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:34.138418 16661 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:34.140528 16783 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.985s	user 0.015s	sys 0.005s
I20260808 01:52:34.141940 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 878) took 1354 ms. Trace:
I20260808 01:52:34.141508 16692 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:34.142501  9476 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 22.
I20260808 01:52:34.143658  9472 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:34.147876 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:34.148598  9559 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
W20260808 01:52:34.149855  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 167103us
I20260808 01:52:34.151495  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:34.127782 16194 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:34.142400 10361 rpcz_store.cc:276] 0808 01:52:32.787932 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:32.891598 (+103666us) service_pool.cc:224] Handling call
0808 01:52:32.892131 (+   533us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:32.892816 (+   685us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:32.892855 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:32.899678 (+  6823us) log.cc:844] Serialized 26 byte log entry
0808 01:52:33.034799 (+135121us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:33.034864 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:33.034942 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:34.139839 (+1104897us) spinlock_profiling.cc:243] Waited 28.8 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:34.141578 (+  1739us) raft_consensus.cc:1685] finished
0808 01:52:34.141602 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:34.141847 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6598,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":121,"spinlock_wait_cycles":66226432,"thread_start_us":129626,"threads_started":1,"wal-append.queue_time_us":241179}
I20260808 01:52:34.156618 16579 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
W20260808 01:52:34.159407 16805 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260808 01:52:34.167426  9476 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: []
W20260808 01:52:34.169855  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:34.169888  9476 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260808 01:52:34.175292  9476 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 14. Current term is 15. Ops: []
W20260808 01:52:34.175679  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:34.179014  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:34.181883 16751 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.812s	user 0.012s	sys 0.001s
I20260808 01:52:34.182247  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:34.183321 16751 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:34.181881  9476 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:34.185300  9474 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:34.185247  9552 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:34.192629  9554 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:34.193455  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:34.195382  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 875) took 1002 ms. Trace:
I20260808 01:52:34.195888  9635 rpcz_store.cc:276] 0808 01:52:33.192605 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:33.207647 (+ 15042us) service_pool.cc:224] Handling call
0808 01:52:33.218627 (+ 10980us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:33.339356 (+120729us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:34.195121 (+855765us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b54004204d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:34.195312 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":276120832}
I20260808 01:52:34.197521  9554 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:34.198415  9559 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 20 index: 11. (term mismatch)
I20260808 01:52:34.200847  9559 pending_rounds.cc:85] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 10
I20260808 01:52:34.203773  9559 pending_rounds.cc:107] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260808 01:52:34.204133  9559 raft_consensus.cc:2889] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:34.202920  9476 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:34.197118 16749 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:52:34.197402 16229 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:34.211347  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:34.197764 16249 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.017s
I20260808 01:52:34.210911  9552 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:34.204914  9472 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
W20260808 01:52:34.231061  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 127101us
I20260808 01:52:34.229384  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:34.239137  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:34.211822 16749 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:34.233856  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:34.257135 16194 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:34.257902 16194 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:34.260538 16194 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:34.260771 16713 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:34.261370 16713 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:34.240579 16753 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:34.268889  9559 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:34.269388  9521 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:34.270139  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:34.271790  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3026) took 1047 ms. Trace:
I20260808 01:52:34.272522  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:34.272908 16669 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:34.274403  9558 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:34.275561 16645 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:34.276330 16645 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:52:34.276400  9555 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260808 01:52:34.277242 16645 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:34.277750 16645 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:34.278115 16645 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:34.267503 16692 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:34.270435  9521 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:34.279795  9550 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: [17.9-19.10]
I20260808 01:52:34.277245 10366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 799) took 1110 ms. Trace:
I20260808 01:52:34.272342  9636 rpcz_store.cc:276] 0808 01:52:33.224577 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:33.226868 (+  2291us) service_pool.cc:224] Handling call
0808 01:52:33.974775 (+747907us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:33.981990 (+  7215us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:34.271546 (+289556us) spinlock_profiling.cc:243] Waited 747 ms on lock 0x7b54004218d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:34.271720 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1719234304}
I20260808 01:52:34.280970 10366 rpcz_store.cc:276] 0808 01:52:33.166501 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:33.184254 (+ 17753us) service_pool.cc:224] Handling call
0808 01:52:34.277043 (+1092789us) spinlock_profiling.cc:243] Waited 278 ms on lock 0x7b5400690354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:34.277159 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":640065792}
I20260808 01:52:34.270205  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:34.291520  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:34.298314  9552 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 20: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:34.278378  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:34.298012  9476 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 17 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260808 01:52:34.299841  9555 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:34.300740  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:34.301256  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:34.301807  9554 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:34.301961  9559 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 19
I20260808 01:52:34.300722  9524 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:34.309012  9550 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 17 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260808 01:52:34.267405 16749 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:34.311563 16749 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:34.311568  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3241) took 1083 ms. Trace:
I20260808 01:52:34.315313  9521 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:34.315834 16249 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:34.316597  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:34.316861 16249 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:34.317484 16249 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:34.317533  9521 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:34.318188 16249 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:34.318902  9556 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:52:34.294121 16692 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:34.319468  9524 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:34.319762 16692 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:34.320330 16692 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:34.320719 16785 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:34.322333 16819 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260808 01:52:34.323269 16819 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:34.323776 16819 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:34.324133 16819 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:34.325753  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:34.328367  9521 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:34.329452  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:34.330945 16751 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:34.331147 16801 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:34.331460 16751 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:34.332020 16751 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:34.332057  9477 rpcz_store.cc:276] 0808 01:52:33.228042 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:33.231212 (+  3170us) service_pool.cc:224] Handling call
0808 01:52:34.184522 (+953310us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b5800005498. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:52:34.184737 (+   215us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:34.193584 (+  8847us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:34.193616 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:34.194344 (+   728us) log.cc:844] Serialized 52 byte log entry
0808 01:52:34.241256 (+ 46912us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:52:34.241302 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:34.241379 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:34.311112 (+ 69733us) raft_consensus.cc:1685] finished
0808 01:52:34.311139 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:34.311489 (+   350us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2786,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":92,"spinlock_wait_cycles":23655680,"thread_start_us":46489,"threads_started":1,"wal-append.queue_time_us":111436}
I20260808 01:52:34.333859 16249 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:34.339409 16249 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:34.341302  9551 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 18. Current term is 19. Ops: []
W20260808 01:52:34.342074  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:34.352066  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:34.354616  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:34.343025  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:34.343971  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:34.359428  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:34.359633 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:34.360944  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:34.346021  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:34.331601 16801 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:34.342556  9521 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:52:34.363011  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:34.361953  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:34.364457  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 90005.5us
W20260808 01:52:34.390479  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48034.7us
I20260808 01:52:34.380282  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:34.380882  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:34.393051  9551 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 20: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:34.355381  9559 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:34.393765  9555 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:52:34.381295  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 61932.6us
I20260808 01:52:34.393195  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:34.387596  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 83178.1us
I20260808 01:52:34.384640 10334 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:34.355950  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:34.397899  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:34.432862  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 78313.6us
I20260808 01:52:34.434386  9601 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:34.436842  9554 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 20: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:34.382606 16570 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:34.443070  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:34.438040 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 53431.5us
I20260808 01:52:34.398326  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:34.440075  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:34.434362  9474 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:34.357617 16194 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:52:34.437878  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:34.446374 10334 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:34.382469 16820 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:34.435464 16595 raft_consensus.cc:2764] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
W20260808 01:52:34.444630 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 57113.3us
I20260808 01:52:34.443063  9601 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:34.450721  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:34.450531  9524 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:34.447405  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:34.398473 16800 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:34.445200  9476 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260808 01:52:34.438167  9558 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:34.447211  9551 raft_consensus.cc:2410] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 18 because replica has last-logged OpId of term: 17 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 15.
I20260808 01:52:34.436115 16801 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:34.472157  9599 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:34.478947  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:34.478960  9554 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:34.358494 16753 consensus_queue.cc:1059] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.089s
W20260808 01:52:34.485872  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47208.2us
I20260808 01:52:34.444672 16763 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:52:34.485457 16660 raft_consensus.cc:3055] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:34.497592 16660 raft_consensus.cc:740] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:34.498772 16660 consensus_queue.cc:260] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:34.501376  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50998.7us
I20260808 01:52:34.503122  9524 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:34.503432  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:34.505697  9524 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:34.506923 16249 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:34.509048 16249 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260808 01:52:34.510042 16249 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260808 01:52:34.511116 16602 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:34.511972 16602 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:34.512377 16602 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:34.512828 16742 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:34.513365 16742 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:34.513784 16602 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:34.514267 16742 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:34.519104 16596 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:34.500499 16660 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:34.521879  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:34.550040  9599 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:34.522843 16808 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:52:34.522950 16810 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:34.524866 16470 consensus_queue.cc:1059] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260808 01:52:34.526013 16754 consensus_queue.cc:1059] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260808 01:52:34.525668  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:34.527717 16669 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:34.568810 10366 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260808 01:52:34.526062 16530 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.003s
I20260808 01:52:34.539139 16570 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260808 01:52:34.533721  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:34.499243  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:34.574748  9559 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 16: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:34.535976  9477 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:34.447137 16594 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:34.448328 16194 raft_consensus.cc:697] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:34.470885 16800 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:34.552161  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46546.5us
W20260808 01:52:34.552966 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 106690us
I20260808 01:52:34.491483 16757 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:34.531674  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:34.600764 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:34.584726 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:34.554591 16808 consensus_queue.cc:1243] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 20.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260808 01:52:34.552706 16671 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:34.606204 10334 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:34.610247  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 78669.3us
I20260808 01:52:34.611812  9477 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260808 01:52:34.553584 16576 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:34.568843 16661 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:34.655591 10366 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
W20260808 01:52:34.613917 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 86364.3us
I20260808 01:52:34.622356  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:34.625610 16713 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:34.614847 16785 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:52:34.628281 16602 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:34.635057  9550 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:52:34.635210 16530 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:34.635742 16669 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:34.598608 16742 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:34.637224 16651 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:34.637050 16194 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:34.649329 16751 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:34.650717 16594 raft_consensus.cc:697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:34.573413 16570 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:34.584372 16810 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Stepping down as leader of term 19
W20260808 01:52:34.661136 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:34.665478 16828 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:34.662214 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:34.739524 10368 raft_consensus.cc:2410] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260808 01:52:34.663223 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:34.673055 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 73605.9us
I20260808 01:52:34.629643 16692 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:34.679332 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:34.746007 10366 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:34.746196  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:34.747608 16692 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:34.680755 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:34.751143 10360 raft_consensus.cc:2376] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19. Re-sending same reply.
I20260808 01:52:34.684785 16669 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:34.685218  9524 leader_election.cc:400] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22.
I20260808 01:52:34.654508 16661 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:34.752584 16661 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:34.753575 16669 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:34.704389  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:34.716651  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 166757us
I20260808 01:52:34.755474  9599 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:34.681864 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:34.755764  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 72021.7us
I20260808 01:52:34.741403 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:34.750027 16801 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:34.743928  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 121672us
I20260808 01:52:34.674939 16530 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:34.603129 16800 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:34.758323  9524 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:34.765105 10366 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
W20260808 01:52:34.677196  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 151607us
I20260808 01:52:34.677414 16290 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:34.652094 16671 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:34.774080  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:34.737437 16570 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:34.773135  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:34.737700 16810 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:34.779088  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:34.793377  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:34.774289 16530 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:34.686173 16592 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.093s
I20260808 01:52:34.822067 16675 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:34.823669 16713 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:34.824641 16713 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:52:34.824978 16736 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:34.825184 16713 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:34.825845 16736 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260808 01:52:34.829969 16645 consensus_meta.cc:287] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.551s	user 0.011s	sys 0.001s
W20260808 01:52:34.831178  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 52028.3us
I20260808 01:52:34.832134  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:34.833895 16801 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:34.811267 16692 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:34.835464 16831 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:34.836351 16668 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 01:52:34.837716 16751 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:34.780215 16757 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:52:34.853591  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:34.778331 16651 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:34.887723  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:34.776388 16576 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:34.756507 16828 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:34.848425 16675 raft_consensus.cc:697] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:34.830482 16645 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:34.896917  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 64880us
W20260808 01:52:34.901975 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 161231us
I20260808 01:52:34.826268 16713 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:34.905134  9473 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260808 01:52:34.794066 16669 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:52:34.891445 16290 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:52:34.889999 16801 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:34.844652 16819 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:34.912588  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 135232us
I20260808 01:52:34.853312 16594 consensus_queue.cc:237] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:34.920444  9599 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:34.920876 10361 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:34.916785  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3238) took 1690 ms. Trace:
I20260808 01:52:34.895942 16836 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:34.923794  9480 raft_consensus.cc:2393] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 18: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:34.921869  9473 rpcz_store.cc:276] 0808 01:52:33.226424 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:33.226804 (+   380us) service_pool.cc:224] Handling call
0808 01:52:34.903296 (+1676492us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b5800005498. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:52:34.903637 (+   341us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:34.913966 (+ 10329us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:34.916710 (+  2744us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":33215872}
I20260808 01:52:34.895102 16651 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:34.889792 16834 consensus_queue.cc:1059] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
I20260808 01:52:34.776386 16830 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.124s
W20260808 01:52:34.944530  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:34.934059 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:34.931671  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:34.924594 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 242813us
I20260808 01:52:34.924682  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:34.970445  9474 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:34.973335 16832 raft_consensus.cc:3055] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:34.973842 16832 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:34.915525 16837 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:34.923346 16836 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:52:34.988921  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:34.987064  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 61822.3us
I20260808 01:52:34.991631 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:35.012892  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 88489.5us
I20260808 01:52:35.028923  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:34.964995 16840 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:34.912938 16810 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:35.040520  9475 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260808 01:52:34.929595 16470 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:52:35.043880  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:35.061875  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:35.050173  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:34.980563 16839 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.068s
I20260808 01:52:35.067962  9477 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: [19.11-21.12]
W20260808 01:52:35.071645 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 102378us
W20260808 01:52:35.077410  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:35.028398 16836 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:35.045349 16831 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:35.082146  9474 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
W20260808 01:52:35.062438 16580 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.963s	user 0.024s	sys 0.046s
I20260808 01:52:35.085227 16843 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260808 01:52:35.085218  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 312156us
I20260808 01:52:35.113592  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:35.114503  9601 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:35.116709  9601 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:35.088815  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 93374.7us
I20260808 01:52:34.956776 16757 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:34.925642 16801 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.045746 16837 raft_consensus.cc:697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:35.124141 16837 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.097539  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:35.128125  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:35.071278 16671 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:35.047083 16842 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:52:35.131455 16842 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:35.130328  9475 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: []
I20260808 01:52:35.136425 16801 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:35.107697  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:35.139102  9477 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:35.141469  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:35.148401  9478 raft_consensus.cc:2368] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19.
W20260808 01:52:35.150216  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 58194.4us
I20260808 01:52:35.155138  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:35.155758  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:35.105966 16580 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.010s	user 0.025s	sys 0.046s
I20260808 01:52:35.102597 16810 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:35.103506 16819 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:35.114225 16835 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:35.113320 16713 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:35.049947 16840 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.166800  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:34.980175 16675 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.170140  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:35.145200 16830 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:35.174053 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59620.2us
I20260808 01:52:35.029105 16576 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 15.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.154279 16651 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:35.098459 16290 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.073s
I20260808 01:52:35.154633 16470 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:35.161092 16580 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:35.183429  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:35.146777 16800 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:35.189682 10366 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 12.
I20260808 01:52:35.177556 16645 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:35.199064  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:35.202457 16852 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:35.202812 16800 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:35.203819  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 85418.5us
I20260808 01:52:35.204032 16651 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:35.204717 16651 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.206615  9551 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260808 01:52:35.207446 16651 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:35.208498  9552 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:52:35.208945  9552 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:35.209826  9552 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:35.210960  9552 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:35.212242 16651 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:35.159121 16785 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.027s
I20260808 01:52:35.220240  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:35.162781 16809 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.025s
I20260808 01:52:35.230170  9552 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260808 01:52:35.236904  9478 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 20: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:35.216761 16651 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:35.203141 16852 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:35.254784  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:35.256726  9445 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:35.263132  9475 raft_consensus.cc:2393] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:35.262318 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:35.235481 16848 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:35.268488 16852 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:35.269595 16848 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.270229  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:35.271034  9552 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:35.181171 16835 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:35.272308  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:35.272339  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:35.235898 16828 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:35.286655  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:35.287823  9478 raft_consensus.cc:2393] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:35.182312 16713 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.181641 16830 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:35.355279  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 98636.8us
I20260808 01:52:35.356343  9445 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:35.318413  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:35.360498 10361 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:35.280776 16645 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 11 FOLLOWER]: Advancing to term 12
W20260808 01:52:35.362118  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:35.379138  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:35.388135  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:35.391288  9445 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:35.255537 16651 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.280108 16852 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.098s
W20260808 01:52:35.406811  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:35.408368  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:35.401089 16853 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260808 01:52:35.409476  9523 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:35.389165 16819 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:35.433017  9475 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:35.397423 16842 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:35.397907 16848 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:35.471343  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 199630us
I20260808 01:52:35.472358  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:35.473774 16867 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:35.486296 16850 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:35.486966 16863 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:35.487092 16850 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.487931 16863 raft_consensus.cc:697] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:35.489136 16863 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.410427 16856 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.098s
I20260808 01:52:35.434552 16809 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:35.489897 16850 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:35.440395 16592 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260808 01:52:35.546865  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 137472us
W20260808 01:52:35.479959 16602 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.805s	user 0.012s	sys 0.003s
I20260808 01:52:35.549876 16831 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:35.495841 16835 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:35.390650 16580 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:35.555999 16857 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.169s	user 0.002s	sys 0.000s
I20260808 01:52:35.557862 16820 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:52:35.560379 16825 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.050s	user 0.003s	sys 0.007s
I20260808 01:52:35.566556 16713 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:35.567456  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1160) took 2355 ms. Trace:
W20260808 01:52:35.569202 16661 consensus_meta.cc:287] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.816s	user 0.012s	sys 0.001s
I20260808 01:52:35.573720 16867 raft_consensus.cc:697] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:35.620568 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:35.623495 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 361445us
I20260808 01:52:35.623780 10364 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:35.480566 16194 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:35.517813 16785 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:35.494410 16651 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:35.516148 16809 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:35.596845 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:35.443738 16819 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:35.602113  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:35.620162 16661 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:35.611698 10368 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
W20260808 01:52:35.613766  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:35.604431 16820 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:35.625519  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:35.626412  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 235707us
I20260808 01:52:35.623513 10362 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 15.
I20260808 01:52:35.604241  9479 rpcz_store.cc:276] 0808 01:52:33.212484 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:33.212704 (+   220us) service_pool.cc:224] Handling call
0808 01:52:33.244469 (+ 31765us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:33.265769 (+ 21300us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:33.265804 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:33.270575 (+  4771us) log.cc:844] Serialized 26 byte log entry
0808 01:52:35.101730 (+1831155us) spinlock_profiling.cc:243] Waited 3.35 ms on lock 0x7b18002d76c0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:52:35.157407 (+ 55677us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:35.157470 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:35.157677 (+   207us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:35.564718 (+407041us) spinlock_profiling.cc:243] Waited 21.3 ms on lock 0x7b24000ef228. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41713594 00007f5d4171169b 00007f5d418cd6d9 00007f5d418cd365 00007f5d418cc8c0 00007f5d4455a638 00007f5d4455ad98 00007f5d44532588 00007f5d4452d95a 00007f5d4452cda6
0808 01:52:35.567184 (+  2466us) raft_consensus.cc:1685] finished
0808 01:52:35.567209 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:35.567352 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7804,"raft.queue_time_us":33508,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":16346,"spinlock_wait_cycles":56776320,"thread_start_us":55417,"threads_started":1,"wal-append.queue_time_us":1930546}
W20260808 01:52:35.636317  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 161968us
I20260808 01:52:35.638669  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:35.701265  9474 raft_consensus.cc:2410] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 21 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260808 01:52:35.597980 16802 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.036s
I20260808 01:52:35.640749  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:35.626253 16602 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:35.635044 16530 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.742s	user 0.009s	sys 0.006s
I20260808 01:52:35.633168 16858 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election won for term 22
W20260808 01:52:35.643154 16570 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.825s	user 0.011s	sys 0.004s
I20260808 01:52:35.636809 16671 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:35.652920 10360 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260808 01:52:35.652208 10365 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:35.602947  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:35.715425  9554 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
W20260808 01:52:35.656512 16660 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.087s	user 0.016s	sys 0.000s
W20260808 01:52:35.656778 16859 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.137s	user 0.000s	sys 0.001s
I20260808 01:52:35.661880 16866 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:35.645502 16865 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.023s
I20260808 01:52:35.667680  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2745) took 1651 ms. Trace:
I20260808 01:52:35.671041  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:35.723167  9560 raft_consensus.cc:2410] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 16 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260808 01:52:35.649893 16194 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.655162 16867 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.684584 16645 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260808 01:52:35.693418  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:35.713181  9552 raft_consensus.cc:2410] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 20 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260808 01:52:35.654559 16809 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:35.737828 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:35.726732 16194 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:35.710747 16870 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:35.720511  9557 rpcz_store.cc:276] 0808 01:52:34.016567 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:34.016768 (+   201us) service_pool.cc:224] Handling call
0808 01:52:34.024799 (+  8031us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:34.202927 (+178128us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:34.202961 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:34.208266 (+  5305us) log.cc:844] Serialized 26 byte log entry
0808 01:52:34.362768 (+154502us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:34.362809 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:34.362984 (+   175us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:35.559670 (+1196686us) spinlock_profiling.cc:243] Waited 27.6 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:35.667316 (+107646us) raft_consensus.cc:1685] finished
0808 01:52:35.667342 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:35.667605 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":42859,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":13703,"spinlock_wait_cycles":63587200,"thread_start_us":235513,"threads_started":2,"wal-append.queue_time_us":231989}
W20260808 01:52:35.720453 16470 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.523s	user 0.013s	sys 0.000s
I20260808 01:52:35.745443  9475 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:52:35.747691  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:35.754276  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:35.757025  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:35.759171  9554 raft_consensus.cc:2410] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 16 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260808 01:52:35.741204 16850 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:35.714457 16602 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.744793 16290 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:35.717065 16570 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:35.712997 16530 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:35.756569 16470 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:35.752341 16645 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:35.719693 16671 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.780874 16602 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:35.781634 16530 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:35.771521 16290 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.782653  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:35.783403  9554 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:52:35.783908 16290 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:35.784451  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:35.764871 16850 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:35.810704  9554 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:35.794251 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:35.794759 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:35.793180 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:35.795899 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 805) took 1119 ms. Trace:
I20260808 01:52:35.814724 10366 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:52:35.815120 10361 rpcz_store.cc:276] 0808 01:52:34.676623 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:34.681342 (+  4719us) service_pool.cc:224] Handling call
0808 01:52:35.795674 (+1114332us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b540037dcd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:35.795805 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":377789184}
I20260808 01:52:35.793627 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:35.814653 10365 raft_consensus.cc:2393] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:35.791318 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:35.816521  9555 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:35.760166 16785 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:35.752970 16870 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:35.824820  9551 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 19
W20260808 01:52:35.827337  9481 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 3198) took 1893 ms (client timeout 1733 ms). Trace:
W20260808 01:52:35.828013  9481 rpcz_store.cc:269] 0808 01:52:33.926754 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:33.955201 (+ 28447us) service_pool.cc:224] Handling call
0808 01:52:35.819912 (+1864711us) spinlock_profiling.cc:243] Waited 23 ms on lock 0x7b54000217d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:35.820064 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":52863232}
I20260808 01:52:35.784646 16290 raft_consensus.cc:2764] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260808 01:52:35.790804 16530 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:35.828255  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 80666.6us
I20260808 01:52:35.836747  9600 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:35.816974 16850 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:35.789090 16830 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:35.844616  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 72690.1us
I20260808 01:52:35.847616  9599 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:35.819852 16858 raft_consensus.cc:697] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:35.835508 16836 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.691s	user 0.011s	sys 0.004s
I20260808 01:52:35.848434  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:35.804634 16645 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.839488 16470 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:35.837699 16841 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:35.861002 16671 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:35.861399 16870 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:35.854709 16194 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:35.855468 16836 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:35.841876 16808 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260808 01:52:35.856510 16810 consensus_meta.cc:287] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.689s	user 0.015s	sys 0.000s
I20260808 01:52:35.872864  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:35.879176  9559 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:35.880167  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2733) took 2264 ms. Trace:
I20260808 01:52:35.880704  9556 rpcz_store.cc:276] 0808 01:52:33.614279 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:33.614420 (+   141us) service_pool.cc:224] Handling call
0808 01:52:35.878195 (+2263775us) spinlock_profiling.cc:243] Waited 677 ms on lock 0x7b5400217ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:35.878335 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1557952768}
I20260808 01:52:35.878509 16808 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:35.883385 16808 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:52:35.884649 16826 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 1.353s	user 0.003s	sys 0.001s
W20260808 01:52:35.887043 16823 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.491s	user 0.005s	sys 0.010s
I20260808 01:52:35.877264 16841 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.889444 16841 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:35.882357  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:35.892140  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:35.843180 16530 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.902315 10363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 886) took 2559 ms. Trace:
I20260808 01:52:35.871030 16819 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:35.876785 16194 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:35.903934 10363 rpcz_store.cc:276] 0808 01:52:33.331942 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:33.366121 (+ 34179us) service_pool.cc:224] Handling call
0808 01:52:33.388994 (+ 22873us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:33.422900 (+ 33906us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:33.422934 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:34.338959 (+916025us) spinlock_profiling.cc:243] Waited 1.14 ms on lock 0x7b2c0002f0f8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:52:34.339648 (+   689us) log.cc:844] Serialized 52 byte log entry
0808 01:52:34.391624 (+ 51976us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:34.391694 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:34.422379 (+ 30685us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:35.883839 (+1461460us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:35.890892 (+  7053us) raft_consensus.cc:1685] finished
0808 01:52:35.890915 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:35.891089 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6127,"raft.queue_time_us":16140,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":65245,"spinlock_wait_cycles":25646848,"thread_start_us":51440,"threads_started":1,"wal-append.queue_time_us":139941}
I20260808 01:52:35.913007  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:35.921502 16871 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:35.922729 16832 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:35.924621 16832 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:35.926949 16773 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:35.927567 16773 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:35.927990 16773 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:35.932274 16645 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:35.935003  9550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3568) took 1595 ms (client timeout 1633 ms). Trace:
W20260808 01:52:35.935539  9550 rpcz_store.cc:269] 0808 01:52:34.339693 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:34.340532 (+   839us) service_pool.cc:224] Handling call
0808 01:52:35.934760 (+1594228us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b5400217fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:35.934922 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":628312448}
I20260808 01:52:35.936519 16836 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:35.936929  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:35.937641 16570 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:35.898965 16470 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:35.939337  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:35.940121  9557 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:52:35.940634 16470 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:35.907332 16785 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.937673  9550 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:52:35.948542  9550 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
W20260808 01:52:35.949101 16879 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260808 01:52:35.950127  9550 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
W20260808 01:52:35.953248 16827 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 1.444s	user 0.004s	sys 0.000s
W20260808 01:52:35.957269  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:35.916447 16819 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:35.959358 16819 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:35.961087 16850 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:35.967445  9551 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
W20260808 01:52:35.907159 16839 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.752s	user 0.006s	sys 0.016s
I20260808 01:52:35.912590 16194 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:35.974786  9557 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260808 01:52:35.974022  9599 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:35.977185  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 106965us
I20260808 01:52:35.986203  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:35.984088  9556 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:35.878763 16878 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260808 01:52:35.992322  9557 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:35.996191  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:35.997079  9559 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:36.000371  9599 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:36.008796  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:35.974030 16839 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.821s	user 0.006s	sys 0.017s
I20260808 01:52:35.994282 16871 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:36.007867 16810 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.007s
I20260808 01:52:35.999039 16836 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:52:36.018062 16836 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:36.010958 16660 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:36.011725  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:36.023303  9556 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:36.025977  9559 pending_rounds.cc:85] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 12
I20260808 01:52:36.041960 16841 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:36.042603  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:36.041853  9559 pending_rounds.cc:107] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 20.13
I20260808 01:52:36.042737 16841 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:36.043659  9559 raft_consensus.cc:2889] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:36.047478 16841 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:35.994850 16470 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:35.794143  9368 catalog_manager.cc:5697] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 15 to 17, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 17 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:52:36.001168 16819 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:36.052278  9473 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:36.011027 16757 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:36.018334 16809 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:36.040529 16810 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:36.025525 16645 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:36.018778 16836 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:52:36.065572  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 69660.1us
I20260808 01:52:36.014218 16871 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:36.071982  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:36.074996  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:36.061942 16720 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.543s	user 0.017s	sys 0.025s
I20260808 01:52:36.053237 16660 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:35.978366 16858 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:36.055540 16470 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:36.106812  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:36.067040 16757 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:52:36.078881 16720 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.560s	user 0.018s	sys 0.025s
I20260808 01:52:36.061678 16837 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.025s
I20260808 01:52:36.121999 16840 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:36.083042 16645 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:52:36.176229  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:36.126770 16809 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:36.127660  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3579) took 1066 ms. Trace:
I20260808 01:52:36.193691  9555 rpcz_store.cc:276] 0808 01:52:35.060970 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:35.061120 (+   150us) service_pool.cc:224] Handling call
0808 01:52:35.079257 (+ 18137us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:35.820268 (+741011us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:36.127357 (+307089us) spinlock_profiling.cc:243] Waited 737 ms on lock 0x7b540015b154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:36.127587 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1694915328}
I20260808 01:52:36.079591 16808 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:36.124629 16851 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.761s	user 0.000s	sys 0.003s
I20260808 01:52:36.135195 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:36.074156 16810 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:36.137281 16871 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:36.116595 16785 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.034s
I20260808 01:52:36.133765 10360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 808) took 1143 ms. Trace:
I20260808 01:52:36.111938 16843 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:36.147570 16757 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:36.144474 16470 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:36.139107 16530 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:36.135298 16660 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:36.140868 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:36.154722 16834 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:36.223803 16824 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.708s	user 0.003s	sys 0.011s
W20260808 01:52:36.227553  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:36.231330 16829 raft_consensus.cc:3055] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:36.206521  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:36.285540  9555 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
W20260808 01:52:36.242595  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:36.240392 10360 rpcz_store.cc:276] 0808 01:52:34.990436 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:34.990774 (+   338us) service_pool.cc:224] Handling call
0808 01:52:35.652534 (+661760us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:35.713993 (+ 61459us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:36.133499 (+419506us) spinlock_profiling.cc:243] Waited 661 ms on lock 0x7b54005122d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:36.133689 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1521046400}
W20260808 01:52:36.249341  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:36.146900 16820 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:36.246457 10334 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:36.269966 10366 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
W20260808 01:52:36.270745  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 155462us
I20260808 01:52:36.294166  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:36.205672 16850 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:36.300585 16645 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:36.301133 16645 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:36.301550 16645 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:36.303300  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:36.304998 16820 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:36.308858 16808 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:36.311556 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 65243.2us
I20260808 01:52:36.265707 16470 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:36.315444  9476 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260808 01:52:36.320662  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:36.320850 16850 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:52:36.324028 16850 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:36.324488 16850 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:36.314726 16871 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:36.330858  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 185247us
I20260808 01:52:36.285142 16829 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:36.266013 16843 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:36.351369  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:36.352134  9550 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:36.328591 16470 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:36.330291 16290 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:36.361516  9444 connection.cc:256] RPC call timeout handler was delayed by -1.3142115469672717s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1637s for a response
I20260808 01:52:36.342996 16713 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.015s
I20260808 01:52:36.284292 16834 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:36.327262 16871 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:36.363875  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0694115469814278s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19090000000000001s for a response
W20260808 01:52:36.373808  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0262115469668061s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1957s for a response
I20260808 01:52:36.374706  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:36.354611 16596 raft_consensus.cc:3055] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:36.372078 16290 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:36.391937  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:36.392653 16661 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:36.392961 16290 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:36.393572 16870 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:36.394068 16870 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:36.394459 16870 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:36.364845 16867 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:36.373859 16720 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:36.374148 16785 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:36.387502 16596 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:36.431053  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:36.409229 16763 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.876s	user 0.014s	sys 0.022s
I20260808 01:52:36.410377 16579 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:36.441282 16763 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.908s	user 0.014s	sys 0.023s
I20260808 01:52:36.459357  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:36.460367  9559 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:36.461285  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:36.461972  9638 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:36.459780  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:36.460289  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:36.465149  9636 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:36.466138  9634 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: []
I20260808 01:52:36.466809  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:36.468308  9629 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 12. (term mismatch)
I20260808 01:52:36.468852  9629 pending_rounds.cc:85] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 11
I20260808 01:52:36.469161  9629 pending_rounds.cc:107] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260808 01:52:36.469470  9629 raft_consensus.cc:2889] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:36.472677  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:36.416617 16867 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:36.447571 16470 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:36.484521 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:36.485551 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:36.485925  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:36.484594  9600 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:36.451296 16661 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:36.454334 16530 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:36.423739 16720 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:36.489681 10360 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:52:36.506858  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:36.509137  9480 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 3240) took 3281 ms (client timeout 2170 ms). Trace:
I20260808 01:52:36.427049 16785 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:36.509243  9472 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:36.497272  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:36.513700  9637 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
W20260808 01:52:36.509730  9480 rpcz_store.cc:269] 0808 01:52:33.227712 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:33.229842 (+  2130us) service_pool.cc:224] Handling call
0808 01:52:34.325558 (+1095716us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b5800005498. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:52:36.508871 (+2183313us) spinlock_profiling.cc:243] Waited 597 ms on lock 0x7b540019d2d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:36.509057 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1399512704}
I20260808 01:52:36.514237  9638 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:36.521895  9630 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:36.525214  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:36.460052 16763 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:36.479625 16808 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:36.457453 16810 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:36.534499 16833 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:52:36.562403 16867 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:36.562785 16833 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:36.563252 16833 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:52:36.564031 16899 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260808 01:52:36.508215 16661 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:36.566340  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3220) took 1381 ms. Trace:
I20260808 01:52:36.454399 16579 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:36.547169  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:36.547279 10368 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 15.
I20260808 01:52:36.547142  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:36.574118  9632 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
W20260808 01:52:36.557400 16832 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.629s	user 0.012s	sys 0.005s
W20260808 01:52:36.511650 16753 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.985s	user 0.023s	sys 0.029s
I20260808 01:52:36.557051  9481 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 14. Current term is 15. Ops: []
W20260808 01:52:36.535439 16773 consensus_meta.cc:287] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.607s	user 0.009s	sys 0.006s
I20260808 01:52:36.557027  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:36.426261 16829 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:36.566936 16661 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:36.558696 16808 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:36.473409 16858 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.071s
I20260808 01:52:36.570261  9473 rpcz_store.cc:276] 0808 01:52:35.185005 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:35.186024 (+  1019us) service_pool.cc:224] Handling call
0808 01:52:35.820311 (+634287us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:36.056345 (+236034us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:36.565314 (+508969us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b54000217d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:36.566137 (+   823us) spinlock_profiling.cc:243] Waited 600 ms on lock 0x7b54000217d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:36.566274 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1912748544}
I20260808 01:52:36.577168  9637 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:36.550328  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:36.559026 16876 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260808 01:52:36.511502 16530 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:36.592712  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:36.615095  9554 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:36.593791  9634 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:36.595820 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:36.617064  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:36.534965 16878 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.045s
I20260808 01:52:36.618126 10332 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:36.610903  9630 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:36.593827 16753 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.058s	user 0.024s	sys 0.029s
I20260808 01:52:36.621907  9631 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:36.562403 16720 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:36.584679 16843 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:36.623018  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 145594us
I20260808 01:52:36.634187  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:36.625432 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:36.627439  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:36.644861  9559 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:36.645787  9629 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:36.649116  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:36.650292  9637 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
W20260808 01:52:36.655664  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:36.649804  9634 raft_consensus.cc:2376] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15. Re-sending same reply.
I20260808 01:52:36.652956  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:36.651988  9555 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:52:36.665941 10360 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:52:36.667999  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:36.677999  9600 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:36.669091  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:36.671770 16530 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:36.569042 16596 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:36.674719  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:36.681484  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:36.682099  9560 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:36.624647 16889 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:36.677278  9634 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:36.691583  9632 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
W20260808 01:52:36.685292  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:36.688647  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:36.690105  9633 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:36.696080  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:36.695170  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:36.692672  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:36.697124  9633 raft_consensus.cc:2376] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20. Re-sending same reply.
I20260808 01:52:36.690387  9636 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:36.698464  9632 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:36.648682 16834 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:36.653004 16720 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:36.699920  9630 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:36.701911  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:36.659122 16829 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:36.712425  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:36.713161  9560 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:36.709286  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:36.667997 16871 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:36.704401 16896 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.267s	user 0.001s	sys 0.001s
I20260808 01:52:36.714010  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:36.638803 16754 consensus_queue.cc:1059] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.058s
I20260808 01:52:36.691859 16757 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:36.723426  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:36.731630  9559 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 12.
I20260808 01:52:36.699936 16290 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:36.725534  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:36.668068 16894 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.045s
I20260808 01:52:36.697139 16858 raft_consensus.cc:3055] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:36.737360 16858 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:36.727607  9554 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:36.739182  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:36.738997 16858 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:36.741096 16858 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:36.713948  9630 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
W20260808 01:52:36.733407 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 83705.1us
I20260808 01:52:36.733496  9632 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:36.744441  9630 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:36.745649  9632 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260808 01:52:36.747144  9632 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260808 01:52:36.748594  9632 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: [16.7-17.8]
I20260808 01:52:36.706975 16720 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:36.750290  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:36.747687  9630 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260808 01:52:36.704943 16878 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:36.761314  9550 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:36.780830  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:36.781658  9633 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:52:36.788368 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:36.724964 16916 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:36.749470 16871 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:36.750156 16570 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:36.793290  9630 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: [14.9-15.10]
I20260808 01:52:36.765354 16720 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:36.764374 16922 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:36.811605 16929 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:36.821534  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 990) took 1236 ms. Trace:
I20260808 01:52:36.822161  9551 rpcz_store.cc:276] 0808 01:52:35.585429 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:35.585587 (+   158us) service_pool.cc:224] Handling call
0808 01:52:36.821233 (+1235646us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b540015ac54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:36.821437 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":391092480}
I20260808 01:52:36.765522 16878 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:36.764506 16757 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:36.833972  9551 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:36.824276  9560 raft_consensus.cc:2410] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 16 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260808 01:52:36.825184  9559 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260808 01:52:36.837857  9633 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:52:36.838096  9560 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:36.793300 16785 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:36.842993  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 55006us
I20260808 01:52:36.764178 16889 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.050s
I20260808 01:52:36.843128  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:36.848956  9551 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260808 01:52:36.852393  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:36.849895  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:36.814198 16570 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:36.862636  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 124409us
I20260808 01:52:36.846004 16579 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:36.864799  9560 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260808 01:52:36.869748  9551 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260808 01:52:36.850266 16808 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:36.872435  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:36.880183  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:36.881181  9632 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:36.847820 16920 consensus_queue.cc:1059] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.029s
W20260808 01:52:36.885318 16194 consensus_meta.cc:287] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.886s	user 0.012s	sys 0.004s
I20260808 01:52:36.885918 16194 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:36.888778 16194 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:36.893960  9633 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 17: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:52:36.899716 16915 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.135s	user 0.000s	sys 0.000s
I20260808 01:52:36.901463 16757 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:36.911916  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:36.913247  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 319216us
W20260808 01:52:36.914536  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 58004.6us
I20260808 01:52:36.915613  9524 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:36.922474  9632 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:36.832007 16922 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:36.604882 16773 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:36.934463  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 59226.8us
W20260808 01:52:36.940448  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:36.941098  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:36.941625  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:36.942169  9632 raft_consensus.cc:2376] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16. Re-sending same reply.
I20260808 01:52:36.943687  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:36.945957  9552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3585) took 1577 ms (client timeout 1668 ms). Trace:
I20260808 01:52:36.955118  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3221) took 1756 ms. Trace:
I20260808 01:52:36.956727 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:36.946434  9552 rpcz_store.cc:269] 0808 01:52:35.368600 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:35.390133 (+ 21533us) service_pool.cc:224] Handling call
0808 01:52:36.945714 (+1555581us) spinlock_profiling.cc:243] Waited 304 ms on lock 0x7b54000b4ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:36.945888 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":698152704}
W20260808 01:52:36.959728  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 44210.6us
I20260808 01:52:36.960762 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:36.961308  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:36.962399 16908 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:36.963897  9524 leader_election.cc:400] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19.
I20260808 01:52:36.965229  9632 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260808 01:52:36.965521  9476 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260808 01:52:36.955628  9472 rpcz_store.cc:276] 0808 01:52:35.198455 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:35.219401 (+ 20946us) service_pool.cc:224] Handling call
0808 01:52:36.509030 (+1289629us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:36.513216 (+  4186us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:36.954891 (+441675us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b540019d2d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:36.955050 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2964929280}
I20260808 01:52:36.969610  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:36.936230 16911 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.134s	user 0.001s	sys 0.001s
I20260808 01:52:36.660269 16763 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:36.935582 16934 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:36.972813  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:36.923190 16749 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:36.864480 16930 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.077s
I20260808 01:52:36.940129 16871 raft_consensus.cc:697] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:36.989882  9524 leader_election.cc:403] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:52:36.977774  9632 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: [15.8-15.8]
W20260808 01:52:37.003551  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:37.004865  9550 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 17 index: 16. Preceding OpId from leader: term: 18 index: 16. (term mismatch)
I20260808 01:52:37.011835  9550 pending_rounds.cc:85] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 15
I20260808 01:52:37.012490  9550 pending_rounds.cc:107] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 17.16
I20260808 01:52:37.012867  9550 raft_consensus.cc:2889] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:37.014993  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:37.016068  9632 raft_consensus.cc:2376] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18. Re-sending same reply.
I20260808 01:52:37.018846  9632 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:36.951244 16290 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:37.057011  9637 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 13. (term mismatch)
I20260808 01:52:37.057482  9637 pending_rounds.cc:85] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 12
I20260808 01:52:37.057714  9637 pending_rounds.cc:107] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 22.13
I20260808 01:52:37.057938  9637 raft_consensus.cc:2889] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:36.976596 16921 raft_consensus.cc:3055] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:36.974839 16530 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:37.070891  9632 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260808 01:52:37.071199  9560 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260808 01:52:37.065797  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:37.068467  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:36.972071 16908 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:36.947117 16912 consensus_queue.cc:1059] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260808 01:52:37.090487  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:37.091444  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 79967.8us
I20260808 01:52:37.072505  9637 raft_consensus.cc:2376] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20. Re-sending same reply.
I20260808 01:52:36.921643 16878 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:37.080864  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:37.085794 16871 consensus_queue.cc:237] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:37.089951 16850 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.765s	user 0.008s	sys 0.004s
I20260808 01:52:37.118757 16850 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:37.121845 16850 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:37.091424  9552 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:37.008793 16194 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:36.976413 16867 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.094s
I20260808 01:52:37.056517 16736 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:37.110541  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:37.059330 16930 consensus_queue.cc:1243] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 15.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
W20260808 01:52:37.125623  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 48558us
I20260808 01:52:37.136721  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:37.141172  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:37.144045  9472 raft_consensus.cc:2368] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19.
W20260808 01:52:37.147635  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 174898us
I20260808 01:52:37.065797 16921 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:37.079000 16530 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:37.149799  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:37.029563 16749 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:37.161368  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:37.157188  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:37.166572  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:37.157341  9637 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:37.101110 16908 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:37.140499 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 147569us
I20260808 01:52:37.092088 16894 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.081s
I20260808 01:52:37.142474 16850 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:37.182178 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 215778us
I20260808 01:52:37.159934 16937 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:37.186259 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:37.186911  9475 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 12. (term mismatch)
I20260808 01:52:37.158015 16945 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:37.187520  9475 pending_rounds.cc:85] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 11
W20260808 01:52:37.188212 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:37.012758 16934 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:37.194228 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:37.196126 10360 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:37.194145 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:37.191061  9475 pending_rounds.cc:107] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20260808 01:52:37.197739  9475 raft_consensus.cc:2889] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:37.167258 16530 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:37.194435  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:37.201444  9637 raft_consensus.cc:2376] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18. Re-sending same reply.
I20260808 01:52:37.200403  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:37.202720  9632 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
W20260808 01:52:37.204448  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 281999us
I20260808 01:52:37.144888 16736 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:37.204718  9445 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:37.206327 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:37.207242  9523 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:37.207610  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 57913.8us
I20260808 01:52:37.186115 16808 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:37.218966  9523 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:37.219838 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:37.220816 10360 raft_consensus.cc:2376] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19. Re-sending same reply.
I20260808 01:52:37.222258 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:37.193109 16945 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:37.195194 16878 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:37.225651  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:37.181615 16834 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
I20260808 01:52:37.207324 16858 raft_consensus.cc:2764] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
W20260808 01:52:37.216885 16870 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.822s	user 0.015s	sys 0.000s
I20260808 01:52:37.188385 16937 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:37.249161  9632 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260808 01:52:37.290956  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:37.292086  9632 raft_consensus.cc:2393] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:37.293047  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:37.250134  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:37.225160 16944 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19.
I20260808 01:52:37.235422 16808 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:37.255447 16570 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:37.295238  9637 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:52:37.261919 16942 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260808 01:52:37.195611 16290 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:37.263080 16922 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:37.265847 16934 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:37.267180 16948 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:37.270512 16749 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:37.161123 16943 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.053s
I20260808 01:52:37.271500  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:37.273411 16935 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.408s	user 0.003s	sys 0.000s
I20260808 01:52:37.278084  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:37.281242 16919 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.450s	user 0.001s	sys 0.000s
I20260808 01:52:37.192902 16850 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:37.240154 16939 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.081s
I20260808 01:52:37.240687 16945 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:37.300763 16763 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:52:37.288393 16645 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.986s	user 0.014s	sys 0.003s
W20260808 01:52:37.313180  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 349430us
W20260808 01:52:37.320837  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 49578.6us
I20260808 01:52:37.309089 16570 raft_consensus.cc:2764] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260808 01:52:37.330572  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:37.299011 16773 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:37.307261 16942 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:37.300225 16753 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:37.323135 16858 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:37.372718 16937 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:37.301380 16720 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:37.304662 16870 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:37.373183  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:37.387488  9634 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
W20260808 01:52:37.392776  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 172043us
I20260808 01:52:37.327488 16948 raft_consensus.cc:697] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 18 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:37.386221  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52106.3us
I20260808 01:52:37.379127 16858 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:37.367997 16942 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:37.319644 16922 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.058s
I20260808 01:52:37.397449  9523 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:37.375087 16836 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.265s	user 0.016s	sys 0.001s
W20260808 01:52:37.404246  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 73750us
I20260808 01:52:37.406067  9474 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 18: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:37.389709 16952 raft_consensus.cc:3055] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:37.413513  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3601) took 1385 ms. Trace:
I20260808 01:52:37.391400 16937 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:37.390013 16773 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:37.365097 16645 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:37.418695  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:37.419901  9524 leader_election.cc:400] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19.
I20260808 01:52:37.440145  9524 leader_election.cc:403] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:52:37.439754  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:37.441817  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:37.416779 16836 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:37.400067 16867 consensus_queue.cc:1243] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260808 01:52:37.380744 16753 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:37.380997 16763 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:37.414255  9554 rpcz_store.cc:276] 0808 01:52:36.028494 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:36.740623 (+712129us) service_pool.cc:224] Handling call
0808 01:52:36.741113 (+   490us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:36.741556 (+   443us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:36.741598 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:36.742420 (+   822us) log.cc:844] Serialized 26 byte log entry
0808 01:52:36.841338 (+ 98918us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:36.841396 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:36.841473 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:37.272613 (+431140us) spinlock_profiling.cc:243] Waited 3.51 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:37.406450 (+133837us) raft_consensus.cc:1685] finished
0808 01:52:37.406488 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:37.413387 (+  6899us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":69697,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":164,"spinlock_wait_cycles":8079232,"thread_start_us":75740,"threads_started":1,"wal-append.queue_time_us":122452}
W20260808 01:52:37.429759 16924 log.cc:927] Time spent T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.461s	user 0.000s	sys 0.002s
I20260808 01:52:37.404234 16858 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:37.456426 10360 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260808 01:52:37.459194  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:37.414075 16952 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:37.413040 16194 raft_consensus.cc:697] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:37.427479 16736 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:37.479293  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:37.433517 16773 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:37.483994  9629 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 23: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:37.487784  9476 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 20: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:52:37.459218  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:37.477484  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:37.481801  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:37.485639  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:37.485944  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:37.464921 16916 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.005s	sys 0.011s
I20260808 01:52:37.490284  9479 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:37.461874 16939 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:37.461127 16944 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:37.523499  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:37.466727 16934 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19.
W20260808 01:52:37.528638  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 51253.3us
I20260808 01:52:37.534859  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:37.545468  9633 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:52:37.534757  9480 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:37.466089 16937 raft_consensus.cc:697] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:37.425429 16749 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.057s
I20260808 01:52:37.545476  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:37.430428 16949 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.105s
W20260808 01:52:37.520556 16916 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.005s	sys 0.012s
I20260808 01:52:37.493786 16773 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:37.679824  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:37.680893  9633 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:52:37.698920  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 301543us
W20260808 01:52:37.705802  9478 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 1169) took 2080 ms (client timeout 1579 ms). Trace:
W20260808 01:52:37.706597  9478 rpcz_store.cc:269] 0808 01:52:35.625062 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:35.625245 (+   183us) service_pool.cc:224] Handling call
0808 01:52:37.705450 (+2080205us) spinlock_profiling.cc:243] Waited 755 ms on lock 0x7b54002c0954. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:37.705678 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1737522048}
I20260808 01:52:37.709718  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:37.681257 16941 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.371s	user 0.000s	sys 0.001s
I20260808 01:52:37.731266  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:37.739519  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:37.600364 16958 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.220s
I20260808 01:52:37.602628 16959 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:37.660223 16916 raft_consensus.cc:697] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:37.796146  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:37.535794 16944 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:37.777164  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3047) took 1327 ms. Trace:
I20260808 01:52:37.742573 16939 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:37.813683  9479 raft_consensus.cc:2410] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 19 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 8.
I20260808 01:52:37.786979  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:37.815068  9478 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:37.815586  9551 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:37.818953 16943 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:37.819775 16943 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:37.790916  9633 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:52:37.714996 16949 consensus_queue.cc:1243] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 19.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260808 01:52:37.709915 16922 raft_consensus.cc:2764] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
W20260808 01:52:37.742239 16896 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
I20260808 01:52:37.833674  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:37.725889 16934 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.088s
W20260808 01:52:37.769330 16530 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.551s	user 0.006s	sys 0.008s
I20260808 01:52:37.909061 16530 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:37.912436 16530 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:37.844869 16954 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.245s	user 0.000s	sys 0.001s
I20260808 01:52:37.847867 16937 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:37.829891 16938 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:37.863196 16956 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:37.865185 16833 consensus_meta.cc:287] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.301s	user 0.016s	sys 0.000s
I20260808 01:52:37.804420 16960 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.062s
I20260808 01:52:37.867084  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:37.938421  9633 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:37.829351 16939 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:37.868804  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:37.940522  9551 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:37.770273  9634 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:37.942230  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3056) took 1288 ms. Trace:
I20260808 01:52:37.942916  9634 rpcz_store.cc:276] 0808 01:52:36.653521 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:36.722078 (+ 68557us) service_pool.cc:224] Handling call
0808 01:52:37.372991 (+650913us) spinlock_profiling.cc:243] Waited 6.42 ms on lock 0x7b5800231698. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:52:37.942138 (+569147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14763776}
I20260808 01:52:37.945487  9634 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260808 01:52:37.871347  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:37.947233  9559 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:37.948937  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:37.950186  9559 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
W20260808 01:52:37.862397 16829 consensus_meta.cc:287] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.142s	user 0.012s	sys 0.002s
I20260808 01:52:37.873160 16968 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:37.877418 16943 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:37.882355  9557 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:37.886752 16753 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:37.888595  9478 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:52:37.888847  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:37.956569 10366 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260808 01:52:37.891690 16858 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:37.898519 16763 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:37.901612 16916 consensus_queue.cc:237] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:37.803494 16194 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:37.794147 16959 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:37.907203 16720 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:37.811985  9635 rpcz_store.cc:276] 0808 01:52:36.449830 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:36.450008 (+   178us) service_pool.cc:224] Handling call
0808 01:52:36.457235 (+  7227us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:36.457761 (+   526us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:36.457804 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:36.500088 (+ 42284us) log.cc:844] Serialized 26 byte log entry
0808 01:52:36.683632 (+183544us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:36.683692 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:36.683794 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:37.274193 (+590399us) spinlock_profiling.cc:243] Waited 2.02 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:37.726943 (+452750us) raft_consensus.cc:1685] finished
0808 01:52:37.726978 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:37.777025 (+ 50047us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13140,"raft.queue_time_us":3282,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":4399,"spinlock_wait_cycles":4640128,"thread_start_us":110270,"threads_started":1,"wal-append.queue_time_us":249001}
I20260808 01:52:37.913555 16836 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:37.919092 16949 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:37.892166 16645 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:37.608196 16948 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:38.000281  9600 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:37.832769 16850 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:37.951447 16939 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:37.959259  9637 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17. Ops: []
W20260808 01:52:37.954430  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 456427us
W20260808 01:52:38.020920  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:37.960021  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:38.023450  9478 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 21: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:38.024853  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:38.026117  9478 raft_consensus.cc:2410] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 22 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260808 01:52:37.961076  9557 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:37.961791  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:38.028332  9551 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:37.806861 16870 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:37.978449  9634 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: [16.13-17.14]
I20260808 01:52:38.038439  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:38.040407  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:37.937935 16938 raft_consensus.cc:697] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 18 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:37.933506  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:38.016980  9473 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260808 01:52:38.008828  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:38.022423  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:38.035904  9474 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:38.041152  9633 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:52:38.040460  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:38.045214  9524 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:38.047739  9557 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
W20260808 01:52:38.056332  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 56184.4us
I20260808 01:52:38.053750 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:37.941134 16833 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:38.066257  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:38.057598  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:38.066653  9559 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
W20260808 01:52:38.052626 16908 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.866s	user 0.014s	sys 0.000s
I20260808 01:52:38.118197  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:38.119127  9474 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:37.938861 16956 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:38.127020  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3041) took 1682 ms. Trace:
I20260808 01:52:38.128881  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1179) took 1753 ms. Trace:
I20260808 01:52:38.132241  9631 rpcz_store.cc:276] 0808 01:52:36.444314 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:36.445224 (+   910us) service_pool.cc:224] Handling call
0808 01:52:36.463467 (+ 18243us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:36.628649 (+165182us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:38.126701 (+1498052us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b540003b6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:38.126943 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":363824512}
I20260808 01:52:38.132957  9481 rpcz_store.cc:276] 0808 01:52:36.375434 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:36.375620 (+   186us) service_pool.cc:224] Handling call
0808 01:52:36.389595 (+ 13975us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:36.593622 (+204027us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:38.128587 (+1534965us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54002bacd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:38.128810 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":384467712}
I20260808 01:52:37.957111 16968 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:38.137223 16945 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.770s	user 0.013s	sys 0.000s
I20260808 01:52:38.138244 16968 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:38.138952  9633 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:38.151448 16970 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.220s	user 0.001s	sys 0.000s
I20260808 01:52:38.154994 16952 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.16, Last appended by leader: 16, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:38.156442 16952 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:38.163512  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:38.173988  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:52:38.175118 10368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 1109) took 1691 ms (client timeout 1690 ms). Trace:
W20260808 01:52:38.175722 10368 rpcz_store.cc:269] 0808 01:52:36.483905 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:36.485233 (+  1328us) service_pool.cc:224] Handling call
0808 01:52:38.174774 (+1689541us) spinlock_profiling.cc:243] Waited 45.6 ms on lock 0x7b54004c3454. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:38.175008 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":104847232}
I20260808 01:52:38.178277 16930 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:38.178841 16930 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:38.178473 16938 consensus_queue.cc:237] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:38.179910 16930 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:38.181062 16930 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
W20260808 01:52:38.181155 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:38.183880 16858 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:38.054091 16850 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:38.184815 16850 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:38.185357  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:38.185828  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:38.186700  9559 raft_consensus.cc:2376] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19. Re-sending same reply.
I20260808 01:52:38.187239 16850 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:38.188129  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:38.060504 16958 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:38.189837 16958 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:38.190255 16958 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:38.190291  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:38.192739  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:38.195959 16959 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:38.055931 16530 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:38.197074 16530 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:52:38.199466 16530 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:38.130146 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:38.038748 16949 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:38.208029  9556 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260808 01:52:38.208731  9636 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 23: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:38.186053  9473 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:38.056277 16749 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:38.158223 16808 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:38.134712  9551 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:52:38.209219 10367 raft_consensus.cc:2410] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260808 01:52:38.153543 16921 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:38.137918 16945 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:38.234227 16878 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.988s	user 0.007s	sys 0.010s
I20260808 01:52:38.228601  9552 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
W20260808 01:52:38.254406  9601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.010s	sys 0.047s
W20260808 01:52:38.271394  9601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.011s	sys 0.047s
W20260808 01:52:38.272095  9601 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 2.170s (SENT) [suppressed 24 similar messages]
I20260808 01:52:38.212752  9633 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260808 01:52:38.261808 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:38.261857  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:38.274204  9473 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:38.262969 10360 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: [18.16-18.16]
I20260808 01:52:38.275904  9478 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:38.229241 16749 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:38.276333  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 102453us
I20260808 01:52:38.203578 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:38.207520 16713 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.054s
I20260808 01:52:38.252640 16968 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260808 01:52:38.285846 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 82397.9us
W20260808 01:52:38.277697  9475 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.008s	sys 0.004s
W20260808 01:52:38.288458  9475 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.009s	sys 0.005s
I20260808 01:52:38.278667 10361 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEARNER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
W20260808 01:52:38.293128  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.780s	user 0.005s	sys 0.012s
W20260808 01:52:38.293548  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.781s	user 0.005s	sys 0.012s
W20260808 01:52:38.293987  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 781506us
I20260808 01:52:38.132607 16908 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:38.307197 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:38.307559  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:38.272943 16921 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:38.313274 16944 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:38.316039 16949 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:38.317401 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:38.277920  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.005s	sys 0.008s
I20260808 01:52:38.320894 10332 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:38.288671 16850 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:38.273607 16977 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:38.324828  9521 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:38.284435 16749 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:38.287851 16962 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260808 01:52:38.345288  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.006s	sys 0.008s
W20260808 01:52:38.363117  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 817730us
I20260808 01:52:38.359457  9521 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:38.365046  9521 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:38.321307  9474 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:38.344749 16713 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:38.372612  9521 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:38.390815  9521 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:38.329656 16943 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:38.352985 16829 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260808 01:52:38.376451  9598 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.001s	sys 0.014s
W20260808 01:52:38.402321  9598 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.002s	sys 0.015s
W20260808 01:52:38.376748  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 104699us
I20260808 01:52:38.394484  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:38.342833 16904 raft_consensus.cc:3055] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:38.341768 16819 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:38.400524  9474 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260808 01:52:38.368693 16949 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:38.366909 16962 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:38.411756  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:38.370908 16983 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:38.353358 16905 consensus_queue.cc:1059] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260808 01:52:38.356890 16763 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.053s
I20260808 01:52:38.389506 16944 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:38.424365  9474 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:38.406055 16922 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election won for term 16
W20260808 01:52:38.445393 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 93034.3us
I20260808 01:52:38.401875 16829 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:38.462527 10332 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:38.464075  9560 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: [18.10-20.11]
I20260808 01:52:38.486203 16713 raft_consensus.cc:697] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:38.469336  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:38.405887 16904 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:38.470094  9558 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:38.367758 16977 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:38.476217  9557 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260808 01:52:38.477105  9601 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:38.421208 16962 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:38.482455  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:38.483315  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:38.500276  9552 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:38.397002 16850 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.037s
I20260808 01:52:38.504557 10366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 1107) took 2021 ms. Trace:
I20260808 01:52:38.505201 10366 rpcz_store.cc:276] 0808 01:52:36.483068 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:36.483446 (+   378us) service_pool.cc:224] Handling call
0808 01:52:36.511866 (+ 28420us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:37.957165 (+1445299us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:38.504175 (+547010us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b5400594054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:38.504455 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3321382656}
I20260808 01:52:38.466782  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:38.507290  9551 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 18: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:38.428738 16878 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:38.509727 16878 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:38.510304 16878 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:38.515398  9630 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:38.516408  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2904) took 1986 ms. Trace:
I20260808 01:52:38.516898  9630 rpcz_store.cc:276] 0808 01:52:36.530016 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:36.863917 (+333901us) service_pool.cc:224] Handling call
0808 01:52:36.864231 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:38.515146 (+1650915us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b5400039ba0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490b1a0 00007f5d449955a0 00007f5d44996611 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c
0808 01:52:38.516038 (+   892us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:38.516325 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3813248}
I20260808 01:52:38.406643 16785 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.066s
W20260808 01:52:38.486873  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 77915.3us
I20260808 01:52:38.406643 16943 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:38.519287  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:38.520375 16943 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:38.521361 16943 raft_consensus.cc:2764] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election decision vote started in defunct term 20: won
I20260808 01:52:38.520445 16570 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.003s
I20260808 01:52:38.008726  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:52:38.537237  9368 catalog_manager.cc:5697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 9 to 14, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 14 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:52:38.478611 16829 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 12 FOLLOWER]: Advancing to term 13
W20260808 01:52:38.553186  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 53837.4us
I20260808 01:52:38.473520 16973 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:38.418249 16819 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:38.582854 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 120477us
I20260808 01:52:38.480370 16808 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:38.493746 16942 consensus_meta.cc:287] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.080s	user 0.008s	sys 0.008s
I20260808 01:52:38.582585  9524 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:38.588703  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:38.485126 16922 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:38.579926  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:38.590996 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:38.474036 16867 consensus_queue.cc:1059] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.107s
I20260808 01:52:38.610826 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:38.493120 16833 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:38.493104 16836 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:38.475157 16959 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.106s
I20260808 01:52:38.610970 10364 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260808 01:52:38.608707  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 131758us
I20260808 01:52:38.579423 16858 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:38.491591 16944 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.113s
I20260808 01:52:38.605194 16960 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:38.615535 16971 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260808 01:52:38.581384 16992 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:38.611721 16530 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:38.607867 16943 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:38.654547  9601 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:38.655782  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 136570us
I20260808 01:52:38.611239 16973 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:38.662863  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:38.670322 17008 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:38.658368 16938 raft_consensus.cc:3055] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Stepping down as leader of term 15
W20260808 01:52:38.671032 16995 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.177s	user 0.001s	sys 0.000s
I20260808 01:52:38.671341 16938 raft_consensus.cc:740] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:38.657619 16595 consensus_queue.cc:1059] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.018s
I20260808 01:52:38.663166  9521 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:38.633008 16806 raft_consensus.cc:3055] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:38.637070 16908 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:38.603943 16570 consensus_queue.cc:1059] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.047s
I20260808 01:52:38.647240 16763 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election won for term 13
I20260808 01:52:38.643332 16944 consensus_queue.cc:1243] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 20.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260808 01:52:38.607390 16808 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:38.660260 16943 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:38.700708 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 94100.3us
I20260808 01:52:38.607817 16942 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:38.673717 16974 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:38.655303 16945 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:38.657073 16949 raft_consensus.cc:697] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:38.682626  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:38.698920 16978 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 14.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.315s
I20260808 01:52:38.633919 16836 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:38.685765  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:38.658769 16956 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:38.698170 16976 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.143s	user 0.002s	sys 0.000s
I20260808 01:52:38.646754 16858 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:38.698601 16773 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.988s	user 0.017s	sys 0.000s
W20260808 01:52:38.752712  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 70167.1us
I20260808 01:52:38.672737 16938 consensus_queue.cc:260] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:38.771669  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:38.684414 16785 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:38.684072 16806 raft_consensus.cc:740] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:38.684324 17008 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:38.737732 16943 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:38.699018 16692 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.070s
I20260808 01:52:38.742303 16763 raft_consensus.cc:697] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:38.761175 16833 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:52:38.776170  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:38.665819 16713 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:38.777817  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 92147.1us
I20260808 01:52:38.655889 16960 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:38.649749 16992 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:38.767331 16958 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.576s	user 0.015s	sys 0.000s
I20260808 01:52:38.781845  9474 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:38.787734  9632 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:38.700647 16993 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.363s
I20260808 01:52:38.794245  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:38.791963  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:38.797017 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:38.798599  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:38.804126 16990 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:38.806557 16977 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:38.783277 16973 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:38.812522 16942 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:38.815142 16994 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.234s	user 0.002s	sys 0.000s
W20260808 01:52:38.819787 17000 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
I20260808 01:52:38.780718 16956 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:38.780354 16974 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:38.824101 16958 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:38.826124  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:38.828166 16808 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:38.828485 16858 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:38.829766  9559 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 18: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:38.743443 16984 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.361s
I20260808 01:52:38.835419 16806 consensus_queue.cc:260] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:38.839973  9635 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260808 01:52:38.836048 16984 raft_consensus.cc:3055] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:38.836088  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:38.825417 16974 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:38.867048 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:38.775905 16785 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:38.860924  9634 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:38.856675 16194 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election won for term 19
W20260808 01:52:38.837054 16991 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.184s	user 0.001s	sys 0.001s
I20260808 01:52:38.836980 16938 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:38.870918  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:38.872769  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:38.870918  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:38.895809  9637 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
W20260808 01:52:38.897660  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:38.897704  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:38.900297  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:38.900832 16993 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:38.901221  9637 raft_consensus.cc:2376] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18. Re-sending same reply.
I20260808 01:52:38.901556 16993 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:38.877158  9480 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.007s	sys 0.008s
I20260808 01:52:38.876187  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:38.860807  9630 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 18: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:38.872061  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:38.775553 16773 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:38.746323 16922 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:38.837162 16806 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:38.862587 16944 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:38.906009  9480 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.008s	sys 0.008s
I20260808 01:52:38.860584 16984 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:38.903296  9634 raft_consensus.cc:2376] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18. Re-sending same reply.
I20260808 01:52:38.930219  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:38.865901 16996 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:38.907522  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:38.971577  9637 raft_consensus.cc:2376] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20. Re-sending same reply.
W20260808 01:52:38.942978  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:38.879897 16942 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:38.950157  9521 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:38.951293  9476 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 20: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:38.958093  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 1000) took 2722 ms. Trace:
I20260808 01:52:38.958947 16785 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:38.960472 17015 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:38.961558  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 3235) took 1496 ms. Trace:
I20260808 01:52:38.984056  9478 rpcz_store.cc:276] 0808 01:52:37.465302 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:38.040202 (+574900us) service_pool.cc:224] Handling call
0808 01:52:38.961304 (+921102us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b54002bb6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:38.961457 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":525755136}
I20260808 01:52:38.985996  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:38.986948  9478 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:38.893209 16720 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.073s
W20260808 01:52:38.963158  9373 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50068 (request call id 239) took 68535 ms (client timeout 15000 ms). Trace:
I20260808 01:52:38.953075  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:38.908877 10364 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260808 01:52:39.000962  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:38.976158  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:38.976812  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:38.973392  9550 rpcz_store.cc:276] 0808 01:52:36.235176 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:36.778137 (+542961us) service_pool.cc:224] Handling call
0808 01:52:36.778750 (+   613us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:37.055480 (+276730us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:38.957590 (+1902110us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b540029efd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:38.957926 (+   336us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":453191936}
I20260808 01:52:38.961848 16990 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:38.908987 16194 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:38.947789 16819 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:38.925050 16977 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
W20260808 01:52:39.003245  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 50262.5us
I20260808 01:52:38.943148  9368 catalog_manager.cc:5697] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 11 to 15, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 15 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:52:39.027504 16871 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:39.027940  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:39.028746  9479 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:39.028403  9368 catalog_manager.cc:5697] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 15 to 17. New cstate: current_term: 17 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:39.029336 16871 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260808 01:52:39.031003 17023 raft_consensus.cc:3055] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Stepping down as leader of term 17
W20260808 01:52:39.031193 16997 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.335s	user 0.000s	sys 0.002s
I20260808 01:52:39.031556 17023 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:38.892808 16949 consensus_queue.cc:237] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.032711 17023 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:39.033896  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:39.031265  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:39.035367 16931 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:39.036072 16931 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.036540 10365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 895) took 1853 ms. Trace:
I20260808 01:52:39.036966  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:38.983088 16996 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:39.038864 16961 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:39.039140 16985 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:39.039674 16996 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:39.040092 16985 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:39.040596 16985 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:38.875708 16904 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.040951 16985 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:39.041824 16904 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:39.004246  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:38.892971 17012 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.087s
I20260808 01:52:39.043056  9635 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:52:39.044417  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:39.045009  9635 raft_consensus.cc:2376] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18. Re-sending same reply.
I20260808 01:52:39.046056  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:39.047832 16960 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:39.048609 16720 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260808 01:52:39.049377  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:39.050287  9635 raft_consensus.cc:2393] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:39.009286  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3271) took 1575 ms. Trace:
I20260808 01:52:38.989097 16942 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:39.051179  9475 rpcz_store.cc:276] 0808 01:52:37.434159 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:37.434419 (+   260us) service_pool.cc:224] Handling call
0808 01:52:37.467968 (+ 33549us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:37.476024 (+  8056us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:37.476068 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:37.522750 (+ 46682us) log.cc:844] Serialized 26 byte log entry
0808 01:52:38.288994 (+766244us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:38.289041 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:38.289114 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:39.009035 (+719921us) raft_consensus.cc:1685] finished
0808 01:52:39.009059 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:39.009217 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":245335,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":71063,"thread_start_us":588445,"threads_started":1,"wal-append.queue_time_us":827116}
I20260808 01:52:39.053071  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:39.012424  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:39.000283 17015 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:38.971515 16570 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.046s
I20260808 01:52:39.055123  9475 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
W20260808 01:52:39.057155 16878 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.546s	user 0.011s	sys 0.004s
I20260808 01:52:39.057698  9550 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260808 01:52:39.012060 16916 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260808 01:52:39.058234 16952 consensus_meta.cc:287] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.901s	user 0.017s	sys 0.001s
I20260808 01:52:38.964700 17007 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:39.058915 17007 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:39.059372 17007 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:39.059896  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:38.964847 16988 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:39.053129 16942 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:39.054433  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:39.061089  9550 raft_consensus.cc:2410] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260808 01:52:39.084631  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:39.086077  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:38.941979 16944 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.086885  9368 catalog_manager.cc:5697] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 12 to 16, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 16 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:39.088864 16944 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:39.039618 16961 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.062752  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:38.980477 17010 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:39.071412  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:39.034094 17023 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:39.114004  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3275) took 1645 ms. Trace:
I20260808 01:52:39.004714  9478 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:39.015319 16785 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:39.127588 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:38.989880 16934 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:39.036983 10365 rpcz_store.cc:276] 0808 01:52:37.183207 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:37.185049 (+  1842us) service_pool.cc:224] Handling call
0808 01:52:38.182869 (+997820us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:38.207881 (+ 25012us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:38.207915 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:38.208517 (+   602us) log.cc:844] Serialized 26 byte log entry
0808 01:52:38.431161 (+222644us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:38.431218 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:38.431299 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:38.837682 (+406383us) raft_consensus.cc:1685] finished
0808 01:52:38.837707 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:39.036228 (+198521us) spinlock_profiling.cc:243] Waited 997 ms on lock 0x7b54004c3450. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:39.036470 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24477,"raft.queue_time_us":30522,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":11751,"spinlock_wait_cycles":2293789440,"thread_start_us":142059,"threads_started":1,"wal-append.queue_time_us":363089}
I20260808 01:52:39.075093  9554 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 19 index: 16. (term mismatch)
I20260808 01:52:39.127393  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:39.132323  9554 pending_rounds.cc:85] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 15
I20260808 01:52:39.133004  9554 pending_rounds.cc:107] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 18.16
I20260808 01:52:39.064370 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:39.077979 16836 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:39.078444 17024 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260808 01:52:39.078805 16194 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.038488 16931 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:39.011531 16989 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:39.114722  9480 rpcz_store.cc:276] 0808 01:52:37.468696 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:38.317416 (+848720us) service_pool.cc:224] Handling call
0808 01:52:38.317807 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:38.318110 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 14
0808 01:52:38.923869 (+605759us) log.cc:844] Serialized 14 byte log entry
0808 01:52:39.113238 (+189369us) log.cc:844] Serialized 14 byte log entry
0808 01:52:39.113464 (+   226us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:39.113525 (+    61us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:52:39.113623 (+    98us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:39.113784 (+   161us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:39.113934 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7212,"prepare.queue_time_us":315926,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"thread_start_us":786522,"threads_started":2}
I20260808 01:52:39.120590  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:39.005983  9521 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:39.133278  9554 raft_consensus.cc:2889] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:39.128562 10367 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
W20260808 01:52:39.136495 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 72268.7us
I20260808 01:52:39.169525 10332 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:39.172952  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 167025us
I20260808 01:52:39.173652  9550 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:52:39.057724 16878 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:38.963212 16763 consensus_queue.cc:237] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:39.132287 17010 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:39.164520  9635 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:39.089514  9368 catalog_manager.cc:5697] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index -1 to 9, term changed from 10 to 15, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194), NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 15 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:52:39.157912 16889 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:52:39.155742 16989 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:39.186866 10334 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.011s	sys 0.010s
I20260808 01:52:39.175801 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:39.139171 16959 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:39.131134 16934 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:39.176316  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:39.193425  9550 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:39.137717  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:39.004225 16962 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.138636 16850 raft_consensus.cc:3055] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:39.016248 16990 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.182662  9599 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:39.186664  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:39.187443 10334 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.011s	sys 0.011s
W20260808 01:52:39.201716 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 660026us
I20260808 01:52:39.178732 16922 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:39.175598  9521 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:39.156458 16836 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:39.204188 16836 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:39.204537  9521 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:39.206259 16952 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:39.131433 16871 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
I20260808 01:52:38.972086 17008 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:39.127844 16916 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 17.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.051s
I20260808 01:52:39.182999 17012 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:39.194641  9475 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
W20260808 01:52:39.184643 16930 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.003s	user 0.017s	sys 0.001s
I20260808 01:52:39.225581  9478 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:39.207417  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:39.199237  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 65156.8us
I20260808 01:52:39.233884  9479 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:39.264595  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:39.265815  9479 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:39.185683 16889 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:39.269775  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:39.189529 16989 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:39.272608  9478 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:52:39.199826 16713 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:39.280037  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:39.197502 16934 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:39.209074 16962 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:39.206403 16836 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:39.197186 16850 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:39.207149 16942 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:39.292574  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:39.209199 16993 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:39.188053 16988 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.091s
I20260808 01:52:39.263453 16922 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:39.281092 17015 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:39.276836 16889 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:52:39.216588 16998 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:52:39.291236  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:52:39.351595  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 60449us
I20260808 01:52:39.202999 16959 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:39.266816 16944 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:39.333247  9475 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260808 01:52:39.206907 16952 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:39.280246 16992 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:39.286782 16749 consensus_meta.cc:287] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.901s	user 0.009s	sys 0.007s
I20260808 01:52:39.337589 16977 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260808 01:52:39.001227  9373 rpcz_store.cc:269] 0808 01:51:30.427110 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:30.434258 (+  7148us) service_pool.cc:224] Handling call
0808 01:52:38.962797 (+68528539us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:52:26.535215 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:52:26.535998 (+   783us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:52:26.536063 (+    65us) write_op.cc:435] Acquired schema lock
0808 01:52:26.536081 (+    18us) tablet.cc:662] Decoding operations
0808 01:52:26.537293 (+  1212us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:52:26.537516 (+   223us) write_op.cc:641] Partition lock acquired for write op
0808 01:52:26.537547 (+    31us) tablet.cc:685] Acquiring locks for 11 operations
0808 01:52:26.743252 (+205705us) tablet.cc:701] Row locks acquired
0808 01:52:26.743279 (+    27us) write_op.cc:260] PREPARE: finished
0808 01:52:26.743564 (+   285us) write_op.cc:270] Start()
0808 01:52:26.743801 (+   237us) write_op.cc:276] Timestamp: P: 1786153946743467 usec, L: 0
0808 01:52:26.743848 (+    47us) op_driver.cc:348] REPLICATION: starting
0808 01:52:26.744824 (+   976us) log.cc:844] Serialized 4422 byte log entry
0808 01:52:27.857067 (+1112243us) op_driver.cc:464] REPLICATION: finished
0808 01:52:28.274311 (+417244us) write_op.cc:301] APPLY: starting
0808 01:52:28.308252 (+ 33941us) tablet.cc:1367] starting BulkCheckPresence
0808 01:52:29.009075 (+700823us) spinlock_profiling.cc:243] Waited 4.04 ms on lock 0x7b180032aa60. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188ce62 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:52:29.167733 (+158658us) tablet.cc:1370] finished BulkCheckPresence
0808 01:52:29.167780 (+    47us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:52:29.990182 (+822402us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:52:30.289417 (+299235us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=22,key_file_lookups=22,delta_file_lookups=22,mrs_lookups=0
0808 01:52:30.316560 (+ 27143us) write_op.cc:312] APPLY: finished
0808 01:52:30.370078 (+ 53518us) log.cc:844] Serialized 105 byte log entry
0808 01:52:30.665616 (+295538us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:52:30.690773 (+ 25157us) write_op.cc:454] Released schema lock
0808 01:52:30.750766 (+ 59993us) write_op.cc:341] FINISH: Updating metrics
0808 01:52:30.939830 (+189064us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:52:30.947151 (+  7321us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":28638,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":359608,"apply.run_cpu_time_us":76489,"apply.run_wall_time_us":2589917,"cfile_cache_hit":46,"cfile_cache_hit_bytes":50173,"num_ops":11,"prepare.queue_time_us":373883,"prepare.run_cpu_time_us":59802,"prepare.run_wall_time_us":495722,"raft.queue_time_us":493442,"raft.run_cpu_time_us":9135,"raft.run_wall_time_us":191357,"replication_time_us":1112997,"spinlock_wait_cycles":9286528,"thread_start_us":1242554,"threads_started":5,"wal-append.queue_time_us":1001009,"wal-append.run_cpu_time_us":52103,"wal-append.run_wall_time_us":2713248}]]}
W20260808 01:52:39.354489  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:39.340241 16921 consensus_meta.cc:287] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.996s	user 0.012s	sys 0.004s
I20260808 01:52:39.355911 16962 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260808 01:52:39.361785 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 145672us
I20260808 01:52:39.375869  9475 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260808 01:52:39.342026 16942 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:39.318118 16871 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260808 01:52:39.270869 16931 consensus_queue.cc:1059] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.079s
W20260808 01:52:39.360646 16829 consensus_meta.cc:287] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.772s	user 0.014s	sys 0.004s
I20260808 01:52:39.374276 16998 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:52:39.394158  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 78172.7us
I20260808 01:52:39.360625 16806 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:39.406627  9444 leader_election.cc:336] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.990s (SENT)
I20260808 01:52:39.407253  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:39.372726 16962 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:39.288152 17010 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 17.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.409325  9478 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 18 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:39.392269 16978 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:39.422891  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:39.361694 16944 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.428613 16977 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:39.431283  9477 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 15 index: 15. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260808 01:52:39.432180  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:39.433559 17034 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260808 01:52:39.435655  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 3231) took 1972 ms. Trace:
I20260808 01:52:39.414636 16962 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:39.436803  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:39.442351  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:39.436828  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:39.450392  9560 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:39.365080 16749 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:39.404160 16961 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:39.421770 17025 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.250s	user 0.000s	sys 0.001s
I20260808 01:52:39.330344  9368 catalog_manager.cc:5697] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 11 to 21. New cstate: current_term: 21 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:39.434783 16942 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.452647  9552 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 12. (term mismatch)
I20260808 01:52:39.491606  9552 pending_rounds.cc:85] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 11
I20260808 01:52:39.491982  9552 pending_rounds.cc:107] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260808 01:52:39.492235  9552 raft_consensus.cc:2889] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:39.450953  9634 raft_consensus.cc:2376] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21. Re-sending same reply.
I20260808 01:52:39.495689  9634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "17e220e6a25747939bdf18f26b732407"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:39.496378  9634 raft_consensus.cc:606] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:39.496724  9634 raft_consensus.cc:614] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:39.422729 16931 consensus_queue.cc:1243] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 15.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260808 01:52:39.448203  9472 rpcz_store.cc:276] 0808 01:52:37.463665 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:37.472202 (+  8537us) service_pool.cc:224] Handling call
0808 01:52:39.435405 (+1963203us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b5400392bd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:39.435582 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3304597760}
I20260808 01:52:39.458992  9558 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:39.425439 16978 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.449335 16992 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260808 01:52:39.463295 16939 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.408s	user 0.010s	sys 0.006s
I20260808 01:52:39.408612 16829 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:39.446586 16878 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:39.464210 16958 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:39.396342 16850 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:39.519169  9556 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:39.449208 16952 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.422788 17012 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.035s
I20260808 01:52:39.476648  9368 catalog_manager.cc:5697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 12 to 15. New cstate: current_term: 15 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:39.412124 16806 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.449592 16977 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:39.454631 17030 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
W20260808 01:52:39.505432  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:39.502333  9472 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260808 01:52:39.499879  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 93279.3us
I20260808 01:52:39.506315  9560 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:39.513378  9555 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 18. Current term is 19. Ops: [18.16-18.16]
W20260808 01:52:39.514222 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 100857us
I20260808 01:52:39.530087 10334 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:39.531862 16836 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:39.532508 16836 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.536216  9560 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260808 01:52:39.546733  9521 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:39.502868  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 17e220e6a25747939bdf18f26b732407 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:39.563452 16773 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:39.565127 17033 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260808 01:52:39.571885  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:39.572676  9630 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:39.576843 16829 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:39.420084 16922 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.582789  9368 catalog_manager.cc:5697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:39.585469 16934 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:39.554090 16943 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.762s	user 0.006s	sys 0.009s
I20260808 01:52:39.535113 16836 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:39.453388 16959 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.529440 16990 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:39.584967  9368 catalog_manager.cc:5697] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 10 to 14. New cstate: current_term: 14 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:39.553853 17012 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:39.504033 16939 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.616214  9630 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:39.606114  9558 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:39.587790 16958 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:39.627569  9368 catalog_manager.cc:5697] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 13 to 17. New cstate: current_term: 17 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:39.632942 16939 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:39.633553  9600 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:39.634335  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:39.635821  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:39.635982 16977 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:39.636440  9634 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:52:39.636593 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:39.637319 16785 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.502s	user 0.009s	sys 0.001s
I20260808 01:52:39.638141 16810 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:39.639132 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:39.639209 16810 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:39.629853  9368 catalog_manager.cc:5697] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:39.642051  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 3238) took 1159 ms. Trace:
W20260808 01:52:39.607967 16917 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 2.758s	user 0.011s	sys 0.015s
I20260808 01:52:39.637491  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:39.610882 17007 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.551s	user 0.010s	sys 0.004s
I20260808 01:52:39.643586 17007 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.644104  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:39.644825  9634 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:39.633037 16934 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:39.641741  9368 catalog_manager.cc:5697] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 9 to 12, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 12 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:52:39.648464  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2914) took 1675 ms. Trace:
I20260808 01:52:39.648921  9633 rpcz_store.cc:276] 0808 01:52:37.972992 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:38.047358 (+ 74366us) service_pool.cc:224] Handling call
0808 01:52:38.047752 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:38.273294 (+225542us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:39.648173 (+1374879us) spinlock_profiling.cc:243] Waited 91 ms on lock 0x7b54000df054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:39.648400 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":209390976}
I20260808 01:52:39.650260 16806 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:39.554288 16998 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.654443 16820 raft_consensus.cc:3055] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:39.655023 16820 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:39.655902 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:39.656009 16820 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.657311 16820 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:39.659399 16858 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:39.659858 16858 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:39.660780 16858 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.607668  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:39.638070 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:39.665227 10367 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:52:39.648643 17040 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260808 01:52:39.637423 10360 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:39.603966  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2780) took 1135 ms. Trace:
I20260808 01:52:39.642460  9474 rpcz_store.cc:276] 0808 01:52:38.482968 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:38.550425 (+ 67457us) service_pool.cc:224] Handling call
0808 01:52:39.641860 (+1091435us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b5400404cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:39.641986 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":442194688}
I20260808 01:52:39.618738  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:39.646279 17007 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:39.636080 10361 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260808 01:52:39.650425  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:39.669953  9600 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:39.673748  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:39.671622  9637 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 18: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:39.619671  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:39.671620  9557 rpcz_store.cc:276] 0808 01:52:38.468851 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:38.470636 (+  1785us) service_pool.cc:224] Handling call
0808 01:52:38.471058 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:38.498401 (+ 27343us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:39.603600 (+1105199us) spinlock_profiling.cc:243] Waited 1.78 ms on lock 0x7b5400383954. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:39.603886 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4093184}
W20260808 01:52:39.676712  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 58144.5us
I20260808 01:52:39.677999 16858 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:39.662540 16989 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.687817  9550 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 21: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:39.681048  9634 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:39.616050 17012 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:39.690550  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:39.684075  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:39.691970  9634 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:39.686741  9630 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 18: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:39.693390  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:39.694316  9634 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:52:39.692816  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:39.702190  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:39.699982  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:39.675832 16990 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:52:39.600631 16943 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.676079 16958 raft_consensus.cc:697] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:39.679358 16962 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:39.773519 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:39.850365 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:39.775797  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:39.774048  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2763) took 1960 ms. Trace:
W20260808 01:52:39.778132  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 76046.9us
W20260808 01:52:39.788808  9476 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 3277) took 2303 ms (client timeout 1643 ms). Trace:
I20260808 01:52:39.913919  9554 rpcz_store.cc:276] 0808 01:52:37.814002 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:37.814206 (+   204us) service_pool.cc:224] Handling call
0808 01:52:37.848582 (+ 34376us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:39.166610 (+1318028us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:39.773763 (+607153us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54001ac554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:39.773978 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2787811072}
I20260808 01:52:39.914475  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:39.918218  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:39.835908  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:39.772909  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:39.741467  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:39.789255  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:39.920387  9637 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:39.789209  9475 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:39.835112  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:39.787493  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2779) took 1319 ms. Trace:
I20260808 01:52:39.921815  9559 rpcz_store.cc:276] 0808 01:52:38.468464 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:38.468989 (+   525us) service_pool.cc:224] Handling call
0808 01:52:39.787205 (+1318216us) spinlock_profiling.cc:243] Waited 334 ms on lock 0x7b5400325fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:39.787403 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":768101760}
I20260808 01:52:39.834962  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:39.841158  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 172656us
I20260808 01:52:39.685500 16921 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:52:39.747329  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:39.705857  9634 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:39.776274 10334 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:39.914801  9476 rpcz_store.cc:269] 0808 01:52:37.485056 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:37.523072 (+ 38016us) service_pool.cc:224] Handling call
0808 01:52:39.788501 (+2265429us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b540038ccd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:39.788707 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3271933312}
I20260808 01:52:39.925160  9476 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []
W20260808 01:52:39.870775 16985 consensus_meta.cc:287] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.829s	user 0.012s	sys 0.002s
I20260808 01:52:39.846302  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:39.706110 16939 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:39.696782 17012 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:39.835369  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:39.923616  9474 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:39.789628  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:39.737859 16806 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:39.926120  9473 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 18: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:39.904856  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:39.926045  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 79845.8us
I20260808 01:52:39.963372  9477 raft_consensus.cc:2410] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 20 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
W20260808 01:52:39.928956 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 152777us
I20260808 01:52:39.966912 16852 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
W20260808 01:52:39.930138  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 182874us
I20260808 01:52:39.972723  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:39.973634  9481 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:39.929924  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:39.758832 16937 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:39.835024 16773 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:39.938175  9557 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 21: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:39.856427 16972 consensus_queue.cc:1059] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.085s
I20260808 01:52:39.704445 16834 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.211s
I20260808 01:52:39.939093 16921 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:39.933570 16867 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:39.942906 16943 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:39.871563 16904 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.829s	user 0.016s	sys 0.000s
I20260808 01:52:39.944363 16878 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20260808 01:52:40.058470  9636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 885) took 2808 ms (client timeout 1579 ms). Trace:
W20260808 01:52:40.059000  9636 rpcz_store.cc:269] 0808 01:52:37.249994 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:37.277740 (+ 27746us) service_pool.cc:224] Handling call
0808 01:52:40.058241 (+2780501us) spinlock_profiling.cc:243] Waited 884 ms on lock 0x7b540025bcd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:40.058391 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2033975680}
I20260808 01:52:40.060379  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:40.061105  9636 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:40.062153  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:39.647848  9368 catalog_manager.cc:5697] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:39.947211 16983 raft_consensus.cc:3055] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Stepping down as leader of term 20
W20260808 01:52:39.947546 16889 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.589s	user 0.002s	sys 0.011s
W20260808 01:52:39.949106 17049 log.cc:927] Time spent T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260808 01:52:39.952001 16962 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:39.952811  9554 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: []
W20260808 01:52:39.952963 17023 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.824s	user 0.010s	sys 0.003s
I20260808 01:52:39.934064  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:39.953440 16988 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:40.117445  9480 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260808 01:52:39.954146 17046 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:39.934983 16958 consensus_queue.cc:237] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.955892 16833 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260808 01:52:39.956830 17048 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260808 01:52:39.956895  9634 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:52:39.928684 16993 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:39.959419  9637 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:39.962011 10367 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:52:39.816448 17007 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:39.816689 16977 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:39.706188 16920 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.229s
I20260808 01:52:39.751706 16944 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:39.975409  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 55073us
I20260808 01:52:39.934420 16985 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:40.140136  9523 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:40.105268 16983 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:39.984716 16937 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:39.935460 16998 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:39.932751 16939 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:39.980245  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:39.984076  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:40.158479  9473 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:40.153079 17007 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:40.159530  9474 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260808 01:52:40.159837  9473 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260808 01:52:39.987054  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:40.139564 16993 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:40.063853  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:40.161463 17007 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:40.054402 17010 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.007s	sys 0.038s
I20260808 01:52:40.161975  9636 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
W20260808 01:52:40.162081 17010 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.007s	sys 0.039s
I20260808 01:52:40.117137  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:39.979674 16929 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.123s
I20260808 01:52:40.125419 16833 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:40.163832 16833 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:40.060016 16867 raft_consensus.cc:697] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:39.983211 17016 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.130s
I20260808 01:52:40.102906 16836 raft_consensus.cc:3055] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:40.126282  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:40.172581  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:40.173455  9634 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 21: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:40.133630  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:39.977628 16806 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:39.988267 16852 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.128s
I20260808 01:52:40.076285 16943 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:40.142401  9633 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
W20260808 01:52:40.065126  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 81605.7us
I20260808 01:52:40.162978 16993 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:40.170873 16836 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:40.186640 16836 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:40.187947 16836 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:40.169726  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:40.128118 16975 raft_consensus.cc:2764] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260808 01:52:40.192207  9636 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:40.174572  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:40.193519  9637 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 18: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:40.177177  9479 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:40.178730  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:40.195034  9634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4f04078e3beb42eda250da45f7485537"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:40.195509  9557 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: [12.12-12.12]
I20260808 01:52:40.181551  9476 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:40.115669 16810 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.057s
I20260808 01:52:40.112471 16889 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:40.169694 16939 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:40.189118  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3091) took 1237 ms. Trace:
I20260808 01:52:40.167840 16998 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:40.191138  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:40.193632  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:40.222044  9636 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 21: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:40.223407  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:40.195636  9634 raft_consensus.cc:606] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:40.136171 16988 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:40.226970  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:40.196656  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:40.197327  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:40.199704 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:40.165939 16867 consensus_queue.cc:237] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:40.176986 17046 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:40.207367  9629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2891) took 3699 ms (client timeout 1990 ms). Trace:
W20260808 01:52:40.239776  9629 rpcz_store.cc:269] 0808 01:52:36.508082 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:36.647563 (+139481us) service_pool.cc:224] Handling call
0808 01:52:40.207127 (+3559564us) spinlock_profiling.cc:243] Waited 827 ms on lock 0x7b540032d054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:40.207291 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1902811008}
I20260808 01:52:40.239893  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:40.153795 16977 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:40.176582 16962 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:40.191591 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 214832us
I20260808 01:52:40.182499 16980 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:40.203565 16929 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:40.225764  9636 pending_rounds.cc:85] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 11
I20260808 01:52:40.184922 16974 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:40.203181  9635 rpcz_store.cc:276] 0808 01:52:38.951639 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:39.201691 (+250052us) service_pool.cc:224] Handling call
0808 01:52:40.169539 (+967848us) spinlock_profiling.cc:243] Waited 1.28 ms on lock 0x7b5800231698. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:52:40.189044 (+ 19505us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2932992}
I20260808 01:52:40.241273  9559 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:40.244345  9634 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260808 01:52:40.253368  9634 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:40.246610  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:40.254910  9634 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260808 01:52:40.224602  9554 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:40.251307  9636 pending_rounds.cc:107] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 20.12
I20260808 01:52:40.189076 16943 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:40.256656  9636 raft_consensus.cc:2889] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:40.257440  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:40.263770  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:40.268405  9560 raft_consensus.cc:2393] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 13: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:40.269147 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:40.269497  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:40.291580  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:40.248942 17046 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:40.292598  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:40.294395  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:40.295739  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:40.455736  9560 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:40.184517 16921 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:40.268231 17057 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.193s	user 0.001s	sys 0.000s
I20260808 01:52:40.301847  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:40.303431 17069 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:52:40.304067 16889 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:40.304351 17061 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260808 01:52:40.305341 17060 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260808 01:52:40.306108 16998 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:40.306493 17064 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
W20260808 01:52:40.307878 17068 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260808 01:52:40.308447 16988 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:40.310140 17054 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.207s	user 0.000s	sys 0.001s
I20260808 01:52:40.311069  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:40.312964 17070 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.016s
I20260808 01:52:40.254390 16974 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:52:40.305423 17056 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
W20260808 01:52:40.386736 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:40.387774 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 118725us
I20260808 01:52:40.388088  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:40.389564 16806 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:40.390969 16937 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:40.392266  9551 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:40.271919  9635 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: [18.10-20.11]
I20260808 01:52:40.294940 16932 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:40.252985 16962 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:40.256853 16929 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:40.253521 16980 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:40.409824 16977 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:40.490419  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:40.490602  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:40.597710  9554 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17. Ops: []
W20260808 01:52:40.573810  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:40.440797 16850 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:40.459539 16990 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:40.410704 16922 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:40.621850  9473 raft_consensus.cc:2410] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 17 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260808 01:52:40.299746  9368 catalog_manager.cc:5697] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 11 to 13, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 13 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:52:40.499466  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:40.458555 16858 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.769s	user 0.013s	sys 0.001s
W20260808 01:52:40.585933 16905 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.102s	user 0.013s	sys 0.017s
I20260808 01:52:40.622043  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2921) took 1088 ms. Trace:
I20260808 01:52:40.466480 16944 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:40.639118  9474 raft_consensus.cc:2410] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260808 01:52:40.638607  9368 catalog_manager.cc:5697] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 10 to 19. New cstate: current_term: 19 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:40.639843 16944 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:40.640653  9368 catalog_manager.cc:5697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 9 to 12. New cstate: current_term: 12 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:40.642270 16944 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:40.486876 16579 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:40.643278 16962 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:40.615618 16932 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:40.643914 16579 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:40.644999 16932 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.8, Last appended by leader: 8, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:40.646423 16579 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:40.634553  9635 rpcz_store.cc:276] 0808 01:52:39.533924 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:40.251766 (+717842us) service_pool.cc:224] Handling call
0808 01:52:40.267906 (+ 16140us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:40.621620 (+353714us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:40.621873 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:40.649051  9635 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260808 01:52:40.649920  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3096) took 1100 ms. Trace:
I20260808 01:52:40.650338  9635 rpcz_store.cc:276] 0808 01:52:39.549426 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:40.648323 (+1098897us) service_pool.cc:224] Handling call
0808 01:52:40.648761 (+   438us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:40.649607 (+   846us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:40.649857 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:40.652040  9635 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:40.652839  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3098) took 1103 ms. Trace:
W20260808 01:52:40.634218 16905 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.156s	user 0.014s	sys 0.019s
I20260808 01:52:40.655128  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3089) took 1704 ms. Trace:
I20260808 01:52:40.656381 17005 raft_consensus.cc:3055] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:40.656854 17005 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:40.657754 17005 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:40.633819 16922 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:40.602905 16820 consensus_meta.cc:287] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.944s	user 0.015s	sys 0.003s
I20260808 01:52:40.634446 16990 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:40.655725  9636 rpcz_store.cc:276] 0808 01:52:38.951092 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:40.222789 (+1271697us) service_pool.cc:224] Handling call
0808 01:52:40.225479 (+  2690us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:40.289287 (+ 63808us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:40.289328 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:40.290065 (+   737us) log.cc:844] Serialized 52 byte log entry
0808 01:52:40.462737 (+172672us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:40.542483 (+ 79746us) log.cc:844] Serialized 14 byte log entry
0808 01:52:40.543081 (+   598us) log.cc:844] Serialized 14 byte log entry
0808 01:52:40.543290 (+   209us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:40.543386 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:40.654768 (+111382us) raft_consensus.cc:1685] finished
0808 01:52:40.654802 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:40.655041 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13467,"prepare.queue_time_us":86315,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":102,"raft.queue_time_us":45383,"raft.run_cpu_time_us":534,"raft.run_wall_time_us":114235,"thread_start_us":172189,"threads_started":1,"wal-append.queue_time_us":318132}
I20260808 01:52:40.601529 16985 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:40.672727  9636 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260808 01:52:40.673460  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3100) took 1123 ms. Trace:
I20260808 01:52:40.673836  9636 rpcz_store.cc:276] 0808 01:52:39.550287 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:40.670048 (+1119761us) service_pool.cc:224] Handling call
0808 01:52:40.670387 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:40.673170 (+  2783us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:40.673395 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:40.675148  9636 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23. Ops: []
I20260808 01:52:40.675731  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2932) took 1066 ms. Trace:
I20260808 01:52:40.676028  9636 rpcz_store.cc:276] 0808 01:52:39.609067 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:40.674605 (+1065538us) service_pool.cc:224] Handling call
0808 01:52:40.674982 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:40.675480 (+   498us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:40.675678 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:40.660605  9635 rpcz_store.cc:276] 0808 01:52:39.549915 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:40.651471 (+1101556us) service_pool.cc:224] Handling call
0808 01:52:40.651837 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:40.652562 (+   725us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:40.652781 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:40.491807 17073 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:40.512236 16773 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:40.680982 17046 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:40.681355 16922 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:40.682916  9635 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260808 01:52:40.709510  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2935) took 1099 ms. Trace:
I20260808 01:52:40.709965  9635 rpcz_store.cc:276] 0808 01:52:39.610054 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:40.682029 (+1071975us) service_pool.cc:224] Handling call
0808 01:52:40.682483 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:40.709091 (+ 26608us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:40.709409 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:40.683511  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2933) took 1074 ms. Trace:
I20260808 01:52:40.711166  9636 rpcz_store.cc:276] 0808 01:52:39.609446 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:40.676757 (+1067311us) service_pool.cc:224] Handling call
0808 01:52:40.677040 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:40.679678 (+  2638us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:40.679711 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:40.679756 (+    45us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:40.679787 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:40.679856 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:40.683389 (+  3533us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1821,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":112}
I20260808 01:52:40.673153 16943 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:40.670425 16932 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
W20260808 01:52:40.691476 17074 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260808 01:52:40.669457  9368 catalog_manager.cc:5697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 6 to 10. New cstate: current_term: 10 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:52:40.696404 16929 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:40.700450 16833 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.536s	user 0.013s	sys 0.000s
W20260808 01:52:40.702368 16836 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.514s	user 0.013s	sys 0.001s
I20260808 01:52:40.669239 17069 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:40.659983 16977 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:40.707091  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3094) took 1158 ms. Trace:
I20260808 01:52:40.727308  9634 rpcz_store.cc:276] 0808 01:52:39.548971 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:40.255766 (+706795us) service_pool.cc:224] Handling call
0808 01:52:40.256046 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:40.257094 (+  1048us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:52:40.270989 (+ 13895us) log.cc:844] Serialized 14 byte log entry
0808 01:52:40.706706 (+435717us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:40.706766 (+    60us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:40.706814 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:40.706886 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:40.707019 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":13567,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":62,"raft.queue_time_us":129,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":4865,"thread_start_us":416913,"threads_started":1}
I20260808 01:52:40.624307 16974 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:40.718459  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:40.730404  9551 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:52:40.707207 17005 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:40.721769 10367 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:57156
I20260808 01:52:40.733914  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 902) took 1786 ms. Trace:
I20260808 01:52:40.733950 10367 raft_consensus.cc:606] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:40.735538 10367 raft_consensus.cc:614] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:40.734378  9637 rpcz_store.cc:276] 0808 01:52:38.947349 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:40.208589 (+1261240us) service_pool.cc:224] Handling call
0808 01:52:40.245416 (+ 36827us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:40.245817 (+   401us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:40.245882 (+    65us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:40.258020 (+ 12138us) log.cc:844] Serialized 26 byte log entry
0808 01:52:40.621253 (+363233us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:52:40.636732 (+ 15479us) log.cc:844] Serialized 14 byte log entry
0808 01:52:40.637036 (+   304us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:40.637147 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:40.733654 (+ 96507us) raft_consensus.cc:1685] finished
0808 01:52:40.733679 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:40.733848 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":15265,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":68,"raft.queue_time_us":4670,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":1759,"thread_start_us":244775,"threads_started":1,"wal-append.queue_time_us":453033}
I20260808 01:52:40.712653 16773 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:40.725013 16943 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:40.714990 17073 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:40.725911 16833 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:40.750878 16939 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.519s	user 0.014s	sys 0.000s
I20260808 01:52:40.761929  9554 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260808 01:52:40.747049 17012 consensus_meta.cc:287] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.770s	user 0.009s	sys 0.005s
W20260808 01:52:40.763146  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 626249dbd6eb4d66bfa8cce2e716cbe2 (from 7201605bb2f54c949a4ebe6e9b3b4b2a to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:40.736922 16977 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:40.733490 17069 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:40.751048 16943 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:40.776526 17069 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:40.777781 16943 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:40.778453  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3622) took 1693 ms. Trace:
I20260808 01:52:40.778930  9550 rpcz_store.cc:276] 0808 01:52:39.085052 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:39.269647 (+184595us) service_pool.cc:224] Handling call
0808 01:52:40.778229 (+1508582us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x7b54001aca54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:40.778386 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":826872064}
I20260808 01:52:40.778465 16836 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:40.781250 16836 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:40.783392 16977 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:40.783579 16836 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:40.784947  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:40.785715 16978 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:40.785058 16990 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:40.766338 16939 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:40.790390  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 903) took 1726 ms. Trace:
I20260808 01:52:40.800629  9629 rpcz_store.cc:276] 0808 01:52:39.063759 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:40.247345 (+1183586us) service_pool.cc:224] Handling call
0808 01:52:40.259519 (+ 12174us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:40.260028 (+   509us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:40.260070 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:40.267562 (+  7492us) log.cc:844] Serialized 26 byte log entry
0808 01:52:40.724695 (+457133us) raft_consensus.cc:1645] Marking committed up to 16
0808 01:52:40.724762 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:40.724867 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:40.790122 (+ 65255us) raft_consensus.cc:1685] finished
0808 01:52:40.790147 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:40.790324 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8392,"raft.queue_time_us":4466,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":29347,"thread_start_us":271066,"threads_started":1,"wal-append.queue_time_us":478473}
I20260808 01:52:40.794493  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 889) took 3468 ms. Trace:
I20260808 01:52:40.802501  9632 rpcz_store.cc:276] 0808 01:52:37.323553 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:37.941219 (+617666us) service_pool.cc:224] Handling call
0808 01:52:37.941572 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:38.797537 (+855965us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:40.791964 (+1994427us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b540008d754. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:40.792217 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1945488128}
I20260808 01:52:40.803647  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3287) took 3179 ms. Trace:
I20260808 01:52:40.804229  9479 rpcz_store.cc:276] 0808 01:52:37.623929 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:39.273710 (+1649781us) service_pool.cc:224] Handling call
0808 01:52:39.283447 (+  9737us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:40.193940 (+910493us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:40.803353 (+609413us) spinlock_profiling.cc:243] Waited 893 ms on lock 0x7b54000933d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:40.803572 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2054727936}
I20260808 01:52:40.805963 16833 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:40.813040 16749 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:40.813416  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3627) took 1719 ms. Trace:
I20260808 01:52:40.812160 16929 raft_consensus.cc:2955] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Committing config change with OpId 15.11: config changed from index 9 to 11, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:52:40.824101  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:40.824327 16871 raft_consensus.cc:3055] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:40.824863 16871 raft_consensus.cc:740] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:40.826014  9632 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7c508218361c446898fede7c5682fa9e"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:40.827365 16984 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:40.828760 16984 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
W20260808 01:52:40.831032  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:40.813942  9558 rpcz_store.cc:276] 0808 01:52:39.094261 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:39.504344 (+410083us) service_pool.cc:224] Handling call
0808 01:52:39.605857 (+101513us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:39.624463 (+ 18606us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:40.813175 (+1188712us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b5400383950. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:40.813339 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":232669952}
I20260808 01:52:40.837232  9632 raft_consensus.cc:606] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:40.837842  9632 raft_consensus.cc:614] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:40.843782  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:40.845651  9551 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260808 01:52:40.847581 17063 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.668s	user 0.007s	sys 0.002s
I20260808 01:52:40.848845  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:40.770148 17012 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:40.795629 16713 raft_consensus.cc:3055] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:40.801990 16939 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:40.870944 16960 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:40.866788 16713 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:52:40.862341  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:40.825891 16871 consensus_queue.cc:260] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:40.881982  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:40.885885 10364 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: [15.14-15.14]
I20260808 01:52:40.887279  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:40.884204  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:40.871933 16960 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:40.904222 16871 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:40.901244 16713 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:40.905200 16713 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:40.901434 16939 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:40.911226 16773 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:40.911861 16939 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:40.913008 17073 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:40.914085 16939 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:40.902552 16785 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:40.915313 16785 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:40.917668 16974 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:40.932977 16989 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:40.919072 16829 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:40.935277 16829 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:40.946260  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:40.947271  9551 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:52:40.939163 16829 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:41.011687  9557 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
W20260808 01:52:41.010943  9521 leader_election.cc:336] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.543s (SENT)
I20260808 01:52:41.014545  9481 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:41.015411 16785 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:41.017343 16960 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:41.019238 16974 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:41.019568 17012 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:41.020296 16829 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:41.020911 16829 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:41.024277 16829 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:41.049126  9631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3074) took 2256 ms (client timeout 2867 ms). Trace:
W20260808 01:52:41.049834  9631 rpcz_store.cc:269] 0808 01:52:38.793062 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:38.793214 (+   152us) service_pool.cc:224] Handling call
0808 01:52:41.047444 (+2254230us) spinlock_profiling.cc:243] Waited 949 ms on lock 0x7b540008c354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:41.049004 (+  1560us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2183319936}
W20260808 01:52:41.059756  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:41.060623  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:41.068377  9551 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260808 01:52:41.070326  9474 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260808 01:52:41.058584 16974 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:41.065166 16984 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:41.064054 16960 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:41.082298 16984 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:41.108271 16984 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:41.115483 16983 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:41.116235 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:41.118940 16983 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:41.116763 10360 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: []
I20260808 01:52:41.117429 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:41.081276 16974 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:41.127034 16974 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:41.093575 16960 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:41.130245 16960 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:41.131243 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:41.132633  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2776) took 2667 ms. Trace:
I20260808 01:52:41.133280  9550 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:52:41.142053  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3288) took 3460 ms. Trace:
I20260808 01:52:41.142474  9480 rpcz_store.cc:276] 0808 01:52:37.681850 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:39.279554 (+1597704us) service_pool.cc:224] Handling call
0808 01:52:39.313945 (+ 34391us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:40.118055 (+804110us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:41.141738 (+1023683us) spinlock_profiling.cc:243] Waited 803 ms on lock 0x7b540021d254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:41.141977 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1846967552}
I20260808 01:52:41.145893 16980 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:41.144373 10332 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:41.147339  9633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2917) took 1614 ms (client timeout 2078 ms). Trace:
I20260808 01:52:41.148389 16720 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:52:41.150870 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:41.147979  9633 rpcz_store.cc:269] 0808 01:52:39.532768 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:39.673480 (+140712us) service_pool.cc:224] Handling call
0808 01:52:41.147075 (+1473595us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x7b540025b7d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:41.147247 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":929094528}
I20260808 01:52:41.153141 16829 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:41.153578 16829 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:41.153493 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:41.154138 16829 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:41.155328 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:41.158103 16596 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:41.158968 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:41.161500 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:41.163422 10362 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 23. Current term is 24. Ops: []
W20260808 01:52:41.163661  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 7c508218361c446898fede7c5682fa9e (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:41.164492 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:41.157132 10363 raft_consensus.cc:2955] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Committing config change with OpId 15.11: config changed from index 9 to 11, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:52:41.144435 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:41.133373  9556 rpcz_store.cc:276] 0808 01:52:38.465428 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:38.465605 (+   177us) service_pool.cc:224] Handling call
0808 01:52:38.488168 (+ 22563us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:39.519833 (+1031665us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:41.132232 (+1612399us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400409554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:41.132536 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2370591616}
I20260808 01:52:41.165656 16720 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
W20260808 01:52:41.163475  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:41.174252 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:41.178426 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:41.180085 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:41.180251  9600 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:41.180857 10362 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:41.181813 10364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 1120) took 1501 ms. Trace:
I20260808 01:52:41.182260 10364 rpcz_store.cc:276] 0808 01:52:39.680042 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:39.680244 (+   202us) service_pool.cc:224] Handling call
0808 01:52:39.680729 (+   485us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:40.886452 (+1205723us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:41.181490 (+295038us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b5400594054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:41.181736 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2770925952}
I20260808 01:52:41.183904  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3243) took 2685 ms. Trace:
I20260808 01:52:41.184357  9475 rpcz_store.cc:276] 0808 01:52:38.499044 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:39.376993 (+877949us) service_pool.cc:224] Handling call
0808 01:52:39.788729 (+411736us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:39.926402 (+137673us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:41.183688 (+1257286us) spinlock_profiling.cc:243] Waited 385 ms on lock 0x7b540038ccd0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:41.183839 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":885607296}
I20260808 01:52:41.186367 17007 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:41.187111 17016 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:41.187611 17007 raft_consensus.cc:3055] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:52:41.188200 17007 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:41.189215 17007 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:41.192134 16834 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:52:41.193151 16834 raft_consensus.cc:697] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:52:41.195291  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 56478.2us
I20260808 01:52:41.187707 17016 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:41.196252 17016 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:41.204490 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:41.206620 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:41.208544  9559 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260808 01:52:41.211447  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:41.218604 17007 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:52:41.223488 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:41.228864 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:41.229671  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1196) took 1303 ms. Trace:
I20260808 01:52:41.230123  9481 rpcz_store.cc:276] 0808 01:52:39.926756 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:40.102378 (+175622us) service_pool.cc:224] Handling call
0808 01:52:40.120150 (+ 17772us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:41.015536 (+895386us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:41.229400 (+213864us) spinlock_profiling.cc:243] Waited 894 ms on lock 0x7b540021e154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:41.229604 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2056321408}
I20260808 01:52:41.231876  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:41.232862  9474 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 21: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:41.233673  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:41.275774 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:41.276125  9475 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:41.281523 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:41.236910  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:41.293855  9550 raft_consensus.cc:2410] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260808 01:52:41.295083  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:41.295855  9550 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:41.296900  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:41.298192  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:41.237954 17058 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 1.125s	user 0.006s	sys 0.002s
I20260808 01:52:41.238039  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:41.229399 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:41.194377 16834 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:41.242208  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:41.315717  9552 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:41.242873 16960 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:41.243460 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 73057.1us
I20260808 01:52:41.244045 17046 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
W20260808 01:52:41.344933  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:41.345657  9524 connection.cc:256] RPC call timeout handler was delayed by -1.4508599980035797s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1718s for a response
I20260808 01:52:41.246198  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:41.249436  9445 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:41.250327 10362 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:41.254103  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:41.255622  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:41.349581  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 100235us
W20260808 01:52:41.258178 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:41.351176 10360 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:41.350210  9554 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
W20260808 01:52:41.273149  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 106501us
I20260808 01:52:41.313087  9475 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:52:41.268326 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:41.314607  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 1020) took 1074 ms. Trace:
I20260808 01:52:41.322549  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 1016) took 2262 ms. Trace:
I20260808 01:52:41.354274  9555 rpcz_store.cc:276] 0808 01:52:39.060171 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:39.490795 (+430624us) service_pool.cc:224] Handling call
0808 01:52:39.491296 (+   501us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:39.529407 (+ 38111us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:41.322238 (+1792831us) spinlock_profiling.cc:243] Waited 21.8 ms on lock 0x7b540029efd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:41.322477 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50098944}
I20260808 01:52:41.335116 16984 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:52:41.336062 16785 consensus_queue.cc:1059] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260808 01:52:41.375370  9524 connection.cc:256] RPC call timeout handler was delayed by -1.3700866099970881s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2761s for a response
I20260808 01:52:41.311633 16871 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:41.376084  9523 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:41.376515  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:41.378793  9524 connection.cc:256] RPC call timeout handler was delayed by -1.2910884770099074s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.195s for a response
W20260808 01:52:41.384917  9524 connection.cc:256] RPC call timeout handler was delayed by -1.2836812500027008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16040000000000001s for a response
I20260808 01:52:41.342079 10368 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "df6982ec010444a8854140f7b64e985d"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:57156
W20260808 01:52:41.345120  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:41.386500  9634 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260808 01:52:41.351787  9556 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:41.246871 17079 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260808 01:52:41.319586  9557 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 20: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:41.356658 10334 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:41.354060 16952 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:41.385167  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:41.405794 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 49256.6us
I20260808 01:52:41.387616  9636 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260808 01:52:41.386515  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.407709  9552 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:52:41.359323 16921 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:52:41.413213  9629 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:41.413240 16810 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:41.415495  9631 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:41.415812 16660 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:41.415943  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:41.416815  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:41.416599 16660 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:41.417568  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.413440 17005 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:52:41.418540 17005 raft_consensus.cc:697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:41.415841 16194 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.004s
W20260808 01:52:41.397676  9524 connection.cc:256] RPC call timeout handler was delayed by -1.179851219028933s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17930000000000001s for a response
W20260808 01:52:41.421037  9524 connection.cc:256] RPC call timeout handler was delayed by -1.1054531930130906s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1531s for a response
I20260808 01:52:41.400739 16990 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:41.337347  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:41.401419  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.422255 16990 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:41.424523 16990 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:41.424640  9633 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260808 01:52:41.416103  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:41.426291  9637 raft_consensus.cc:2376] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18. Re-sending same reply.
I20260808 01:52:41.405889  9630 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:52:41.375604 16713 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:41.340021 10332 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:41.415297 16810 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:41.421554  9524 connection.cc:256] RPC call timeout handler was delayed by -1.0688463969854638s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1592s for a response
I20260808 01:52:41.354179  9560 rpcz_store.cc:276] 0808 01:52:40.240553 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:40.614979 (+374426us) service_pool.cc:224] Handling call
0808 01:52:40.630775 (+ 15796us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:40.672972 (+ 42197us) raft_consensus.cc:1534] Early marking committed up to index 16
0808 01:52:40.917673 (+244701us) log.cc:844] Serialized 14 byte log entry
0808 01:52:41.062137 (+144464us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:41.062231 (+    94us) raft_consensus.cc:1645] Marking committed up to 16
0808 01:52:41.062300 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:41.314247 (+251947us) spinlock_profiling.cc:243] Waited 41.9 ms on lock 0x7b5400217ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:41.314324 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:41.314524 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24783,"prepare.queue_time_us":244610,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":1547,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":29342,"spinlock_wait_cycles":96259840,"thread_start_us":338017,"threads_started":2,"wal-append.queue_time_us":157407}
I20260808 01:52:41.428221  9632 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:52:41.437474 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 97604.7us
I20260808 01:52:41.436165  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.441005  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:41.418032  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.441314  9636 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
W20260808 01:52:41.444386  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:41.392938  9638 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
W20260808 01:52:41.429323  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 92090us
I20260808 01:52:41.448729  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:41.412122  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.449661  9630 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260808 01:52:41.402140 16934 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:41.435513  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1185) took 2365 ms. Trace:
I20260808 01:52:41.458539  9478 rpcz_store.cc:276] 0808 01:52:39.070304 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:39.421085 (+350781us) service_pool.cc:224] Handling call
0808 01:52:39.421593 (+   508us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:39.452619 (+ 31026us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:39.452651 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:39.453157 (+   506us) log.cc:844] Serialized 26 byte log entry
0808 01:52:39.815857 (+362700us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:52:39.815907 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:39.815985 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:41.237193 (+1421208us) spinlock_profiling.cc:243] Waited 3.83 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:41.413215 (+176022us) raft_consensus.cc:1685] finished
0808 01:52:41.413243 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:41.435419 (+ 22176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29103,"raft.queue_time_us":29358,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":6995,"spinlock_wait_cycles":8819968,"thread_start_us":271130,"threads_started":1,"wal-append.queue_time_us":456321}
I20260808 01:52:41.415447 16932 raft_consensus.cc:3055] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Stepping down as leader of term 12
I20260808 01:52:41.431125  9633 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260808 01:52:41.451617  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.466571  9637 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260808 01:52:41.467270  9630 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:41.437431  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:41.469964  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:41.447412 16808 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:41.462214  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.473469  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:41.474525  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.475528  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.475098  9633 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 6. Preceding OpId from leader: term: 17 index: 8. (index mismatch)
I20260808 01:52:41.440124 17008 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 15.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.039s
I20260808 01:52:41.388494 16939 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.072s
I20260808 01:52:41.480592  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.443811 16785 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:41.486536  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.487383  9632 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:41.483983  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.482730  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:41.495960  9631 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:41.496896  9634 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:41.488552  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:41.398840 16871 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:41.353600 16960 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:41.482385  9638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2897) took 4963 ms (client timeout 1627 ms). Trace:
W20260808 01:52:41.507264  9638 rpcz_store.cc:269] 0808 01:52:36.510345 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:36.536307 (+ 25962us) service_pool.cc:224] Handling call
0808 01:52:41.473655 (+4937348us) spinlock_profiling.cc:243] Waited 4.74 s on lock 0x7b5400421dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:41.473798 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10893178240}
I20260808 01:52:41.480054 16808 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:41.429134 16952 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:41.412567 16773 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.070s
W20260808 01:52:41.504915  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:41.479844 16983 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:52:41.481222 16974 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:41.506479  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.515228  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:41.515848  9633 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:41.464485 16932 raft_consensus.cc:740] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:41.502487  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.525552  9636 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:41.526930  9634 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:41.528153  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:41.530104  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:41.531277  9630 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:41.541254  9638 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260808 01:52:41.556596  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.446941 16713 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:41.501247 16984 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:41.579830  9521 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:41.581334  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:41.557492  9631 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:41.419634 17005 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:41.558243 16660 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:41.587713 16808 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:41.588995 16952 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:41.492712 16934 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:41.589865  9558 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260808 01:52:41.503475 16989 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260808 01:52:41.593031  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:41.609131  9476 raft_consensus.cc:2410] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 19 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260808 01:52:41.594625  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:41.596388  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:41.609968  9473 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:41.609997  9481 raft_consensus.cc:2410] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260808 01:52:41.596866 16810 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:41.597579  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:41.599902  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:41.612360 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:41.613312  9630 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260808 01:52:41.618701  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:41.615165  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:41.620997  9481 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:52:41.641433 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:41.663555  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:41.619426  9473 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:41.591475 16984 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:41.520217 16983 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:41.520377 16974 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:41.687580 17099 log.cc:927] Time spent T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Append to log took a long time: real 0.091s	user 0.002s	sys 0.000s
I20260808 01:52:41.689162  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:41.676225 16983 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:41.691640 16871 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:41.576684 16829 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:41.676568 16984 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
W20260808 01:52:41.743543  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 255114us
I20260808 01:52:41.744575  9524 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:41.746042  9524 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:41.616361 17016 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:41.758901 10367 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 17: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:52:41.734330 17007 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.515s	user 0.013s	sys 0.000s
W20260808 01:52:41.755358  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 348925us
I20260808 01:52:41.679369 16974 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:41.750823 16810 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:41.763129  9524 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:41.749106 16929 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:52:41.771447  9524 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:41.772079 10361 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 18: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:41.668934 16934 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:41.782183 10368 raft_consensus.cc:606] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:41.791428 10368 raft_consensus.cc:614] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:41.781219 17007 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:41.782881 16960 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:41.760109 16958 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election won for term 19
W20260808 01:52:41.792225  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 411734us
I20260808 01:52:41.675197 16932 consensus_queue.cc:260] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:41.816915  9601 connection.cc:256] RPC call timeout handler was delayed by -1.3011842069972772s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1622s for a response
I20260808 01:52:41.818370  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.780215 16929 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:41.799521  9630 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260808 01:52:41.803099  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.841638  9634 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:41.769585 17010 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:41.818468  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.839459  9631 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260808 01:52:41.795125  9445 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:41.798312  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 349664us
I20260808 01:52:41.809072 16808 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:41.809516 16660 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:41.817729  9601 connection.cc:256] RPC call timeout handler was delayed by -1.2201842070207931s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17120000000000002s for a response
I20260808 01:52:41.773115 16810 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:41.856168  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:41.816689 17012 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:41.862105  9637 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
W20260808 01:52:41.863209  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 68326.8us
I20260808 01:52:41.866506  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:41.867538  9481 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:41.870105  9481 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
W20260808 01:52:41.859928 10363 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.006s	sys 0.016s
W20260808 01:52:41.871279 10363 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.006s	sys 0.016s
I20260808 01:52:41.847924 16929 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
W20260808 01:52:41.821269 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 341979us
I20260808 01:52:41.877166  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:52:41.882157 10332 leader_election.cc:336] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.586s (SENT)
I20260808 01:52:41.885419  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:41.861135 16660 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:41.859226 16932 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 12 FOLLOWER]: Advancing to term 13
I20260808 01:52:41.858722 17010 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:41.883569 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:41.858647 16952 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:41.893841  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:41.874179 16871 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:41.882027 16988 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:41.853951 16983 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:41.911655  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:41.858605  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:41.918010  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 59546.8us
I20260808 01:52:41.921808  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:41.924256 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:41.891297 16833 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:41.861274 16808 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:41.930572  9551 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 19 index: 9. (index mismatch)
I20260808 01:52:41.874975 16960 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
I20260808 01:52:41.936304 16660 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:41.936097 17010 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:41.850816 16773 consensus_queue.cc:1175] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" responder_term: 13 status { last_received { term: 12 index: 12 } last_committed_idx: 11 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 12. Current term is 13. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 01:52:41.938468 16713 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:41.902805 16952 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:42.019600  9634 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:41.940567  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:41.941922  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:41.944958  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:41.944808  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:42.028843  9556 raft_consensus.cc:2410] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260808 01:52:41.910085 16988 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:42.041944  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:42.023229  9631 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:41.949803  9601 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:41.952986  9477 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 10. (term mismatch)
I20260808 01:52:42.045099  9477 pending_rounds.cc:85] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 9
I20260808 01:52:42.045524  9477 pending_rounds.cc:107] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 15.10
I20260808 01:52:42.045874  9477 raft_consensus.cc:2889] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:41.956332  9637 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:41.915974 16975 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:41.960588  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:41.906435 16980 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
I20260808 01:52:41.965773  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:41.970815  9473 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260808 01:52:41.971937  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:41.973328  9560 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:41.970991 16829 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:42.031062 17016 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:42.031772 16810 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:42.032527 16983 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:42.036123 16808 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:42.041081 16952 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:41.951244  9481 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260808 01:52:42.121934  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:42.123124  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:42.124238  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:42.044318  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:42.127950  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:42.087064 16194 raft_consensus.cc:2764] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260808 01:52:42.116596 16975 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:42.136494 10368 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260808 01:52:42.136629 16975 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:42.059851  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:42.137986  9551 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:42.065294 16713 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:41.971190 17114 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.111s
I20260808 01:52:41.950644 16833 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:42.069202  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 127388us
W20260808 01:52:42.078751  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 121896us
I20260808 01:52:42.193919  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:42.090032  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:42.019688 17083 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260808 01:52:42.107000  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:42.201617  9473 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:42.108736  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:42.109376  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:41.908363 17008 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:42.067998 16988 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:42.119397  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:42.120347  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 540557us
I20260808 01:52:42.132688 16960 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:52:42.134131  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1201) took 1040 ms. Trace:
I20260808 01:52:42.132952  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:42.138777 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:42.229122  9479 rpcz_store.cc:276] 0808 01:52:41.093575 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:41.093884 (+   309us) service_pool.cc:224] Handling call
0808 01:52:41.134082 (+ 40198us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:41.134490 (+   408us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:41.134520 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:41.135113 (+   593us) log.cc:844] Serialized 26 byte log entry
0808 01:52:41.597887 (+462774us) raft_consensus.cc:1645] Marking committed up to 16
0808 01:52:42.119037 (+521150us) log.cc:844] Serialized 14 byte log entry
0808 01:52:42.132312 (+ 13275us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:42.133260 (+   948us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:42.133424 (+   164us) raft_consensus.cc:1685] finished
0808 01:52:42.133499 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:42.133972 (+   473us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":79297,"prepare.queue_time_us":89095,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":14343,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":237,"thread_start_us":894638,"threads_started":2,"wal-append.queue_time_us":657348}
W20260808 01:52:42.139741  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 359025us
W20260808 01:52:42.141896 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:42.163461 16921 consensus_queue.cc:1059] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:52:42.164770 16929 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:42.175740 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 271681us
I20260808 01:52:42.178969 10363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 913) took 1026 ms. Trace:
W20260808 01:52:42.182699  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 188416us
I20260808 01:52:42.191835 16835 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:42.273339  9600 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:42.202510  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:42.198531  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:42.209108  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:42.203464  9472 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:42.210465 17007 raft_consensus.cc:697] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:42.215701  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:42.140280 16983 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:42.220292  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:42.198271 16833 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:42.196177 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:42.054175  9634 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:52:42.240477 10363 rpcz_store.cc:276] 0808 01:52:41.152079 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:41.156414 (+  4335us) service_pool.cc:224] Handling call
0808 01:52:41.156757 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:41.156998 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 13
0808 01:52:41.207951 (+ 50953us) log.cc:844] Serialized 14 byte log entry
0808 01:52:41.872071 (+664120us) log.cc:844] Serialized 14 byte log entry
0808 01:52:41.872830 (+   759us) log.cc:844] Serialized 14 byte log entry
0808 01:52:42.175814 (+302984us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:42.176350 (+   536us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:42.176998 (+   648us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:42.178743 (+  1745us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:42.178891 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":81415,"prepare.queue_time_us":533694,"prepare.run_cpu_time_us":163,"prepare.run_wall_time_us":1743,"raft.queue_time_us":84525,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":82577,"spinlock_wait_cycles":14689152,"thread_start_us":881104,"threads_started":2}
I20260808 01:52:42.196177 16829 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:42.203209  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:42.318327  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:42.319190  9479 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:42.209931 16194 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:42.295809  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:42.299845  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 101447us
W20260808 01:52:42.292541 16974 consensus_meta.cc:287] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.520s	user 0.005s	sys 0.007s
I20260808 01:52:42.302311  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:42.197937 16713 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:42.252313 16958 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:42.322212  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:42.336114  9601 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:42.339816  9600 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:42.356382 10332 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:42.364962  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:42.307588 16871 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:42.366290  9552 raft_consensus.cc:2393] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:42.368798 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:42.303726 17113 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:42.378057  9551 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:42.389746  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:42.386791  9479 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:42.403072  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:42.406857  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:42.408929  9557 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 23: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:42.412035  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:42.298964 17135 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.126s
I20260808 01:52:42.480619  9479 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 17 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260808 01:52:42.480525  9477 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 8. (term mismatch)
I20260808 01:52:42.588151  9477 pending_rounds.cc:85] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 7
I20260808 01:52:42.588501  9477 pending_rounds.cc:107] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 15.8
I20260808 01:52:42.332675 17008 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.159s
I20260808 01:52:42.333014 16829 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:42.352877 16980 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.139s
I20260808 01:52:42.354997 16974 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:42.366988 17139 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.179s
W20260808 01:52:42.529409  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 229028us
W20260808 01:52:42.516363 17058 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
W20260808 01:52:42.536481  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:42.535498  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 261583us
I20260808 01:52:42.484897 16194 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:42.398929 16871 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:42.483765 17016 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.068s
I20260808 01:52:42.498383 16660 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.056s
W20260808 01:52:42.563913  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 343678us
I20260808 01:52:42.529541 17140 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260808 01:52:42.563659 16984 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.804s	user 0.014s	sys 0.000s
I20260808 01:52:42.510746 17007 consensus_queue.cc:237] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:42.580803 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 224411us
I20260808 01:52:42.582883  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:42.646864 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:42.583365  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:42.648236  9631 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:42.584836  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:42.649724  9630 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:42.288736 16834 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.085s
I20260808 01:52:42.332429 16921 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:42.588802  9477 raft_consensus.cc:2889] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:42.589032  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:42.598330  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:42.597513  9476 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
W20260808 01:52:42.599795  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0564886380161624s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1628s for a response
I20260808 01:52:42.625059 17012 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:42.391206 17114 raft_consensus.cc:3055] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:52:42.602264  9635 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260808 01:52:42.628352  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3103) took 1237 ms. Trace:
W20260808 01:52:42.602849 17120 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.312s	user 0.001s	sys 0.001s
I20260808 01:52:42.592247 16929 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:42.639348  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:42.629840 16810 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:42.677537  9474 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260808 01:52:42.677320 10332 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:42.677254  9473 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
W20260808 01:52:42.680030 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 283868us
I20260808 01:52:42.559835 16958 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:42.684643  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0438059400185011s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20800000000000002s for a response
W20260808 01:52:42.675004 17136 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.296s	user 0.001s	sys 0.000s
I20260808 01:52:42.706292 16988 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:42.637040 17161 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:42.685015 17138 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260808 01:52:42.684077 17118 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260808 01:52:42.723985 17124 log.cc:927] Time spent T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.244s	user 0.000s	sys 0.001s
I20260808 01:52:42.702404  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:42.651644 16713 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:42.741267 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 64080us
I20260808 01:52:42.752539  9552 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 13.
I20260808 01:52:42.651298 17134 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:42.727037  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:42.692636 17114 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:42.726532  9475 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.638s	user 0.011s	sys 0.026s
I20260808 01:52:42.692322 17012 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
W20260808 01:52:42.758502  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 129634us
W20260808 01:52:42.758662  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 121236us
W20260808 01:52:42.760764  9475 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.673s	user 0.012s	sys 0.026s
I20260808 01:52:42.761296  9524 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:42.651054 16984 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:42.756431  9552 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260808 01:52:42.691233 16921 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:42.654788 17132 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.116s
W20260808 01:52:42.781420 17151 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
I20260808 01:52:42.781255 16829 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:42.734149 16990 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
I20260808 01:52:42.750065 16194 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:42.783772 10361 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 1121) took 1556 ms (client timeout 1571 ms). Trace:
W20260808 01:52:42.854256 10361 rpcz_store.cc:269] 0808 01:52:41.227662 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:41.228543 (+   881us) service_pool.cc:224] Handling call
0808 01:52:42.783463 (+1554920us) spinlock_profiling.cc:243] Waited 542 ms on lock 0x7b5400594cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:42.783674 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1247174528}
W20260808 01:52:42.799235 17141 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.243s	user 0.001s	sys 0.000s
I20260808 01:52:42.800223 16974 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:42.801570 17137 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
W20260808 01:52:42.804106 17121 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.236s	user 0.001s	sys 0.001s
I20260808 01:52:42.740886 17159 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:42.806604 16921 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:42.812798 17132 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:42.864643 17132 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:42.814247 16988 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:42.819041 17140 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:42.743897 17161 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:42.821076 17150 log.cc:927] Time spent T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260808 01:52:42.764583 17046 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260808 01:52:42.823354 17142 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.185s	user 0.000s	sys 0.000s
I20260808 01:52:42.787250 17134 raft_consensus.cc:697] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:42.836601 17114 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:42.849442 17116 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.816s	user 0.004s	sys 0.005s
I20260808 01:52:42.889344 17161 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:42.885397 17134 consensus_queue.cc:237] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:42.875715 17140 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
W20260808 01:52:42.941792  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:42.942152  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet df6982ec010444a8854140f7b64e985d (from 7201605bb2f54c949a4ebe6e9b3b4b2a to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:42.944132  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 1
W20260808 01:52:42.941558  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0226225700171199s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3022s for a response
I20260808 01:52:42.948393  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:42.953158  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:42.955922  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:42.959046  9478 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260808 01:52:42.962167  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1207) took 1439 ms. Trace:
W20260808 01:52:42.962719  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 200539us
W20260808 01:52:42.967649 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 226842us
W20260808 01:52:42.968086  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 206666us
I20260808 01:52:42.968716  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:42.860610 16932 consensus_meta.cc:287] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.957s	user 0.011s	sys 0.005s
I20260808 01:52:42.972281 16932 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:42.973104  9633 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:42.972951  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:42.973398 16932 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:42.962895  9480 rpcz_store.cc:276] 0808 01:52:41.522979 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:41.523112 (+   133us) service_pool.cc:224] Handling call
0808 01:52:41.584939 (+ 61827us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:41.595365 (+ 10426us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:41.595406 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:41.612901 (+ 17495us) log.cc:844] Serialized 26 byte log entry
0808 01:52:41.925754 (+312853us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:41.925799 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:41.925873 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:42.961564 (+1035691us) raft_consensus.cc:1685] finished
0808 01:52:42.961605 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:42.961938 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9961,"raft.queue_time_us":13591,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":9014,"thread_start_us":312488,"threads_started":1,"wal-append.queue_time_us":704248}
I20260808 01:52:42.980904 17046 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:42.981694 16984 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:42.983922 16984 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260808 01:52:42.983593 17046 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:42.985422 16984 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:52:42.986163 16984 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:42.987033 16984 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:42.987367 16974 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:42.989037 17046 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:42.989871 17046 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:42.812935 16983 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.003s
I20260808 01:52:42.992316 17005 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:42.994434 16867 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:42.995388 16867 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:42.996122 16867 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:42.773247 16660 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:42.994575 17113 raft_consensus.cc:3055] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:42.996019 17114 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:42.830183 16871 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:42.771304 16934 consensus_queue.cc:1059] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
I20260808 01:52:42.857946 16194 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:43.049849 16194 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:42.977978 16932 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:42.961578  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3312) took 1380 ms. Trace:
I20260808 01:52:42.980345 16921 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:43.044116 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 59368.7us
I20260808 01:52:43.052323 16921 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:43.044502 17121 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.167s	user 0.000s	sys 0.009s
I20260808 01:52:43.057242  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:43.058714  9480 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:52:43.059166  9473 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260808 01:52:43.061110  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 1028) took 1733 ms. Trace:
W20260808 01:52:43.045159  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 75073us
I20260808 01:52:43.028929 17046 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:43.072997 16974 raft_consensus.cc:3055] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:43.075601 16660 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:43.077800  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:43.087518 16989 consensus_queue.cc:1059] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:43.089294 16932 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:43.090646 16932 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:43.092849 17149 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:43.094272 17149 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:52:43.094841 17129 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:43.095486 17129 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:43.095999 17129 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:43.087677 16660 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:43.099277 17149 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:52:43.100863 16660 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:43.101923 16660 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:43.042085 17113 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:43.106735  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:43.107802  9473 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:52:43.107417 17113 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:43.103497  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:43.111207 16660 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:43.112475 17070 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:43.117867 17154 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:52:43.120187 17154 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:43.120685 17130 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:52:43.123107 17113 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:43.113992 16934 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:43.061589  9554 rpcz_store.cc:276] 0808 01:52:41.327804 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:41.423340 (+ 95536us) service_pool.cc:224] Handling call
0808 01:52:41.423720 (+   380us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:41.441055 (+ 17335us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:52:41.483229 (+ 42174us) log.cc:844] Serialized 14 byte log entry
0808 01:52:41.740954 (+257725us) spinlock_profiling.cc:243] Waited 4.52 ms on lock 0x7b1800160c00. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4889671f 00007f5d4188ceb1 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d
0808 01:52:42.311934 (+570980us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:42.312880 (+   946us) log.cc:844] Serialized 26 byte log entry
0808 01:52:42.313192 (+   312us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:42.314046 (+   854us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:42.314120 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:43.060450 (+746330us) raft_consensus.cc:1685] finished
0808 01:52:43.060587 (+   137us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:43.060825 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":165862,"prepare.queue_time_us":56770,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":742849,"raft.run_cpu_time_us":364,"raft.run_wall_time_us":23195,"spinlock_wait_cycles":10393344,"thread_start_us":730790,"threads_started":2,"wal-append.queue_time_us":1005600}
I20260808 01:52:43.130369 16921 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:43.133347  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:43.142391 16934 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260808 01:52:43.144317  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:43.145200 10334 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:43.145548  9551 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260808 01:52:43.146445 16932 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:43.147104 17070 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260808 01:52:43.147620  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:52:43.149703  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
W20260808 01:52:43.150890  9601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.010s	sys 0.011s
W20260808 01:52:43.151465  9601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.780s	user 0.010s	sys 0.011s
I20260808 01:52:43.147631 17070 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:43.156222  9557 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:43.157307 16785 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:43.177098 16785 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:43.156035  9599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.738s	user 0.010s	sys 0.011s
W20260808 01:52:43.178187  9599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.795s	user 0.011s	sys 0.011s
I20260808 01:52:43.179333 16810 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:43.178920 16785 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:43.180552 16785 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:43.164661  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2946) took 1249 ms. Trace:
I20260808 01:52:43.181190  9633 rpcz_store.cc:276] 0808 01:52:41.915227 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:41.915414 (+   187us) service_pool.cc:224] Handling call
0808 01:52:41.932205 (+ 16791us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:42.973925 (+1041720us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:43.164184 (+190259us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54003f9b54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:43.164492 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2391960320}
I20260808 01:52:43.182238 10368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 831) took 1025 ms. Trace:
I20260808 01:52:43.182760 10368 rpcz_store.cc:276] 0808 01:52:42.156734 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:42.156953 (+   219us) service_pool.cc:224] Handling call
0808 01:52:42.157340 (+   387us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:42.400034 (+242694us) raft_consensus.cc:1534] Early marking committed up to index 16
0808 01:52:42.503532 (+103498us) log.cc:844] Serialized 14 byte log entry
0808 01:52:42.597180 (+ 93648us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:42.597257 (+    77us) raft_consensus.cc:1645] Marking committed up to 16
0808 01:52:42.597316 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:43.181727 (+584411us) spinlock_profiling.cc:243] Waited 242 ms on lock 0x7b5400553054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:43.181798 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:43.182025 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3303,"prepare.queue_time_us":110051,"prepare.run_cpu_time_us":5995,"prepare.run_wall_time_us":8448,"raft.queue_time_us":19952,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":5519,"spinlock_wait_cycles":557338880,"thread_start_us":87093,"threads_started":1,"wal-append.queue_time_us":310956}
I20260808 01:52:43.191896 17079 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:43.192721 10362 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260808 01:52:43.192001 16835 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
I20260808 01:52:43.192749 17079 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:43.172420 16720 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:43.194762 16720 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:43.195556 17079 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:43.196467 16720 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:43.084141 16974 raft_consensus.cc:740] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:43.159770 17046 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:43.161998  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:43.199844 16974 consensus_queue.cc:260] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:43.198468 16720 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:43.178498 16808 raft_consensus.cc:3055] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:43.180133 16810 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:43.197292 17079 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:43.154402  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 818327us
I20260808 01:52:43.204558 17070 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:43.051371  9475 rpcz_store.cc:276] 0808 01:52:41.581212 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:41.581366 (+   154us) service_pool.cc:224] Handling call
0808 01:52:41.581699 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:41.581939 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 13
0808 01:52:42.087857 (+505918us) log.cc:844] Serialized 14 byte log entry
0808 01:52:42.959463 (+871606us) log.cc:844] Serialized 14 byte log entry
0808 01:52:42.960149 (+   686us) log.cc:844] Serialized 14 byte log entry
0808 01:52:42.960466 (+   317us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:42.960563 (+    97us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:42.960701 (+   138us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:42.960862 (+   161us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:42.961239 (+   377us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":196678,"prepare.queue_time_us":181241,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":77,"raft.queue_time_us":298695,"raft.run_cpu_time_us":513,"raft.run_wall_time_us":513,"thread_start_us":1122579,"threads_started":2,"wal-append.queue_time_us":712252}
I20260808 01:52:43.257999 10368 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
W20260808 01:52:43.256704  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 67261.7us
I20260808 01:52:43.211230 16808 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:52:43.268931  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 433870us
W20260808 01:52:43.271102  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 137968us
I20260808 01:52:43.273272  9524 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:43.273917  9601 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:43.275312 17135 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260808 01:52:43.282094 16833 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.949s	user 0.009s	sys 0.020s
I20260808 01:52:43.282766 16833 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:43.283329  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:43.284097  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:43.284232  9559 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:43.202584 17132 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:43.285425 16833 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:43.285863 17132 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:43.148420 16984 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:43.315985  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:43.320675  9473 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 22. Current term is 23. Ops: []
W20260808 01:52:43.321769  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 353386us
I20260808 01:52:43.322868  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:43.324936  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:43.326934 17115 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:43.326761 17007 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.094s
W20260808 01:52:43.329909 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:43.330984 16808 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:43.332742 16808 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:43.334842 17133 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:43.335487  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 61710.2us
I20260808 01:52:43.335777 17133 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:43.338095 10367 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 909) took 2194 ms (client timeout 1582 ms). Trace:
W20260808 01:52:43.338515 10367 rpcz_store.cc:269] 0808 01:52:41.143887 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:41.144148 (+   261us) service_pool.cc:224] Handling call
0808 01:52:43.337583 (+2193435us) spinlock_profiling.cc:243] Waited 587 ms on lock 0x7b5400620f54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:43.337835 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1350294784}
I20260808 01:52:43.339825 16810 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:43.341022 10365 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
W20260808 01:52:43.344928 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:43.326839  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:43.352896  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 56653.6us
I20260808 01:52:43.353358  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:43.309274 16974 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:43.355657 16958 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:43.356313 16984 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:43.361646  9600 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:43.364933 17010 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:43.367244 17159 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260808 01:52:43.367823 17012 consensus_meta.cc:287] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.588s	user 0.003s	sys 0.018s
I20260808 01:52:43.368331 17012 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:43.292539 16194 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.077s
W20260808 01:52:43.359151  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 197411us
W20260808 01:52:43.372229  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:43.370927  9524 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:43.371112  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 73254.7us
I20260808 01:52:43.336678  9601 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:43.371178 17170 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260808 01:52:43.382388  9599 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:43.383656 17046 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:43.384588 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:43.384456 17046 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:43.385566 10363 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 18: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:43.386375 17115 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:43.386927 17115 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:43.387027  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:43.387446 17115 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:43.388995 16871 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:52:43.389531  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 74351.9us
I20260808 01:52:43.390251 16871 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:43.391716  9554 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:43.370636 16194 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:43.393652 16194 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:43.393395 16939 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:43.395095 16194 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:43.396059 10366 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260808 01:52:43.396220 16939 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:43.397040 16939 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:43.397723 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:43.399829  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2810) took 1069 ms. Trace:
I20260808 01:52:43.401449 16983 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:43.390935  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:43.404331 10369 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 24: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:43.366124 17010 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:43.417698  9559 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:52:43.409032 16983 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:43.338533 17133 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:43.411398 16194 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:43.428910 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:43.439296 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:43.440624 17012 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:43.444408 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:43.447948  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:43.400310  9552 rpcz_store.cc:276] 0808 01:52:42.329814 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:42.770793 (+440979us) service_pool.cc:224] Handling call
0808 01:52:42.771221 (+   428us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:42.998543 (+227322us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:42.998655 (+   112us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:43.105069 (+106414us) log.cc:844] Serialized 26 byte log entry
0808 01:52:43.260243 (+155174us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:52:43.260307 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:43.291109 (+ 30802us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:43.399347 (+108238us) raft_consensus.cc:1685] finished
0808 01:52:43.399382 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:43.399598 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":226205,"raft.queue_time_us":279276,"raft.run_cpu_time_us":400,"raft.run_wall_time_us":400,"thread_start_us":154625,"threads_started":1,"wal-append.queue_time_us":281708}
I20260808 01:52:43.430841 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:43.465197  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:43.473680 10366 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 17: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:43.480921 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:43.482533 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:43.482864  9560 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 13.
I20260808 01:52:43.483498 10365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 1126) took 2194 ms. Trace:
W20260808 01:52:43.483620  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:43.484782  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:43.483232 10369 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
W20260808 01:52:43.488857 17140 consensus_meta.cc:287] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.590s	user 0.010s	sys 0.005s
I20260808 01:52:43.483922 10365 rpcz_store.cc:276] 0808 01:52:41.278973 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:41.279165 (+   192us) service_pool.cc:224] Handling call
0808 01:52:43.340428 (+2061263us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:43.341562 (+  1134us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:43.466450 (+124888us) spinlock_profiling.cc:243] Waited 2.06 s on lock 0x7b5400620f50. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:43.473479 (+  7029us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4739118848}
I20260808 01:52:43.489473 17140 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:43.452878 16983 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:43.452163 17130 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:43.500944 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:43.473106 16810 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:43.423947 16867 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:43.423156 17129 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:43.519203  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:43.531716  9559 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24. Ops: []
I20260808 01:52:43.534299 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:43.408583 16939 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:43.507681 17133 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:43.548177  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:43.553942 10362 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
W20260808 01:52:43.532574 16952 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.213s	user 0.017s	sys 0.004s
I20260808 01:52:43.557013  9554 raft_consensus.cc:2410] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260808 01:52:43.538476 16934 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260808 01:52:43.564991  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 147706us
W20260808 01:52:43.568110  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 231552us
I20260808 01:52:43.570338 17156 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:43.570173 17167 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:43.572525  9600 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:43.565737  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 175041us
I20260808 01:52:43.566254  9555 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 22. Current term is 23. Ops: []
I20260808 01:52:43.576843  9559 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 17. Current term is 18. Ops: []
W20260808 01:52:43.578279 17107 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.408s	user 0.060s	sys 0.090s
W20260808 01:52:43.578805 17107 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.514s	user 0.060s	sys 0.090s
W20260808 01:52:43.570127 17110 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.377s	user 0.006s	sys 0.000s
I20260808 01:52:43.585460 17158 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:43.586587  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:43.598623  9554 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 18. Current term is 19. Ops: [18.16-18.16]
I20260808 01:52:43.574572 16871 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:43.592123 16810 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:43.562561 17070 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:43.607865  9554 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:43.605849 17172 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:52:43.613314 10332 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:43.614880  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 150516us
I20260808 01:52:43.616303 17010 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:43.619642 17140 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:43.621913  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2807) took 1293 ms. Trace:
I20260808 01:52:43.628981  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 1031) took 1291 ms. Trace:
I20260808 01:52:43.628791  9557 rpcz_store.cc:276] 0808 01:52:42.328714 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:42.744111 (+415397us) service_pool.cc:224] Handling call
0808 01:52:42.764616 (+ 20505us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:43.177090 (+412474us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:43.621544 (+444454us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b54001dfe54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:43.621812 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":899169664}
I20260808 01:52:43.566000 16952 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:43.607823 17149 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:52:43.598488 17133 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:43.638645  9521 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:43.643509 17158 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:43.661572 17158 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:43.662730  9472 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 21: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:43.645054 17129 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:43.663203 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:43.645547  9601 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:43.598045 17167 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:43.664489 10367 raft_consensus.cc:2410] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 22 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
W20260808 01:52:43.665122  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 61535us
I20260808 01:52:43.647670  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3265) took 1678 ms. Trace:
I20260808 01:52:43.653924  9558 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:43.655071 16850 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:43.627731 17159 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
I20260808 01:52:43.666311 17167 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:43.665804  9474 rpcz_store.cc:276] 0808 01:52:41.969055 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:42.395338 (+426283us) service_pool.cc:224] Handling call
0808 01:52:42.491784 (+ 96446us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:42.722770 (+230986us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:43.647165 (+924395us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b5400403dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:43.647483 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":426240256}
I20260808 01:52:43.666790 17113 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:43.667506 17159 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:43.668044 17159 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:43.668569  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:43.656880 16867 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:43.665525 10368 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 15. (term mismatch)
I20260808 01:52:43.673842 10368 pending_rounds.cc:85] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting all ops after (but not including) 14
I20260808 01:52:43.670006 16835 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:43.674256 10368 pending_rounds.cc:107] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting uncommitted NO_OP operation due to leader change: 18.15
I20260808 01:52:43.685858 10368 raft_consensus.cc:2889] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:43.630203  9551 rpcz_store.cc:276] 0808 01:52:42.331789 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:43.146571 (+814782us) service_pool.cc:224] Handling call
0808 01:52:43.205306 (+ 58735us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:43.206113 (+   807us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:52:43.206146 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:43.266002 (+ 59856us) log.cc:844] Serialized 26 byte log entry
0808 01:52:43.531292 (+265290us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:43.531375 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:43.531519 (+   144us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:43.623121 (+ 91602us) raft_consensus.cc:1685] finished
0808 01:52:43.623146 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:43.623324 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35173,"raft.queue_time_us":3639,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":43111,"thread_start_us":208158,"threads_started":1,"wal-append.queue_time_us":257503}
I20260808 01:52:43.696178 17133 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:43.700555 17112 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.638s	user 0.005s	sys 0.001s
W20260808 01:52:43.701347  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 73887.6us
W20260808 01:52:43.702385 17122 log.cc:927] Time spent T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 1.378s	user 0.000s	sys 0.002s
I20260808 01:52:43.703859  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:43.672150 17046 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:43.707417  9601 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:43.708849  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 70375.7us
W20260808 01:52:43.709551 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 96403.5us
I20260808 01:52:43.710716  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:43.711755 17149 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:43.712263 17149 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:43.712749 17149 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:43.712790 16934 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:43.713681 16934 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:43.713843 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:43.715000 17172 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:43.716117 17046 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:43.716779 17046 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:43.722941 16952 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:43.724790  9523 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:43.726635  9523 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:43.727236 16952 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:43.717952 17046 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:43.737373 16810 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260808 01:52:43.739223  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:43.743462 16867 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:43.746124 16867 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:43.743712 17070 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:43.748164  9601 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:43.751034  9601 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:43.751226  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:43.753861  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:43.787587  9477 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:43.788697 17182 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:52:43.790130 17182 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Stepping down as leader of term 19
W20260808 01:52:43.754773  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:43.790596 17182 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:43.755643  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:43.791363  9524 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:43.792034  9480 raft_consensus.cc:2410] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260808 01:52:43.758039 17167 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:52:43.764456  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3314) took 2173 ms. Trace:
I20260808 01:52:43.768604 17115 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:43.770143 16835 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:43.796227  9478 rpcz_store.cc:276] 0808 01:52:41.591413 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:41.592010 (+   597us) service_pool.cc:224] Handling call
0808 01:52:41.592455 (+   445us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:42.960264 (+1367809us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:43.763919 (+803655us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b5400404cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:43.764203 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3140276480}
I20260808 01:52:43.796907 16835 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:43.797466 16835 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:43.797941 17115 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:43.771967  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3655) took 1864 ms. Trace:
I20260808 01:52:43.773272 17140 consensus_queue.cc:1059] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:52:42.686529  9629 rpcz_store.cc:276] 0808 01:52:41.390762 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:41.418623 (+ 27861us) service_pool.cc:224] Handling call
0808 01:52:41.418984 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:41.419219 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 12
0808 01:52:41.895033 (+475814us) log.cc:844] Serialized 14 byte log entry
0808 01:52:42.627967 (+732934us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:42.628056 (+    89us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:52:42.628103 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:42.628175 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:42.628294 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":168994,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":277790,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":20872,"thread_start_us":1172581,"threads_started":3,"wal-append.queue_time_us":673830}
I20260808 01:52:43.798873  9558 rpcz_store.cc:276] 0808 01:52:41.907190 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:41.907573 (+   383us) service_pool.cc:224] Handling call
0808 01:52:41.927578 (+ 20005us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:43.666044 (+1738466us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:43.771399 (+105355us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b5400313dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:43.771874 (+   475us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3969163520}
I20260808 01:52:43.798622 16835 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:43.779942 17171 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:52:43.809036 16835 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:43.783373 16785 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:43.669447 17159 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:43.757380 16808 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:52:43.807878 17016 raft_consensus.cc:3055] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:43.777628 16934 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:43.812814  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 86273.6us
I20260808 01:52:43.817795  9477 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:52:43.820272 10334 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:43.833611  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:43.814261 17016 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:52:43.836387  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:43.821684 17129 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:43.691270 16974 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.165s
I20260808 01:52:43.835085 17005 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader election won for term 14
I20260808 01:52:43.791999 17182 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:43.835071 17130 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:43.847669 16810 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260808 01:52:43.857429 17190 log.cc:927] Time spent T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.290s	user 0.006s	sys 0.001s
I20260808 01:52:43.858534 16808 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:52:43.858994 17134 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:43.860342  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:43.899489 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:43.861251  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:43.901384 16867 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:43.902078 16867 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:43.902537 16867 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:43.882557 17133 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260808 01:52:43.870956 17115 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
I20260808 01:52:43.861340  9524 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:43.905733 16810 raft_consensus.cc:3055] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:43.906181 16810 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:43.907048 16952 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:43.907675 16952 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:43.907409 16810 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:43.883417 17016 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:43.908828 16952 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:43.863502 17132 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.048s
I20260808 01:52:43.880702 16983 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:43.940276 17132 consensus_queue.cc:1243] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 19.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260808 01:52:43.942308  9550 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:43.861517 17005 raft_consensus.cc:697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:43.867365  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:43.942248 16983 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:43.864795  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:43.913111 17134 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:43.868491  9559 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 19. Current term is 20. Ops: []
W20260808 01:52:43.906565  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 45371us
I20260808 01:52:43.903156  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:43.870762  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 160140us
I20260808 01:52:43.846122 17129 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:43.945591  9560 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:43.867738 17171 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.045s
W20260808 01:52:43.864413 17196 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.168s	user 0.001s	sys 0.000s
I20260808 01:52:43.962057  9552 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:43.944161  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:43.965848  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:43.982348  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:43.990018  9551 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:52:43.903156 17176 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.092s
I20260808 01:52:43.981843 17129 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:43.980722 16939 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:44.028538  9559 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 18. Current term is 19. Ops: []
W20260808 01:52:44.029863  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 278984us
I20260808 01:52:43.943840 17005 consensus_queue.cc:237] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:44.014612 17167 raft_consensus.cc:3055] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:52:44.020533 17182 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
W20260808 01:52:44.047839  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:44.049927  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 70779.3us
I20260808 01:52:44.051817  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:43.981344 17149 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:44.055833  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 152788us
I20260808 01:52:44.047887 16974 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.010s
I20260808 01:52:44.045248 17167 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:44.025313 16934 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:44.062237  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 194222us
I20260808 01:52:44.046150 16194 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:44.041473 16983 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260808 01:52:44.073709  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:44.077890  9551 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:44.088475 16785 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:44.089398 16785 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:44.074045 16194 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:44.098610 16194 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:44.099328 16194 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:44.099972 16194 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:44.106954 16194 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:44.111621 16194 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260808 01:52:44.120867  9478 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260808 01:52:44.094838 17107 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.027s
I20260808 01:52:44.112073 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:44.095031 16934 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:44.131615  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:44.132586  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:44.121096 17197 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260808 01:52:44.134666  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:44.134024  9480 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 11. (term mismatch)
I20260808 01:52:44.123648 10367 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:44.142822  9480 pending_rounds.cc:85] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 10
I20260808 01:52:44.148519  9480 pending_rounds.cc:107] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 20.11
I20260808 01:52:44.148945  9480 raft_consensus.cc:2889] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:44.151975  9472 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:44.154481  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:44.156314  9473 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:52:44.141148 16934 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:44.162571  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:44.155541  9477 raft_consensus.cc:2376] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21. Re-sending same reply.
I20260808 01:52:44.125795 17171 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:52:44.177277  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:44.165215  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 99171.3us
I20260808 01:52:44.189133  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:44.193646  9560 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:52:44.179165 16934 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:44.182499 17171 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:44.091411 17134 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:44.223341  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:44.230837 16808 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:44.231909 16808 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:44.239189  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 99914.4us
I20260808 01:52:44.245740  9472 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
W20260808 01:52:44.246536  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 176269us
I20260808 01:52:44.205768 17176 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:52:44.257618  9601 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:44.258901  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:44.259569 17208 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:44.260716 16785 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:44.269687  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:44.230870  9481 raft_consensus.cc:2955] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Committing config change with OpId 15.11: config changed from index 9 to 11, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:52:44.247067 16983 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260808 01:52:44.276206  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 207980us
I20260808 01:52:44.268747 17149 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:44.289510 17016 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:44.293427  9472 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260808 01:52:44.292906  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:44.303627  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 101997us
I20260808 01:52:44.287822 16983 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:44.245823 16867 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:44.314538  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:44.315193  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 65208.5us
I20260808 01:52:44.276100 17129 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:44.317555  9480 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:44.324110  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:44.311980 16808 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:44.330753  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:44.329030  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:44.329851  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:44.331713  9475 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:44.316215 16983 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:44.331756  9474 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:44.312590 17176 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:44.335369 17176 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:44.335882 17176 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:44.339128 17209 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:44.340773  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:44.341943  9476 raft_consensus.cc:2410] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 20 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 8.
W20260808 01:52:44.345017  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 87509.8us
I20260808 01:52:44.300212 17070 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.022s
I20260808 01:52:44.346294  9472 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:44.356752  9556 raft_consensus.cc:2368] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 21. Current term is 22.
W20260808 01:52:44.364192  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 44711us
I20260808 01:52:44.367003  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:44.368099  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 51202us
I20260808 01:52:44.369078 17159 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:44.370859 16785 consensus_queue.cc:1059] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:44.367095 17210 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:52:44.375082 17210 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:44.375655 17210 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:52:44.376232 17129 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:44.378504 17115 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260808 01:52:44.380174  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:44.370361  9601 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:44.393955  9475 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:44.294090 17167 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:44.299217 17171 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:44.414003  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:44.333359 17208 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260808 01:52:44.479640 17171 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:44.370836 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:44.417878  9480 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
W20260808 01:52:44.437198  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66956us
I20260808 01:52:44.438575 16833 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260808 01:52:44.484561 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:44.484722  9475 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
W20260808 01:52:44.485234 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:44.452921 16867 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:44.455333  9521 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:44.455926  9479 raft_consensus.cc:3055] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:44.457415  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:44.458308  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:44.460302 17206 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.217s	user 0.004s	sys 0.000s
I20260808 01:52:44.466002 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:44.466877 17133 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 01:52:44.471787 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:44.475720 17167 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:44.365820 17012 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:44.478178 16810 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:44.479101 17207 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.178s	user 0.001s	sys 0.001s
W20260808 01:52:44.484601  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:44.493714  9473 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
W20260808 01:52:44.503221  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 91454us
I20260808 01:52:44.495913  9479 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:44.506341  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 104842us
I20260808 01:52:44.509172 10367 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:44.382602 16808 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.112s
I20260808 01:52:44.510509 10366 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 20: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:44.512513 17012 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:44.514554 17107 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260808 01:52:44.525838  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 70592.1us
I20260808 01:52:44.526218  9479 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:44.527899  9479 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:44.532259  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:44.533367 17182 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:44.533690  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:44.521445  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:44.541515  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:44.544245  9551 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:44.478442 16934 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:44.568241  9601 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:44.540673 17107 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:44.554476 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:44.579773 17107 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:44.570163  9472 raft_consensus.cc:1217] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Deduplicated request from leader. Original: 15.9->[17.10-20.12]   Dedup: 17.10->[18.11-20.12]
I20260808 01:52:44.584575 17113 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:52:44.585182 17113 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:44.570528  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:44.592381  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:44.592823  9473 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:44.515540 17129 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.047s
I20260808 01:52:44.594017  9472 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: [17.10-20.12]
I20260808 01:52:44.597239  9474 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:44.601827  9552 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:44.566644 16983 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:44.590986 16974 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260808 01:52:44.605623  9521 connection.cc:256] RPC call timeout handler was delayed by -1.4950402829854283s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2117s for a response
I20260808 01:52:44.547408 16952 raft_consensus.cc:2764] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260808 01:52:44.609570 16833 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:44.620466 17113 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:44.573732 17176 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:44.629046  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 107715us
I20260808 01:52:44.630251 17209 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:44.641587  9551 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:44.610208 17172 raft_consensus.cc:3055] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:44.671017  9552 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260808 01:52:44.673056 10367 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
W20260808 01:52:44.678661  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 110493us
I20260808 01:52:44.695740  9555 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:44.679616  9599 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:44.651211 16939 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:44.705245  9478 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260808 01:52:44.684345  9523 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:44.699715 10332 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.617s (SENT) [suppressed 23 similar messages]
I20260808 01:52:44.664903 17208 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:44.670710 17172 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:44.648561 16833 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:44.736799  9473 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
W20260808 01:52:44.739269  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 89303.2us
I20260808 01:52:44.707906 16808 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:44.715968 17129 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:52:44.673738 17209 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:44.771024  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:44.774278  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 240780us
I20260808 01:52:44.731284 16835 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:44.812799  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:44.783355  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 48018.5us
I20260808 01:52:44.835130  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:44.767804 17113 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:52:44.779739 16808 raft_consensus.cc:697] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:44.793170 17129 raft_consensus.cc:697] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:44.876583 17212 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
W20260808 01:52:44.911908 17219 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260808 01:52:44.782554 16974 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.107s
I20260808 01:52:44.916021  9479 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 11. (term mismatch)
I20260808 01:52:44.922747  9479 pending_rounds.cc:85] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 10
W20260808 01:52:44.922502 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 222949us
I20260808 01:52:44.923019  9479 pending_rounds.cc:107] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 19.11
I20260808 01:52:44.923277  9479 raft_consensus.cc:2889] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:52:44.887816 17197 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260808 01:52:44.876044 17133 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:44.914417 16833 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:44.894096 17182 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:52:44.795099 17210 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:44.799366 16835 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:44.934792 17046 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:44.935186 16785 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 20 current term: 20 is higher.
I20260808 01:52:44.937402 16720 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:44.938704 17129 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:44.939881 17012 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:44.940361 10332 leader_election.cc:400] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 21. Current term is 22.
I20260808 01:52:45.024566 10332 leader_election.cc:403] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260808 01:52:45.024830 17012 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:44.941043 17211 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:44.942157 17176 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:44.943005 17016 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:44.896270 17172 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:44.952981 16194 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:44.953131  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:44.953785  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:45.034360 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:52:45.025501 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 85488.1us
I20260808 01:52:44.954156  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:44.953214  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:44.932981 16974 consensus_queue.cc:1243] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 18.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260808 01:52:45.026459 17016 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:44.956789  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:44.961377  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 126457us
I20260808 01:52:44.962116 16934 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:44.973825 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:45.045326 16194 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:44.975948  9475 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260808 01:52:45.046041 16194 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:52:44.976922 17217 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.311s	user 0.001s	sys 0.000s
W20260808 01:52:44.987066  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 163437us
W20260808 01:52:44.995697  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:45.051888  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:44.999488  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:45.008517 17210 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:45.010213 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:45.010902  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:44.992093 17204 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260808 01:52:45.025306 17167 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 21. Current term is 22.
I20260808 01:52:45.025978 17211 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:44.954881  9368 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55058 (request call id 131) took 29627 ms (client timeout 15000 ms). Trace:
I20260808 01:52:45.066375 17209 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:45.029035 17205 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:45.027446 17172 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:45.047034  9478 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:45.016404 17046 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:45.045615 10361 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:45.051018  9477 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24. Ops: []
I20260808 01:52:45.026907 17133 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:45.017130 17182 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:45.046666 17176 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:45.060176  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:45.026981 17149 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:45.076395  9551 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260808 01:52:45.086386  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:45.078823  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:45.079111  9560 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:45.090934  9556 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260808 01:52:45.095708 16194 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.079586  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:45.098181  9477 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:52:45.098806  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47627us
I20260808 01:52:45.078415  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.082486  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:45.101438  9524 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:45.113102 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 129104us
I20260808 01:52:45.118003 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:45.122882  9629 raft_consensus.cc:2368] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 18. Current term is 19.
I20260808 01:52:45.127740  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:45.033759 16720 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:45.074946 16952 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:45.132339  9524 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:45.066663 17211 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:45.139843  9554 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:52:45.068044 17016 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:45.143420  9472 raft_consensus.cc:1217] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Deduplicated request from leader. Original: 14.13->[15.14-15.14]   Dedup: 15.14->[]
I20260808 01:52:45.144084  9472 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 15. Current term is 16. Ops: [15.14-15.14]
I20260808 01:52:45.139011  9638 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:45.113029 16934 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:52:45.120016 17210 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:45.080734 16850 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.080s
I20260808 01:52:45.153092  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:45.153937  9478 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:52:45.153981  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:45.155328 10367 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 24: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:45.159034 16835 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:45.159570 16720 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:45.161178  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.161919  9634 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:45.076406 16867 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.070s
I20260808 01:52:45.072300 17133 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.113407 16974 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.036s
I20260808 01:52:45.068159 17205 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.079298 17176 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:45.115960 17167 consensus_queue.cc:1059] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
W20260808 01:52:45.148290 17171 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.668s	user 0.015s	sys 0.001s
I20260808 01:52:45.133415 17232 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:45.067493 16808 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:45.164687 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:45.173353  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:45.175442 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 112373us
I20260808 01:52:45.150785 17134 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:45.218842  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:45.219679  9635 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:45.221643  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:45.222462  9630 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:45.146394 17149 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:45.174578  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:45.175757  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.150882 17210 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:45.183039  9637 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260808 01:52:45.162488 17158 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:45.190771  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:45.233294 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:45.193991  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:45.162530 17223 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:45.188990 17133 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:45.206348  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:45.073642 17182 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:45.240125  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 94725.7us
W20260808 01:52:45.239133  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:45.150313 16952 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:45.238691  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.247242  9474 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:52:45.226662 16194 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:45.236979  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:45.254838  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:45.199329 17232 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:45.198134 16867 consensus_queue.cc:1243] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 21.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260808 01:52:45.264676  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:45.262261  9634 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:45.254315  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.178544 17046 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:45.273619  9474 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:45.263350 10334 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:45.233031 17016 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:45.238211 17171 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:45.282212  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:45.283007  9472 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:52:45.284198  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:45.245684 16934 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:45.274471  9633 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
W20260808 01:52:45.359442 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 96195.2us
I20260808 01:52:45.262884 17211 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:52:45.360018  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:45.384682  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:45.385645  9474 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:45.379971  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.265851 17235 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:45.398763 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 113499us
I20260808 01:52:45.232046 17210 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:45.407338  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:45.397482  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 136075us
I20260808 01:52:45.407465  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:45.266358 17232 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:45.258046 17133 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.125s
I20260808 01:52:45.279443 17237 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:45.285009 17208 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:45.418897  9472 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
W20260808 01:52:45.403515 17113 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.505s	user 0.020s	sys 0.002s
I20260808 01:52:45.265461 17167 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.143s
I20260808 01:52:45.253261 17223 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.359493 17134 raft_consensus.cc:697] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:45.442550  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:45.442700  9524 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:45.427124  9472 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260808 01:52:45.441766  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.428718  9474 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 9. (term mismatch)
I20260808 01:52:45.456815  9524 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:45.362399 17171 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.406010 17046 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:45.454552  9474 pending_rounds.cc:85] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 8
I20260808 01:52:45.467026  9474 pending_rounds.cc:107] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 17.9
I20260808 01:52:45.467453  9474 raft_consensus.cc:2889] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:45.462652  9632 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:45.458945 10334 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:45.420516 16810 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:45.466395  9478 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:45.472105  9635 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:45.452382 16974 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:52:45.288539 17016 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:45.483021  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:45.484829  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:45.485949  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:45.486164  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:45.486900  9635 raft_consensus.cc:2410] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 24 because replica has last-logged OpId of term: 23 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260808 01:52:45.469179 17046 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:45.491303  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:45.496407  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:52:45.501245 17240 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:45.443490 17237 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:45.504344 17107 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.924s	user 0.015s	sys 0.000s
I20260808 01:52:45.504927 17237 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:45.504846 17107 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.506383  9480 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: [16.8-16.8]
I20260808 01:52:45.506886 16934 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.508284  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3297) took 1036 ms. Trace:
I20260808 01:52:45.508826  9480 rpcz_store.cc:276] 0808 01:52:44.471977 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:44.583747 (+111770us) service_pool.cc:224] Handling call
0808 01:52:44.584227 (+   480us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:45.507108 (+922881us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:45.507872 (+   764us) spinlock_profiling.cc:243] Waited 922 ms on lock 0x7b540039a154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:45.508203 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2119946624}
I20260808 01:52:45.511018 17149 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:45.511693 16952 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:45.512413 16952 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:45.511976 17149 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:45.513013 16952 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:45.513213 17149 consensus_queue.cc:1243] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 19.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260808 01:52:45.480610  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.515250  9636 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:45.517102 17242 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.022s
I20260808 01:52:45.423206 17235 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.519927 16939 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:45.520656 16939 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:45.521584 17235 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:45.444130 17208 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.522889 16974 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:45.524896 17208 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:45.524515 16974 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.517130 17176 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:45.066953  9368 rpcz_store.cc:269] 0808 01:52:15.327489 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:15.328149 (+   660us) service_pool.cc:224] Handling call
0808 01:52:44.954559 (+29626410us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:52:41.097731 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:52:41.098455 (+   724us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:52:41.098510 (+    55us) write_op.cc:435] Acquired schema lock
0808 01:52:41.098537 (+    27us) tablet.cc:662] Decoding operations
0808 01:52:41.100292 (+  1755us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:52:41.100413 (+   121us) write_op.cc:641] Partition lock acquired for write op
0808 01:52:41.100443 (+    30us) tablet.cc:685] Acquiring locks for 18 operations
0808 01:52:41.102013 (+  1570us) tablet.cc:701] Row locks acquired
0808 01:52:41.102031 (+    18us) write_op.cc:260] PREPARE: finished
0808 01:52:41.102229 (+   198us) write_op.cc:270] Start()
0808 01:52:41.102388 (+   159us) write_op.cc:276] Timestamp: P: 1786153961102152 usec, L: 0
0808 01:52:41.102427 (+    39us) op_driver.cc:348] REPLICATION: starting
0808 01:52:41.103552 (+  1125us) log.cc:844] Serialized 7107 byte log entry
0808 01:52:42.522244 (+1418692us) op_driver.cc:464] REPLICATION: finished
0808 01:52:42.816226 (+293982us) write_op.cc:301] APPLY: starting
0808 01:52:42.816400 (+   174us) tablet.cc:1367] starting BulkCheckPresence
0808 01:52:42.866254 (+ 49854us) tablet.cc:1370] finished BulkCheckPresence
0808 01:52:42.866390 (+   136us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:52:42.899865 (+ 33475us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:52:42.900398 (+   533us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=36,key_file_lookups=36,delta_file_lookups=36,mrs_lookups=0
0808 01:52:42.900491 (+    93us) write_op.cc:312] APPLY: finished
0808 01:52:43.073901 (+173410us) log.cc:844] Serialized 164 byte log entry
0808 01:52:43.377549 (+303648us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:52:43.437409 (+ 59860us) write_op.cc:454] Released schema lock
0808 01:52:43.437596 (+   187us) write_op.cc:341] FINISH: Updating metrics
0808 01:52:43.438323 (+   727us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:52:43.438368 (+    45us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6957414,"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":263436,"apply.run_cpu_time_us":85329,"apply.run_wall_time_us":622054,"cfile_cache_hit":74,"cfile_cache_hit_bytes":79475,"num_ops":18,"prepare.queue_time_us":221597,"prepare.run_cpu_time_us":27018,"prepare.run_wall_time_us":355890,"raft.queue_time_us":664625,"raft.run_cpu_time_us":16433,"raft.run_wall_time_us":327251,"replication_time_us":1390839,"thread_start_us":1012129,"threads_started":5,"wal-append.queue_time_us":746718,"wal-append.run_cpu_time_us":23704,"wal-append.run_wall_time_us":2405738}]]}
W20260808 01:52:45.526250  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:45.526963  9560 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:45.543322  9638 raft_consensus.cc:2410] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 20 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
W20260808 01:52:45.546484  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 89782.6us
I20260808 01:52:45.514353 17149 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:45.488760 17012 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:45.529079 17046 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:45.567090 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:45.563786 17205 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:45.518553 17171 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:45.494349 16810 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.514117 16952 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.556043 17012 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:45.552284 17149 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:45.589480 17211 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:45.566663 17240 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:45.588629 17012 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:45.612744 17242 consensus_queue.cc:1059] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:52:45.614665  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:45.617318  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:45.624249  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:45.627559  9632 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:52:45.628676  9480 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:45.633767  9630 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:52:45.569136 17239 consensus_queue.cc:1059] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20260808 01:52:45.648118  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:45.653820  9472 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 20: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:45.654575  9630 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260808 01:52:45.566560 17134 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.660778  9636 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260808 01:52:45.584628 17243 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.063s
I20260808 01:52:45.665303  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:45.636194 17149 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:45.674263 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:45.601631 17211 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:45.688884  9633 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:52:45.693977  9630 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260808 01:52:45.674867 16850 raft_consensus.cc:3055] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:45.699849  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:45.705417  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:45.729861  9472 raft_consensus.cc:2410] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 20 because replica has last-logged OpId of term: 18 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 15.
I20260808 01:52:45.732638  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.704020  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:45.702134 16850 raft_consensus.cc:740] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:45.705832 16808 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:45.702170 17246 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:45.735790  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:45.740559  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.731892 17016 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:45.741793  9472 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260808 01:52:45.731683 17149 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:45.748620 17243 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:45.749330 17243 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.700515 17223 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:45.745738  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:45.746487  9631 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:45.738876 17205 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:45.706239 17242 raft_consensus.cc:3055] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:45.747748 17262 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:45.749790 16194 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:45.733316 17239 raft_consensus.cc:3055] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:45.676532 17240 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.770944  9635 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: [14.9-16.10]
I20260808 01:52:45.769951 16194 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:45.752833 16810 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:45.772918 16194 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:45.773109  9472 pending_rounds.cc:85] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 9
W20260808 01:52:45.763710 17252 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.080s	user 0.001s	sys 0.001s
I20260808 01:52:45.778942  9472 pending_rounds.cc:107] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 18.10
I20260808 01:52:45.780828  9472 raft_consensus.cc:2889] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:45.760300 17242 raft_consensus.cc:740] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:45.776603 16850 consensus_queue.cc:260] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:45.783829  9557 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7c508218361c446898fede7c5682fa9e"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:47726
I20260808 01:52:45.784723 16939 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.745623 16808 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:45.797688  9630 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:45.769793 17239 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:45.799201  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:45.790915  9557 raft_consensus.cc:606] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Received request to transfer leadership
I20260808 01:52:45.801995  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:45.746583 17246 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:45.801836 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 327852us
I20260808 01:52:45.808425  9630 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:45.809049  9635 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
W20260808 01:52:45.810261 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:45.811595  9631 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:45.812670  9633 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: [17.11-18.12]
I20260808 01:52:45.805822 16850 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:45.775564 17262 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.829525  9480 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
W20260808 01:52:45.831676  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:45.835109 16763 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:45.835942 17172 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.767s	user 0.012s	sys 0.005s
I20260808 01:52:45.835841 16763 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:45.836987  9634 raft_consensus.cc:2410] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 19 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260808 01:52:45.837824  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:45.838518 16983 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:45.839087 17239 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.839300  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:45.846009  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:45.847291  9521 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.108s	user 0.056s	sys 0.115s
I20260808 01:52:45.838361 16763 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:45.852280  9521 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.113s	user 0.056s	sys 0.115s
W20260808 01:52:45.852905  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 1.11411e+06us
I20260808 01:52:45.857649 17210 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:45.839880 17046 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:45.859120 17262 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:45.861212 17046 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:45.863111 17243 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:45.842213 16808 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:45.842769 17242 consensus_queue.cc:260] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:45.865273  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:45.865757 17242 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
W20260808 01:52:45.842118 17256 log.cc:927] Time spent T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260808 01:52:45.867848 16939 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:45.869293 17246 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:45.869563  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.870191 17239 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:45.870388  9637 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:52:45.871763  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.871846 17012 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:45.873087  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.873700  9637 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:52:45.874657  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.876029  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:45.883078  9629 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:45.883678 17107 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:45.884809 17241 raft_consensus.cc:3055] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:52:45.885260 17241 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:45.882400  9637 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:52:45.887086 16871 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:45.881496 17133 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260808 01:52:45.887569 16871 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:45.874929 17012 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:45.879567 17262 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:45.888599 16871 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.889374 17262 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.889658  9635 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:45.889742 16871 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:45.887415  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.897604 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:45.891629  9637 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:52:45.899595  9554 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7c508218361c446898fede7c5682fa9e"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:47726
I20260808 01:52:45.899809  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.900296  9554 raft_consensus.cc:606] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Received request to transfer leadership
I20260808 01:52:45.901455 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:45.886215 17241 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:45.902738  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3282) took 1781 ms. Trace:
I20260808 01:52:45.902787 17241 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:45.903189  9481 rpcz_store.cc:276] 0808 01:52:44.121677 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:44.122250 (+   573us) service_pool.cc:224] Handling call
0808 01:52:44.126158 (+  3908us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:44.151747 (+ 25589us) raft_consensus.cc:1534] Early marking committed up to index 12
0808 01:52:44.519761 (+368014us) log.cc:844] Serialized 14 byte log entry
0808 01:52:44.970273 (+450512us) spinlock_profiling.cc:243] Waited 3.48 ms on lock 0x7b18004a0200. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d41889e05 00007f5d483aada7 00007f5d4499dd87 00007f5d449ad49f 00007f5d449ad418
0808 01:52:45.127568 (+157295us) log.cc:844] Serialized 14 byte log entry
0808 01:52:45.127847 (+   279us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:45.144551 (+ 16704us) log.cc:844] Serialized 26 byte log entry
0808 01:52:45.144696 (+   145us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:45.145147 (+   451us) log.cc:844] Serialized 14 byte log entry
0808 01:52:45.146600 (+  1453us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:45.156599 (+  9999us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:45.876299 (+719700us) spinlock_profiling.cc:243] Waited 56.2 ms on lock 0x7b18004a0200. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:52:45.878883 (+  2584us) spinlock_profiling.cc:243] Waited 2.44 ms on lock 0x7b18004a0200. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:52:45.902489 (+ 23606us) raft_consensus.cc:1685] finished
0808 01:52:45.902511 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:45.902649 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9234,"prepare.queue_time_us":60100,"prepare.run_cpu_time_us":139,"prepare.run_wall_time_us":5315,"raft.queue_time_us":21970,"raft.run_cpu_time_us":662,"raft.run_wall_time_us":69855,"spinlock_wait_cycles":142827648,"thread_start_us":384968,"threads_started":2,"wal-append.queue_time_us":386514}
W20260808 01:52:45.913738  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:45.906397  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:45.907240  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:45.920469  9599 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:45.921226  9629 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:52:45.924325  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.925716  9632 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:52:45.926554 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:45.926864  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.927297 10362 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:45.929706  9551 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260808 01:52:45.929436  9521 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:45.932144  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:45.933256  9551 raft_consensus.cc:2410] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 22 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260808 01:52:45.933974  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.940213 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:45.941419  9444 connection.cc:256] RPC call timeout handler was delayed by -1.4215442749846261s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16090000000000002s for a response
W20260808 01:52:45.941968  9444 connection.cc:256] RPC call timeout handler was delayed by -1.2928442749835085s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17520000000000002s for a response
W20260808 01:52:45.942345  9444 connection.cc:256] RPC call timeout handler was delayed by -1.123644274979597s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.194s for a response
W20260808 01:52:45.942665  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0867442750022747s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19810000000000003s for a response
W20260808 01:52:45.942993  9444 connection.cc:256] RPC call timeout handler was delayed by -1.021044274995802s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2054s for a response
I20260808 01:52:45.944418  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:45.946144 10367 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:45.943203  9636 raft_consensus.cc:2955] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Committing config change with OpId 15.11: config changed from index 9 to 11, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:52:45.948140  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.949111  9630 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:52:45.955930  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.957679  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:45.958864  9629 raft_consensus.cc:2410] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260808 01:52:45.959390  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:45.960140  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:45.976301  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:45.977766  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:45.978628  9555 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 24.
I20260808 01:52:45.914288 16867 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:45.977334  9552 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:52:45.972216  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:45.984356  9637 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:45.908193 17210 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:45.930938 17149 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:45.991189  9629 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260808 01:52:45.987861 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:45.903872 17232 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:46.003677  9632 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: [18.11-20.12]
I20260808 01:52:46.013013  9551 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:46.007017 17149 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:46.009455 17182 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:46.010491  9629 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: []
I20260808 01:52:46.011129 10361 raft_consensus.cc:2393] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 20: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:46.012388  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:45.947132 17133 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:46.039115  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:45.945533 17159 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260808 01:52:46.017057  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 93801.9us
I20260808 01:52:46.021720  9637 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260808 01:52:46.043426  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:46.018186  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:46.044783  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:46.045156  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:46.044682  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:46.046715  9555 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:46.047689  9629 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 10. (term mismatch)
I20260808 01:52:46.033941  9601 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:46.034650 16808 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:45.960570 17176 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:46.053314  9557 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:45.988040 17134 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:46.048228  9629 pending_rounds.cc:85] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 9
I20260808 01:52:46.060323  9629 pending_rounds.cc:107] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 19.10
I20260808 01:52:46.060604  9629 raft_consensus.cc:2889] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:46.063298 16810 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:46.063937 16810 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:46.069101  9559 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260808 01:52:46.066362 16810 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:46.073172  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:46.076334 17149 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:46.076742 17232 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:46.077270  9556 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:46.014461 16867 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:46.078891  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:46.080348  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:46.080803 16867 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:46.012692 17245 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:46.082774 17245 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:46.083547 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:46.085301 17245 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:46.085178 17211 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:46.084256  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:46.030781 17242 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.056s
I20260808 01:52:46.087538 17176 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:46.088591 17242 raft_consensus.cc:993] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4: : Instructing follower 6e7856bed1be43acaa821290d506ee97 to start an election
I20260808 01:52:46.089250 17242 raft_consensus.cc:1081] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Signalling peer 6e7856bed1be43acaa821290d506ee97 to start an election
I20260808 01:52:46.089457 17211 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:46.095249  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:46.095572  9599 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:46.105314 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:46.114192  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:46.055918 16785 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:46.130501 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:46.131734  9521 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:46.054867 17158 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:46.056463 17159 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:46.135401  9629 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 16 index: 14. (term mismatch)
I20260808 01:52:46.139411  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:46.142508  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:46.144630  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:46.135902  9629 pending_rounds.cc:85] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 13
I20260808 01:52:46.154242  9629 pending_rounds.cc:107] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 15.14
I20260808 01:52:46.154560  9629 raft_consensus.cc:2889] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:52:46.154749  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:46.149876 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:46.150123 10361 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260808 01:52:46.150605  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:46.149619 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:46.125300 17246 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:46.155380 10363 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:46.195823  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:46.197288  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:46.198422  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 3327) took 1206 ms. Trace:
I20260808 01:52:46.198776 17211 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:46.199024  9476 rpcz_store.cc:276] 0808 01:52:44.992258 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:44.999145 (+  6887us) service_pool.cc:224] Handling call
0808 01:52:46.198155 (+1199010us) spinlock_profiling.cc:243] Waited 594 ms on lock 0x7b5400399254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:46.198305 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1365714304}
I20260808 01:52:46.199461 17211 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:46.133054 16835 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:46.201049  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:46.093835 16808 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:46.207129 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:46.229851  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:46.140328 16785 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:46.151149 17232 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:46.140602 17159 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:46.211242  9601 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:46.112002 17271 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.081s
I20260808 01:52:46.213973 16194 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:46.237819 16194 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:46.238759  9559 raft_consensus.cc:2410] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 17 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
W20260808 01:52:46.214579  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:46.124722 17242 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:46.222841  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:46.229732  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:46.240612  9556 raft_consensus.cc:2410] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 21 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260808 01:52:46.148908 17158 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:46.059572 17133 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:46.225258  9557 raft_consensus.cc:2410] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 20 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260808 01:52:46.227301  9550 raft_consensus.cc:2410] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260808 01:52:46.139395 17210 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.094s
I20260808 01:52:46.145785 17172 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:46.233093  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 90705.7us
I20260808 01:52:46.257995  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:46.236063  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:46.192942  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:46.246256  9479 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 21: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:46.247388  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:46.249161  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:46.249229 17259 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:46.228785 16835 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:46.224965 10362 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:46.262166  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:46.233860 17134 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260808 01:52:46.232088 17239 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:46.291527 10367 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
W20260808 01:52:46.255045 17272 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260808 01:52:46.246817 17251 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:46.301394  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 90267.5us
I20260808 01:52:46.303050 16194 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:46.271682 17246 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260808 01:52:46.274853 16835 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:46.275828 17265 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:46.300367  9552 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 12. (term mismatch)
I20260808 01:52:46.315997  9552 pending_rounds.cc:85] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 11
I20260808 01:52:46.316305  9552 pending_rounds.cc:107] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 20.12
I20260808 01:52:46.316572  9552 raft_consensus.cc:2889] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:46.279152  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:46.314885  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:46.304625 17251 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:46.320703  9521 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:46.325517  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:46.263979 17176 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:46.275077 17232 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.041s
I20260808 01:52:46.274403 17259 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:46.341223  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:46.301774 17239 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:46.273077 17172 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:46.347033  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:46.342541  9476 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:46.326933 17265 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:46.329191 17241 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:46.331463 17242 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
I20260808 01:52:46.326529 17134 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260808 01:52:46.317185 16934 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.032s
I20260808 01:52:46.379743  9601 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:46.343955 16785 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:46.393294  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:46.401491  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:46.407577  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:46.413923  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:46.415282  9476 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
W20260808 01:52:46.421587  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:46.422923  9601 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:46.424322  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2971) took 1164 ms. Trace:
I20260808 01:52:46.408746 16810 raft_consensus.cc:2764] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260808 01:52:46.409035 17210 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:46.407066 17246 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:52:46.427228  9634 rpcz_store.cc:276] 0808 01:52:45.255258 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:45.280842 (+ 25584us) service_pool.cc:224] Handling call
0808 01:52:46.414496 (+1133654us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b540003b1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:46.418735 (+  4239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1036102912}
W20260808 01:52:46.420313  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 475914us
W20260808 01:52:46.437006  9360 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 318436us
I20260808 01:52:46.440085  9601 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:46.435868 17265 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:46.451802  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:46.454521  9476 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:52:46.458112  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:46.458902  9634 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:46.501368  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:46.501488 10334 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:46.504585  9634 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:46.508725  9557 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7c508218361c446898fede7c5682fa9e"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:47726
I20260808 01:52:46.510154  9557 raft_consensus.cc:606] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Received request to transfer leadership
I20260808 01:52:46.460712 17210 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260808 01:52:46.509688 17276 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.185s	user 0.000s	sys 0.001s
I20260808 01:52:46.512440  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:46.499755 17208 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:46.514927  9476 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 20
I20260808 01:52:46.515270  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:46.503087 17182 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260808 01:52:46.510805 17275 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
I20260808 01:52:46.508725 17239 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:46.527043  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:46.508428 17289 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:46.510605 17158 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:46.534157 17278 log.cc:927] Time spent T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260808 01:52:46.559333  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:46.560109  9472 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:46.561223  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:46.534466 17289 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:46.563122 17289 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:52:46.543931 17281 log.cc:927] Time spent T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
W20260808 01:52:46.547569 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:46.516245 17285 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:46.550355  9635 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 22. Current term is 23. Ops: []
I20260808 01:52:46.552062  9637 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260808 01:52:46.562458  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:46.566074  9477 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:46.569813  9635 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:46.572180  9634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:46.572906  9634 raft_consensus.cc:606] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:46.573637  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:46.575775  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2960) took 1388 ms. Trace:
I20260808 01:52:46.574838  9635 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260808 01:52:46.576159  9638 rpcz_store.cc:276] 0808 01:52:45.187151 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:45.205988 (+ 18837us) service_pool.cc:224] Handling call
0808 01:52:46.575540 (+1369552us) spinlock_profiling.cc:243] Waited 303 ms on lock 0x7b54004218d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:46.575703 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":696311168}
I20260808 01:52:46.576779  9632 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 19. Current term is 20. Ops: [18.12-19.13]
I20260808 01:52:46.577608  9638 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260808 01:52:46.577756  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:46.578261  9632 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260808 01:52:46.578413  9472 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:46.579187  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3328) took 1579 ms. Trace:
I20260808 01:52:46.572329 17285 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:46.580018  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:46.579567  9473 rpcz_store.cc:276] 0808 01:52:44.999817 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:45.051202 (+ 51385us) service_pool.cc:224] Handling call
0808 01:52:45.064485 (+ 13283us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:45.571250 (+506765us) raft_consensus.cc:1534] Early marking committed up to index 15
0808 01:52:45.571301 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:45.579254 (+  7953us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:52:45.579305 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:46.578971 (+999666us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b54002c0954. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:46.579017 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:46.579130 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":52305,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":75016,"spinlock_wait_cycles":370440960,"thread_start_us":345152,"threads_started":1}
I20260808 01:52:46.581543 17285 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:46.581584  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:46.585618 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:46.585752  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 145864us
I20260808 01:52:46.587729  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:46.590639  9477 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:52:46.590927  9638 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
W20260808 01:52:46.592458 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 91077.6us
I20260808 01:52:46.592046 17246 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
I20260808 01:52:46.601115  9474 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260808 01:52:46.570382 17172 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:46.606338  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:46.611682 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:46.643685  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 116745us
I20260808 01:52:46.655426  9474 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: []
W20260808 01:52:46.656886  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 69266.7us
I20260808 01:52:46.659198 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:46.660000 10362 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:52:46.660393  9476 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:52:46.661851  9474 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260808 01:52:46.608364 17285 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:52:46.664060 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:46.658217  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:46.659282 16871 consensus_meta.cc:287] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.769s	user 0.015s	sys 0.000s
I20260808 01:52:46.672699 17242 raft_consensus.cc:3055] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:52:46.675616  9523 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:46.597599 16835 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:46.693364  9600 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:46.661543 17210 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:46.662601 17288 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:46.707685  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 2974) took 1212 ms. Trace:
I20260808 01:52:46.708470  9631 rpcz_store.cc:276] 0808 01:52:45.494913 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:45.813158 (+318245us) service_pool.cc:224] Handling call
0808 01:52:45.814453 (+  1295us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:45.832496 (+ 18043us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:45.832529 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:45.845641 (+ 13112us) log.cc:844] Serialized 26 byte log entry
0808 01:52:46.248395 (+402754us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:46.248452 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:46.248557 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:46.707356 (+458799us) raft_consensus.cc:1685] finished
0808 01:52:46.707390 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:46.707595 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":152,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":1522,"thread_start_us":333262,"threads_started":1,"wal-append.queue_time_us":496753}
I20260808 01:52:46.713845  9629 raft_consensus.cc:2368] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22.
I20260808 01:52:46.717926  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:46.723714 16934 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:46.723584 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:46.683022 17242 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:46.703815 17171 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:46.706358 17208 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260808 01:52:46.746351  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 67537.8us
W20260808 01:52:46.748353 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 153828us
W20260808 01:52:46.758749 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:46.761874  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:46.735379 17271 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:46.713594 17297 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:52:46.704224 17232 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
I20260808 01:52:46.735783 17210 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:46.784010  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:46.739346 17295 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:46.787477  9479 raft_consensus.cc:3055] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:46.788019  9479 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:46.796056  9479 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:46.799772 17232 consensus_queue.cc:1243] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 22.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
W20260808 01:52:46.803601  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 69776.7us
I20260808 01:52:46.782138 17297 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:46.782315 17271 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:52:46.807893  9478 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 3335) took 1655 ms (client timeout 1914 ms). Trace:
W20260808 01:52:46.808629  9478 rpcz_store.cc:269] 0808 01:52:45.152824 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:45.156447 (+  3623us) service_pool.cc:224] Handling call
0808 01:52:46.807670 (+1651223us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b540021d754. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:46.807818 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":552995072}
W20260808 01:52:46.810318  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 117047us
W20260808 01:52:46.810783  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 152710us
I20260808 01:52:46.812129  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:46.754405 17171 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:46.822700  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:46.843838  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:46.792474 16871 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:52:46.812002 17271 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:46.792965 17295 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:46.844340  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:46.847810  9478 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260808 01:52:46.791047 17240 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:46.843663  9479 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:46.860376  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46808.5us
I20260808 01:52:46.781641 17208 consensus_queue.cc:1059] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.032s
I20260808 01:52:46.863306  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3133) took 1318 ms. Trace:
I20260808 01:52:46.808640 17297 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:52:46.886878  9478 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260808 01:52:46.885975  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:46.890204  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:46.863008 17205 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260808 01:52:46.893889 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 133782us
I20260808 01:52:46.873637  9636 rpcz_store.cc:276] 0808 01:52:45.544900 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:45.942353 (+397453us) service_pool.cc:224] Handling call
0808 01:52:45.942734 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:45.943057 (+   323us) raft_consensus.cc:1534] Early marking committed up to index 13
0808 01:52:46.450786 (+507729us) log.cc:844] Serialized 14 byte log entry
0808 01:52:46.731828 (+281042us) log.cc:844] Serialized 14 byte log entry
0808 01:52:46.861270 (+129442us) log.cc:844] Serialized 14 byte log entry
0808 01:52:46.862474 (+  1204us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:46.862549 (+    75us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:46.862622 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:46.862754 (+   132us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:46.862956 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29423,"prepare.queue_time_us":26615,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":86,"raft.queue_time_us":93183,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":10789,"spinlock_wait_cycles":24041856,"thread_start_us":248442,"threads_started":2,"wal-append.queue_time_us":292519}
I20260808 01:52:46.867457 17246 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:46.866155 17295 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:46.868534 17285 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260808 01:52:46.900360  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 56119.7us
I20260808 01:52:46.910187  9479 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 12. (term mismatch)
I20260808 01:52:46.911165  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:46.911594  9479 pending_rounds.cc:85] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 11
I20260808 01:52:46.914834  9479 pending_rounds.cc:107] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 18.12
I20260808 01:52:46.907788 16835 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:46.915138  9479 raft_consensus.cc:2889] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:52:46.916296 17304 log.cc:927] Time spent T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260808 01:52:46.916970  9479 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
W20260808 01:52:46.909898 17305 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260808 01:52:46.867614 17210 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:46.923341  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3330) took 1872 ms. Trace:
I20260808 01:52:46.923913  9475 rpcz_store.cc:276] 0808 01:52:45.050535 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:45.053592 (+  3057us) service_pool.cc:224] Handling call
0808 01:52:45.074386 (+ 20794us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:45.074943 (+   557us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:45.074986 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:45.075736 (+   750us) log.cc:844] Serialized 26 byte log entry
0808 01:52:45.195124 (+119388us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:52:45.254950 (+ 59826us) log.cc:844] Serialized 14 byte log entry
0808 01:52:45.669094 (+414144us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:45.669705 (+   611us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:45.913044 (+243339us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b1800364000. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4889671f 00007f5d4188ceb1 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d
0808 01:52:46.767805 (+854761us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b1800364000. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:52:46.923004 (+155199us) raft_consensus.cc:1685] finished
0808 01:52:46.923037 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:46.923253 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7083,"prepare.queue_time_us":64894,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":153256,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":43086,"spinlock_wait_cycles":73974912,"thread_start_us":523581,"threads_started":2,"wal-append.queue_time_us":324149}
I20260808 01:52:46.882004 17240 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:46.931316  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:46.933110  9478 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:46.936404  9601 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:46.914927 17159 consensus_meta.cc:287] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.670s	user 0.014s	sys 0.002s
W20260808 01:52:46.909168 17300 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260808 01:52:46.940272  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:46.909828 17310 raft_consensus.cc:3055] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:46.907563 17311 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260808 01:52:46.947548 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52250.5us
I20260808 01:52:46.948395 10334 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:46.930769 17208 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:46.950465 17309 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:46.951259 17303 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:46.939142 17172 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:46.941934 16835 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:46.945225 17246 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:46.957602  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:46.944906 17310 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:46.969086 17310 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:46.960193 17303 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:46.942863 17159 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:46.971525 17262 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:46.971475 17303 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:46.972797 17159 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:46.977499 16934 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:46.978493 17285 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:46.979115  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:46.980455  9557 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:46.980664  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:46.982024  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:46.983279  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:46.984279  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:46.963236 17298 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260808 01:52:46.962862 17289 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:46.980538  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:46.959434 17246 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:46.986917 17246 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:46.989285 17251 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:46.953112 17312 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:46.990661 17312 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:46.993047 17312 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:46.955991 17172 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:46.995190 17172 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:46.948056 17311 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:46.966529 17240 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:46.995857 17311 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:46.967551 17171 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:46.979115 17285 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:46.999769 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:47.000797 10368 raft_consensus.cc:2410] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260808 01:52:47.001104 17285 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:47.003530  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45619.9us
I20260808 01:52:47.004693 16785 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260808 01:52:46.964417 17307 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260808 01:52:47.005223 16785 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:47.006860 10334 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:47.024544  9478 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
W20260808 01:52:47.024479 10332 connection.cc:256] RPC call timeout handler was delayed by -1.2411120079923421s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.219s for a response
W20260808 01:52:47.027616  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:46.958055 16835 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:47.033229  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:47.038990  9474 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:46.985773 17310 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:47.039880  9635 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: []
I20260808 01:52:47.040400  9474 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
W20260808 01:52:47.026141 17287 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.567s	user 0.000s	sys 0.004s
I20260808 01:52:46.981880  9472 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7c508218361c446898fede7c5682fa9e"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:50124
I20260808 01:52:47.057181  9472 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:52:47.057684  9472 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:47.052978 10368 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260808 01:52:47.053975  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:47.060303  9552 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:52:47.060716  9474 raft_consensus.cc:2410] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 22 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 8.
I20260808 01:52:47.063143  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:47.020093 16939 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
I20260808 01:52:47.023761 16785 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:47.075030  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:47.032781 17240 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:47.042266 16835 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:47.077152  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:47.083340  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 124240us
W20260808 01:52:47.076994 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:47.096777 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:47.101682 17308 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:47.103586 17308 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:47.114847  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3313) took 1073 ms. Trace:
I20260808 01:52:47.115478  9480 rpcz_store.cc:276] 0808 01:52:46.041800 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:46.600599 (+558799us) service_pool.cc:224] Handling call
0808 01:52:46.647682 (+ 47083us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:46.648091 (+   409us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:46.648125 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:46.648703 (+   578us) log.cc:844] Serialized 26 byte log entry
0808 01:52:46.791858 (+143155us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:46.791996 (+   138us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:46.792149 (+   153us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:47.114473 (+322324us) raft_consensus.cc:1685] finished
0808 01:52:47.114506 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:47.114747 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":161,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":10345,"thread_start_us":115257,"threads_started":1,"wal-append.queue_time_us":169851}
W20260808 01:52:47.118988  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 44112.7us
I20260808 01:52:47.120196  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:47.122161  9521 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:47.024322 17158 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
I20260808 01:52:47.108863 17312 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:47.108824 17241 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:47.149230  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2830) took 1068 ms. Trace:
W20260808 01:52:47.150396 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 143626us
I20260808 01:52:47.145263  9472 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:47.126652 17232 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260808 01:52:47.156883  9523 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:47.160296 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:47.152580  9550 rpcz_store.cc:276] 0808 01:52:46.080810 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:46.127803 (+ 46993us) service_pool.cc:224] Handling call
0808 01:52:47.139700 (+1011897us) spinlock_profiling.cc:243] Waited 59 ms on lock 0x7b54003147d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:47.149128 (+  9428us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":135719808}
W20260808 01:52:47.162197  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:47.130066 17239 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:47.153000 16934 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:47.196303 17337 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:47.196062 17242 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:47.197198 17335 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:47.196995 17337 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:47.204203 17330 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:47.204771 17335 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:47.206898 17264 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:52:47.207554 17335 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:47.208779 17211 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:47.205847 17337 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:47.209715 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:47.210300  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:47.215407  9631 raft_consensus.cc:2376] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20. Re-sending same reply.
I20260808 01:52:47.216681  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:47.217381  9631 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:47.220860 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:47.221894 17259 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:47.223093 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:47.230926  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:47.232014  9480 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:52:47.237048 10334 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:47.247503  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3156) took 1115 ms. Trace:
W20260808 01:52:47.248703 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 146855us
I20260808 01:52:47.241631  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2825) took 1162 ms. Trace:
I20260808 01:52:47.251317  9559 rpcz_store.cc:276] 0808 01:52:46.079298 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:46.080100 (+   802us) service_pool.cc:224] Handling call
0808 01:52:47.241367 (+1161267us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b540002d0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:47.241538 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":362217728}
I20260808 01:52:47.244477  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:47.260689  9634 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:52:47.247941  9629 rpcz_store.cc:276] 0808 01:52:46.132143 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:46.248775 (+116632us) service_pool.cc:224] Handling call
0808 01:52:47.247243 (+998468us) spinlock_profiling.cc:243] Waited 453 ms on lock 0x7b540003b6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:47.247396 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1041869568}
W20260808 01:52:47.266855  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:47.272998  9631 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:52:47.250188 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:47.241600 16952 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.006s
I20260808 01:52:47.305384  9472 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:47.307330 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: true dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:47.152202 17328 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:47.128279 17240 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:47.308200 10363 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:47.309901  9631 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:47.316349 10332 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:47.238472 17314 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:47.143559 17171 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.151s
I20260808 01:52:47.320828 10332 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:47.190817 17239 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:47.189194 17289 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:47.320142  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3134) took 1748 ms. Trace:
I20260808 01:52:47.309259 17337 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260808 01:52:47.304543  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 165713us
W20260808 01:52:47.242585 17273 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.180s	user 0.004s	sys 0.006s
I20260808 01:52:47.288170 16808 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:47.354966 17242 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:47.357092  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3155) took 1225 ms. Trace:
I20260808 01:52:47.163620 17312 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:47.360316  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 316472us
I20260808 01:52:47.173741 17241 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:47.360574 17251 consensus_queue.cc:1059] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 01:52:47.361492 16939 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260808 01:52:47.320154 17328 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:47.323107  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3131) took 1853 ms. Trace:
W20260808 01:52:47.329524 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:47.295944 17271 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:47.364981  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:47.304636 17279 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.904s	user 0.004s	sys 0.002s
I20260808 01:52:47.366303  9554 raft_consensus.cc:2393] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 20: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:47.321341 17314 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:47.366770 17271 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:47.366984 17314 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:47.327464 17239 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:47.368845 17259 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:47.363754  9633 rpcz_store.cc:276] 0808 01:52:45.469706 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:45.747916 (+278210us) service_pool.cc:224] Handling call
0808 01:52:45.812368 (+ 64452us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:45.813082 (+   714us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:47.322842 (+1509760us) spinlock_profiling.cc:243] Waited 64 ms on lock 0x7b540032d550. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:47.323008 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":147148032}
I20260808 01:52:47.327232  9630 rpcz_store.cc:276] 0808 01:52:45.571898 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:45.961124 (+389226us) service_pool.cc:224] Handling call
0808 01:52:46.021025 (+ 59901us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:46.110297 (+ 89272us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:46.110349 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:46.111201 (+   852us) log.cc:844] Serialized 52 byte log entry
0808 01:52:46.341149 (+229948us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:46.515988 (+174839us) log.cc:844] Serialized 14 byte log entry
0808 01:52:46.528168 (+ 12180us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:46.528277 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:47.002165 (+473888us) spinlock_profiling.cc:243] Waited 2.52 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:47.312235 (+310070us) raft_consensus.cc:1685] finished
0808 01:52:47.319883 (+  7648us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:47.320075 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":71533,"prepare.queue_time_us":232826,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":97719,"raft.run_cpu_time_us":357,"raft.run_wall_time_us":15259,"spinlock_wait_cycles":5804800,"thread_start_us":266606,"threads_started":2,"wal-append.queue_time_us":209475}
I20260808 01:52:47.326035 16934 raft_consensus.cc:3055] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:47.372722 16934 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:47.373425  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:47.374208  9480 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:47.349162 17133 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:47.351312 17337 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:47.375362 17337 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:52:47.376456  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:47.374055 16934 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:47.377970  9476 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:52:47.379175  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:47.379722 17323 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:47.379448  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3310) took 1749 ms. Trace:
I20260808 01:52:47.393162 16939 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:47.393814 16939 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:47.400738 17302 log.cc:927] Time spent T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.531s	user 0.003s	sys 0.000s
I20260808 01:52:47.366019  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:47.380362  9481 rpcz_store.cc:276] 0808 01:52:45.629786 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:45.905402 (+275616us) service_pool.cc:224] Handling call
0808 01:52:45.905773 (+   371us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:46.026936 (+121163us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:52:46.038995 (+ 12059us) log.cc:844] Serialized 14 byte log entry
0808 01:52:46.245025 (+206030us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:46.247225 (+  2200us) log.cc:844] Serialized 26 byte log entry
0808 01:52:46.249016 (+  1791us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:52:46.249068 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:46.250030 (+   962us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:47.228262 (+978232us) spinlock_profiling.cc:243] Waited 18.1 ms on lock 0x7b24000ef228. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41713594 00007f5d4171169b 00007f5d418cd6d9 00007f5d418cd365 00007f5d418cc9ad 00007f5d445162fe 00007f5d4452da7c 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411
0808 01:52:47.379053 (+150791us) raft_consensus.cc:1685] finished
0808 01:52:47.379205 (+   152us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:47.379373 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":120758,"prepare.queue_time_us":670,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":126387,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":15237,"spinlock_wait_cycles":41591168,"thread_start_us":204877,"threads_started":1,"wal-append.queue_time_us":293232}
I20260808 01:52:47.419338 17289 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:47.419759 17312 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:47.420094  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:47.420878 16934 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:47.421247 17129 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:47.425208 16983 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:47.290709 17211 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:47.290182 17205 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:47.425441 17311 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:47.431006 17318 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:47.430862  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:47.357539  9635 rpcz_store.cc:276] 0808 01:52:46.131887 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:46.576759 (+444872us) service_pool.cc:224] Handling call
0808 01:52:46.577038 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:47.046485 (+469447us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:47.356790 (+310305us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x7b540008cd54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:47.357025 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1063938944}
W20260808 01:52:47.420101  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 55186.2us
I20260808 01:52:47.444854 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:47.445264 10363 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:47.483120 10364 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:47.448709  9601 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:47.450434 16194 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:47.451311  9475 pending_rounds.cc:85] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 15
I20260808 01:52:47.504889  9475 pending_rounds.cc:107] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 22.16
I20260808 01:52:47.505247  9475 raft_consensus.cc:2889] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:47.457144  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:47.506616  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 49568us
I20260808 01:52:47.459263 17262 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:47.445712 17323 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:47.462105  9480 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
W20260808 01:52:47.511478  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 62897us
I20260808 01:52:47.462688  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:47.463244  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:47.520221  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:47.450027 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:47.469856 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:47.475178  9552 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
W20260808 01:52:47.502064  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:47.508572 16194 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:47.514760  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:47.515970  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:47.526903  9480 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260808 01:52:47.529130  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:47.431988 16952 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
I20260808 01:52:47.521178  9559 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 22
I20260808 01:52:47.530936  9524 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:47.459949  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 337762us
W20260808 01:52:47.531492 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:47.475889 16939 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:47.534472  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:47.528877 17241 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:47.593173  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:47.594393  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:47.587809 16194 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:47.596709  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:47.597719  9552 raft_consensus.cc:2410] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260808 01:52:47.598057 16194 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:47.512717 17302 log.cc:927] Time spent T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260808 01:52:47.598577 16194 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:47.587066  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:47.536264  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:47.601009  9633 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:47.613590  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:47.614321  9554 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:47.611116  9472 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260808 01:52:47.616080  9474 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
W20260808 01:52:47.618074 16785 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.534s	user 0.016s	sys 0.003s
W20260808 01:52:47.514647 17339 log.cc:927] Time spent T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260808 01:52:47.536389  9473 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: []
I20260808 01:52:47.619570  9629 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:47.469336 17289 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:47.622192  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:47.498557 17251 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.130s
W20260808 01:52:47.591681 17332 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260808 01:52:47.512181 17312 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:47.592464 17240 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:47.680655  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:47.481324 17211 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:47.682009  9552 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
W20260808 01:52:47.640051  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 105728us
I20260808 01:52:47.641827 10367 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260808 01:52:47.642940  9559 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
W20260808 01:52:47.638864  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0488089009886608s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1532s for a response
W20260808 01:52:47.637158  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 117646us
I20260808 01:52:47.482954 17308 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:47.591570 17262 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260808 01:52:47.711606 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 390837us
I20260808 01:52:47.476197 17129 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:47.515405 17321 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:52:47.639948  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 104673us
I20260808 01:52:47.726667 16194 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:47.510061 17324 raft_consensus.cc:2764] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260808 01:52:47.536649 17288 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.144s
I20260808 01:52:47.717038  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:47.517624 17271 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.124s
I20260808 01:52:47.718457  9601 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:47.713347 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:47.648206 16939 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:47.744567 17323 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:47.720480 17265 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.053s
W20260808 01:52:47.768848 17343 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
W20260808 01:52:47.771700  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 44867.8us
I20260808 01:52:47.775825 17338 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:47.628585 16785 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:47.748726 17317 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.153s
W20260808 01:52:47.749197  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:47.712123 17262 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:47.751238  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:47.752292  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:47.756258  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:47.509654 17205 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:47.758136 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 44858.7us
I20260808 01:52:47.758440  9633 raft_consensus.cc:2410] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 17 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260808 01:52:47.762929  9629 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:47.802639  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:47.802752  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:47.803882  9629 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:47.803923  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:47.804809  9635 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:47.805673  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:47.806326  9635 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:47.763799  9554 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:47.715618 17328 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:47.768276 16939 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:47.812476  9631 raft_consensus.cc:606] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:47.788584 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:47.748600 17158 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:47.747916  9444 leader_election.cc:400] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 18. Current term is 19.
W20260808 01:52:47.758951 17172 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.763s	user 0.006s	sys 0.009s
W20260808 01:52:47.807202 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:47.816943  9444 leader_election.cc:403] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260808 01:52:47.813126  9631 raft_consensus.cc:614] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:47.818475  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:47.822940  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66793us
I20260808 01:52:47.824321 10332 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:47.791350 17308 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:52:47.827339  9635 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 22: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:47.681248 17318 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.119s
I20260808 01:52:47.823908  9601 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:47.832448  9635 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260808 01:52:47.822229  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:47.802632 17339 log.cc:927] Time spent T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260808 01:52:47.834535 16194 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:52:47.837167 16194 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:52:47.834923  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:47.801622 17262 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:47.801220 17317 consensus_queue.cc:1243] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 20.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260808 01:52:47.839108  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:47.841619  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:47.843314  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:47.820609 17246 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.833s	user 0.014s	sys 0.001s
W20260808 01:52:47.820986 17295 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.913s	user 0.014s	sys 0.000s
I20260808 01:52:47.881981  9480 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:52:47.845556 16194 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:47.832798 17297 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.940s	user 0.009s	sys 0.005s
I20260808 01:52:47.794292 16983 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.042s
I20260808 01:52:47.873538  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:47.975191  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 201403us
I20260808 01:52:48.001075  9557 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
W20260808 01:52:48.001736 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:47.905516  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:47.912015  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:48.005538  9481 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
W20260808 01:52:48.005977  9426 auto_rebalancer.cc:912] Removing replace marker failed after 3 inline attempts; will retry on next rebalancer loop iteration: Service unavailable: leader transfer in progress
I20260808 01:52:47.992045 17289 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:48.006799 17319 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:48.007997 17336 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:47.914043 17246 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:48.008659 17336 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:47.914949  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 167277us
I20260808 01:52:48.009996  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:48.010842 17311 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:48.010990 17259 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:48.011430 17259 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:48.011859 17259 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:48.017112 17129 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:48.021118 17259 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:48.022193 17242 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.667s	user 0.008s	sys 0.003s
I20260808 01:52:48.023624 17259 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:47.882984 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 58764.9us
I20260808 01:52:48.045020  9521 leader_election.cc:400] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22.
I20260808 01:52:47.917728  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:48.006455  9426 auto_rebalancer.cc:921] Could not move replica: Remote error: Service unavailable: leadership transfer for 7c508218361c446898fede7c5682fa9e already in progress
W20260808 01:52:47.970065  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 323078us
I20260808 01:52:48.002288  9477 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:47.833474 17308 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:47.819406 17158 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:47.848634 17134 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:47.881639  9474 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:47.915408  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:48.062780  9521 leader_election.cc:403] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:52:47.848538 17239 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:48.027141  9633 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 21: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:52:48.069875  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59970.6us
I20260808 01:52:48.027405 17211 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:48.031512 17205 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:48.030962 17246 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:48.034288  9629 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:47.812659 17321 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:48.045655 10332 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:48.062249  9478 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
W20260808 01:52:48.067783  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 245673us
I20260808 01:52:47.847071 17251 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 18. Current term is 19.
I20260808 01:52:48.070823  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:48.061239  9426 auto_rebalancer.cc:342] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 7c508218361c446898fede7c5682fa9e already in progress
I20260808 01:52:48.061985 17158 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:48.121188  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47764us
I20260808 01:52:48.121320 17232 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22.
I20260808 01:52:48.122314 17242 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:48.162207  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 91520.8us
I20260808 01:52:48.123879 17239 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:48.127306 16785 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:48.128468 16939 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:48.162228 17336 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:48.130229 17289 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:48.130424  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:47.830950 17172 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:47.847671 17328 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:52:47.837924 17288 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.236s
I20260808 01:52:48.141940  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:48.182608  9601 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:48.157469 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 111949us
I20260808 01:52:48.183194 16194 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:52:48.183831 16194 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:48.183925 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:48.184214 16194 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:52:48.044355 17295 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:48.146698  9599 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:48.195343  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 48774.8us
I20260808 01:52:48.160548  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:48.161775  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 3b8e0e8fda6744ad877514b11658b9d9 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:48.000221 17315 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.359s
I20260808 01:52:48.061744 17308 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:52:48.156370 17346 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:52:47.997231 17347 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:47.878643 17314 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:48.179207 17328 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:48.194257  9601 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:48.061504 17297 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:48.264611  9637 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:48.265784 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:48.074064 17134 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:48.192552 17337 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.817s	user 0.014s	sys 0.006s
I20260808 01:52:48.203467 17296 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260808 01:52:48.284267  9632 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 18: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:52:48.285297 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 101430us
W20260808 01:52:48.276582 16934 consensus_meta.cc:287] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.810s	user 0.012s	sys 0.001s
I20260808 01:52:48.202447 17342 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:48.299135  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:48.305769 17328 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:48.180474 17205 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.101s
I20260808 01:52:48.280670 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:48.344604 10369 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:48.320245 17208 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:48.178602 17232 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.100s
I20260808 01:52:48.203161 17251 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:48.178771 17242 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:48.275930 16939 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.047s
W20260808 01:52:48.288507 16835 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.161s	user 0.010s	sys 0.007s
I20260808 01:52:48.185091 17289 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:48.265575 17129 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:48.299175  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 105016us
I20260808 01:52:48.338496  9559 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 14. (term mismatch)
I20260808 01:52:48.359156  9559 pending_rounds.cc:85] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 13
I20260808 01:52:48.359467  9559 pending_rounds.cc:107] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260808 01:52:48.359702  9559 raft_consensus.cc:2889] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:48.339836  9634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "439040d168824496acc076b1c280bde3"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:48.337284  9473 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:48.193854 17321 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260808 01:52:48.333796 17310 consensus_meta.cc:287] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.263s	user 0.013s	sys 0.006s
W20260808 01:52:48.350214  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 187173us
I20260808 01:52:48.351356 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:48.343274 17335 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.131s	user 0.002s	sys 0.020s
I20260808 01:52:48.336874 17342 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:48.376075  9524 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:48.366240  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:48.367678  9635 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260808 01:52:48.371050  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:48.347095 17358 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.084s
I20260808 01:52:48.372345 16194 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:48.349107 17208 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:48.323969 17337 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:48.378274  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:48.386329  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:48.386883  9629 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:48.388257 17349 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:48.388671 17172 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:48.378129 17335 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:48.389369 17349 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:52:48.378319 17342 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:48.358002 17251 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:48.390965 17335 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:48.391847  9550 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 18: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:48.392992 17251 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:48.393874 17312 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:48.394145 17251 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:48.394387 17312 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:48.394966 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:48.395195 17312 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:48.396253 17337 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:48.397235 10361 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 24. Current term is 25. Ops: [24.14-24.14]
W20260808 01:52:48.399384 17354 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.135s	user 0.000s	sys 0.000s
I20260808 01:52:48.400504 17289 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:48.372864 17246 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:48.402119  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3363) took 1036 ms. Trace:
I20260808 01:52:48.402046 17246 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:48.402616  9475 rpcz_store.cc:276] 0808 01:52:47.429238 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:47.429454 (+   216us) service_pool.cc:224] Handling call
0808 01:52:47.451041 (+ 21587us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:47.514150 (+ 63109us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:47.514195 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:47.514760 (+   565us) log.cc:844] Serialized 26 byte log entry
0808 01:52:47.984019 (+469259us) raft_consensus.cc:1645] Marking committed up to 16
0808 01:52:48.147002 (+162983us) log.cc:844] Serialized 14 byte log entry
0808 01:52:48.158262 (+ 11260us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:48.158331 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:48.401876 (+243545us) raft_consensus.cc:1685] finished
0808 01:52:48.401901 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:48.402052 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10566,"prepare.queue_time_us":139019,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":119,"spinlock_wait_cycles":19438336,"thread_start_us":619882,"threads_started":2,"wal-append.queue_time_us":617004}
I20260808 01:52:48.404847  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:48.405490  9475 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:48.394269  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:48.408699  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:48.409581  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3166) took 1357 ms. Trace:
I20260808 01:52:48.409262 17312 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:48.410392 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:48.412344 17340 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.825s	user 0.006s	sys 0.002s
I20260808 01:52:48.414022  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 1059) took 2483 ms. Trace:
I20260808 01:52:48.419212 17312 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:48.414629  9558 rpcz_store.cc:276] 0808 01:52:45.930142 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:45.930263 (+   121us) service_pool.cc:224] Handling call
0808 01:52:45.941335 (+ 11072us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:45.988994 (+ 47659us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:45.989048 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:47.322750 (+1333702us) spinlock_profiling.cc:243] Waited 6.12 ms on lock 0x7b2c000f57d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:52:47.325863 (+  3113us) log.cc:844] Serialized 26 byte log entry
0808 01:52:47.455770 (+129907us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:47.455830 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:47.455937 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:48.411685 (+955748us) spinlock_profiling.cc:243] Waited 8.05 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:48.413686 (+  2001us) raft_consensus.cc:1685] finished
0808 01:52:48.413719 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:48.413915 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13271,"raft.queue_time_us":46123,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":19362,"spinlock_wait_cycles":32600320,"thread_start_us":124370,"threads_started":1,"wal-append.queue_time_us":168546}
W20260808 01:52:48.420814  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:48.422215 17314 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:48.361821 17321 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:48.457161 17264 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Stepping down as leader of term 18
W20260808 01:52:48.464507 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:48.443681 10366 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260808 01:52:48.424211  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:48.455982  9481 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260808 01:52:48.442536 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:48.425935  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:48.436317  9474 raft_consensus.cc:1217] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Deduplicated request from leader. Original: 16.8->[17.9-21.11]   Dedup: 17.9->[19.10-21.11]
I20260808 01:52:48.482250  9474 pending_rounds.cc:85] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 9
I20260808 01:52:48.444523 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:48.482540  9474 pending_rounds.cc:107] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 20.10
I20260808 01:52:48.483441  9474 raft_consensus.cc:2889] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:48.416828 16939 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:52:48.417621 17364 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260808 01:52:48.362020 16835 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:48.424432 17208 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:48.409353  9473 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:48.386124 17288 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:48.450835  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:48.519942  9630 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:48.417845 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:48.418046 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:48.454536 17295 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:48.390763 17134 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:48.452865 17297 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:48.448529 17242 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:48.531042  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 2984) took 1022 ms. Trace:
I20260808 01:52:48.406615  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:48.531797  9475 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:48.532393  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3717) took 1070 ms. Trace:
I20260808 01:52:48.532795  9550 rpcz_store.cc:276] 0808 01:52:47.462214 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:47.462378 (+   164us) service_pool.cc:224] Handling call
0808 01:52:48.532175 (+1069797us) spinlock_profiling.cc:243] Waited 871 ms on lock 0x7b54000fe954. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:48.532326 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2003929472}
I20260808 01:52:48.461581  9479 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:48.535768  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:48.429666 17262 consensus_meta.cc:287] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.518s	user 0.010s	sys 0.000s
I20260808 01:52:48.539027 17262 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:48.467677  9523 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:48.534530 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 1160) took 1179 ms. Trace:
W20260808 01:52:48.541487  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 73980.9us
I20260808 01:52:48.531469  9633 rpcz_store.cc:276] 0808 01:52:47.508528 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:47.802337 (+293809us) service_pool.cc:224] Handling call
0808 01:52:48.530793 (+728456us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b540034fb54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:48.530961 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":514088320}
I20260808 01:52:48.542577  9638 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: [20.10-20.10]
I20260808 01:52:48.410004  9637 rpcz_store.cc:276] 0808 01:52:47.051715 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:47.111544 (+ 59829us) service_pool.cc:224] Handling call
0808 01:52:47.111983 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:48.308590 (+1196607us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:48.409325 (+100735us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b540027e054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:48.409508 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2524177664}
I20260808 01:52:48.545496  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:48.547868  9559 raft_consensus.cc:2368] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18.
W20260808 01:52:48.549659  9632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3161) took 1503 ms (client timeout 1668 ms). Trace:
I20260808 01:52:48.549893  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:48.550611  9560 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 14.
I20260808 01:52:48.552676  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:48.541697 10361 rpcz_store.cc:276] 0808 01:52:47.355168 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:47.355399 (+   231us) service_pool.cc:224] Handling call
0808 01:52:47.355967 (+   568us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:48.397724 (+1041757us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:48.534200 (+136476us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400512cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:48.534426 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2393939328}
I20260808 01:52:48.553826  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:48.550153  9632 rpcz_store.cc:269] 0808 01:52:47.046287 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:47.046446 (+   159us) service_pool.cc:224] Handling call
0808 01:52:48.549310 (+1502864us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54001b3ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:48.549451 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2747772800}
I20260808 01:52:48.554713  9554 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 22: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:48.558358  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:48.559099  9473 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:52:48.559260  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:48.509471 16939 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:48.560833  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:48.560979 17134 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:48.561620  9475 raft_consensus.cc:2410] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 21 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260808 01:52:48.561573 17134 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:48.560523  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:48.560544  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:48.566037  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:48.566943  9554 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:48.570922  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:48.473495  9636 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:48.464452  9476 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:48.480032 16194 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:48.463042 17312 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:48.480907  9629 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:52:48.579545  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:48.581238  9629 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 23. Current term is 24. Ops: []
I20260808 01:52:48.484701 17264 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:48.413543 17246 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:48.470966 17251 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:48.486539  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:48.596918  9481 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:48.487284 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:48.598095 10366 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:48.485443  9478 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:48.598803  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:48.599406  9552 raft_consensus.cc:3055] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:48.497864 17342 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:48.599705  9552 raft_consensus.cc:740] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:48.500536 10368 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 19: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:48.601490 17342 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:48.502434 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:52:48.506479 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 155203us
I20260808 01:52:48.597760  9629 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260808 01:52:48.602908  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:48.507702  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:48.606750 17324 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260808 01:52:48.595813 17312 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:48.608770 17324 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 01:52:48.609549 16835 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:48.509999  9635 raft_consensus.cc:2410] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 21 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260808 01:52:48.363994 17129 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:48.613204 17308 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:48.583150  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 246029us
I20260808 01:52:48.606065 10332 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:48.599713  9478 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: [18.10-19.11]
I20260808 01:52:48.591362  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:48.618481  9632 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
W20260808 01:52:48.618853  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:48.591228  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:48.619830  9632 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:48.619979  9630 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:48.621373  9630 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260808 01:52:48.600666  9552 consensus_queue.cc:260] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:48.607017 17246 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:48.622807  9552 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:48.623178 17246 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:48.624233 17262 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:48.624122 17246 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:48.593256  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:48.529071  9369 catalog_manager.cc:5697] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 8 to 14, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 14 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:52:48.557955 16983 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:52:48.644982  9550 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260808 01:52:48.500252 17321 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:48.625815  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:48.613652 17129 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:48.568145 17133 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:48.580705 17317 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:48.652052  9473 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
W20260808 01:52:48.653210 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47231us
I20260808 01:52:48.660490  9552 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:48.666189  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:48.527086 17288 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:48.665392  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:48.579387 17171 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:48.673354  9633 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 24. Current term is 25. Ops: []
I20260808 01:52:48.674664  9524 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:48.675822  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:48.681262  9473 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:52:48.683116  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:48.800164 10332 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:48.781680  9550 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:48.782579  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:48.818207  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3165) took 1766 ms. Trace:
I20260808 01:52:48.823405  9481 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:48.823755  9634 raft_consensus.cc:606] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:48.824290  9634 raft_consensus.cc:614] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:48.819972  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:48.651364 17285 raft_consensus.cc:3055] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Stepping down as leader of term 17
W20260808 01:52:48.785933  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 160196us
I20260808 01:52:48.560555 17158 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:48.784638  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:48.805723  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:48.844482  9477 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:48.807390  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:48.807591  9629 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: []
I20260808 01:52:48.650091 17312 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:48.811621 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:48.806885 17265 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:48.814284 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:48.812951 17313 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:48.861323  9521 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:48.627053 17251 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:48.592028 16939 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:48.618889 17335 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:48.823400  9638 rpcz_store.cc:276] 0808 01:52:47.051350 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:47.069966 (+ 18616us) service_pool.cc:224] Handling call
0808 01:52:47.070353 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:48.543311 (+1472958us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:48.817860 (+274549us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b540008c354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:48.818110 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3384935296}
I20260808 01:52:48.870160  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:48.871160  9638 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:52:48.686090 17317 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:48.830842 17285 raft_consensus.cc:740] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:48.817406  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 451212us
I20260808 01:52:48.847796 17314 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:48.857540 17149 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Stepping down as leader of term 21
W20260808 01:52:48.908929  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 439040d168824496acc076b1c280bde3 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
W20260808 01:52:48.913580  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 716532us
W20260808 01:52:48.868584 17328 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.517s	user 0.012s	sys 0.000s
I20260808 01:52:48.827518  9600 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:48.830111  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 155565us
I20260808 01:52:48.803797 17210 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:52:48.785090 17259 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:48.684473 17345 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:48.843036 17129 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.058s
I20260808 01:52:48.850945  9558 raft_consensus.cc:2410] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 22 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260808 01:52:48.687044 17133 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:48.818590 17240 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:48.872422  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:48.954910  9638 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:48.822458  9551 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:52:48.842666 17232 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:48.821036  9554 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:52:48.891932  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:48.963058  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 135644us
I20260808 01:52:48.963420  9475 raft_consensus.cc:2393] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 14: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:48.963970  9600 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:48.964797 17211 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:48.965793 17211 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:48.893823  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:48.827033  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:48.897022  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:48.806875 17171 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:48.898559  9636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3162) took 1851 ms (client timeout 2068 ms). Trace:
W20260808 01:52:48.904738  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 43517.9us
I20260808 01:52:48.971773  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:48.900915  9477 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:48.974589  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:48.976158  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:48.825286  9637 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:48.979272  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:48.917500  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:48.928027  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:48.983253  9633 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:52:48.940769  9445 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:48.943159  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:48.848661  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:48.987284  9481 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:52:48.861578 17315 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.061s
I20260808 01:52:48.969997  9480 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 19
I20260808 01:52:48.875335 17239 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.067s
W20260808 01:52:48.970625  9636 rpcz_store.cc:269] 0808 01:52:47.046669 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:47.062824 (+ 16155us) service_pool.cc:224] Handling call
0808 01:52:48.898300 (+1835476us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b540003c5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:48.898469 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3017591680}
W20260808 01:52:48.916865  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:48.971068  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:49.005661  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 178717us
I20260808 01:52:49.010828 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:48.999042  9521 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:49.002347  9475 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:49.021566  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:48.886850 17335 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:48.992784  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 99155us
W20260808 01:52:48.994163  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 53470.6us
I20260808 01:52:48.909276 17149 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:49.031540  9557 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260808 01:52:49.032852  9481 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:48.962235 17265 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:48.961431 17342 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:49.059316  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 60445.5us
I20260808 01:52:48.996439 17129 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:52:49.045173  9634 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 18: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:48.969259 17232 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:49.001827  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:48.842434 17264 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:48.962464 17210 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:49.060159  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:49.046849  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:48.876004 17313 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:49.004833 17211 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:48.996160 17345 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:49.004114 16939 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:49.048173 17295 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:48.931384 17328 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:49.073640  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 71930.5us
I20260808 01:52:48.997394 17372 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:49.048190  9475 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:49.352077  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3355) took 1018 ms. Trace:
I20260808 01:52:49.352488  9475 rpcz_store.cc:276] 0808 01:52:48.333359 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:49.047468 (+714109us) service_pool.cc:224] Handling call
0808 01:52:49.047940 (+   472us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:49.351775 (+303835us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:49.352005 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:48.952824 17259 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:48.921216  9599 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:49.095997  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:49.098287  9551 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 25.
I20260808 01:52:49.099340  9481 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:49.364988  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3356) took 1021 ms. Trace:
I20260808 01:52:49.365396  9481 rpcz_store.cc:276] 0808 01:52:48.343735 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:49.068096 (+724361us) service_pool.cc:224] Handling call
0808 01:52:49.099158 (+ 31062us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:49.364677 (+265519us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:49.364907 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:48.990363 17240 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:49.059149 17324 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:48.997128 17311 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.075s
I20260808 01:52:49.105737  9473 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 24.
I20260808 01:52:49.031213 17336 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.047s
I20260808 01:52:49.068361 17242 raft_consensus.cc:3055] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Stepping down as leader of term 19
W20260808 01:52:49.226513  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:49.396627 10334 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:49.197743 17374 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.386s	user 0.000s	sys 0.001s
I20260808 01:52:49.200711 17315 consensus_queue.cc:1059] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.089s
I20260808 01:52:49.284624 17158 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:49.285990 17171 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:49.291354 17373 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.381s	user 0.000s	sys 0.001s
I20260808 01:52:49.318178 17210 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:49.301534  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3375) took 1204 ms. Trace:
I20260808 01:52:49.416570  9474 rpcz_store.cc:276] 0808 01:52:48.097198 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:48.435536 (+338338us) service_pool.cc:224] Handling call
0808 01:52:48.436033 (+   497us) raft_consensus.cc:1471] Updating replica for 3 ops
0808 01:52:48.502376 (+ 66343us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:52:48.502409 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:48.503035 (+   626us) log.cc:844] Serialized 52 byte log entry
0808 01:52:48.814535 (+311500us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:52:48.814602 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:48.814735 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:49.301269 (+486534us) raft_consensus.cc:1685] finished
0808 01:52:49.301291 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:49.301460 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19640,"raft.queue_time_us":5086,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":8223,"thread_start_us":274425,"threads_started":1,"wal-append.queue_time_us":555089}
I20260808 01:52:49.111899 17134 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:49.067698 16974 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.122s
I20260808 01:52:49.100636 17368 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:49.419483 17368 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:49.419611 17335 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:49.420763 17380 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.224s	user 0.000s	sys 0.001s
I20260808 01:52:49.420686 17368 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:49.421151 17349 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:52:49.082088 17129 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:49.100651 16808 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:49.030748 17285 consensus_queue.cc:260] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:49.102741 16983 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:49.072714 17312 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:49.071990 17288 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:49.189711 17264 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:49.186311 16939 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:49.072265 17342 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:48.961206  9369 catalog_manager.cc:5697] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 9 to 13. New cstate: current_term: 13 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:52:49.101447 17211 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:49.192586  9633 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 24.
I20260808 01:52:49.074162 17232 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:49.200556  9480 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:52:49.452364 17134 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Stepping down as leader of term 23
W20260808 01:52:49.334805  9472 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.032s	sys 0.093s
W20260808 01:52:49.495776  9472 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.033s	sys 0.093s
I20260808 01:52:49.348446  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:49.357393 17208 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.830s	user 0.005s	sys 0.005s
I20260808 01:52:49.359580 17133 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:49.369940  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:49.387585  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 466474us
I20260808 01:52:49.313875 17308 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:49.389668 10367 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
I20260808 01:52:49.184351 17149 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:49.398185  9477 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:52:49.403551 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:49.394415 17242 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:49.407346 17313 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:49.391189 17379 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.203s	user 0.001s	sys 0.000s
I20260808 01:52:49.434218 17129 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:49.401187 17372 raft_consensus.cc:2764] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260808 01:52:49.362825 16952 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.068s
I20260808 01:52:49.368518 17370 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:49.386644 17324 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:49.552925  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 204607us
I20260808 01:52:49.543515  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3351) took 1237 ms. Trace:
I20260808 01:52:49.555809  9599 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:49.393826 17205 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:49.436491 17349 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:49.389947 17295 raft_consensus.cc:697] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:52:49.185767 10366 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.559s	user 0.015s	sys 0.000s
I20260808 01:52:49.452103 16983 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:49.459794 17240 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:49.448938 17314 consensus_meta.cc:287] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.537s	user 0.015s	sys 0.000s
W20260808 01:52:49.560696 17371 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.005s	sys 0.015s
I20260808 01:52:49.557446  9475 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:52:49.450086  9637 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:49.529461  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3381) took 1028 ms. Trace:
I20260808 01:52:49.592494  9481 rpcz_store.cc:276] 0808 01:52:48.501331 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:49.397933 (+896602us) service_pool.cc:224] Handling call
0808 01:52:49.528963 (+131030us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:49.529133 (+   170us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:49.529153 (+    20us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:49.529189 (+    36us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:49.529215 (+    26us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:49.529267 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:49.529389 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:52:49.516700  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:49.459615 17345 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:49.594651  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:49.477841 16939 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:49.597676  9551 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 15 index: 15. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260808 01:52:49.564899  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3382) took 1015 ms. Trace:
I20260808 01:52:49.598642  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:49.453938 16808 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:49.469394 17285 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
W20260808 01:52:49.571616  9629 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.043s	sys 0.067s
I20260808 01:52:49.573132 10366 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:49.555140  9478 rpcz_store.cc:276] 0808 01:52:48.306198 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:48.616023 (+309825us) service_pool.cc:224] Handling call
0808 01:52:48.616402 (+   379us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:48.616822 (+   420us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:48.616850 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:48.617568 (+   718us) log.cc:844] Serialized 52 byte log entry
0808 01:52:48.866674 (+249106us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:48.866757 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:48.866841 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:49.543215 (+676374us) raft_consensus.cc:1685] finished
0808 01:52:49.543243 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:49.543423 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10301,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":7339,"thread_start_us":248695,"threads_started":1,"wal-append.queue_time_us":749192}
W20260808 01:52:49.578169  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 482226us
I20260808 01:52:49.598773  9473 rpcz_store.cc:276] 0808 01:52:48.549114 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:49.409647 (+860533us) service_pool.cc:224] Handling call
0808 01:52:49.562541 (+152894us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:49.564387 (+  1846us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:49.564432 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:49.564501 (+    69us) raft_consensus.cc:1645] Marking committed up to 16
0808 01:52:49.564549 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:49.564639 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:49.564809 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:49.600996  9524 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:49.495652 17134 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:49.608302 10332 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.002s	sys 0.013s
W20260808 01:52:49.637845  9629 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.044s	sys 0.067s
W20260808 01:52:49.608099 10365 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.720s	user 0.011s	sys 0.006s
I20260808 01:52:49.562067 17324 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:49.647555 10366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 1168) took 1152 ms. Trace:
I20260808 01:52:49.651230  9635 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:49.651978  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1266) took 1031 ms. Trace:
I20260808 01:52:49.652571  9478 rpcz_store.cc:276] 0808 01:52:48.620677 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:49.649242 (+1028565us) service_pool.cc:224] Handling call
0808 01:52:49.651433 (+  2191us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:49.651622 (+   189us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:49.651645 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:49.651688 (+    43us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:49.651716 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:49.651775 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:49.651914 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:49.648609 10366 rpcz_store.cc:276] 0808 01:52:48.486873 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:48.487028 (+   155us) service_pool.cc:224] Handling call
0808 01:52:49.639584 (+1152556us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:52:49.591691 17371 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.006s	sys 0.015s
W20260808 01:52:49.637904 10332 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.005s	sys 0.015s
I20260808 01:52:49.591950 17149 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:52:49.608676  9636 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.010s	sys 0.005s
W20260808 01:52:49.689021  9636 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.013s	sys 0.007s
W20260808 01:52:49.639297  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 244001us
I20260808 01:52:49.594988 17314 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:49.643415  9524 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:49.691813  9600 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:49.660933  9369 catalog_manager.cc:5697] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 6 to 9, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 9 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:52:49.637058 17323 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.003s	sys 0.012s
I20260808 01:52:49.662842 17310 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:49.717527  9473 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 23. Current term is 24. Ops: [23.14-23.14]
W20260808 01:52:49.663048 17384 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
I20260808 01:52:49.669289 17265 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:49.670706 17296 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:49.671023 17387 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260808 01:52:49.571491 17370 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:49.671928 17358 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:49.672533 17134 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:49.672746 17295 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:49.675544 17321 consensus_meta.cc:287] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.005s	user 0.011s	sys 0.002s
I20260808 01:52:49.675962 16808 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:49.637651 17240 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.072s
I20260808 01:52:49.679934 17288 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:49.760928 17288 raft_consensus.cc:993] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: : Instructing follower 6e7856bed1be43acaa821290d506ee97 to start an election
I20260808 01:52:49.681005 17171 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:49.763209 17171 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:49.591573 16983 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:49.769475  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 946) took 1089 ms. Trace:
W20260808 01:52:49.640305 10365 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.763s	user 0.013s	sys 0.008s
I20260808 01:52:49.688222 17242 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:49.637094  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:49.771397 17358 raft_consensus.cc:697] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:49.772478  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 135448us
I20260808 01:52:49.773270 17134 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:52:49.773797  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:49.573589 17336 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:49.756963 17210 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:49.775013 17336 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:49.775624 17336 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:49.775506 17210 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:49.778024 17210 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:49.674024 17349 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:49.779394 17211 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:49.780073 17211 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:49.780583 17211 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:49.716935 17239 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:49.613356 17308 raft_consensus.cc:2764] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
W20260808 01:52:49.695070  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 51722.6us
I20260808 01:52:49.799430  9524 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:49.659989 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:49.602540 17205 raft_consensus.cc:697] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:49.571951 17208 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:49.718391 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:49.726106  9475 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:52:49.599048 17133 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.102s
W20260808 01:52:49.733362  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.638s	user 0.016s	sys 0.024s
I20260808 01:52:49.688648 17328 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:49.700996 17323 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.004s	sys 0.012s
W20260808 01:52:49.828871  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.016s	sys 0.024s
I20260808 01:52:49.692301 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:49.792191  9473 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: []
W20260808 01:52:49.829303  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 733767us
I20260808 01:52:49.829917  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3362) took 1048 ms. Trace:
I20260808 01:52:49.830300  9473 rpcz_store.cc:276] 0808 01:52:48.781553 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:49.751565 (+970012us) service_pool.cc:224] Handling call
0808 01:52:49.792005 (+ 40440us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:49.829637 (+ 37632us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:49.829858 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:52:49.770388 17251 consensus_meta.cc:287] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.882s	user 0.005s	sys 0.005s
I20260808 01:52:49.727948 17259 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:49.803383  9369 catalog_manager.cc:5697] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 11 to 12, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 12 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:52:49.805426 17158 raft_consensus.cc:2764] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election decision vote started in defunct term 23: won
I20260808 01:52:49.757211 17321 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:49.809413 17317 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:49.730794 17310 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:49.781524 17211 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:49.811373 17370 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:49.755049 17296 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:49.772560 17358 consensus_queue.cc:237] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:49.769985  9632 rpcz_store.cc:276] 0808 01:52:48.680125 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:49.355877 (+675752us) service_pool.cc:224] Handling call
0808 01:52:49.356266 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:49.356652 (+   386us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:49.356679 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:49.357237 (+   558us) log.cc:844] Serialized 26 byte log entry
0808 01:52:49.559864 (+202627us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:52:49.585679 (+ 25815us) log.cc:844] Serialized 14 byte log entry
0808 01:52:49.639368 (+ 53689us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:49.639462 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:49.769228 (+129766us) raft_consensus.cc:1685] finished
0808 01:52:49.769251 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:49.769404 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53396,"prepare.queue_time_us":25693,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":74670,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":263,"thread_start_us":202122,"threads_started":1,"wal-append.queue_time_us":409482}
I20260808 01:52:49.753800 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:49.829105  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3188) took 1202 ms. Trace:
I20260808 01:52:49.834951 17240 raft_consensus.cc:3055] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:49.831377  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3348) took 1545 ms. Trace:
I20260808 01:52:49.827282  9478 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 24. Current term is 25. Ops: [24.13-24.13]
W20260808 01:52:49.835721 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 143573us
I20260808 01:52:49.842474 17371 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:52:49.820380 16808 raft_consensus.cc:1081] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Signalling peer 6e7856bed1be43acaa821290d506ee97 to start an election
I20260808 01:52:49.798432 17372 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:49.859098  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3346) took 1589 ms. Trace:
I20260808 01:52:49.908313  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3186) took 1538 ms. Trace:
I20260808 01:52:49.908726  9630 rpcz_store.cc:276] 0808 01:52:48.370205 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:48.622128 (+251923us) service_pool.cc:224] Handling call
0808 01:52:48.622395 (+   267us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:48.622596 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 10
0808 01:52:48.922984 (+300388us) log.cc:844] Serialized 14 byte log entry
0808 01:52:49.907955 (+984971us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:49.908014 (+    59us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:49.908060 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:49.908128 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:49.908255 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":173354,"prepare.queue_time_us":446085,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":6245,"raft.queue_time_us":74996,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":4782,"thread_start_us":1207205,"threads_started":3,"wal-append.queue_time_us":654047}
I20260808 01:52:49.907799  9472 rpcz_store.cc:276] 0808 01:52:48.269979 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:48.546220 (+276241us) service_pool.cc:224] Handling call
0808 01:52:48.566845 (+ 20625us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:48.567160 (+   315us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:48.567199 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:48.594091 (+ 26892us) log.cc:844] Serialized 26 byte log entry
0808 01:52:49.496602 (+902511us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:49.496663 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:49.496758 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:49.858771 (+362013us) raft_consensus.cc:1685] finished
0808 01:52:49.858804 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:49.859024 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":640384,"thread_start_us":678419,"threads_started":1,"wal-append.queue_time_us":846036}
W20260808 01:52:49.847026  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:49.846732  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:49.872720  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 181000us
I20260808 01:52:49.766584 17171 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:49.877568  9629 rpcz_store.cc:276] 0808 01:52:48.626620 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:48.985529 (+358909us) service_pool.cc:224] Handling call
0808 01:52:48.985967 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:48.986338 (+   371us) raft_consensus.cc:1534] Early marking committed up to index 12
0808 01:52:49.639001 (+652663us) log.cc:844] Serialized 14 byte log entry
0808 01:52:49.828729 (+189728us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:49.828780 (+    51us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:52:49.828824 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:49.828900 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:49.829043 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4216,"prepare.queue_time_us":292959,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"raft.queue_time_us":2173,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":2571,"thread_start_us":839529,"threads_started":2}
I20260808 01:52:49.831527  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:49.922139  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 90661.4us
I20260808 01:52:49.824446 17398 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:49.883549  9638 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:49.886488  9476 rpcz_store.cc:276] 0808 01:52:48.285689 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:48.573861 (+288172us) service_pool.cc:224] Handling call
0808 01:52:48.574378 (+   517us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:48.574936 (+   558us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:48.575085 (+   149us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:48.647724 (+ 72639us) log.cc:844] Serialized 26 byte log entry
0808 01:52:49.056214 (+408490us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:49.056255 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:49.056321 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:49.831100 (+774779us) raft_consensus.cc:1685] finished
0808 01:52:49.831132 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:49.831314 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":903,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":28499,"thread_start_us":244340,"threads_started":1,"wal-append.queue_time_us":611864}
I20260808 01:52:49.939076  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:49.890940  9520 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.010s	sys 0.015s
I20260808 01:52:49.811271 10365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 1167) took 1326 ms. Trace:
I20260808 01:52:49.924886  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:49.840646 17239 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:49.939297  9476 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 13. Current term is 14. Ops: []
I20260808 01:52:49.940032  9552 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
W20260808 01:52:49.929435  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 288993us
I20260808 01:52:49.940101 10365 rpcz_store.cc:276] 0808 01:52:48.485012 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:48.485175 (+   163us) service_pool.cc:224] Handling call
0808 01:52:48.518072 (+ 32897us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:48.518655 (+   583us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:48.518722 (+    67us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:48.530865 (+ 12143us) log.cc:844] Serialized 26 byte log entry
0808 01:52:48.877015 (+346150us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b180032dbe0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:52:49.771240 (+894225us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:49.771307 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:49.771486 (+   179us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:49.810797 (+ 39311us) raft_consensus.cc:1685] finished
0808 01:52:49.810835 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:49.811170 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":201,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":4714,"spinlock_wait_cycles":50956672,"thread_start_us":893848,"threads_started":1,"wal-append.queue_time_us":1203983}
I20260808 01:52:49.841940 17251 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:49.946388  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3178) took 1577 ms. Trace:
I20260808 01:52:49.944300 10367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 848) took 1746 ms. Trace:
I20260808 01:52:49.952564 10367 rpcz_store.cc:276] 0808 01:52:48.198380 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:48.198771 (+   391us) service_pool.cc:224] Handling call
0808 01:52:48.282961 (+ 84190us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:49.556826 (+1273865us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:49.943955 (+387129us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400449b54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:49.944211 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2539457408}
I20260808 01:52:49.886233 17240 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:49.949899  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1268) took 1038 ms. Trace:
I20260808 01:52:49.867085 17403 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.099s
I20260808 01:52:49.967474  9476 rpcz_store.cc:276] 0808 01:52:48.911895 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:49.938520 (+1026625us) service_pool.cc:224] Handling call
0808 01:52:49.939005 (+   485us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:49.949523 (+ 10518us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:49.949827 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:52:49.939992  9520 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.850s	user 0.011s	sys 0.015s
I20260808 01:52:49.972133 10361 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4f04078e3beb42eda250da45f7485537"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:57156
I20260808 01:52:49.943473 17398 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:49.977131 10361 raft_consensus.cc:606] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:49.944464 17370 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:49.925990 17371 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:52:49.968958  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 152065us
I20260808 01:52:49.900653 17346 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.072s
I20260808 01:52:49.984786 10361 raft_consensus.cc:614] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:49.984683  9475 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:49.951779 17314 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:49.949386 17239 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:49.966337 17394 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:49.952080  9635 rpcz_store.cc:276] 0808 01:52:48.368622 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:48.611442 (+242820us) service_pool.cc:224] Handling call
0808 01:52:48.611786 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:49.654799 (+1043013us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:49.946013 (+291214us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540003b1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:49.946299 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2382800768}
I20260808 01:52:50.000433 17342 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:49.945164 17407 consensus_queue.cc:1059] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.082s
I20260808 01:52:49.987450 17398 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:50.018163  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:50.030975  9632 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
W20260808 01:52:50.002606  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 155992us
I20260808 01:52:50.010036  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:50.032532  9637 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:50.003693  9629 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260808 01:52:49.993831  9476 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260808 01:52:50.035878  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:50.036823  9476 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
W20260808 01:52:50.021595  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 96824.7us
I20260808 01:52:50.039516  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:49.950722 17205 consensus_queue.cc:237] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:50.028550  9601 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:50.038275  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:50.017706 17396 log.cc:927] Time spent T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
I20260808 01:52:49.968111 17409 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.062s
W20260808 01:52:50.059248 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 83171.4us
I20260808 01:52:50.064082  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:50.001080 17239 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:50.085273  9476 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 20: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:50.072355  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3192) took 1444 ms. Trace:
I20260808 01:52:50.024432 17368 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260808 01:52:50.065527 17264 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.580s	user 0.007s	sys 0.008s
W20260808 01:52:50.082238  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 53826.4us
I20260808 01:52:50.042517 17133 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:50.009466 17129 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.068s
W20260808 01:52:50.006232 17262 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.135s	user 0.020s	sys 0.019s
I20260808 01:52:50.097692  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:50.101447  9476 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:50.100471 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:50.086525  9636 rpcz_store.cc:276] 0808 01:52:48.627843 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:48.999081 (+371238us) service_pool.cc:224] Handling call
0808 01:52:48.999519 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:48.999808 (+   289us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:52:49.053988 (+ 54180us) log.cc:844] Serialized 14 byte log entry
0808 01:52:50.012379 (+958391us) log.cc:844] Serialized 14 byte log entry
0808 01:52:50.071981 (+ 59602us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:50.072039 (+    58us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:50.072093 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:50.072166 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:50.072294 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":79403,"prepare.queue_time_us":287185,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":102,"raft.queue_time_us":148111,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":12539,"thread_start_us":957551,"threads_started":2,"wal-append.queue_time_us":797117}
I20260808 01:52:49.983736 17406 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.066s
I20260808 01:52:50.007665 17314 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:50.099634  9637 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:52:50.078645 17208 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:50.109984 17310 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:50.110414 17251 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:50.095733 17308 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election won for term 25
W20260808 01:52:50.102658 17262 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.241s	user 0.021s	sys 0.020s
I20260808 01:52:50.077418 17372 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:50.128057  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 84379.3us
I20260808 01:52:50.096254 17133 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:50.142437  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 4f04078e3beb42eda250da45f7485537 (from 7201605bb2f54c949a4ebe6e9b3b4b2a to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:50.159785  9476 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:52:50.009961 17240 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:50.012270 17394 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:50.113502 16808 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:50.143748  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3383) took 1320 ms. Trace:
I20260808 01:52:50.120204 17321 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:50.113438 17432 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260808 01:52:50.153861  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 93527.8us
I20260808 01:52:50.139127 17129 consensus_queue.cc:1243] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 23.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260808 01:52:50.138479 17337 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:52:50.143312 17375 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.950s	user 0.000s	sys 0.008s
I20260808 01:52:50.138913 17296 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:50.142248 17395 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:50.169167  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:50.153592 17308 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:50.162459  9472 rpcz_store.cc:276] 0808 01:52:48.823107 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:49.940281 (+1117174us) service_pool.cc:224] Handling call
0808 01:52:49.966831 (+ 26550us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:49.967059 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 12
0808 01:52:50.087857 (+120798us) log.cc:844] Serialized 14 byte log entry
0808 01:52:50.143385 (+ 55528us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:50.143441 (+    56us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:52:50.143490 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:50.143565 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:50.143687 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":28760,"thread_start_us":149037,"threads_started":2}
I20260808 01:52:50.145355 17328 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.023s
I20260808 01:52:50.203639  9637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:50.162787  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:50.225508  9637 raft_consensus.cc:606] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:50.225962  9637 raft_consensus.cc:614] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:50.226491  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:50.227782  9476 raft_consensus.cc:2410] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 20 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260808 01:52:50.162026 17422 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:50.228929  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:50.180127  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:50.099943 17371 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:50.168309 17430 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:52:50.171530 17240 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
W20260808 01:52:50.204432 17405 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.221s	user 0.001s	sys 0.000s
W20260808 01:52:50.234978  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 54928.2us
I20260808 01:52:50.239360  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:50.240855  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:50.237828  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:50.244297  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:50.174523 17395 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:50.246869  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:50.230103 17259 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:52:50.166502 16808 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:50.232678 17437 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:52:50.266551  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:50.177599 17372 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:50.242208 17251 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:50.336567  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:50.338421  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:50.338647  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:50.352495  9636 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
W20260808 01:52:50.339928 10334 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.916s (SENT) [suppressed 10 similar messages]
I20260808 01:52:50.276151  9560 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260808 01:52:50.341722  9555 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:50.341373  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:50.342430  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:50.343142  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:50.344055  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3169) took 1977 ms. Trace:
I20260808 01:52:50.179061 17337 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:50.262883 17410 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
I20260808 01:52:50.270854 17395 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:50.347950  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:50.351015  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:50.255666 17149 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.558s	user 0.012s	sys 0.000s
I20260808 01:52:50.354218  9550 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 20: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:50.354172  9631 rpcz_store.cc:276] 0808 01:52:48.366544 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:48.374121 (+  7577us) service_pool.cc:224] Handling call
0808 01:52:48.374490 (+   369us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:48.376717 (+  2227us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:48.376767 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:48.377432 (+   665us) log.cc:844] Serialized 26 byte log entry
0808 01:52:48.809380 (+431948us) raft_consensus.cc:1645] Marking committed up to 8
0808 01:52:48.809462 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:48.809565 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:50.002744 (+1193179us) spinlock_profiling.cc:243] Waited 3.33 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:50.263307 (+260563us) raft_consensus.cc:1685] finished
0808 01:52:50.263333 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:50.343935 (+ 80602us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12824,"raft.queue_time_us":4048,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":93397,"spinlock_wait_cycles":7657728,"thread_start_us":339201,"threads_started":1,"wal-append.queue_time_us":567101}
I20260808 01:52:50.372385  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:50.373268  9631 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:50.346698  9559 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:52:50.357208  9629 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 20: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:50.359644  9632 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: [15.14-15.14]
I20260808 01:52:50.262599 17133 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:50.436003  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:50.346339 17437 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:50.238302 17422 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:50.456307  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:50.457671  9555 raft_consensus.cc:2393] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 20: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:50.439288  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:50.447098  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:50.459311  9550 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 18: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:50.464042  9631 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
W20260808 01:52:50.470090 17324 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.782s	user 0.011s	sys 0.000s
I20260808 01:52:50.341796 17308 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:50.472354  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:50.349177  9369 catalog_manager.cc:5697] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 10 to 11, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 11 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:52:50.475466  9369 catalog_manager.cc:5697] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 11 to 13, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 13 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:52:50.477450  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:50.474364  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:50.479547  9557 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:50.481077  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:50.484912 17415 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.273s	user 0.000s	sys 0.001s
W20260808 01:52:50.485970 17419 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.286s	user 0.001s	sys 0.000s
I20260808 01:52:50.478394  9555 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:50.487840  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:50.488153 17423 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.321s	user 0.000s	sys 0.001s
I20260808 01:52:50.489512  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:50.482002 17427 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.365s	user 0.003s	sys 0.007s
W20260808 01:52:50.482060 17285 consensus_meta.cc:287] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.840s	user 0.011s	sys 0.000s
I20260808 01:52:50.491356 17285 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election won for term 20
W20260808 01:52:50.491755 17426 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.250s	user 0.001s	sys 0.000s
I20260808 01:52:50.483309 17443 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:50.492612 17443 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:50.493781 17440 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.145s	user 0.000s	sys 0.000s
W20260808 01:52:50.494729  9586 heartbeater.cc:646] Failed to heartbeat to 127.7.165.254:43565 (4 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.7.165.254:43565 timed out after 15.000s (SENT)
I20260808 01:52:50.495774 17443 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:50.483458 17421 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
W20260808 01:52:50.497942 17378 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 1.304s	user 0.003s	sys 0.010s
I20260808 01:52:50.498437  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1263) took 1878 ms. Trace:
I20260808 01:52:50.499015  9481 rpcz_store.cc:276] 0808 01:52:48.619889 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:49.639610 (+1019721us) service_pool.cc:224] Handling call
0808 01:52:49.652289 (+ 12679us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:49.684597 (+ 32308us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:49.684651 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:49.690295 (+  5644us) log.cc:844] Serialized 26 byte log entry
0808 01:52:49.974165 (+283870us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:52:49.974316 (+   151us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:49.974623 (+   307us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:50.498106 (+523483us) raft_consensus.cc:1685] finished
0808 01:52:50.498142 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:50.498353 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":35134,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":27432,"thread_start_us":219489,"threads_started":1,"wal-append.queue_time_us":518434}
I20260808 01:52:50.503489 17358 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:50.503690  9632 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260808 01:52:50.504300 17358 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:50.507465 17358 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:50.486378  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:50.510031  9557 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:50.511734  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:50.512674  9631 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:50.360102 17434 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260808 01:52:50.490409 17416 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.267s	user 0.001s	sys 0.000s
I20260808 01:52:50.465401  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:50.545571  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:50.572862  9629 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:50.436219 17439 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260808 01:52:50.547164 17134 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.773s	user 0.013s	sys 0.001s
I20260808 01:52:50.376343 17259 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.170s
I20260808 01:52:50.547262 17314 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:50.546550 17434 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:50.612706  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3345) took 2411 ms. Trace:
I20260808 01:52:50.613199  9479 rpcz_store.cc:276] 0808 01:52:48.201217 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:48.536856 (+335639us) service_pool.cc:224] Handling call
0808 01:52:48.537190 (+   334us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:48.537608 (+   418us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:48.537644 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:48.538112 (+   468us) log.cc:844] Serialized 26 byte log entry
0808 01:52:48.875214 (+337102us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:48.875263 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:48.875341 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:50.273647 (+1398306us) spinlock_profiling.cc:243] Waited 3.54 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:50.598892 (+325245us) raft_consensus.cc:1685] finished
0808 01:52:50.598916 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:50.612621 (+ 13705us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8889,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":12645,"spinlock_wait_cycles":8137088,"thread_start_us":314374,"threads_started":1,"wal-append.queue_time_us":525652}
I20260808 01:52:50.582263 17439 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:50.622589  9479 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260808 01:52:50.623672  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3387) took 1026 ms. Trace:
W20260808 01:52:50.625140  9524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.003s	sys 0.009s
W20260808 01:52:50.625653  9524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.003s	sys 0.009s
I20260808 01:52:50.624241  9479 rpcz_store.cc:276] 0808 01:52:49.597143 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:50.619599 (+1022456us) service_pool.cc:224] Handling call
0808 01:52:50.620026 (+   427us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:50.623264 (+  3238us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:50.623590 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:52:50.606604 16983 consensus_meta.cc:287] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.838s	user 0.009s	sys 0.004s
I20260808 01:52:50.647966  9479 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:50140
I20260808 01:52:50.648900  9479 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:52:50.590391 17408 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:50.609453 17434 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:50.675551  9557 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
W20260808 01:52:50.653064 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 313153us
I20260808 01:52:50.526063 17324 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:50.607100 16835 raft_consensus.cc:3055] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:52:50.655632  9479 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:52:50.684445  9444 connection.cc:256] RPC call timeout handler was delayed by -1.2406817940063775s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.168s for a response
W20260808 01:52:50.685170  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1947817939799279s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17310000000000003s for a response
I20260808 01:52:50.639374 17372 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:50.633275 17285 raft_consensus.cc:697] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:50.617192  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 457791us
I20260808 01:52:50.659560  9629 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
W20260808 01:52:50.629942 17370 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.006s	sys 0.006s
W20260808 01:52:50.626012  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 638082us
W20260808 01:52:50.685580  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1767817939980887s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1751s for a response
W20260808 01:52:50.727551  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1470817939843982s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1784s for a response
W20260808 01:52:50.727967  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1182817939843517s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1816s for a response
W20260808 01:52:50.728276  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0795817939797416s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1859s for a response
I20260808 01:52:50.728667  9631 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:50.729063  9444 leader_election.cc:400] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18.
I20260808 01:52:50.729532  9444 leader_election.cc:403] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260808 01:52:50.713099 17285 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:50.627389 17439 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:50.661428 17158 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.005s	sys 0.011s
W20260808 01:52:50.743376  9477 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 3361) took 1962 ms (client timeout 2045 ms). Trace:
I20260808 01:52:50.731630  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:50.748008  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:50.733392  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 947) took 2052 ms. Trace:
I20260808 01:52:50.662511 17251 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:50.751766 10360 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260808 01:52:50.751041  9633 rpcz_store.cc:276] 0808 01:52:48.680512 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:49.495205 (+814693us) service_pool.cc:224] Handling call
0808 01:52:49.496357 (+  1152us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:49.497040 (+   683us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:49.497070 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:49.540773 (+ 43703us) log.cc:844] Serialized 52 byte log entry
0808 01:52:49.828457 (+287684us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:49.867420 (+ 38963us) log.cc:844] Serialized 14 byte log entry
0808 01:52:49.871767 (+  4347us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:49.871848 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:50.733113 (+861265us) raft_consensus.cc:1685] finished
0808 01:52:50.733142 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:50.733319 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4697,"prepare.queue_time_us":38583,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":59,"raft.queue_time_us":4647,"raft.run_cpu_time_us":363,"raft.run_wall_time_us":30832,"thread_start_us":192323,"threads_started":1,"wal-append.queue_time_us":408742}
I20260808 01:52:50.753441  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:50.760708  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:50.745879  9477 rpcz_store.cc:269] 0808 01:52:48.781265 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:48.978934 (+197669us) service_pool.cc:224] Handling call
0808 01:52:50.743023 (+1764089us) spinlock_profiling.cc:243] Waited 402 ms on lock 0x7b540009a6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:50.743272 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":925199104}
I20260808 01:52:50.778594  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:50.780543  9444 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:50.783149  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:50.784982  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:50.677735 16835 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:52:50.745134 17158 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.006s	sys 0.011s
I20260808 01:52:50.656154 16983 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:50.734750 17370 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.006s	sys 0.007s
I20260808 01:52:50.731612 17134 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:52:50.891134  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:50.727406 17406 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:50.787876  9479 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:50.791785 17336 consensus_meta.cc:287] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.015s	user 0.017s	sys 0.000s
I20260808 01:52:50.902444  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:50.903759  9557 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 18: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:50.789803  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:50.782965 17342 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18.
I20260808 01:52:50.913044 17342 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:50.913936 17342 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:50.914336 17342 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:50.914837 17342 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:50.916378  9554 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:50.917716 16983 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:50.641196 17437 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:50.919227 17437 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:52:50.924170  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:50.926079  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:50.927048 17461 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:50.927508  9552 raft_consensus.cc:2410] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260808 01:52:50.927584 17461 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:50.928248  9479 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:50.750977 17447 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:50.909504 17336 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:50.909757 16835 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:50.910521 17445 log.cc:927] Time spent T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.161s	user 0.000s	sys 0.000s
I20260808 01:52:50.910727 17441 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:50.910842 17442 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:50.912040 17457 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:50.928347 17461 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:50.928691  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:50.964917 17406 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:50.933163 16939 consensus_meta.cc:287] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.248s	user 0.013s	sys 0.000s
I20260808 01:52:50.937760 17251 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:50.675129 17408 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:50.750144 17443 consensus_queue.cc:1059] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.149s
I20260808 01:52:51.056084  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3187) took 2682 ms. Trace:
I20260808 01:52:51.056757  9638 rpcz_store.cc:276] 0808 01:52:48.370417 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:48.969630 (+599213us) service_pool.cc:224] Handling call
0808 01:52:48.970080 (+   450us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:49.938077 (+967997us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:51.052862 (+1114785us) spinlock_profiling.cc:243] Waited 913 ms on lock 0x7b54004213d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:51.053162 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2100095360}
I20260808 01:52:50.942291  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:50.951766 17422 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:50.952565 17394 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:50.728132  9369 catalog_manager.cc:5697] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 6 to 12, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 12 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:52:50.955574 17259 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:51.043982 17323 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:50.799584 17446 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:50.957259 17158 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260808 01:52:50.959659 17232 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.458s	user 0.014s	sys 0.002s
I20260808 01:52:50.732160 17372 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:50.872547 17264 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:50.873519 17370 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:50.973513 17265 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.221s	user 0.014s	sys 0.000s
I20260808 01:52:51.024431  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:51.031478  9479 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22. Ops: []
I20260808 01:52:51.043889  9474 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
W20260808 01:52:51.004459 17239 consensus_meta.cc:287] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.919s	user 0.011s	sys 0.002s
I20260808 01:52:50.906359 17134 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.051972  9601 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:51.108857  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 57045.5us
I20260808 01:52:51.057996  9552 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 22: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:50.944360 17324 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:51.084527  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:51.114866  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:51.087318 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 335097us
I20260808 01:52:51.053642 16835 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
W20260808 01:52:51.091540  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 306632us
I20260808 01:52:51.091629 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:51.093310 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:51.100778  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 113803us
I20260808 01:52:51.111649  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:51.124337  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:51.110365  9601 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:51.046854 17447 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:51.055832 16939 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.095989 17394 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:51.128679  9600 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:51.130303  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:51.135871  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 90465c2f160641a7901ab0815e723ce1 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:51.136509  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260808 01:52:51.137148  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:51.137954  9552 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:52:51.135774  9476 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 21: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:51.128368  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:51.131240  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:51.140059  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:52:51.132985 17312 raft_consensus.cc:3055] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Stepping down as leader of term 20
W20260808 01:52:51.133373 17466 log.cc:927] Time spent T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260808 01:52:51.141103  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:52:51.108011 17262 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:51.109543 17265 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260808 01:52:51.111238 17407 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:51.114585 17319 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:51.091275 17323 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.097656 17232 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:51.143587 17312 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:51.157542 17262 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:51.157938 17469 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260808 01:52:51.082142 17446 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:51.057675 17442 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:51.166935 10367 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260808 01:52:51.057081 17441 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.127472 17211 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260808 01:52:51.210767  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 82217us
I20260808 01:52:51.129585 17311 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election won for term 21
W20260808 01:52:51.218221  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:51.221798  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 1077) took 1302 ms. Trace:
W20260808 01:52:51.222038  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 97814.5us
I20260808 01:52:51.135293 16939 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:51.106725 17264 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:51.142521 17394 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.234174 10360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 951) took 1718 ms. Trace:
I20260808 01:52:51.148814 17324 consensus_queue.cc:1059] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.092s
I20260808 01:52:51.108525 17342 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.114136 17239 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:51.236701  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:51.242735  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1271) took 1832 ms. Trace:
I20260808 01:52:51.264427  9481 rpcz_store.cc:276] 0808 01:52:49.410257 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:50.501287 (+1091030us) service_pool.cc:224] Handling call
0808 01:52:50.501745 (+   458us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:50.523710 (+ 21965us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:50.523759 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:50.523824 (+    65us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:52:50.523881 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:51.242322 (+718441us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b54002bacd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:51.242403 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:51.242618 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49826688}
I20260808 01:52:51.248569  9554 rpcz_store.cc:276] 0808 01:52:49.918969 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:49.919180 (+   211us) service_pool.cc:224] Handling call
0808 01:52:49.919646 (+   466us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:50.916838 (+997192us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:51.221520 (+304682us) spinlock_profiling.cc:243] Waited 996 ms on lock 0x7b540002e254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:51.221727 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2291762944}
I20260808 01:52:51.249754  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:51.223869 17285 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:51.224051 17295 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:51.260712  9557 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:51.263237  9552 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260808 01:52:51.248718 17211 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:51.260809 10360 rpcz_store.cc:276] 0808 01:52:49.516240 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:49.516422 (+   182us) service_pool.cc:224] Handling call
0808 01:52:49.571709 (+ 55287us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:50.758117 (+1186408us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:51.222572 (+464455us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400690fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:51.234055 (+ 11483us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2713266432}
I20260808 01:52:51.286010 17232 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:51.286926 17470 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
W20260808 01:52:51.288118 17398 consensus_meta.cc:287] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.227s	user 0.009s	sys 0.005s
I20260808 01:52:51.288196 17422 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:51.290531 17262 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:51.238893 17443 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:51.294788  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:51.240042 17403 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:51.238626 17473 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260808 01:52:51.312050  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 183727us
I20260808 01:52:51.210826 17319 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.284113 17447 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:51.215613 17407 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:51.322264  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:51.328688 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 169829us
I20260808 01:52:51.300208 17211 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:51.338114  9552 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260808 01:52:51.291360 17450 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:51.351082 17447 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:51.357841 17447 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:51.358382 17447 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:51.367528  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:51.299355 16939 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260808 01:52:51.367959  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:51.368315  9633 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:52:51.371562 17465 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.058s
I20260808 01:52:51.372309 17436 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:51.373266  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:51.402642  9481 raft_consensus.cc:2376] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19. Re-sending same reply.
I20260808 01:52:51.373538 17408 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:51.319458 17422 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.253093 17370 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:51.377931  9637 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:51.377857  9633 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:51.366323 17336 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:51.320815 17443 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.344866 17311 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:51.334141 17457 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.073s
I20260808 01:52:51.133387  9369 catalog_manager.cc:5697] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 4 to 11, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 11 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:52:51.321031 17403 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.424757  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:51.426363 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47354.8us
I20260808 01:52:51.430884 10334 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:51.413334  9633 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:51.314329 17398 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:51.432299  9633 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:51.433917  9635 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 22: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:51.331287 17312 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:51.451059  9559 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 24. Current term is 25. Ops: []
I20260808 01:52:51.419981 17480 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:51.463264  9554 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:51.463503  9637 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:52:51.377799 17473 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.066s
I20260808 01:52:51.432529 17372 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:51.470209  9637 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260808 01:52:51.449369 17406 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:51.411832 17459 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.130s
I20260808 01:52:51.467434 17312 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:51.527491 17477 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:51.528313 17477 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.488118 17323 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:51.538457  9633 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:52:51.539083 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:51.540942  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3357) took 2760 ms. Trace:
I20260808 01:52:51.541854 10369 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:51.541545  9478 rpcz_store.cc:276] 0808 01:52:48.780146 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:49.653753 (+873607us) service_pool.cc:224] Handling call
0808 01:52:49.826920 (+173167us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:49.886956 (+ 60036us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:51.540692 (+1653736us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b540023aad0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:51.540869 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":397215616}
I20260808 01:52:51.544523 17394 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:51.559056 17484 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.103s
I20260808 01:52:51.563540 17422 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:51.565855 17481 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:51.567061 17319 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:51.566824 17481 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.567754  9369 catalog_manager.cc:5697] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 10 to 11, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 11 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:52:51.571096 17443 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:51.520473 17311 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:51.560755 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:51.589676  9551 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 13. Current term is 14. Ops: []
I20260808 01:52:51.586829  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 3389) took 1572 ms. Trace:
I20260808 01:52:51.611109 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:51.616866 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 186044us
I20260808 01:52:51.634518  9476 rpcz_store.cc:276] 0808 01:52:50.014059 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:50.239080 (+225021us) service_pool.cc:224] Handling call
0808 01:52:51.586468 (+1347388us) spinlock_profiling.cc:243] Waited 895 ms on lock 0x7b54002a6cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:51.586730 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2059289984}
I20260808 01:52:51.581355  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3001) took 1014 ms. Trace:
I20260808 01:52:51.611958  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:51.636065  9631 rpcz_store.cc:276] 0808 01:52:50.566999 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:50.583075 (+ 16076us) service_pool.cc:224] Handling call
0808 01:52:50.583639 (+   564us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:50.729319 (+145680us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:51.580812 (+851493us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b540008dc54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:51.581252 (+   440us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":332850560}
I20260808 01:52:51.600042  9479 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:51.637542  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:51.596585  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:51.558027 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:51.596298  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:51.638602  9478 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 22.
I20260808 01:52:51.583498 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:51.640604  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:51.610632 10369 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260808 01:52:51.641360  9478 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:51.521203 17264 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:51.635066 10361 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:52:51.643267  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:51.667068  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:51.659340  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:51.668371  9476 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:51.669333  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:51.670135  9476 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:52:51.689134  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:51.664614 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:51.692821  9481 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 19: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:52:51.541917 17434 consensus_meta.cc:287] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.816s	user 0.008s	sys 0.006s
W20260808 01:52:51.579669 17395 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.145s	user 0.014s	sys 0.001s
W20260808 01:52:51.591135 17412 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.580s	user 0.009s	sys 0.010s
I20260808 01:52:51.521975 17486 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election won for term 19
W20260808 01:52:51.540524 17437 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.621s	user 0.010s	sys 0.005s
I20260808 01:52:51.699788  9478 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:51.539548 17342 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:51.667845 17240 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.426s	user 0.018s	sys 0.000s
I20260808 01:52:51.478295 17480 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.539373 17459 consensus_queue.cc:1243] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 22.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260808 01:52:51.592347 17432 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:51.534868 17446 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:51.585211 16808 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:51.600898 17403 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:51.603052 17398 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:51.544859 17450 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:51.716712  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:51.717722  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:51.532490 17477 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:51.719234  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:51.751437  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:51.751959  9481 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:51.607640 17407 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:51.586738 17372 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.129s
I20260808 01:52:51.689560 17481 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:51.741806 17486 raft_consensus.cc:697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:51.703996 17494 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:52:51.574549 17461 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:51.750267  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:51.742017 17439 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.952s	user 0.016s	sys 0.000s
I20260808 01:52:51.737092  9555 raft_consensus.cc:2410] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 20 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260808 01:52:51.591038 17158 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.137s
W20260808 01:52:51.769238 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 82287.6us
I20260808 01:52:51.747366 17442 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:51.737856 17432 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:51.695711 17434 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:51.751165 16835 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.621s	user 0.008s	sys 0.006s
W20260808 01:52:51.765358  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:51.767205  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 428112us
I20260808 01:52:51.713140 17395 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.751045 16974 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:51.570555  9369 catalog_manager.cc:5697] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 6 to 13. New cstate: current_term: 13 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:52:51.757745  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:51.762250 17494 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:51.784329  9481 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 22: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:51.784396 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:51.785706 17346 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:51.792315 17406 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:51.793097 17395 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:51.793503 17406 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:51.794054 17406 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:51.770742 17439 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:51.794960  9474 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:51.797009 17439 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:51.796738 17432 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:51.798023 17486 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.787225 16974 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.800136  9480 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:51.801538 17494 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:51.754371 17450 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:51.801314 17406 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.803596 17406 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:51.801810 17129 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.003s
I20260808 01:52:51.805254 17129 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:51.806201 17129 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:51.782917 17398 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:51.807232 17398 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.809540 17398 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:51.811748  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3745) took 1344 ms. Trace:
I20260808 01:52:51.811882  9477 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:51.807811 17129 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:51.812417  9552 rpcz_store.cc:276] 0808 01:52:50.467636 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:51.342389 (+874753us) service_pool.cc:224] Handling call
0808 01:52:51.365944 (+ 23555us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:51.366128 (+   184us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:52:51.749707 (+383579us) log.cc:844] Serialized 14 byte log entry
0808 01:52:51.808609 (+ 58902us) log.cc:844] Serialized 14 byte log entry
0808 01:52:51.811056 (+  2447us) log.cc:844] Serialized 14 byte log entry
0808 01:52:51.811287 (+   231us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:51.811350 (+    63us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:51.811407 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:51.811494 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:51.811662 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17364,"raft.queue_time_us":21698,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":309,"thread_start_us":382937,"threads_started":2}
I20260808 01:52:51.818594 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:51.819628 10366 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 21: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:51.821493 16974 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:51.822382  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3363) took 3040 ms. Trace:
I20260808 01:52:51.823304  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:51.824230  9560 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:51.824288  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 3000) took 1321 ms. Trace:
I20260808 01:52:51.824923  9629 rpcz_store.cc:276] 0808 01:52:50.503239 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:50.545159 (+ 41920us) service_pool.cc:224] Handling call
0808 01:52:51.824073 (+1278914us) spinlock_profiling.cc:243] Waited 19.1 ms on lock 0x7b54000df554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:51.824214 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":44026112}
I20260808 01:52:51.826457 17240 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:51.828979  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3193) took 3200 ms. Trace:
I20260808 01:52:51.822851  9475 rpcz_store.cc:276] 0808 01:52:48.781809 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:49.826034 (+1044225us) service_pool.cc:224] Handling call
0808 01:52:49.826486 (+   452us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:49.987760 (+161274us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:51.822105 (+1834345us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b54002bbbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:51.822310 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":363117312}
I20260808 01:52:51.829552  9634 rpcz_store.cc:276] 0808 01:52:48.628105 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:49.068810 (+440705us) service_pool.cc:224] Handling call
0808 01:52:49.069166 (+   356us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:49.517837 (+448671us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:49.517868 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:49.542930 (+ 25062us) log.cc:844] Serialized 26 byte log entry
0808 01:52:49.841588 (+298658us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:49.882256 (+ 40668us) log.cc:844] Serialized 14 byte log entry
0808 01:52:49.882616 (+   360us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:49.942951 (+ 60335us) spinlock_profiling.cc:243] Waited 448 ms on lock 0x7b54001b44d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:49.943029 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:51.589908 (+1646879us) spinlock_profiling.cc:243] Waited 2.39 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:51.593241 (+  3333us) spinlock_profiling.cc:243] Waited 4.5 ms on lock 0x7b180043fa80. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:52:51.828608 (+235367us) raft_consensus.cc:1685] finished
0808 01:52:51.828647 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:51.828873 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35025,"prepare.queue_time_us":45973,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":5861,"raft.queue_time_us":1890,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":15655,"spinlock_wait_cycles":1046653056,"thread_start_us":203020,"threads_started":1,"wal-append.queue_time_us":376799}
I20260808 01:52:51.827117 17240 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:51.840783 17240 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:51.814407 17346 raft_consensus.cc:3055] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:51.841862 17346 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:51.810626 17447 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:51.845101 17447 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:51.845352 17453 raft_consensus.cc:3055] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:51.846055 17453 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:51.844992 17319 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:51.803975 17450 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:51.798527 17432 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:51.866498 17319 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.867218  9369 catalog_manager.cc:5697] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 11 to 14, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 14 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:52:51.873605  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:51.877867  9560 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:52:51.857039 17129 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:51.875373  9552 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:52:51.856309 17441 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:51.898463 17319 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:51.912501  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:51.931911 17480 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:51.932319 17337 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:51.885474 17453 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:51.933240 17453 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:51.934091 17134 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:51.945436  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:51.948402  9472 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 3397) took 1843 ms (client timeout 1961 ms). Trace:
W20260808 01:52:51.949024  9472 rpcz_store.cc:269] 0808 01:52:50.105399 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:50.223796 (+118397us) service_pool.cc:224] Handling call
0808 01:52:51.948077 (+1724281us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54002dc3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:51.948313 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2511824640}
I20260808 01:52:51.896454  9369 catalog_manager.cc:5697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 6 to 12. New cstate: current_term: 12 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:52:51.951912  9369 catalog_manager.cc:5697] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 8 to 11, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 11 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:52:51.954849 17240 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:51.955611 17240 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.958828 17240 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:51.966836  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:51.967476  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:51.968063  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:51.967985  9472 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
W20260808 01:52:51.968672  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:51.969085  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:51.969858  9472 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260808 01:52:51.969666 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:51.974063  9481 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
W20260808 01:52:51.994024 17425 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.766s	user 0.009s	sys 0.013s
I20260808 01:52:51.977604 17337 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:52:51.998488 17409 raft_consensus.cc:3055] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Stepping down as leader of term 18
W20260808 01:52:51.995857 17211 consensus_meta.cc:287] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.630s	user 0.007s	sys 0.008s
I20260808 01:52:51.956130 17346 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:52.009577  9475 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 14. Current term is 15. Ops: []
I20260808 01:52:52.001946 17337 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:52.002800 17409 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:52.017815 17409 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:52.019074 17409 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:52.021382 17494 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:51.954531  9369 catalog_manager.cc:5697] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 10 to 12, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 12 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:52:52.022990 17337 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:52:52.023928  9369 catalog_manager.cc:5697] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 9 to 12, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 12 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:52:52.027347  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2858) took 1658 ms. Trace:
I20260808 01:52:52.027944  9557 rpcz_store.cc:276] 0808 01:52:50.433127 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:51.110778 (+677651us) service_pool.cc:224] Handling call
0808 01:52:52.027110 (+916332us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b54000b45d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:52.027264 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":276361088}
I20260808 01:52:52.029700  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:52.009851 17211 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:52.033316 17211 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:52.034358 17251 raft_consensus.cc:3055] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:52.034996 17494 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:52.039692  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:52.041205  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:52.042156  9475 raft_consensus.cc:2393] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:52.033195 17239 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:52.035039 17251 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:52.051755  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.025982  9369 catalog_manager.cc:5697] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 8 to 10, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 10 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:52:52.054186  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:52.043254 17479 consensus_queue.cc:1059] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.008s
I20260808 01:52:52.056860  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:52.057340 17479 raft_consensus.cc:3055] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Stepping down as leader of term 14
I20260808 01:52:52.057881 17479 raft_consensus.cc:740] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:52.059240 17479 consensus_queue.cc:260] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:52.060881 17479 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:52.063766  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:52.065479  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1267) took 3242 ms. Trace:
I20260808 01:52:52.067183  9475 raft_consensus.cc:2393] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 17: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:52.066998  9473 rpcz_store.cc:276] 0808 01:52:48.822632 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:49.865874 (+1043242us) service_pool.cc:224] Handling call
0808 01:52:50.743290 (+877416us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:50.743628 (+   338us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:50.743671 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:50.743742 (+    71us) raft_consensus.cc:1645] Marking committed up to 16
0808 01:52:50.743794 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:50.743881 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:52.065286 (+1321405us) spinlock_profiling.cc:243] Waited 857 ms on lock 0x7b540009a6d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:52.065406 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1971651456}
I20260808 01:52:52.068967  9473 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:52.070400  9473 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260808 01:52:52.059242  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:52.073129  9472 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260808 01:52:52.074479  9477 raft_consensus.cc:3055] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:52.075551  9477 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:52.063642 17251 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:52.083243  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2860) took 1629 ms. Trace:
I20260808 01:52:52.083773  9554 rpcz_store.cc:276] 0808 01:52:50.453393 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:51.266836 (+813443us) service_pool.cc:224] Handling call
0808 01:52:51.267281 (+   445us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:51.463859 (+196578us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:52.082643 (+618784us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b540002d0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:52.083162 (+   519us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":450154368}
I20260808 01:52:52.085386  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:52.088650  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:52.089412  9556 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:52.093616 17434 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:52.090518  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:52.096621  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:52.097977  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:52.098861  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.098963  9554 raft_consensus.cc:2410] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 21 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260808 01:52:52.103972  9476 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:52:52.104594  9476 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:52.076576  9477 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:52.115803  9477 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:52:52.118010  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.105623  9476 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:52.124462  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:52.125247  9556 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:52:52.125262  9476 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:52.117152  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:52.107684 17251 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:52.108935 17314 raft_consensus.cc:3055] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 LEADER]: Stepping down as leader of term 13
I20260808 01:52:52.133325  9369 catalog_manager.cc:5697] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:52:52.135481  9369 catalog_manager.cc:5697] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 9 to 14. New cstate: current_term: 14 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
W20260808 01:52:52.136392 17511 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260808 01:52:52.138134  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3748) took 1669 ms. Trace:
I20260808 01:52:52.138541  9560 rpcz_store.cc:276] 0808 01:52:50.468675 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:51.883738 (+1415063us) service_pool.cc:224] Handling call
0808 01:52:51.884247 (+   509us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:51.887660 (+  3413us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:51.887695 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:51.888209 (+   514us) log.cc:844] Serialized 26 byte log entry
0808 01:52:51.975491 (+ 87282us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:51.975544 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:51.975642 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:52.137848 (+162206us) raft_consensus.cc:1685] finished
0808 01:52:52.137874 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:52.138051 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13816,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":695,"thread_start_us":71561,"threads_started":1,"wal-append.queue_time_us":97631}
I20260808 01:52:52.109305 17319 consensus_queue.cc:1059] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
I20260808 01:52:52.140643  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:52.132274 17314 raft_consensus.cc:740] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:52.145463 17314 consensus_queue.cc:260] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:52.146975 17314 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 13 FOLLOWER]: Advancing to term 14
I20260808 01:52:52.151670  9369 catalog_manager.cc:5697] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 6 to 8, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 8 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:52:52.155638  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.155408 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:52.165645  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:52.166394  9556 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:52:52.166991 17461 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:52.167471  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:52.168167  9476 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20260808 01:52:52.168592  9476 pending_rounds.cc:85] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 11
I20260808 01:52:52.168877  9476 pending_rounds.cc:107] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 21.12
I20260808 01:52:52.168728  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:52.169183  9476 raft_consensus.cc:2889] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:52:52.169226  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.169416  9556 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:52:52.171083  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:52.178056  9524 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:52.183172  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:52.183346  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:52.184249 17409 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:52:52.184927  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:52.194257  9556 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:52.187234  9481 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:52.189369  9557 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:52:52.186609 17436 raft_consensus.cc:3055] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:52:52.192739  9524 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:52.203617 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 48335.2us
I20260808 01:52:52.204888 10332 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:52.204986  9524 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:52.206884 17319 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:52.207434 17319 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:52.207828 17319 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:52.191980 17265 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:52.209218 17265 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:52.209319 17337 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:52.201983 17436 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:52.211714 17265 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:52.212213  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:52.211797 17436 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:52.204900  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:52.213431 10332 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:52.214917  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:52.202550 17239 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:52.215633 17436 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:52.215746 17239 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:52.182467 17129 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:52.220597  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:52.217258 17323 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260808 01:52:52.218366 17394 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:52.212908 17451 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:52:52.199216 17409 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:52.182183 17432 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:52.216244 17239 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:52.219434 10367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 1175) took 1115 ms. Trace:
I20260808 01:52:52.225358  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:52.232147 10367 rpcz_store.cc:276] 0808 01:52:51.103828 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:51.104035 (+   207us) service_pool.cc:224] Handling call
0808 01:52:51.104518 (+   483us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:51.239269 (+134751us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:52.219112 (+979843us) spinlock_profiling.cc:243] Waited 58.2 ms on lock 0x7b5400512cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:52.219359 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":133759872}
I20260808 01:52:52.216549  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:52.214088 17453 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:52.229405  9476 raft_consensus.cc:2393] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 20: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:52.233022 17409 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:52.234359 17453 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:52.239091 17453 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:52.239485  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:52:52.231734 17394 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:52.232007 17451 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:52.236857  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:52.246928  9481 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:52:52.225091 17129 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:52.234184 17265 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:52:52.239552 17403 raft_consensus.cc:3055] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:52:52.247931 17394 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:52.256460 17240 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:52.259872 17428 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:52:52.261543 17453 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:52.263789  9524 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:52.247682 17451 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:52:52.266522 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.267539 17133 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:52.280457 17311 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:52.301471 17311 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:52.301764 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:52.270650  9552 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 11. (term mismatch)
W20260808 01:52:52.270157  9555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3732) took 1933 ms (client timeout 2036 ms). Trace:
I20260808 01:52:52.302875  9552 pending_rounds.cc:85] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 10
I20260808 01:52:52.303295  9552 pending_rounds.cc:107] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 19.11
W20260808 01:52:52.303004  9555 rpcz_store.cc:269] 0808 01:52:50.336695 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:50.489175 (+152480us) service_pool.cc:224] Handling call
0808 01:52:52.269884 (+1780709us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b540016a3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:52.270085 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2866622208}
I20260808 01:52:52.271199  9477 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260808 01:52:52.303881  9552 raft_consensus.cc:2889] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:52.304785  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:52.305147 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:52.305886  9555 raft_consensus.cc:2410] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 21 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260808 01:52:52.305764 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:52.306458 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:52.278210  9599 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:52.306969  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:52.279774  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.272703  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 1075) took 2354 ms. Trace:
I20260808 01:52:52.308506  9555 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 20: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:52.308203  9551 rpcz_store.cc:276] 0808 01:52:49.918304 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:49.919141 (+   837us) service_pool.cc:224] Handling call
0808 01:52:49.919391 (+   250us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:51.634611 (+1715220us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:52.272326 (+637715us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b54000ff854. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:52.272620 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3840768768}
I20260808 01:52:52.308964 17453 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:52.310114  9551 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: []
I20260808 01:52:52.304850 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:52.305712 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:52.279505  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.234561 17432 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:52.280320 17477 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:52.333376  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:52.282879 17483 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:52.306568 10363 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
W20260808 01:52:52.313190  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 49557.2us
I20260808 01:52:52.341714  9629 raft_consensus.cc:2368] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21.
I20260808 01:52:52.321604  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3739) took 1969 ms. Trace:
I20260808 01:52:52.269088 17129 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:52.347399 10363 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24. Ops: []
I20260808 01:52:52.310050  9555 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260808 01:52:52.315402 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:52.310837 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:52.318151 17358 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:52.295567 17403 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:52.319901 17432 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:52.321631  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.321882 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:52.322573 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:52.325124 17134 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:52.381383 17134 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:52.382405 17134 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:52.382982 17370 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:52.383744 17134 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:52.388895  9558 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:52.390228 17406 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:52.303890 17311 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:52.326948 17259 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:52.392225  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.392107 17259 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:52.393159  9632 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:52.327088 16808 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:52.281751 17481 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:52.281114 17346 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:52.395862 17239 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:52.395722 16808 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:52.395735 17481 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:52.401309 17481 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:52.330713  9476 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:52.402822 17259 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:52.331070  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.330535  9554 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:52.280220 17501 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:52.412611  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.414120  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3751) took 1944 ms. Trace:
I20260808 01:52:52.414840  9552 rpcz_store.cc:276] 0808 01:52:50.469752 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:51.887287 (+1417535us) service_pool.cc:224] Handling call
0808 01:52:52.270085 (+382798us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:52.304516 (+ 34431us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:52.413907 (+109391us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b540016a3d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:52.414053 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":848747904}
I20260808 01:52:52.414431  9634 raft_consensus.cc:2410] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 26 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260808 01:52:52.342648  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.313732  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:52.333765  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.344225 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:52.334393  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.342319  9559 rpcz_store.cc:276] 0808 01:52:50.352777 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:51.255583 (+902806us) service_pool.cc:224] Handling call
0808 01:52:51.256029 (+   446us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:51.451805 (+195776us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:52.321329 (+869524us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b540002cbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:52.321520 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":447912192}
I20260808 01:52:52.282498 17308 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:52.314968  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:52.351296 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.318708 10366 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: []
I20260808 01:52:52.356254  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:60546 (request call id 31) took 1885 ms. Trace:
W20260808 01:52:52.430218  9558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2849) took 2155 ms (client timeout 1642 ms). Trace:
I20260808 01:52:52.356849  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:52.358242 17472 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 1.127s	user 0.001s	sys 0.003s
I20260808 01:52:52.353171 17358 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:52.359761 17242 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:52.360708  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:52.361497  9521 leader_election.cc:336] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.668s (SENT)
I20260808 01:52:52.367628  9551 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260808 01:52:52.374707  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:52.365026 17129 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:52.390882 17406 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:52.399443 17457 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 22 current term: 22 is higher.
I20260808 01:52:52.326586 17473 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:52.331182  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:52.394481  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:52.410367  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.426879  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.429678  9523 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:52.457845  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.359658  9477 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:52.458918  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:52.369705  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 121290us
I20260808 01:52:52.459981  9445 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:52.439374  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 77918.3us
I20260808 01:52:52.464864 10366 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
W20260808 01:52:52.465830  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:52.467839 17285 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 19 current term: 19 is higher.
I20260808 01:52:52.468557  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.468675 17432 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260808 01:52:52.435462 17337 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:52.430745  9558 rpcz_store.cc:269] 0808 01:52:50.264692 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:50.264844 (+   152us) service_pool.cc:224] Handling call
0808 01:52:52.419401 (+2154557us) spinlock_profiling.cc:243] Waited 2 s on lock 0x7b5400382a54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:52.419562 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4604096384}
I20260808 01:52:52.445636  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:52.511567  9478 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:52.512022  9633 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:52.360960 17288 consensus_queue.cc:1059] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.082s
I20260808 01:52:52.517365  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.518409  9637 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:52.461300  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:52.433697  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:52.519330  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 58114.6us
I20260808 01:52:52.521917  9523 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:52.447126  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 90354.5us
I20260808 01:52:52.351363 17477 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:52.430526  9557 rpcz_store.cc:276] 0808 01:52:50.475326 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:52.355123 (+1879797us) service_pool.cc:224] Handling call
0808 01:52:52.356183 (+  1060us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:52.470244  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.353468 17483 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:52.535454  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.474951 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:52.478858  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.486816 17409 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:52.545908  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:52.546969  9474 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:52.442925 17358 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:52.440100  9556 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:52:52.493060 10362 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:52.456202 17129 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:52.508167 17403 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:52.353118 17428 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.085s
W20260808 01:52:52.436612  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 122931us
I20260808 01:52:52.426190 17501 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:52.437201 17422 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:52.471789 16808 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:52.536723  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.584847  9629 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
W20260808 01:52:52.585166  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:52.585785  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.529392 17285 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:52.537685  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.455049 17447 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:52.604264  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:52.455528 17493 raft_consensus.cc:3055] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:52.541836  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:52.616254  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 74506.7us
I20260808 01:52:52.548332  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:52.619204  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.619870  9475 raft_consensus.cc:2393] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 15: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:52.546958  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:52.620213  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.620517  9634 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:52.550043  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:52.618139  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:52.621851  9477 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:52:52.551148  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:52.555935  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.565512  9599 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:52.398502 17239 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:52.518618 16939 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:52:52.573819  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:52.607969  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:52.578387 17403 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:52:52.583007 17337 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260808 01:52:52.661095 17337 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:52.661561 17337 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:52.662870 17398 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:52.662694 17337 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.17, Last appended by leader: 17, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:52.663388 17398 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:52.663825 17398 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:52.664147 17337 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:52.668681 17483 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:52.530094 16983 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:52:52.681448 16983 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:52.531454 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:52.682456 10366 raft_consensus.cc:2393] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 15: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:52.528458 17133 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:52.588490  9635 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:52.439535 17308 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:52.611876  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:52.599979 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:52.623867  9631 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:52.688680  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:52.535095 17394 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:52.689678  9637 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:52.689896  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.690501  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.690989  9479 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:52.691609  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.624332  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:52.625474  9636 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
W20260808 01:52:52.626157  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 60753.3us
W20260808 01:52:52.693351  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 69109.3us
I20260808 01:52:52.694532  9599 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:52.606483  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:52.529553 17240 consensus_queue.cc:1059] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.074s
I20260808 01:52:52.639546  9550 raft_consensus.cc:2410] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260808 01:52:52.641319 17409 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:52.644157 17295 raft_consensus.cc:3055] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:52:52.614905 17285 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:52.615617 17493 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:52.648638 17129 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:52.584481 17457 consensus_queue.cc:1175] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "259bcb3f0d254e8b9effac3277e654e0" responder_term: 16 status { last_received { term: 14 index: 13 } last_committed_idx: 13 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: [15.14-15.14]" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 01:52:52.538230 17265 consensus_queue.cc:1059] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.077s
I20260808 01:52:52.651041 17450 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:52:52.570469 17479 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.086s
I20260808 01:52:52.648691 17251 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:52.657081 17319 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:52.657958 17406 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:52.669996  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.583271 17371 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:52:52.677778  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:52.701665  9634 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
W20260808 01:52:52.688563 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 88753.8us
I20260808 01:52:52.706591 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:52.706840  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:52.709307  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:52.710035  9479 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260808 01:52:52.626667 16939 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:52.708436  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:52.712747  9633 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:52.676496 17422 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260808 01:52:52.717252  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.717418  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:52.712803  9474 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:52.781922  9474 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22. Ops: []
I20260808 01:52:52.707654  9629 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:52:52.686175 16983 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:52:52.724946  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:52.731490 10368 raft_consensus.cc:2393] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 21: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:52.708480 17443 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:52.711760 17469 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:52.676177 17480 consensus_queue.cc:1059] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.046s
I20260808 01:52:52.714783 17158 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:52.820632  9474 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:52.820798  9556 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260808 01:52:52.821738  9445 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:52.714381 17483 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:52:52.711673 17295 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:52.754153  9635 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:52.760686  9637 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:52:52.765092  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.826211  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:52.827540  9474 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:52.775326  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.828037  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.828799  9634 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:52:52.829804  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.829972  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:52.782882  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 76129us
I20260808 01:52:52.836889  9599 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:52.783895  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 106227us
I20260808 01:52:52.837280  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:52.838276  9634 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:52.839273  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:52.841104  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:52.715127 17436 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.050s
I20260808 01:52:52.844427 17311 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260808 01:52:52.725674 17475 consensus_queue.cc:1175] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" responder_term: 16 status { last_received { term: 15 index: 14 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 01:52:52.787691  9638 raft_consensus.cc:2376] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16. Re-sending same reply.
I20260808 01:52:52.845613 17311 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:52.795218 10360 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260808 01:52:52.780895 16939 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:52.787585 16835 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260808 01:52:52.770551  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 61319.6us
I20260808 01:52:52.844533  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:52.856194  9474 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:52.788674  9480 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260808 01:52:52.780320 17422 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:52.825508  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:52.846211  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.847667  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.773725 10367 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:52.853165  9599 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:52.856002  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:52.863636  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:52.765650 17251 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:52.791718 17288 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:52.831472 17450 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:52:52.869261  9599 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:52.781031 17461 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:52:52.872310  9638 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 22: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:52.868713  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:52.878255  9475 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 23
I20260808 01:52:52.880542 17358 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:52.883283 17522 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260808 01:52:52.847738 17158 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:52.847340 17469 raft_consensus.cc:697] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:52.862191 17483 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:52:52.884356  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.873674 17495 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260808 01:52:52.900079  9638 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260808 01:52:52.901695  9638 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:52.870605 17422 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:52.876500 17481 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:52.906651 16939 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:52.908003  9476 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: []
I20260808 01:52:52.911079  9633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:52.912616  9475 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
W20260808 01:52:52.867316  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 401026us
I20260808 01:52:52.875195  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:52.915213 17308 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:52.919591 17285 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:52.920480  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:52.909170 16939 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:52.922580  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 50379.7us
W20260808 01:52:52.924018  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 54872.7us
I20260808 01:52:52.925817  9638 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260808 01:52:52.927603  9638 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260808 01:52:52.929929  9634 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:52:52.934657  9599 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:52.940055  9634 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:52.942443  9634 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: [20.10-20.10]
I20260808 01:52:52.943768  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:52.944536  9634 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260808 01:52:52.947391  9634 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:52.850301 17443 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:52.937505 17246 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:52:52.929471 17368 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:52.933033 17450 raft_consensus.cc:697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:52.940397 17358 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:52.941356 17447 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:52.910540 17398 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:52.886346 17158 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:52.990164  9478 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:52.997748  9629 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
W20260808 01:52:52.999805  9444 connection.cc:256] RPC call timeout handler was delayed by -1.2237670010072179s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1642s for a response
W20260808 01:52:53.000242  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1247670009906869s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17520000000000002s for a response
W20260808 01:52:53.000520  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0959670009906404s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1784s for a response
W20260808 01:52:53.001605  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:53.001401  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:53.008005  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:52.944675 16835 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.032s
I20260808 01:52:52.942286 17483 consensus_queue.cc:1059] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260808 01:52:52.968721 17246 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:52.888895 17295 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:52.968032 17461 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:52.888371 17469 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:53.009953  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:52.999047 17368 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:53.010386 17308 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260808 01:52:53.046337  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 220926us
I20260808 01:52:53.049417  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:53.019549 17259 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260808 01:52:53.020191 17447 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
W20260808 01:52:53.052021  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:53.059916  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 116247us
I20260808 01:52:53.052008  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:53.060725  9599 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:53.060873  9472 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:53.061460  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:53.060901  9524 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:53.053465  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:52:53.055083  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 214115us
I20260808 01:52:53.056325  9478 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
W20260808 01:52:53.058285  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:53.026785 17246 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:53.063252  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:53.063772  9555 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260808 01:52:53.063303  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:53.066239  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:53.066483  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:53.035351 17295 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:52:53.046337 17432 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:53.074280  9560 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:53.074533  9554 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:53.102836 17450 consensus_queue.cc:237] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:53.103997  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:53.105731  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:53.106478 17461 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 21.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.106874 17370 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:52:53.107996 17370 consensus_queue.cc:1243] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 16.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260808 01:52:53.108820 17285 raft_consensus.cc:3055] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:53.109313 17285 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:53.077265  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:53.077428  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:53.079064 17398 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:53.080058 17529 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260808 01:52:53.084393  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:53.087252 16835 raft_consensus.cc:3055] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:53.057614 17358 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:53.092211 17453 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:53.092876  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:53.135072  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:53.062796  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:53.093955 17368 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:53.110875  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47122.7us
I20260808 01:52:53.057895 17409 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260808 01:52:53.114956  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 54127us
I20260808 01:52:53.114358  9560 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:52:53.142660  9557 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260808 01:52:53.106542  9554 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:53.120436  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:53.122864  9552 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260808 01:52:53.123019  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:53.120128 16835 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:53.131932  9555 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 21. Ops: []
I20260808 01:52:53.131811  9521 leader_election.cc:400] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21.
I20260808 01:52:53.147284  9521 leader_election.cc:403] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
W20260808 01:52:53.138799  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:53.070757 17436 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:53.069255 17403 raft_consensus.cc:2764] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
W20260808 01:52:53.148408  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:53.136332 17451 consensus_meta.cc:287] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.791s	user 0.007s	sys 0.005s
I20260808 01:52:53.131886 17295 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:53.059010 16939 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:52:53.115310 17308 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:53.142238  9524 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:53.023857 17251 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.171366  9556 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260808 01:52:53.082482 16983 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.146286 17134 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:53.150312 17242 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:53.223976  9550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3738) took 2875 ms (client timeout 1859 ms). Trace:
W20260808 01:52:53.224475  9550 rpcz_store.cc:269] 0808 01:52:50.348866 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:50.465006 (+116140us) service_pool.cc:224] Handling call
0808 01:52:53.223731 (+2758725us) spinlock_profiling.cc:243] Waited 2.09 s on lock 0x7b5400313dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:53.223892 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4797581056}
I20260808 01:52:53.149009 17409 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:53.146075 17265 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:53.161222 17428 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:53.122298 17259 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.060s
I20260808 01:52:53.132995 17358 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.226971  9472 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:53.140863 17483 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.046s
I20260808 01:52:53.184219 17308 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:53.173413 17436 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.133020 17285 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.228614 17409 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:52:53.220100 17533 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:53.183277 17451 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.232393 17134 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:53.238999  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 97741.1us
I20260808 01:52:53.185078 17422 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.243351 17308 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:53.231528 17242 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:53.243512 17311 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:53.249109 17259 consensus_queue.cc:1243] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 20.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260808 01:52:53.231535 17265 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.249661 17531 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:53.259984 17398 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:53.263751 17285 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:53.288286 17543 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:52:53.278505 17242 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:53.289706 17422 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:53.293303 17477 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:53.237522 16835 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.284335 17398 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:53.294571 17323 raft_consensus.cc:3055] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Stepping down as leader of term 16
W20260808 01:52:53.300402 17473 consensus_meta.cc:287] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.782s	user 0.013s	sys 0.000s
I20260808 01:52:53.318671 17372 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:52:53.319233 17372 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:53.320142 17372 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.323072 17443 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:53.333828 17436 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:53.311175 17398 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:53.300117 17481 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:53.313938 17323 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:53.339178 17323 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:53.339838 17239 raft_consensus.cc:3055] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:53.340250 17239 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:53.340601 17323 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:53.341096 17239 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.341640 17473 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:53.342209 17239 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:53.346556 17319 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:53.347405 17358 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:53.352370 17288 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:53.353868 17394 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:53.355811 17537 log.cc:927] Time spent T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Append to log took a long time: real 0.061s	user 0.002s	sys 0.000s
I20260808 01:52:53.339180 17493 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:53.355829 17501 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:53.357580 17493 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:53.332130 17451 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:53.328128 17528 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.292s	user 0.004s	sys 0.000s
I20260808 01:52:53.354514 17265 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:53.369186 17447 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:53.372594 17453 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260808 01:52:53.374676 17428 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21.
I20260808 01:52:53.372141 17308 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.372192 17246 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.378793 17246 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:53.381187 17308 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:53.385872 17409 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.386631 17398 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:53.445708  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:53.456995  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:53.451390  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 309121us
I20260808 01:52:53.457373  9629 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17. Ops: []
I20260808 01:52:53.458711  9524 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:53.459734  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3198) took 1130 ms. Trace:
I20260808 01:52:53.461125  9524 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:53.453305  9636 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
W20260808 01:52:53.454550  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:53.455214  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 306129us
W20260808 01:52:53.447413  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 336723us
W20260808 01:52:53.301069 17133 consensus_meta.cc:287] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.610s	user 0.014s	sys 0.000s
I20260808 01:52:53.471508 17133 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:53.472791 17453 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:52:53.473732 17453 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:53.479281 16835 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:53.479815 17540 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:53.481671 16983 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:53.482165 17398 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:53.489761 17370 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:53.494266 17323 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:53.495025 17447 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:53.495491 17323 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:53.496497 17447 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.497522 10367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 955) took 1196 ms. Trace:
I20260808 01:52:53.500063 17323 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:53.460417  9630 rpcz_store.cc:276] 0808 01:52:52.328645 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:52.331543 (+  2898us) service_pool.cc:224] Handling call
0808 01:52:52.331996 (+   453us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:52.332424 (+   428us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:52.332466 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:52.332947 (+   481us) log.cc:844] Serialized 26 byte log entry
0808 01:52:52.871264 (+538317us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:52.871313 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:52.871386 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:53.327349 (+455963us) spinlock_profiling.cc:243] Waited 7.11 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:53.458705 (+131356us) raft_consensus.cc:1685] finished
0808 01:52:53.458756 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:53.459403 (+   647us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11760,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":20298,"spinlock_wait_cycles":16392448,"thread_start_us":428564,"threads_started":1,"wal-append.queue_time_us":614360}
I20260808 01:52:53.500274  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:53.482194 17428 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:53.502584 17428 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:53.502146 17439 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:53.503178 17428 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:53.503968 10368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 959) took 1199 ms. Trace:
I20260808 01:52:53.504545 10368 rpcz_store.cc:276] 0808 01:52:52.304125 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:52.306220 (+  2095us) service_pool.cc:224] Handling call
0808 01:52:53.503349 (+1197129us) spinlock_profiling.cc:243] Waited 421 ms on lock 0x7b5400553054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:53.503722 (+   373us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":968275200}
I20260808 01:52:53.505177 17439 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election won for term 19
W20260808 01:52:53.505820  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:53.512606 17409 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:53.512945  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:53.514356  9472 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
W20260808 01:52:53.514457  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:53.495155 17129 raft_consensus.cc:3055] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Stepping down as leader of term 22
W20260808 01:52:53.517409  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:53.471915  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:53.487808 17540 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.491168 17370 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.527182  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3223) took 1059 ms. Trace:
I20260808 01:52:53.528712 17370 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:53.527721  9638 rpcz_store.cc:276] 0808 01:52:52.467617 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:52.939246 (+471629us) service_pool.cc:224] Handling call
0808 01:52:52.939571 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:52.939798 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 9
0808 01:52:53.187080 (+247282us) log.cc:844] Serialized 14 byte log entry
0808 01:52:53.526384 (+339304us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:53.526505 (+   121us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:53.526609 (+   104us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:53.526743 (+   134us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:53.526960 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29469,"prepare.queue_time_us":341261,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"spinlock_wait_cycles":1059584,"thread_start_us":538835,"threads_started":2,"wal-append.queue_time_us":202385}
I20260808 01:52:53.498106 10367 rpcz_store.cc:276] 0808 01:52:52.300528 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:52.300774 (+   246us) service_pool.cc:224] Handling call
0808 01:52:53.496874 (+1196100us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b5400594cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:53.497174 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":933901184}
W20260808 01:52:53.533609  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 410544us
I20260808 01:52:53.516660 17129 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:53.515383 17439 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:53.459811 17134 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.547991  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3204) took 1212 ms. Trace:
I20260808 01:52:53.548663  9636 rpcz_store.cc:276] 0808 01:52:52.335378 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:52.866317 (+530939us) service_pool.cc:224] Handling call
0808 01:52:53.547484 (+681167us) spinlock_profiling.cc:243] Waited 556 ms on lock 0x7b54000dfa54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:53.547847 (+   363us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1278337920}
I20260808 01:52:53.554164 17372 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:52:53.555356 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:53.556205 17129 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:53.558314 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:53.574240 10368 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:53.577251  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:53.575066 17556 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:53.578632  9550 raft_consensus.cc:2393] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 21: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:53.555388  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:53.559760  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 958) took 1069 ms. Trace:
W20260808 01:52:53.566790 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:53.568012  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:53.581918 17556 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:53.578917  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:53.586666  9558 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:53.587121  9554 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
I20260808 01:52:53.587277 10367 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24. Ops: []
I20260808 01:52:53.588444  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3209) took 1238 ms. Trace:
W20260808 01:52:53.592551  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:53.580571  9629 rpcz_store.cc:276] 0808 01:52:52.490361 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:53.000613 (+510252us) service_pool.cc:224] Handling call
0808 01:52:53.000942 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:53.458155 (+457213us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:53.559267 (+101112us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b54004204d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:53.559666 (+   399us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1047750016}
W20260808 01:52:53.594614  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:53.595752  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 70037us
I20260808 01:52:53.597702  9524 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:53.599345  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 959) took 1108 ms. Trace:
W20260808 01:52:53.602298  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 280770us
I20260808 01:52:53.602633  9634 rpcz_store.cc:276] 0808 01:52:52.490645 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:53.457348 (+966703us) service_pool.cc:224] Handling call
0808 01:52:53.540397 (+ 83049us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:53.583765 (+ 43368us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:53.583801 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:53.598797 (+ 14996us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:52:53.598850 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:53.598942 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:53.599095 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:53.605749  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:53.601676 17285 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:53.600343 17453 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:53.616317  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:53.588948  9635 rpcz_store.cc:276] 0808 01:52:52.341660 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:52.360378 (+ 18718us) service_pool.cc:224] Handling call
0808 01:52:53.574416 (+1214038us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b54003faa54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:53.580125 (+  5709us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":343930624}
I20260808 01:52:53.579520  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:53.598389 17540 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:53.619796  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 43738.2us
I20260808 01:52:53.595925  9599 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:53.629307  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:53.600242  9633 raft_consensus.cc:606] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:53.630285  9633 raft_consensus.cc:614] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:53.631374 17133 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:52:53.631925  9637 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: [18.12-20.14]
I20260808 01:52:53.632838  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3211) took 1290 ms. Trace:
I20260808 01:52:53.633340  9637 rpcz_store.cc:276] 0808 01:52:52.342169 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:52.915818 (+573649us) service_pool.cc:224] Handling call
0808 01:52:53.582253 (+666435us) spinlock_profiling.cc:243] Waited 8.85 ms on lock 0x7b5800231698. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:52:53.584012 (+  1759us) raft_consensus.cc:1471] Updating replica for 3 ops
0808 01:52:53.632437 (+ 48425us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:53.632766 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20345600}
I20260808 01:52:53.635030 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:53.635748 10368 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: []
I20260808 01:52:53.636400 10367 raft_consensus.cc:2393] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 21: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:53.638025 17129 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:53.625787 17285 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:53.640588 17398 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:53.631747  9478 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 14.
W20260808 01:52:53.638203  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 166608us
I20260808 01:52:53.641501 17398 raft_consensus.cc:697] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:53.649220 17447 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:53.650071  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:53.611410 17439 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:53.659045  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 159166us
I20260808 01:52:53.667260  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34802 (request call id 98) took 1258 ms. Trace:
W20260808 01:52:53.690629  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 95092.1us
I20260808 01:52:53.692087  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3408) took 1106 ms. Trace:
I20260808 01:52:53.691219 16808 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:53.677918 17323 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260808 01:52:53.679631  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:53.719552  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:53.680160 10334 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:53.683477 17285 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:53.688740  9521 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:53.653309 16983 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:52:53.659742 17533 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.031s
I20260808 01:52:53.691207  9633 rpcz_store.cc:276] 0808 01:52:52.388433 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:52.874327 (+485894us) service_pool.cc:224] Handling call
0808 01:52:53.599690 (+725363us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b5800231698. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d499a1a50 00007f5d4998d9c9 00007f5d4998d28c 00007f5d42ee1934
0808 01:52:53.646606 (+ 46916us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":30793088}
I20260808 01:52:53.692727  9477 rpcz_store.cc:276] 0808 01:52:52.585975 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:52.685298 (+ 99323us) service_pool.cc:224] Handling call
0808 01:52:52.685697 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:52.686101 (+   404us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:52.686137 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:52.715054 (+ 28917us) log.cc:844] Serialized 26 byte log entry
0808 01:52:53.510549 (+795495us) spinlock_profiling.cc:243] Waited 4.12 ms on lock 0x7b18003f6d60. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:52:53.574530 (+ 63981us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:53.574609 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:53.574747 (+   138us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:53.691422 (+116675us) raft_consensus.cc:1685] finished
0808 01:52:53.691455 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:53.691793 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5494,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":931,"spinlock_wait_cycles":9479936,"thread_start_us":25330,"threads_started":1,"wal-append.queue_time_us":936647}
W20260808 01:52:53.697288  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 91861.7us
I20260808 01:52:53.706457  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:53.707633  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:53.707789  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:53.709651  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:53.710152  9481 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:53.710907  9635 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: []
I20260808 01:52:53.678009  9523 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:53.712743  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:53.713544 17134 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:53.714298  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:53.715499 17556 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:53.715703  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:53.691293 17543 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:53.690902  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:53.667815 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:53.743046  9473 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:53.739656  9636 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:53.747059  9479 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:53.747290 10367 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:53.748716 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:53.749822 10360 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 24: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:53.750953  9475 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:53.753644 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:53.755227  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:53.759423  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:53.760210 10362 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260808 01:52:53.742264 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:53.760429  9474 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:52:53.761932 10368 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 23: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:52:53.762113  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 3b8e0e8fda6744ad877514b11658b9d9 (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:52:53.765545 17447 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260808 01:52:53.765933 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 85895us
I20260808 01:52:53.766031 17323 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260808 01:52:53.766741 17323 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:53.767252 17323 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:53.767293  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:53.768391  9481 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
W20260808 01:52:53.777736  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 99996.8us
I20260808 01:52:53.649777 17398 consensus_queue.cc:237] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.768440 17134 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:53.759459 17285 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:53.755229 17548 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:52:53.770058  9599 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:53.770892 17239 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:53.773594  9636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "439040d168824496acc076b1c280bde3"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:53.761409  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:53.774732  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:53.760349  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 69715.4us
I20260808 01:52:53.779138  9472 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
W20260808 01:52:53.786564  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:53.790257 17239 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.791843 17533 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:53.792268 17285 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.792573 17556 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:52:53.793176 16808 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 19.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260808 01:52:53.802770  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:53.795186 17549 raft_consensus.cc:3055] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:53.806350 17549 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:53.806809 16983 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:53.807199 17239 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:53.807693 16983 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:53.809473  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 121038us
I20260808 01:52:53.809265 16983 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:53.811815  9521 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:53.815179 17547 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:53.816998 17134 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:53.817129  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:53.817505  9524 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:53.818125  9481 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
W20260808 01:52:53.818236  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 98062.2us
I20260808 01:52:53.819213 17532 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:53.819949 17532 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:53.821108 17457 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:53.821187 17532 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.823552 17547 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:53.823714  9524 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:53.824236 17547 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:53.824769 17547 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:53.825299 10334 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:53.825661 17548 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:53.826244 17548 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:53.832311 17493 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:53.833081 17493 raft_consensus.cc:697] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:53.807888 17549 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.9, Last appended by leader: 9, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.835225  9445 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:53.836352 17549 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:53.839272  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:53.839634 17422 consensus_queue.cc:1175] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" responder_term: 20 status { last_received { term: 19 index: 9 } last_committed_idx: 9 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 01:52:53.826663 17548 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:53.842937 17443 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:53.843776 17443 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.844283  9629 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:53.841773  9633 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260808 01:52:53.800690  9636 raft_consensus.cc:606] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:53.794541 17533 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:52:53.852845  9601 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:53.853627  9635 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:53.854734  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:53.856774  9481 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
W20260808 01:52:53.876572  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 47820.8us
I20260808 01:52:53.872109  9524 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:53.875957 17457 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260808 01:52:53.883096 17358 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:53.883965 17358 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.878929 17439 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:53.890453 17439 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.891770  9523 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:53.834337 17493 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:53.855844 17533 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:53.868050 17285 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:53.870175 17556 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.032s
I20260808 01:52:53.905964  9634 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: [14.9-17.10]
I20260808 01:52:53.906999 17439 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:53.908097  9638 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:53.908255  9636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:34802
W20260808 01:52:53.910434  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 95722.9us
I20260808 01:52:53.912000  9636 raft_consensus.cc:606] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:53.915304  9636 raft_consensus.cc:614] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:52:53.920289 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 78792.3us
W20260808 01:52:53.923395  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 71568.2us
W20260808 01:52:53.926213  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 76596.9us
W20260808 01:52:53.937852  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 136585us
I20260808 01:52:53.937295 10368 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 21 index: 17. (term mismatch)
I20260808 01:52:53.942879 10368 pending_rounds.cc:85] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting all ops after (but not including) 16
I20260808 01:52:53.926092  9473 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:52:53.945408 10368 pending_rounds.cc:107] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting uncommitted NO_OP operation due to leader change: 20.17
I20260808 01:52:53.945886 10368 raft_consensus.cc:2889] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:53.939018  9444 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:53.955008  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:53.961318  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet c85ff498df484fc6ac0913aa57aa868b (from 259bcb3f0d254e8b9effac3277e654e0 to 6e7856bed1be43acaa821290d506ee97) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
W20260808 01:52:53.963276  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:53.969812 17428 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:53.959990 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:53.989120  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:53.941084 17358 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:53.994930  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:53.962405 17453 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:52:53.962083 17133 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:53.994288  9474 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:52:53.988094 17428 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:53.960340 17443 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:54.020879 17239 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:52:54.016018 17133 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:54.034902  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:54.023869 10362 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
W20260808 01:52:54.044157  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:54.013995 17453 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:54.033767 10332 leader_election.cc:336] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.780s (SENT)
I20260808 01:52:54.014300 17422 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:54.029876 17133 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:54.034698 17372 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:52:53.991099 17540 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.045s
I20260808 01:52:54.058840 10332 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:53.996718 17556 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.054s
W20260808 01:52:54.069602  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:54.067524 16835 consensus_meta.cc:287] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.549s	user 0.009s	sys 0.011s
I20260808 01:52:54.070947 17394 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:54.076323  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:54.076483 17398 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:54.082700  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 138571us
I20260808 01:52:54.085940 17285 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:54.090714 17547 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:54.091408  9550 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
W20260808 01:52:54.091771  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 156070us
I20260808 01:52:54.050540 17453 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:54.026482 17543 consensus_queue.cc:1059] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.040s
I20260808 01:52:54.128782  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:54.108291  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 66225.8us
W20260808 01:52:54.112834  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:54.115077  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:54.116968 17398 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:54.119150 17285 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:54.105674 17422 raft_consensus.cc:697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:54.135573  9523 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:54.142319 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:54.145143  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 229718us
I20260808 01:52:54.160163  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:54.161525 17447 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:52:54.162525  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:54.111195 17323 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:54.168896  9524 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:54.162456 17447 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:52:54.164335  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 7d3a904863b545ffa227b964ee3a3f18: Illegal state: Replica 259bcb3f0d254e8b9effac3277e654e0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 18 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:52:54.184043  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 43842.1us
I20260808 01:52:54.106491 17533 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:54.186547  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:54.155853 17428 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:54.175025 17457 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:54.189527  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:54.139183 17394 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:54.181214 17443 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:52:54.200210 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:54.200464  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 126000us
I20260808 01:52:54.204998 17358 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:52:54.205806 17358 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:54.205053 10362 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:54.189383 17428 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:54.196568 17457 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:54.163050 17439 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.045s
W20260808 01:52:54.222541  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 60142.6us
W20260808 01:52:54.223383  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:54.223752  9600 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:54.232929 17346 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:52:54.232687 17422 consensus_queue.cc:237] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:54.232820 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 216626us
I20260808 01:52:54.238300 17295 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:54.213433 17239 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:52:54.216841 17428 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:54.243870  9558 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21. Ops: []
W20260808 01:52:54.237907 10332 leader_election.cc:336] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.739s (SENT)
I20260808 01:52:54.247970 17398 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:54.215464 17443 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:54.248430 10332 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:54.249784 17443 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:54.228399 17457 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:54.253794 17533 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:52:54.258389  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:54.258771 17547 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:54.259878 17285 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:54.259487 17574 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.094s	user 0.000s	sys 0.005s
I20260808 01:52:54.265180  9634 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23. Ops: []
I20260808 01:52:54.267655  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:54.268003  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:54.268778  9551 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:54.269083  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:54.269986  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:54.270490 17447 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:54.270983  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:54.272312  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:54.271289  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:54.274056  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:54.274173  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:54.275422  9560 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 18: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:54.279114  9556 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:54.279748  9559 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260808 01:52:54.279806  9480 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260808 01:52:54.281118  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:54.281595 17532 raft_consensus.cc:3055] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:54.282063 17532 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:54.279155  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:54.278352  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:54.283236 17532 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:54.284119  9474 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:54.285099 17532 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:54.286713  9559 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:54.287143  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:54.287247 17358 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:54.288004  9634 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:52:54.283522  9521 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:54.283267  9478 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:54.291147  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:54.291642 17129 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:52:54.239969 17556 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:54.293735 17556 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:54.294230 17556 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:54.294461  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:54.295310  9480 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:54.297443  9636 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:54.265321  9558 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260808 01:52:54.301229  9474 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
W20260808 01:52:54.301676  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:54.303450  9524 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:54.304945  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:54.305907 17543 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:52:54.306813 17543 consensus_queue.cc:1243] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 19.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260808 01:52:54.307670 16835 raft_consensus.cc:3055] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:54.308138 16835 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:54.309410 16835 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:54.311342  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:54.317036 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 72512.6us
I20260808 01:52:54.307119  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:54.311335  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:54.318060  9551 raft_consensus.cc:2368] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20.
I20260808 01:52:54.317328  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:54.311240  9558 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
W20260808 01:52:54.327119  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:54.221634 17251 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:54.341645  9474 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26. Ops: []
I20260808 01:52:54.339715  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:54.348963  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:54.349990  9481 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:54.352636  9636 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 21: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:54.353432  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:54.354532  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:54.346995  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:54.345949  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:54.365301  9560 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:52:54.373934  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:54.334244 17239 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:54.333745 16983 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:54.361517  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:54.384073  9558 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260808 01:52:54.244601 17295 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:54.387804  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 76612.8us
I20260808 01:52:54.388969  9629 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:52:54.395161  9559 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
W20260808 01:52:54.397606  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 73531.5us
I20260808 01:52:54.323478 17398 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.052s
I20260808 01:52:54.401978  9633 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260808 01:52:54.400914  9477 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:54.400985  9551 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:54.407940 17372 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:54.408784 17372 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:54.411628  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:54.382228 17549 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:54.383630 16983 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:54.418301  9637 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:54.415880  9635 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:52:54.363425 17409 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.037s
W20260808 01:52:54.429574 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 191742us
I20260808 01:52:54.431530  9560 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260808 01:52:54.427397  9630 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260808 01:52:54.437009  9636 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260808 01:52:54.438445  9478 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:52:54.439178  9474 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22. Ops: []
W20260808 01:52:54.444183  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:54.417146 17549 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:54.422822 16983 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:54.449658 17129 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:54.450738 17129 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:54.451526 17582 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:54.454643  9480 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:54.396919 17251 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:54.434803 17323 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:54.473943  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:54.477752 17583 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:54.478236  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:54.479092  9551 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:54.477829  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:54.461912 17394 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:54.489712 17548 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.647s	user 0.007s	sys 0.007s
I20260808 01:52:54.478628 17583 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:54.447575 17549 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:54.504068  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:54.506625  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:54.512948 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:54.517379 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:54.518311 10368 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 16.
I20260808 01:52:54.518903  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:54.523377  9521 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:54.526235 17589 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:54.526875 17394 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:54.527359 16835 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:54.527511 17394 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:54.505254 17548 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:54.545711 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 81538.6us
I20260808 01:52:54.547477 10332 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:54.550093  9524 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:54.552469  9554 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 22: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:52:54.553660  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:54.555855  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:54.557788 17584 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260808 01:52:54.559120  9524 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:54.560051 17398 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:54.560565 17398 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:54.560945 17398 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:54.563651 17295 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:54.564059  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 60259.7us
I20260808 01:52:54.565099 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:54.565039  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:54.569377  9551 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:54.543618 17548 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:54.573741  9550 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
W20260808 01:52:54.575058  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 177359us
I20260808 01:52:54.576229  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:54.576929  9472 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:54.583011  9559 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26. Ops: []
I20260808 01:52:54.585280  9478 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:52:54.584928  9445 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:54.581697  9524 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:54.463022 17428 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:54.570988 17493 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260808 01:52:54.568406 17285 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:54.595433  9599 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:54.576458 17547 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:52:54.605397  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:54.609414  9555 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 24. Current term is 25. Ops: []
I20260808 01:52:54.609313  9552 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 18. Current term is 19. Ops: []
I20260808 01:52:54.611790 17595 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:52:54.612323 17595 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:54.612696 17595 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:54.613595  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:54.614252  9477 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:54.618889  9556 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260808 01:52:54.620018  9473 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
W20260808 01:52:54.620661  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:54.628111  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:54.604295 17548 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:54.625000 10367 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
W20260808 01:52:54.629287  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 64379.1us
I20260808 01:52:54.603613 17129 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:54.549821 17239 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:54.624770  9524 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:54.606268 17439 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260808 01:52:54.637712 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:54.620898 17547 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:54.633729  9478 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260808 01:52:54.643471  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:54.636547 17543 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:54.627278 17295 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:54.734975  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 91593.1us
W20260808 01:52:54.716092  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 131119us
I20260808 01:52:54.675020  9477 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
W20260808 01:52:54.702831 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:54.722219 10368 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
W20260808 01:52:54.792667 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 170734us
I20260808 01:52:54.793052  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:54.793490  9472 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:52:54.651396 17548 raft_consensus.cc:697] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:54.665855 17134 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:54.795907 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:54.768256 17457 consensus_meta.cc:287] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.517s	user 0.016s	sys 0.000s
I20260808 01:52:54.684738 17323 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:54.675925 17439 raft_consensus.cc:2764] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260808 01:52:54.735854 17251 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:54.770062 17493 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:54.735406 17583 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:54.693936 17428 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:54.709231 17591 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:54.816442 17582 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:52:54.710927 17372 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:54.822024  9599 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:54.803540 17134 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:54.713370 17409 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.098s
W20260808 01:52:54.812284 17453 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.672s	user 0.015s	sys 0.000s
I20260808 01:52:54.823002 10334 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:54.830415  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:54.832612 10367 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d2b2a88358244f93970835ea3120c380"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:57156
I20260808 01:52:54.832729  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:54.829700 17582 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:54.813033 17457 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:54.839143 17596 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.199s	user 0.001s	sys 0.000s
I20260808 01:52:54.829502  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:54.853368  9559 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:52:54.855791 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:54.861027  9555 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 14 FOLLOWER]: Advancing to term 15
I20260808 01:52:54.859236  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:54.853178 10367 raft_consensus.cc:606] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:54.857189 17582 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:52:54.859524  9474 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:54.851162 17134 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:54.863351  9473 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
W20260808 01:52:54.863904  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:54.864315  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 54909.6us
W20260808 01:52:54.880815  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:54.862339  9554 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
W20260808 01:52:54.862591 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 43811.8us
I20260808 01:52:54.865661  9556 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
W20260808 01:52:54.860910 17533 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.606s	user 0.016s	sys 0.000s
I20260808 01:52:54.823654 17346 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.042s
W20260808 01:52:54.891669 17532 consensus_meta.cc:287] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.606s	user 0.014s	sys 0.002s
I20260808 01:52:54.892066  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:54.863859 16983 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:54.827630 17295 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:54.899737 17295 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:54.879525 17599 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260808 01:52:54.900916 16983 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:54.870532 17457 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:54.871050  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 7b5ea129398a42fb9981fca652027cd6: Illegal state: Replica 52ccf4dcca1843f7bb29e9172e2d4ab4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 22 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:52:54.856457 17453 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:54.847765 17493 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:54.930272 17493 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:54.872674 17548 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:54.873633 17398 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:54.875101 17593 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:54.875887  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 53971.2us
I20260808 01:52:54.877031 17591 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:54.694947 17547 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:54.884222 17133 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.776s	user 0.015s	sys 0.000s
I20260808 01:52:54.880643 17323 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader election won for term 15
I20260808 01:52:54.884851 17543 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:54.969394 17593 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:54.970320  9474 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 22. Current term is 23. Ops: [20.12-22.13]
I20260808 01:52:54.885829  9557 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260808 01:52:54.970966 17443 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:52:54.908795 17394 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:54.910023 17239 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:54.911826 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:54.914276  9478 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260808 01:52:54.918275 17453 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:54.918840  9555 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:52:54.922310  9551 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 23. Ops: []
I20260808 01:52:54.968703 17457 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.018s
I20260808 01:52:54.927346  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:54.988654  9481 raft_consensus.cc:2410] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260808 01:52:54.872471  9600 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:54.989151 17543 consensus_queue.cc:1059] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:54.989840  9599 leader_election.cc:400] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20.
I20260808 01:52:54.990247  9599 leader_election.cc:403] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
W20260808 01:52:54.990042  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 117684us
I20260808 01:52:54.990653 17543 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:54.918452 17134 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:54.907830  9524 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:54.936882  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:54.976616 17133 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:54.993036  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 85279.9us
I20260808 01:52:54.993220  9477 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:54.994493 17134 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:54.995340 17134 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:52:54.996191 17533 raft_consensus.cc:3055] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:54.996672 17533 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:54.997722 17533 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:55.001133 17556 consensus_meta.cc:287] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.706s	user 0.009s	sys 0.003s
I20260808 01:52:55.001796 17134 raft_consensus.cc:697] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:55.003966 17593 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:54.962859  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:54.965756  9554 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260808 01:52:55.004653 17134 consensus_queue.cc:237] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:55.006362 17532 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 20
I20260808 01:52:55.001844  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:55.007563 17285 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:55.008981 17372 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:55.009313  9481 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260808 01:52:55.009469 17372 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:55.010628 17372 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:55.011399 17533 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:55.012338 17372 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:54.969655 17323 raft_consensus.cc:697] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:54.979714  9630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "df6982ec010444a8854140f7b64e985d"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:34802
I20260808 01:52:54.957376 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:55.019161  9630 raft_consensus.cc:606] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:55.019613  9630 raft_consensus.cc:614] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:52:54.980825 17398 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:54.983167 17591 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:55.002233 17556 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:55.003232 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:54.992184  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:55.013559  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 44916.4us
I20260808 01:52:55.011507  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:52:54.983116 17605 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.105s	user 0.002s	sys 0.000s
I20260808 01:52:55.022095  9477 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:55.025400  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:55.038829  9481 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 22
W20260808 01:52:55.027367 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 71829.7us
W20260808 01:52:55.019686  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47167.2us
I20260808 01:52:54.992393 17543 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:52:55.034149  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:55.037125 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:55.042094 10367 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:52:55.039523 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 64991.5us
W20260808 01:52:55.042905  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:55.043236 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:55.052628  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:55.053455  9473 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 22
I20260808 01:52:55.056382  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:55.026252 17133 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:55.050872 17549 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.504s	user 0.011s	sys 0.002s
I20260808 01:52:55.032166 17583 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:55.064808 16835 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:55.068840 17295 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:55.071449  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:55.072237  9479 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260808 01:52:55.048086 17543 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:52:55.087311  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet df6982ec010444a8854140f7b64e985d (from 259bcb3f0d254e8b9effac3277e654e0 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
W20260808 01:52:55.088471  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:55.024041 17453 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.047s
W20260808 01:52:55.103128  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 45670.6us
I20260808 01:52:55.098094  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 01:52:55.059423 17394 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.037s
I20260808 01:52:55.093286 17583 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:55.115114  9524 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:55.104234 17543 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:52:55.117152  9478 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:52:55.117700  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:55.120954  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:55.115258 17285 consensus_queue.cc:1059] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.015s
I20260808 01:52:55.112717 17295 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:55.133509  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:52:55.134279 17323 consensus_queue.cc:237] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:55.094624 17549 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:55.146313  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 72032us
I20260808 01:52:55.123696 17583 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:55.145769  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:52:55.087036 17592 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.026s
I20260808 01:52:55.126919 17548 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:55.117089 17239 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.030s
I20260808 01:52:55.115433 17409 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.030s
W20260808 01:52:55.172032  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:55.127905 17593 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.042s
I20260808 01:52:55.183472  9481 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
W20260808 01:52:55.186398  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 73087.2us
W20260808 01:52:55.195461  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 51974.5us
I20260808 01:52:55.161237 17457 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:55.223070 17469 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.012s
W20260808 01:52:55.213450 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46169.6us
I20260808 01:52:55.224759  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:55.214661 17618 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:55.229710  9478 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260808 01:52:55.255901  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:55.178524 17548 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:55.260422 17548 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:55.264195 17549 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:55.265767 17549 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:55.268887 17533 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:52:55.269577 17533 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:55.280403 17556 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:55.288663 17595 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.675s	user 0.011s	sys 0.000s
I20260808 01:52:55.289467 17595 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:55.292327 17295 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:55.293690 17457 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:55.294577 17457 raft_consensus.cc:697] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:55.296195 17457 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:55.161303 17582 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:55.300714 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:55.304114  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:55.304807  9550 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:52:55.314903  9550 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 24.
I20260808 01:52:55.316725 10332 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:55.318435 17616 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:52:55.319190 17616 raft_consensus.cc:697] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:55.319248  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:55.320669 17616 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:55.335074  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 79303.9us
I20260808 01:52:55.216509  9524 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:55.355859  9473 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:52:55.389599  9479 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 24. Current term is 25. Ops: [24.14-24.14]
W20260808 01:52:55.392884  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 202885us
I20260808 01:52:55.396026 10367 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:52:55.398118  9523 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:55.413779 10334 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:55.389716 17439 raft_consensus.cc:3055] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:55.391628 17582 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:55.428872  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 215617us
I20260808 01:52:55.432144  9524 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:55.453755  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:55.416862 17358 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:52:55.417148 17409 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:52:55.454560  9479 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
W20260808 01:52:55.460371  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 312944us
I20260808 01:52:55.323657 17533 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:55.497018  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:55.394721 17549 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:55.434777 17439 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:55.459085 16983 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:52:55.501521 17394 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:52:55.507874 17358 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:55.550387 17618 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:55.510419 17409 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:55.585347  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 68226.9us
I20260808 01:52:55.569036 17319 raft_consensus.cc:3055] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:55.602339 17319 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:55.603361 17319 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:55.603981 17556 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:55.604821 17319 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:55.559242 17623 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:55.606244 17623 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:55.607684 17615 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:55.608498 17615 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:55.609294 17623 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:55.609994 17623 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:55.612573 17422 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:52:55.613565 17422 consensus_queue.cc:1243] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 26.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260808 01:52:55.615480 17439 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:55.617229 17439 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:55.555049 16983 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:55.626526 10332 leader_election.cc:336] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.692s (SENT)
I20260808 01:52:55.627223 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:52:55.628285 17617 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
W20260808 01:52:55.631515 17624 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
W20260808 01:52:55.634244 17372 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.621s	user 0.013s	sys 0.001s
W20260808 01:52:55.635258 17610 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.519s	user 0.000s	sys 0.003s
I20260808 01:52:55.635469  9476 raft_consensus.cc:2410] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 20 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 15.
I20260808 01:52:55.639766 17285 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:55.640556 17285 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:55.641391 17533 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:55.604560 17556 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:55.661527  9479 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 24.
W20260808 01:52:55.664403  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 237728us
W20260808 01:52:55.664556  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 45884.2us
I20260808 01:52:55.563809 17394 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:55.658658 17493 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:52:55.678947 17615 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:55.681828 17615 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:52:55.682281 17615 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:55.679509 17285 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:55.683306 17394 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:55.685478 17615 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:55.680198 17556 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:55.666412 17532 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.659s	user 0.010s	sys 0.004s
I20260808 01:52:55.698144  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:55.610566 17623 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:55.700500 17532 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20.
I20260808 01:52:55.687461 17285 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:55.694440  9521 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.269s (SENT) [suppressed 28 similar messages]
I20260808 01:52:55.622273 16983 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:55.564747 17618 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:55.731671  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 64989.6us
I20260808 01:52:55.744098 17372 raft_consensus.cc:2764] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260808 01:52:55.755529 17372 consensus_queue.cc:1059] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260808 01:52:55.759510  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:55.764472 17372 raft_consensus.cc:3055] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:55.735262  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:55.787688  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 52583.5us
W20260808 01:52:55.783205 16835 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.131s	sys 0.252s
I20260808 01:52:55.781769  9476 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260808 01:52:55.787010  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:55.787220  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:55.792106  9630 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:55.784125  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:55.795775  9633 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:55.778297 17285 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:55.799256 17372 raft_consensus.cc:740] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:55.850731  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:55.879129  9638 raft_consensus.cc:2410] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 21 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260808 01:52:55.836046 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:55.881057 10367 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:55.823716  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:55.834440  9476 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260808 01:52:55.853544 17409 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:55.818977  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 320809us
I20260808 01:52:55.847962  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:55.865108  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:55.697249 17594 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.088s
W20260808 01:52:55.842094 17583 consensus_meta.cc:287] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.663s	user 0.009s	sys 0.002s
W20260808 01:52:55.789491 16835 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.132s	sys 0.252s
W20260808 01:52:55.865744 17625 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260808 01:52:55.848685  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:55.895223  9629 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:55.884709  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:55.833076  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:55.790040 17547 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:55.857451  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:55.852988 17615 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:52:55.886576  9637 raft_consensus.cc:2410] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260808 01:52:55.854148  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:55.899734  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:55.934530  9638 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:52:55.903046  9635 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:55.892750 16835 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:55.908151  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:55.892303 17618 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:55.939733  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:55.910939 17629 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:55.911755 17556 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 01:52:55.943152  9632 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:55.913398  9480 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260808 01:52:55.916760  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:55.918607  9559 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22. Ops: []
I20260808 01:52:55.919667  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:55.953055  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:55.919716  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:55.921046  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:55.830201 17595 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:55.830355 17627 log.cc:927] Time spent T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260808 01:52:55.962582 16835 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:55.858796 17623 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:55.937273 17547 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:55.967303 17629 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:55.969707 17629 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:55.910012 17394 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:55.970448 17556 consensus_queue.cc:1243] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 23.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260808 01:52:55.971194 17556 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:55.971719 17556 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:55.975746  9558 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:55.787465 17295 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:55.838550 17549 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:55.838999 17628 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:52:55.855906 17134 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.057s
I20260808 01:52:55.979354  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:55.940528  9560 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:52:55.896847  9633 raft_consensus.cc:2376] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20. Re-sending same reply.
I20260808 01:52:55.959352  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:55.982296  9638 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
W20260808 01:52:55.984674 10332 leader_election.cc:336] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.747s (SENT)
I20260808 01:52:55.898494 17422 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.067s
I20260808 01:52:55.999171  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:55.973403 17394 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:55.969707 17547 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:55.941035 17447 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260808 01:52:55.991633 17623 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:55.978571 17618 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:55.974493 17556 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:55.980546 17628 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:55.837011 17358 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:55.893584 17583 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:55.993058 17629 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:55.941470 17239 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.067s
I20260808 01:52:55.979046 17453 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.037s
I20260808 01:52:55.997458 17295 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:56.006276  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:56.031275 10334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.000s	sys 0.026s
I20260808 01:52:56.086355  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:56.087235 10334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.000s	sys 0.026s
I20260808 01:52:56.087576  9632 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
W20260808 01:52:56.088092 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 677386us
I20260808 01:52:56.006829 17439 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.083s
I20260808 01:52:56.089922 17323 raft_consensus.cc:993] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: : Instructing follower 6e7856bed1be43acaa821290d506ee97 to start an election
I20260808 01:52:56.090509 17323 raft_consensus.cc:1081] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Signalling peer 6e7856bed1be43acaa821290d506ee97 to start an election
I20260808 01:52:56.024371  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.091786  9636 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:56.023856 17422 consensus_queue.cc:1243] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 25.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
W20260808 01:52:56.063539 17543 consensus_meta.cc:287] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.896s	user 0.010s	sys 0.004s
I20260808 01:52:56.095083  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.095039 17543 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:56.096158 17532 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:52:56.096617 17532 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:56.097534 17543 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:56.031474  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:56.067210  9477 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.010s	sys 0.008s
I20260808 01:52:56.024188 17398 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:56.071868 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 87250.6us
I20260808 01:52:56.097574 17532 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:56.078797  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:56.115574  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 421139us
W20260808 01:52:56.068738  9481 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.009s	sys 0.009s
W20260808 01:52:56.117951  9481 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.009s	sys 0.009s
W20260808 01:52:56.130461  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 255290us
I20260808 01:52:56.039677  9555 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
W20260808 01:52:56.101352  9477 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.010s	sys 0.008s
I20260808 01:52:56.132230  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:56.102397  9630 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:52:56.109375  9474 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260808 01:52:56.002741 10367 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:56.119297 17583 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:56.120004 17295 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:56.120676  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:56.122174 17593 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:56.052735 17372 consensus_queue.cc:260] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:56.190078 17593 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:56.124426  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:56.125577  9556 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260808 01:52:56.044137 17628 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:56.197209  9474 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260808 01:52:56.127013  9554 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22. Ops: []
I20260808 01:52:56.010753  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.198531  9638 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:56.009898 17595 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.108s
I20260808 01:52:56.042294 17618 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:56.066223 17285 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:56.069207 17619 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:56.180337  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.178201  9632 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
W20260808 01:52:56.180778  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48654.2us
I20260808 01:52:56.218240  9630 raft_consensus.cc:2376] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20. Re-sending same reply.
W20260808 01:52:56.181372  9521 leader_election.cc:336] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.760s (SENT)
I20260808 01:52:56.183766 10367 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:56.006850 17549 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:52:56.195078  9557 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:56.200718  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:56.201948  9600 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.008s	sys 0.013s
W20260808 01:52:56.246330  9600 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.741s	user 0.008s	sys 0.013s
I20260808 01:52:56.246409  9474 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
W20260808 01:52:56.249203  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:56.114475 17319 consensus_meta.cc:287] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.502s	user 0.012s	sys 0.004s
I20260808 01:52:56.251094 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:56.251386 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:56.251967 10366 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:52:56.207163  9445 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:56.253181  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 46168.9us
I20260808 01:52:56.069089 17629 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:56.053122 17623 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:56.197299 17634 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260808 01:52:56.222977  9521 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:56.224742  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:56.237669  9633 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260808 01:52:56.225071  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.270499  9638 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:56.270596 17623 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:56.270816 17443 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:52:56.271924  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.273015  9638 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:56.274191  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.198977  9476 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d2b2a88358244f93970835ea3120c380"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:37248
I20260808 01:52:56.275025  9476 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:52:56.275565  9476 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:56.206225 17593 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:56.250998 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:56.250849 17295 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:56.204634  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.116688 17398 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:56.255214  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.366705  9630 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:56.220584 17532 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:52:56.253032 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:56.264997 17439 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:52:56.306210  9629 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:56.324292 17618 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:56.326475 17595 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:52:56.363102 17632 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.129s	user 0.000s	sys 0.002s
I20260808 01:52:56.389472 17595 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:56.327296 17285 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:56.391215 17285 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:56.338809 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 60134.6us
I20260808 01:52:56.393527 17285 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:56.394181 17636 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:56.395219 17439 raft_consensus.cc:3055] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:52:56.395635 17439 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:56.340862 17631 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:56.396173  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3432) took 1077 ms. Trace:
I20260808 01:52:56.396497 17631 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:56.396687  9472 rpcz_store.cc:276] 0808 01:52:55.318755 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:55.323690 (+  4935us) service_pool.cc:224] Handling call
0808 01:52:55.324149 (+   459us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:55.417143 (+ 92994us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:52:55.468605 (+ 51462us) log.cc:844] Serialized 14 byte log entry
0808 01:52:56.337905 (+869300us) spinlock_profiling.cc:243] Waited 3.54 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:52:56.366948 (+ 29043us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:56.367008 (+    60us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:56.395854 (+ 28846us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:56.395944 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:56.396088 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3113,"prepare.queue_time_us":59494,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":63,"raft.queue_time_us":122251,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"spinlock_wait_cycles":8141696,"thread_start_us":843695,"threads_started":2,"wal-append.queue_time_us":593302}
I20260808 01:52:56.341271 17629 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:56.344161  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.399401  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.400074  9629 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:56.344717  9638 raft_consensus.cc:2376] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18. Re-sending same reply.
I20260808 01:52:56.400550  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.401188  9632 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:56.401278  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.402567  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.366083 17295 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:56.403177  9638 raft_consensus.cc:2376] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22. Re-sending same reply.
I20260808 01:52:56.296868 17594 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:52:56.228328 17619 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:56.404655 17594 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:56.405597 17594 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:56.406129 17619 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:56.351727 17640 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
W20260808 01:52:56.354399 17643 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260808 01:52:56.396518 17439 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:56.298864 17639 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:52:56.408262 17639 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:56.408599 17439 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:52:56.408720 17639 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:52:56.359989  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:56.410936  9637 raft_consensus.cc:2410] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 21 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
W20260808 01:52:56.411125  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 51252us
I20260808 01:52:56.411554 17595 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:56.225346 17549 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:52:56.369007  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 109026us
I20260808 01:52:56.412230 17595 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:56.413166 17595 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:56.412873 17549 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:56.413734 17595 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:56.387764 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:56.360926  9636 raft_consensus.cc:2376] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19. Re-sending same reply.
I20260808 01:52:56.416159  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:56.416893  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.417374 17650 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:56.417793  9636 raft_consensus.cc:2376] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23. Re-sending same reply.
I20260808 01:52:56.356974  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.419034  9633 raft_consensus.cc:2376] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23. Re-sending same reply.
I20260808 01:52:56.420143  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.392098 17372 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:56.414188 17549 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:56.396198  9634 raft_consensus.cc:2410] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 26 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260808 01:52:56.404136  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.442358  9638 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:56.428148  9474 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:52:56.443159 10366 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:52:56.400524  9557 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:52:56.428921  9630 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:52:56.444586 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:56.449396  9629 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:52:56.451953  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.453900  9636 raft_consensus.cc:2376] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20. Re-sending same reply.
W20260808 01:52:56.383358 17533 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.741s	user 0.007s	sys 0.007s
I20260808 01:52:56.456328  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.457791  9629 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:52:56.244215 17323 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.177s
I20260808 01:52:56.460218  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.466706  9631 raft_consensus.cc:2410] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260808 01:52:56.475849  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.474444  9554 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260808 01:52:56.479707  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:56.462250 17394 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:56.480742 17394 raft_consensus.cc:697] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:56.481220  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.481814 17394 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:56.483513  9558 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260808 01:52:56.483793  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:56.473963  9600 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:56.491914  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:56.493438  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:56.484047 17533 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260808 01:52:56.494546 17629 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:52:56.495048 17533 consensus_queue.cc:1243] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 22.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260808 01:52:56.495304 17629 raft_consensus.cc:697] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:56.495667 17533 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:56.485271  9560 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260808 01:52:56.496256  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:56.496399 17629 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:56.497591  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:56.497680  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:56.498322  9552 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:56.498493  9560 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:56.499402 17533 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:56.496438 17615 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:52:56.451370 17295 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:56.500840 17615 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:56.501291 17631 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:56.501996 17533 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:56.504182 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:56.510121  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:56.516120  9638 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:56.518172  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.520164  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.520848  9633 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:52:56.523850  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 51559.5us
I20260808 01:52:56.524154  9632 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:52:56.525038  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:56.536367 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:56.537652 10366 raft_consensus.cc:2393] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 18: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:52:56.541049  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:56.544358  9630 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:52:56.555694  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:56.556677  9557 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:56.555783  9636 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:56.558492  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.558709  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.560120  9550 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23. Ops: []
I20260808 01:52:56.564735  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:56.565688  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:56.570467  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.563928  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:56.501889 17615 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:56.584225  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.590117  9554 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:56.590708  9630 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 21: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:56.591164  9629 raft_consensus.cc:2368] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 21. Current term is 22.
I20260808 01:52:56.548671 16835 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:56.592103  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.608959  9638 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:52:56.616611  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.583932 17556 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:56.626946  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:56.636256  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:56.637956 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 214056us
I20260808 01:52:56.635390  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:56.585076 17654 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:52:56.647178  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.650136  9630 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
W20260808 01:52:56.647773  9635 consensus_meta.cc:287] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.711s	user 0.012s	sys 0.000s
I20260808 01:52:56.651132  9635 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:52:56.651499  9599 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:52:56.652117  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.652961  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:56.653609  9554 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
W20260808 01:52:56.654736  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 238665us
I20260808 01:52:56.658555  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:56.607110 17619 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.042s
I20260808 01:52:56.664758  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.666378  9636 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:52:56.667188 17653 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:56.668140  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:56.652683 17648 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
W20260808 01:52:56.667356  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 527072us
I20260808 01:52:56.670197  9636 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:52:56.672979  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.616000 16835 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:56.644862 17398 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:56.654906 17615 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:56.660231 17543 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:56.656585 17556 raft_consensus.cc:697] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:56.690798 17543 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:56.697398 17543 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:56.697664 17639 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:56.664849 17319 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260808 01:52:56.711875  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3454) took 1300 ms. Trace:
I20260808 01:52:56.678416  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:52:56.649909 17654 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:56.681447  9630 raft_consensus.cc:2376] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20. Re-sending same reply.
I20260808 01:52:56.671480 17285 raft_consensus.cc:2764] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
W20260808 01:52:56.689520 17647 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260808 01:52:56.660094 17533 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260808 01:52:56.674857  9521 leader_election.cc:336] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.777s (SENT)
I20260808 01:52:56.712384  9477 rpcz_store.cc:276] 0808 01:52:55.411973 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:55.412228 (+   255us) service_pool.cc:224] Handling call
0808 01:52:55.412801 (+   573us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:55.441543 (+ 28742us) raft_consensus.cc:1534] Early marking committed up to index 12
0808 01:52:56.136934 (+695391us) log.cc:844] Serialized 14 byte log entry
0808 01:52:56.562401 (+425467us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:56.605882 (+ 43481us) log.cc:844] Serialized 52 byte log entry
0808 01:52:56.606053 (+   171us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:52:56.606095 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:56.606177 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:56.711610 (+105433us) raft_consensus.cc:1685] finished
0808 01:52:56.711632 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:56.711805 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":88969,"prepare.queue_time_us":95239,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":63,"raft.queue_time_us":47911,"raft.run_cpu_time_us":420,"raft.run_wall_time_us":41599,"thread_start_us":982631,"threads_started":2,"wal-append.queue_time_us":488276}
I20260808 01:52:56.722759  9635 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 20: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:52:56.725509  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 47325.3us
I20260808 01:52:56.656556 17295 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
I20260808 01:52:56.727159 17654 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:56.720846 17439 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:56.729777 17631 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260808 01:52:56.783463  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 124184us
W20260808 01:52:56.734141  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59337.2us
I20260808 01:52:56.784289  9599 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:52:56.784886  9521 leader_election.cc:336] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.830s (SENT)
I20260808 01:52:56.888747  9521 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:56.734714  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:56.889551  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 105356us
I20260808 01:52:56.889809  9636 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
W20260808 01:52:56.785739 17656 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260808 01:52:56.743175  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.893628  9630 raft_consensus.cc:2376] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19. Re-sending same reply.
I20260808 01:52:56.743428 17663 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
W20260808 01:52:56.732857 17648 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260808 01:52:56.744105 17285 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
I20260808 01:52:56.747740 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:56.897321 10366 raft_consensus.cc:2368] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 24. Current term is 25.
I20260808 01:52:56.897408  9629 raft_consensus.cc:2410] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 23 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
W20260808 01:52:56.750517  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:56.751637 17650 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:52:56.775424 17595 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:56.728986  9638 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:52:56.901392  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.897944 17661 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:52:56.902952  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.903764  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3453) took 1499 ms. Trace:
I20260808 01:52:56.904206  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.905041  9638 raft_consensus.cc:2393] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 20: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:56.904166  9481 rpcz_store.cc:276] 0808 01:52:55.404241 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:55.404472 (+   231us) service_pool.cc:224] Handling call
0808 01:52:55.429706 (+ 25234us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:56.129674 (+699968us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:56.129710 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:56.187925 (+ 58215us) log.cc:844] Serialized 52 byte log entry
0808 01:52:56.545789 (+357864us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:56.545850 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:56.545941 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:56.903526 (+357585us) raft_consensus.cc:1685] finished
0808 01:52:56.903549 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:56.903704 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":99368,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":28328,"thread_start_us":966864,"threads_started":2,"wal-append.queue_time_us":422329}
I20260808 01:52:56.905916  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:56.749003 17439 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:56.802526  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:56.907666  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:56.783514 17631 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:56.908622 17631 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:56.808221 17677 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.004s
I20260808 01:52:56.819984 17457 raft_consensus.cc:3055] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Stepping down as leader of term 22
W20260808 01:52:56.775628 17628 consensus_meta.cc:287] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.575s	user 0.007s	sys 0.008s
I20260808 01:52:56.910745 17457 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:56.911074 17628 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:56.912026 17457 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:56.913772 17457 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:56.914091 17628 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:56.835975  9633 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:52:56.745404 17639 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:56.837167 17556 consensus_queue.cc:237] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:56.915549  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.887308 17674 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:56.848497  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:56.920310  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:56.920810  9479 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:52:56.849022  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:56.846668  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:56.849803  9477 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
W20260808 01:52:56.864797 10332 leader_election.cc:336] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.916s (SENT)
W20260808 01:52:56.892429  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:56.906005  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:56.879724 17619 consensus_queue.cc:1243] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 22.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
W20260808 01:52:56.927295 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62556.1us
I20260808 01:52:56.927830  9473 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:52:56.929006 10332 leader_election.cc:336] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.968s (SENT)
I20260808 01:52:56.929546 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:56.930006  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:56.930923  9479 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:52:56.932358  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:56.933852  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:56.935048 17673 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:56.936753  9635 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260808 01:52:56.939877  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:56.941932  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:56.942030  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:56.943012 17674 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:56.945811 17652 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.672s	user 0.000s	sys 0.003s
I20260808 01:52:56.945878 17674 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:56.949198 10332 leader_election.cc:336] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.981s (SENT)
I20260808 01:52:56.949755 10332 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:56.950717 17616 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:56.952039  9475 raft_consensus.cc:2410] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 18 because replica has last-logged OpId of term: 17 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20260808 01:52:56.896216 17629 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260808 01:52:56.797060  9476 consensus_meta.cc:287] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.521s	user 0.009s	sys 0.005s
I20260808 01:52:56.961133  9635 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260808 01:52:56.847590  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:56.919087  9638 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:56.920976  9636 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
W20260808 01:52:56.911098  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 126207us
I20260808 01:52:56.963723  9633 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260808 01:52:56.954509  9476 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:56.918308  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:56.969466  9472 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:56.977216  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:56.979465  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:56.961432 17629 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:56.980145  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:56.996004  9475 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:52:57.003005  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:57.011221  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 226340us
I20260808 01:52:57.002485 10334 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:57.012159  9476 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:57.015251  9638 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23. Ops: []
W20260808 01:52:57.047612 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 74560.1us
I20260808 01:52:57.057237  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3246) took 1196 ms. Trace:
W20260808 01:52:57.057956 10332 leader_election.cc:336] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.684s (SENT)
I20260808 01:52:57.061425 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:57.061335  9634 rpcz_store.cc:276] 0808 01:52:55.860517 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:55.907734 (+ 47217us) service_pool.cc:224] Handling call
0808 01:52:57.048967 (+1141233us) spinlock_profiling.cc:243] Waited 458 ms on lock 0x7b540032d054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:57.057147 (+  8180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1053524864}
W20260808 01:52:57.049853  9521 leader_election.cc:336] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.607s (SENT)
I20260808 01:52:57.063217  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:56.997983  9601 connection.cc:256] RPC call timeout handler was delayed by -1.3676755329943262s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1635s for a response
W20260808 01:52:57.072232  9601 connection.cc:256] RPC call timeout handler was delayed by -1.3397755330079235s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1666s for a response
I20260808 01:52:57.073633  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:57.081940  9479 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
W20260808 01:52:57.072561  9601 connection.cc:256] RPC call timeout handler was delayed by -1.3163755330024287s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16920000000000002s for a response
I20260808 01:52:56.967374 17543 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:57.084666  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:57.102252  9601 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:57.022290 17409 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:57.000370 17629 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:57.099851  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:57.111007  9638 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260808 01:52:57.111582  9480 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:52:57.107570 17443 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:52:57.112702  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:57.113499  9634 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 14. (term mismatch)
I20260808 01:52:57.113971  9634 pending_rounds.cc:85] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 13
I20260808 01:52:57.114187  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:57.047747 17674 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:57.108036  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:57.011009 17619 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:57.100569  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:57.153868  9479 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 21: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:57.157151  9479 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22. Ops: []
I20260808 01:52:57.114235  9634 pending_rounds.cc:107] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 25.14
I20260808 01:52:57.160084  9634 raft_consensus.cc:2889] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:57.085474 17628 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260808 01:52:57.119940 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62036us
I20260808 01:52:57.071594 17618 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:52:57.125375  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:57.125712  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:57.178032  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52475.3us
I20260808 01:52:57.129067 17457 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:52:57.191669 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: true dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:52:57.192437 10366 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:57.131045 17671 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:52:57.192837 10366 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:52:57.140033 17372 consensus_meta.cc:287] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.698s	user 0.011s	sys 0.002s
I20260808 01:52:57.108052 17319 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:52:57.146231  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3267) took 1010 ms. Trace:
I20260808 01:52:57.158612 17674 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:57.149580  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:57.150846 17654 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:57.164522  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:52:57.183512  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 81353.9us
I20260808 01:52:57.207682  9632 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 23: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:57.095978  9636 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:52:57.071275 17594 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.088s
W20260808 01:52:57.139135  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 0b4331dfdd5b43448a400f2eb7b1be70: Illegal state: Replica 6e7856bed1be43acaa821290d506ee97 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 21 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:52:57.104630 17670 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.090s
I20260808 01:52:57.195873 17548 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:52:57.168107 17628 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:57.147305  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3256) took 1062 ms. Trace:
I20260808 01:52:57.228289  9637 rpcz_store.cc:276] 0808 01:52:56.085475 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:56.085629 (+   154us) service_pool.cc:224] Handling call
0808 01:52:57.147010 (+1061381us) spinlock_profiling.cc:243] Waited 315 ms on lock 0x7b540027e054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:57.147234 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":723708672}
I20260808 01:52:57.212714 17674 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:57.208936  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:57.224776  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:57.203493  9629 rpcz_store.cc:276] 0808 01:52:56.136014 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:56.613204 (+477190us) service_pool.cc:224] Handling call
0808 01:52:57.145921 (+532717us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b540003b6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:57.146144 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":598241024}
I20260808 01:52:57.227155 10366 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:57.236009 10365 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260808 01:52:57.236506 10366 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
W20260808 01:52:57.240115  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 190268us
I20260808 01:52:57.241707 17672 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:57.244860  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3251) took 1284 ms. Trace:
I20260808 01:52:57.248003  9601 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:57.205423 17618 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:57.251530 17670 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:52:57.192883 17629 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:57.233166  9634 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:57.245718  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:57.259693  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:57.260509  9632 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
W20260808 01:52:57.241753  9521 leader_election.cc:336] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.631s (SENT)
I20260808 01:52:57.262285  9521 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:57.239545  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:57.258589  9631 rpcz_store.cc:276] 0808 01:52:55.960089 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:55.960417 (+   328us) service_pool.cc:224] Handling call
0808 01:52:57.244381 (+1283964us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b54001b3ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:52:57.244776 (+   395us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1117750656}
I20260808 01:52:57.286204  9445 leader_election.cc:400] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 24. Current term is 25.
I20260808 01:52:57.223464 17548 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:57.198895 17619 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:57.232379 17628 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:57.307840  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:57.267970 17439 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:57.309571  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:57.310230  9445 leader_election.cc:403] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
W20260808 01:52:57.266970 17657 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260808 01:52:57.330263 17654 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:57.324586 17549 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260808 01:52:57.333357  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 85510.6us
I20260808 01:52:57.335016  9601 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:57.335479  9636 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 13. (term mismatch)
I20260808 01:52:57.317133  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:57.336270 17676 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:57.318084  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:57.312103  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:57.337226  9559 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:57.337458  9629 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:52:57.265203 17670 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:57.339529 17670 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:57.341054 17670 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:57.325101  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:57.342636  9560 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:57.159705 17409 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:57.327170  9558 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:52:57.328609  9554 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
W20260808 01:52:57.335100  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 49060.9us
I20260808 01:52:57.335719 17285 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 24. Current term is 25.
I20260808 01:52:57.336552  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:57.316440 17548 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:57.350051  9636 pending_rounds.cc:85] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 12
I20260808 01:52:57.351997  9636 pending_rounds.cc:107] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 22.13
I20260808 01:52:57.352226  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:57.354421  9636 raft_consensus.cc:2889] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:57.349201  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:57.264202  9637 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:52:57.356659  9634 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:57.360504 10366 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:52:57.362167  9637 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260808 01:52:57.368289  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3289) took 1059 ms. Trace:
I20260808 01:52:57.368729  9636 rpcz_store.cc:276] 0808 01:52:56.308787 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:57.321646 (+1012859us) service_pool.cc:224] Handling call
0808 01:52:57.335080 (+ 13434us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:57.367985 (+ 32905us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:57.368234 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:57.367998  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3291) took 1054 ms. Trace:
I20260808 01:52:57.372964  9637 rpcz_store.cc:276] 0808 01:52:56.309386 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:57.357216 (+1047830us) service_pool.cc:224] Handling call
0808 01:52:57.357656 (+   440us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:57.362988 (+  5332us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:57.363614 (+   626us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:57.374404  9637 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:52:57.375130  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3042) took 1022 ms. Trace:
I20260808 01:52:57.375456  9637 rpcz_store.cc:276] 0808 01:52:56.352987 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:57.373849 (+1020862us) service_pool.cc:224] Handling call
0808 01:52:57.374197 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:57.374848 (+   651us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:57.375067 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:57.377306  9637 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260808 01:52:57.378437  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3044) took 1024 ms. Trace:
I20260808 01:52:57.379043  9637 rpcz_store.cc:276] 0808 01:52:56.354195 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:57.376460 (+1022265us) service_pool.cc:224] Handling call
0808 01:52:57.376931 (+   471us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:57.377926 (+   995us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:57.378352 (+   426us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:57.284768 17672 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:57.387003  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:57.378827  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:57.357268 16835 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:57.390455  9637 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260808 01:52:57.398160  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:57.396261  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 59795.9us
I20260808 01:52:57.404217  9554 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:52:57.406025  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3045) took 1051 ms. Trace:
I20260808 01:52:57.406730  9637 rpcz_store.cc:276] 0808 01:52:56.354850 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:57.381279 (+1026429us) service_pool.cc:224] Handling call
0808 01:52:57.381861 (+   582us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:57.405609 (+ 23748us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:57.405931 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:57.384601 17671 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260808 01:52:57.415361  9636 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260808 01:52:57.419493  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3292) took 1109 ms. Trace:
I20260808 01:52:57.419998  9636 rpcz_store.cc:276] 0808 01:52:56.309715 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:57.380955 (+1071240us) service_pool.cc:224] Handling call
0808 01:52:57.414837 (+ 33882us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:57.419092 (+  4255us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:57.419405 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:57.409343  9637 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 11. (term mismatch)
I20260808 01:52:57.424257  9637 pending_rounds.cc:85] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 10
I20260808 01:52:57.425236  9637 pending_rounds.cc:107] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 21.11
I20260808 01:52:57.427924 10365 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260808 01:52:57.426540  9637 raft_consensus.cc:2889] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:57.410037 17594 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:57.435235 17594 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:57.438133 17594 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:57.440179 17629 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:57.440732 17688 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.058s	user 0.002s	sys 0.000s
I20260808 01:52:57.444380 17703 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:57.445977 17619 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:57.447450 17692 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260808 01:52:57.431879  9445 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:57.449805  9636 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 14. (term mismatch)
W20260808 01:52:57.450748  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 72254us
I20260808 01:52:57.426290 17543 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:57.467345  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:57.466198  9636 pending_rounds.cc:85] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 13
W20260808 01:52:57.428526 17631 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.519s	user 0.009s	sys 0.006s
I20260808 01:52:57.458951  9560 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
W20260808 01:52:57.459375  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50418.3us
I20260808 01:52:57.481194  9601 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:57.471719  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3288) took 1163 ms. Trace:
W20260808 01:52:57.487372 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:57.469069  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3294) took 1105 ms. Trace:
I20260808 01:52:57.457305  9559 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:52:57.489395  9558 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:57.470429  9636 pending_rounds.cc:107] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260808 01:52:57.490360  9636 raft_consensus.cc:2889] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:57.462920 17661 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:57.460933 17653 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:52:57.488623  9637 rpcz_store.cc:276] 0808 01:52:56.363457 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:57.408295 (+1044838us) service_pool.cc:224] Handling call
0808 01:52:57.408767 (+   472us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:57.468783 (+ 60016us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:57.469006 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:57.501909  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 968) took 1118 ms. Trace:
I20260808 01:52:57.502599  9636 rpcz_store.cc:276] 0808 01:52:56.383887 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:57.421204 (+1037317us) service_pool.cc:224] Handling call
0808 01:52:57.449293 (+ 28089us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:57.501516 (+ 52223us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:57.501823 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:57.506027  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:57.528616  9637 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260808 01:52:57.523973  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:57.582451 17409 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:57.582540 17682 log.cc:927] Time spent T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260808 01:52:57.588268 17672 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:57.511255 17698 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.080s
I20260808 01:52:57.530886 17619 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:52:57.490581 17285 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:57.486720 17631 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:57.597537 17699 log.cc:927] Time spent T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
I20260808 01:52:57.600243  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3282) took 1373 ms. Trace:
I20260808 01:52:57.600788  9633 rpcz_store.cc:276] 0808 01:52:56.226540 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:56.969970 (+743430us) service_pool.cc:224] Handling call
0808 01:52:56.998083 (+ 28113us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:57.011652 (+ 13569us) raft_consensus.cc:1534] Early marking committed up to index 8
0808 01:52:57.172938 (+161286us) log.cc:844] Serialized 14 byte log entry
0808 01:52:57.317455 (+144517us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:57.324597 (+  7142us) log.cc:844] Serialized 26 byte log entry
0808 01:52:57.324777 (+   180us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:52:57.325414 (+   637us) log.cc:844] Serialized 14 byte log entry
0808 01:52:57.325777 (+   363us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:57.325915 (+   138us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:57.599945 (+274030us) raft_consensus.cc:1685] finished
0808 01:52:57.599978 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:57.600159 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":87243,"prepare.queue_time_us":192273,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":80,"raft.queue_time_us":17758,"raft.run_cpu_time_us":570,"raft.run_wall_time_us":39871,"thread_start_us":238781,"threads_started":2,"wal-append.queue_time_us":238011}
I20260808 01:52:57.612026  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:57.613463  9554 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:57.617882 10365 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260808 01:52:57.578409 17618 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:57.530822 17703 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:57.623652 17703 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:57.624050 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:52:57.627815 17697 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.313s	user 0.000s	sys 0.002s
I20260808 01:52:57.632320  9560 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260808 01:52:57.598496 17631 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:57.636483  9472 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260808 01:52:57.485764 17532 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:52:57.644275 17646 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:52:57.644960 17646 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:57.646265 17672 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:52:57.531402 17629 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.123s
W20260808 01:52:57.533066 17692 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260808 01:52:57.486219 17372 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:57.657577 17372 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:57.659104 17372 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:57.669749  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1335) took 1023 ms. Trace:
I20260808 01:52:57.670430  9478 rpcz_store.cc:276] 0808 01:52:56.646192 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:56.728608 (+ 82416us) service_pool.cc:224] Handling call
0808 01:52:56.771484 (+ 42876us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:56.771847 (+   363us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:56.771889 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:56.792780 (+ 20891us) log.cc:844] Serialized 26 byte log entry
0808 01:52:57.085322 (+292542us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:52:57.085366 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:57.085444 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:57.669391 (+583947us) raft_consensus.cc:1685] finished
0808 01:52:57.669428 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:57.669654 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":250448,"threads_started":1,"wal-append.queue_time_us":385623}
I20260808 01:52:57.577384  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 972) took 1051 ms. Trace:
I20260808 01:52:57.673713  9637 rpcz_store.cc:276] 0808 01:52:56.526318 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:57.498008 (+971690us) service_pool.cc:224] Handling call
0808 01:52:57.517171 (+ 19163us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:57.529530 (+ 12359us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:57.577258 (+ 47728us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:52:57.676035  9637 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260808 01:52:57.678611  9637 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23. Ops: []
I20260808 01:52:57.668089  9632 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
W20260808 01:52:57.691905  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 450439us
I20260808 01:52:57.691851  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:57.693316  9559 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
W20260808 01:52:57.693534  9521 leader_election.cc:336] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.694s (SENT)
W20260808 01:52:57.694631  9521 leader_election.cc:336] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.721s (SENT)
I20260808 01:52:57.695145  9521 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:57.697449 17358 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260808 01:52:57.710281  9634 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:52:57.723822  9559 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260808 01:52:57.482954  9631 rpcz_store.cc:276] 0808 01:52:56.308513 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:57.320607 (+1012094us) service_pool.cc:224] Handling call
0808 01:52:57.327314 (+  6707us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:57.327857 (+   543us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:52:57.334087 (+  6230us) log.cc:844] Serialized 14 byte log entry
0808 01:52:57.471055 (+136968us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:57.471154 (+    99us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:57.471312 (+   158us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:57.471425 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:57.471624 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5340,"prepare.queue_time_us":5539,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":116,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":5934,"thread_start_us":136447,"threads_started":1}
I20260808 01:52:57.731070  9521 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:57.735270  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:57.736301  9631 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:57.738893  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:57.740037  9560 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:52:57.740828  9632 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
W20260808 01:52:57.737896  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 214055us
I20260808 01:52:57.742727  9601 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:57.744776  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:57.747833  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:57.504745 17653 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:57.782878  9473 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260808 01:52:57.776216  9637 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
W20260808 01:52:57.787410  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 44835.4us
I20260808 01:52:57.788632  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:57.491590 17439 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.267s
I20260808 01:52:57.715217 17639 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:57.739096 16835 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:57.796465  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:57.807067  9601 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:57.800256 17654 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:57.770907 17708 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.070s
I20260808 01:52:57.814651  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:57.815825 17708 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:52:57.816670 17708 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:57.816867  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:57.814940 17556 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:57.819586 17654 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:57.749747 17398 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.069s
I20260808 01:52:57.824271 17398 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
W20260808 01:52:57.833261 17674 consensus_meta.cc:287] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.601s	user 0.011s	sys 0.001s
I20260808 01:52:57.803454 17615 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:57.835706 17615 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:57.792429 17653 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:57.803277 17549 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:57.806429 17593 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:57.845389 17593 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:57.848012 17593 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:57.811975 17703 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:57.646194 17646 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:57.862627  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:57.811746 16835 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:57.838361 17615 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:57.833909 17674 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:57.776755 17629 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260808 01:52:57.928510  9552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.011s	sys 0.030s
W20260808 01:52:57.928926  9552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.011s	sys 0.030s
I20260808 01:52:57.867607 17676 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:52:57.807289 17583 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.124s
I20260808 01:52:57.927860 17700 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:57.818033 17708 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:57.971745  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 154991us
I20260808 01:52:57.879491 17639 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:57.966722 17548 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.599s	user 0.013s	sys 0.002s
I20260808 01:52:57.983333  9601 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:57.989656  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:57.994228  9560 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260808 01:52:57.995548  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:52:57.994223  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 271956us
I20260808 01:52:57.999851  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:58.000730  9555 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:52:57.970183 17710 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:58.001598  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:58.002637  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:58.003355  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 272382us
I20260808 01:52:58.009253  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:57.993252 17676 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:57.997577 17543 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:58.017493  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:57.978223 17700 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:58.003439 17629 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:57.984853 17548 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:58.004794 17615 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260808 01:52:58.036077  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:58.037561  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:58.030337 10332 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:58.043206  9478 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 16. (term mismatch)
I20260808 01:52:58.044147  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:58.047818  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:58.048892  9555 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:52:58.055313 17716 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
W20260808 01:52:58.056680 17709 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260808 01:52:58.057965  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:52:58.027948 17628 consensus_meta.cc:287] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.687s	user 0.011s	sys 0.004s
I20260808 01:52:58.059798  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3290) took 1750 ms. Trace:
I20260808 01:52:58.066107 17628 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:58.066862  9629 rpcz_store.cc:276] 0808 01:52:56.309088 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:57.338379 (+1029291us) service_pool.cc:224] Handling call
0808 01:52:57.338829 (+   450us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:57.339324 (+   495us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:52:57.341690 (+  2366us) log.cc:844] Serialized 14 byte log entry
0808 01:52:57.721592 (+379902us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:57.750489 (+ 28897us) log.cc:844] Serialized 26 byte log entry
0808 01:52:57.750742 (+   253us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:52:57.751056 (+   314us) log.cc:844] Serialized 14 byte log entry
0808 01:52:57.786791 (+ 35735us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:57.786912 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:58.059536 (+272624us) raft_consensus.cc:1685] finished
0808 01:52:58.059560 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:58.059711 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40857,"prepare.queue_time_us":9801,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":100,"raft.queue_time_us":57630,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":29915,"spinlock_wait_cycles":37607040,"thread_start_us":364731,"threads_started":1,"wal-append.queue_time_us":540534}
I20260808 01:52:58.059123 17673 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260808 01:52:58.068260 17615 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:58.069573 17615 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:58.063235 17616 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:58.068849 17628 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:58.018380 17726 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:52:58.073226  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:58.071928  9550 raft_consensus.cc:2368] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22.
I20260808 01:52:58.075301  9633 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:52:58.043853  9478 pending_rounds.cc:85] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 15
I20260808 01:52:58.077576  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:58.077863  9478 pending_rounds.cc:107] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 19.16
I20260808 01:52:58.079010  9479 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:52:58.079859  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:58.080778  9633 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 20: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:58.078598  9478 raft_consensus.cc:2889] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:58.043360 17635 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:58.088376  9634 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260808 01:52:58.085721  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:58.090553  9638 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:52:58.093254  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:58.093504 17719 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20260808 01:52:58.095140  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47063.7us
I20260808 01:52:58.096204  9521 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:58.007572 17710 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:58.097816 17674 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:58.099113 17710 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:58.099668 17639 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:58.096873  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3038) took 1842 ms. Trace:
I20260808 01:52:58.101509  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:58.102547  9638 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:52:58.101954  9635 rpcz_store.cc:276] 0808 01:52:56.254625 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:56.980296 (+725671us) service_pool.cc:224] Handling call
0808 01:52:57.022490 (+ 42194us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:57.071810 (+ 49320us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:57.071846 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:57.072411 (+   565us) log.cc:844] Serialized 26 byte log entry
0808 01:52:57.692618 (+620207us) spinlock_profiling.cc:243] Waited 1.58 ms on lock 0x7b18004d1700. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:52:57.926769 (+234151us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:57.926816 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:57.926901 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:58.096520 (+169619us) raft_consensus.cc:1685] finished
0808 01:52:58.096548 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:58.096802 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39343,"raft.queue_time_us":24903,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":27223,"spinlock_wait_cycles":3640192,"thread_start_us":233822,"threads_started":1,"wal-append.queue_time_us":888717}
I20260808 01:52:58.104033  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:58.104873  9638 raft_consensus.cc:2376] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22. Re-sending same reply.
I20260808 01:52:58.105042  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:58.105856  9635 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:52:58.108004  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:58.109418  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:58.110276 17548 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:58.111615 17615 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:58.112613  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:58.094030  9634 raft_consensus.cc:2376] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18. Re-sending same reply.
I20260808 01:52:58.113668  9635 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 22.
I20260808 01:52:58.114192 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:58.114959  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:58.118853 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:58.121117 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:58.125044 10332 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:58.125178  9521 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:58.129036 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:58.129218 17674 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:58.130563 17592 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:52:58.130897 17674 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:58.131346 17592 raft_consensus.cc:697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:58.128150 17725 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:58.132207 17725 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:58.132658 17725 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:58.132516 17592 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:58.134806 17710 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:58.101588 17700 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:58.131079 10332 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:58.131501 17674 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:58.118480  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:58.137446  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:58.140275  9521 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:58.143352 17693 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:52:58.145406 17693 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:58.143846 17710 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:58.146462 17693 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:58.157717  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:58.160662  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:58.161885 17676 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260808 01:52:58.165069 17409 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:52:58.160517 17710 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:52:58.176296 17653 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:58.179443  9474 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:58.181381 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:58.182206 10360 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:58.185837 17726 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:58.138657 17358 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:58.187599  9600 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:52:58.142014 17323 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:52:58.188859  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:58.198524 17726 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:58.189702 17543 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.013s
I20260808 01:52:58.207401 17616 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:52:58.187002 17615 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.022s
I20260808 01:52:58.209244 17663 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:58.209723 17663 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:58.210279 17358 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:58.191721 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:58.211575  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:58.216060 17674 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:58.217657  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:58.219050 17674 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:58.220357 17653 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:58.221236 17726 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:52:58.221323  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 66758.8us
I20260808 01:52:58.223158  9521 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:58.227393 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:58.228399  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:58.228874 17671 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:58.229336 17671 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:58.229739 17671 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:58.229907  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:58.230960  9558 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:58.231664 17663 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:58.233021 17663 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:58.235859 17443 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:58.230093  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:58.237509  9559 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
W20260808 01:52:58.238384 17690 log.cc:927] Time spent T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.817s	user 0.000s	sys 0.008s
I20260808 01:52:58.243490  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:58.245787 17398 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260808 01:52:58.247941  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:58.252961 17725 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:58.256026  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:58.256772  9560 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 21
I20260808 01:52:58.257412  9475 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260808 01:52:58.261147  9481 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
W20260808 01:52:58.240257  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:58.265909  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:58.241231 17543 raft_consensus.cc:3055] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:52:58.267483  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:58.241298  9629 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260808 01:52:58.281447  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:58.218576  9473 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 22
I20260808 01:52:58.267246  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:58.282470  9638 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:58.283155  9635 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:52:58.284175  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:58.268003  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:58.268121  9600 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:58.286530  9558 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:52:58.295176  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:58.304154  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:58.306967  9634 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:58.314077  9560 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
W20260808 01:52:58.296262  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 99697.9us
I20260808 01:52:58.286978 17591 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:58.281230 17543 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:52:58.330276 10332 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:52:58.343984  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 96119.6us
I20260808 01:52:58.347985  9473 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
W20260808 01:52:58.349411  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 81969.4us
I20260808 01:52:58.348985  9638 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:52:58.312338 17447 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:58.352480 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:58.350737  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:58.354442  9521 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:58.375991  9634 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:52:58.351207 17672 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:58.390640  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:58.325297 17548 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260808 01:52:58.358806 17457 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:52:58.392571  9557 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 19: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:52:58.393465  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:52:58.399178  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:58.399052  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:58.387384 17323 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:58.360077 17693 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.037s
I20260808 01:52:58.410070  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:58.409379  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:58.361902 17661 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.041s
I20260808 01:52:58.363195 17447 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:58.397187 17457 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:58.362577 17714 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.040s
I20260808 01:52:58.397878 17639 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:58.431519  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:58.400261 17672 raft_consensus.cc:697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:58.407052 17636 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.088s
I20260808 01:52:58.478314 17653 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:58.501025 17653 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:58.501541 17653 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:58.503574  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3482) took 1379 ms. Trace:
I20260808 01:52:58.504211  9476 rpcz_store.cc:276] 0808 01:52:57.124084 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:57.460802 (+336718us) service_pool.cc:224] Handling call
0808 01:52:57.461203 (+   401us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:57.461747 (+   544us) raft_consensus.cc:1534] Early marking committed up to index 14
0808 01:52:57.716393 (+254646us) log.cc:844] Serialized 14 byte log entry
0808 01:52:57.717152 (+   759us) log.cc:844] Serialized 14 byte log entry
0808 01:52:58.502553 (+785401us) spinlock_profiling.cc:243] Waited 966 us on lock 0x7b180049f780. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d44548f50 00007f5d4453c483 00007f5d44531eda 00007f5d4453242a 00007f5d4499de0a 00007f5d449ad49f 00007f5d449ad418
0808 01:52:58.502969 (+   416us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:58.503054 (+    85us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:52:58.503122 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:58.503226 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:58.503459 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7291,"prepare.queue_time_us":479047,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":66,"raft.queue_time_us":91,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":27010,"spinlock_wait_cycles":2222848,"thread_start_us":240784,"threads_started":1}
W20260808 01:52:58.469848 17734 log.cc:927] Time spent T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260808 01:52:58.398128 17543 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:58.472148 17447 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:58.412027 17677 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:52:58.473011 17695 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:58.477257 17635 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:58.407008 17398 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.073s
I20260808 01:52:58.416446 17720 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.072s
W20260808 01:52:58.488734 17732 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260808 01:52:58.497956 17457 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:58.495440 17636 consensus_queue.cc:1243] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 20.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260808 01:52:58.482498 17443 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260808 01:52:58.479542 17733 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260808 01:52:58.465306 17723 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.104s
I20260808 01:52:58.464707  9634 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
W20260808 01:52:58.487483  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 56061.5us
I20260808 01:52:58.483807 17323 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:58.412724 17725 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:58.555438 17543 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:58.486871 17639 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:58.649827 17639 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:58.650905 17549 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:58.651654 17549 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:58.556860 17677 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:58.457527 17661 consensus_queue.cc:1243] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 23.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260808 01:52:58.506096 17700 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:52:58.591260 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:58.485127 17693 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:52:58.486260 17714 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.148s
I20260808 01:52:58.600220 17723 consensus_queue.cc:1243] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 24.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260808 01:52:58.488597 17672 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:58.605405 17618 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:58.649998  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:52:58.686138  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 78071us
W20260808 01:52:58.694000  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:58.664162 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 73066.1us
W20260808 01:52:58.756928  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 349807us
I20260808 01:52:58.699890  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
W20260808 01:52:58.765283  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 366293us
I20260808 01:52:58.665922 17725 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election won for term 19
W20260808 01:52:58.795468  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:58.665719 17693 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:58.657747 17700 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:58.816843  9521 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:58.654642 17549 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:58.683732 17695 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:52:58.654944 17615 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:58.822283  9475 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: []
W20260808 01:52:58.800392  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 84629.9us
I20260808 01:52:58.833091 17708 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:58.839079  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:58.840612  9444 leader_election.cc:400] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22.
I20260808 01:52:58.840924  9444 leader_election.cc:403] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:52:58.841761  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:58.846956 17703 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:58.847123 17443 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:52:58.847733 17703 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:58.848423 17703 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:58.816512 17591 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:58.818368 17663 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:58.678330 17618 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:58.823506 17700 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:58.843406 17725 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:52:58.853204 17549 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:58.890475  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:58.894068  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:58.895170  9633 raft_consensus.cc:2376] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22. Re-sending same reply.
I20260808 01:52:58.896692  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:58.892511  9634 raft_consensus.cc:2410] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 25 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260808 01:52:58.903579  9633 raft_consensus.cc:2410] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 21 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260808 01:52:58.655869 17677 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:58.683334 17323 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:58.905084  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:58.909576  9635 raft_consensus.cc:2410] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 21 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260808 01:52:58.910642  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:58.911631  9638 raft_consensus.cc:2376] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23. Re-sending same reply.
W20260808 01:52:58.916601 17671 consensus_meta.cc:287] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.686s	user 0.012s	sys 0.004s
I20260808 01:52:58.917258 17671 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:58.919692  9478 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 15. (term mismatch)
I20260808 01:52:58.920162  9478 pending_rounds.cc:85] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 14
I20260808 01:52:58.920405  9478 pending_rounds.cc:107] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 22.15
I20260808 01:52:58.920630  9478 raft_consensus.cc:2889] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:58.921180 17639 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:58.923326 16835 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:58.905965 17628 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:52:58.881623 17725 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:58.852825 17615 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:58.885533 17650 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22.
I20260808 01:52:58.901883 17549 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:58.929883 17628 raft_consensus.cc:697] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:58.939301  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:52:58.939767  9637 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.678s	user 0.016s	sys 0.008s
W20260808 01:52:58.940116  9637 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.016s	sys 0.008s
I20260808 01:52:58.873224 17663 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:58.905719 17636 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.032s
I20260808 01:52:58.952502  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:58.953524  9629 raft_consensus.cc:2376] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22. Re-sending same reply.
I20260808 01:52:58.959849  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3479) took 1986 ms. Trace:
I20260808 01:52:58.944022 17549 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:58.960625  9480 rpcz_store.cc:276] 0808 01:52:56.973522 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:57.172818 (+199296us) service_pool.cc:224] Handling call
0808 01:52:57.173206 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:57.173497 (+   291us) raft_consensus.cc:1534] Early marking committed up to index 12
0808 01:52:57.173919 (+   422us) log.cc:844] Serialized 14 byte log entry
0808 01:52:57.660601 (+486682us) spinlock_profiling.cc:243] Waited 23.8 ms on lock 0x7b4800211158. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:52:58.655793 (+995192us) spinlock_profiling.cc:243] Waited 5.15 ms on lock 0x7b18002eedc0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667 00007f5d449ae5e8
0808 01:52:58.950525 (+294732us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:58.950630 (+   105us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:52:58.950725 (+    95us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:58.959340 (+  8615us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:58.959734 (+   394us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":158,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":4306,"raft.queue_time_us":59335,"raft.run_cpu_time_us":4970,"raft.run_wall_time_us":417408,"spinlock_wait_cycles":66637952,"thread_start_us":280196,"threads_started":1}
W20260808 01:52:58.984747  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 334874us
I20260808 01:52:58.990521  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:58.991588  9629 raft_consensus.cc:2376] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21. Re-sending same reply.
I20260808 01:52:58.992734  9599 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:58.994391  9599 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:58.942111 17398 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.045s
I20260808 01:52:59.005689  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:58.949911 17720 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.045s
I20260808 01:52:58.963130 17443 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:52:59.010901 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 250395us
W20260808 01:52:59.018975  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:52:59.023747  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 182077us
I20260808 01:52:59.025902  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:59.026785  9629 raft_consensus.cc:2376] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20. Re-sending same reply.
I20260808 01:52:59.030750  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:59.040262  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:58.952981 17693 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.050139  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:59.054227  9629 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:59.052510 17663 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:59.031790 17661 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:52:59.052119  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:52:59.056185  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 61841.2us
I20260808 01:52:59.060336  9635 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:52:59.060264  9634 raft_consensus.cc:2393] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 20: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:59.061013  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:59.037056 17639 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:59.017781 17443 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.077111 17543 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:52:59.050261 17372 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:59.078052 17663 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:59.079068 17443 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:59.068586  9638 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:52:59.075804 17618 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:59.078536 17372 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.078712 17663 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:59.073318  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:59.084681  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:59.057971 17661 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:59.107331  9635 raft_consensus.cc:2376] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22. Re-sending same reply.
I20260808 01:52:59.108908  9633 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:52:59.110265 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:59.112149  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:59.032353 17628 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.067128 17756 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.070s
I20260808 01:52:59.115880  9629 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
W20260808 01:52:59.146649  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:52:59.081269 17650 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:59.131944  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 141362us
I20260808 01:52:59.081593 17636 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:59.184072  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:59.105513 17671 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:59.099797 17543 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:52:59.116974 17618 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260808 01:52:59.127365 17749 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260808 01:52:59.145601 17723 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:59.146891 17615 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:59.204656 17723 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:59.187031 17763 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:59.206660 17763 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.186250 17650 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:59.195851 17701 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:59.209302 17663 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:59.201417  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:52:59.210216  9629 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:59.209976 17701 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:59.202653  9638 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260808 01:52:59.212175  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:59.212334 17701 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:59.212841 17708 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:59.167815 17372 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:59.208318 17671 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:52:59.208479 17447 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:59.209180 17763 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:59.240046  9559 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 21 index: 18. (index mismatch)
I20260808 01:52:59.209409 17756 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:52:59.207707 17723 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:59.185498  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 73564.4us
I20260808 01:52:59.213007  9638 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 23: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:52:59.218392  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3493) took 1022 ms. Trace:
I20260808 01:52:59.248484  9481 rpcz_store.cc:276] 0808 01:52:58.195994 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:58.304936 (+108942us) service_pool.cc:224] Handling call
0808 01:52:58.305216 (+   280us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:58.361373 (+ 56157us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:58.361428 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:58.387750 (+ 26322us) log.cc:844] Serialized 26 byte log entry
0808 01:52:58.867712 (+479962us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:52:58.867764 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:58.867845 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:59.218090 (+350245us) raft_consensus.cc:1685] finished
0808 01:52:59.218115 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:59.218320 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55275,"raft.queue_time_us":72053,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":3427,"thread_start_us":449115,"threads_started":1,"wal-append.queue_time_us":590036}
I20260808 01:52:59.221459  9629 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:52:59.185581  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:59.221932  9521 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:59.215482 17701 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:52:59.207821 17748 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260808 01:52:59.145608 17285 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:59.213913 17700 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.232235  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:59.243286  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:59.258939 17723 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:52:59.266330  9552 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:59.241096 17663 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:52:59.268968  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:59.272368  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:59.277977  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:59.278316  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:59.260491 17653 consensus_meta.cc:287] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.758s	user 0.013s	sys 0.003s
I20260808 01:52:59.283531  9521 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:59.277429  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:59.285630  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 53516.1us
I20260808 01:52:59.285739  9556 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:59.289278  9599 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:59.218477 17703 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.292335  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:59.293378  9559 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:59.294534  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:52:59.276742 17756 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:59.246326 17447 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:59.310279  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:59.194550 17636 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:59.331668  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:59.184924 17661 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.302197 17639 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:52:59.347664  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:59.303428 17758 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:59.292208 17723 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:52:59.322017 17457 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.648s	user 0.012s	sys 0.003s
I20260808 01:52:59.268779 17285 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.300678 17615 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:52:59.292985 17653 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.334363  9559 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:52:59.279323 17701 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:59.351058 17639 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:59.292706 17663 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:59.323266 17763 consensus_queue.cc:1059] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
I20260808 01:52:59.336309 17372 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.034s
I20260808 01:52:59.345521 17398 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.021s
I20260808 01:52:59.336555 17765 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.065s
I20260808 01:52:59.385231  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:59.386696  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:59.339722 17764 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.077s
I20260808 01:52:59.362411 17770 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:52:59.380399 17639 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:59.343154 17761 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.078s
I20260808 01:52:59.343515 17756 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.364581 17650 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:59.386534 17661 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:52:59.362879 17457 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.391567 17774 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:52:59.406781 17646 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:52:59.403097 17770 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:52:59.364104 17758 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.433930  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:52:59.434986  9479 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:52:59.420809 17650 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:52:59.448000  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2910) took 2129 ms. Trace:
I20260808 01:52:59.426988 17779 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:59.448501  9551 rpcz_store.cc:276] 0808 01:52:57.311813 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:57.312025 (+   212us) service_pool.cc:224] Handling call
0808 01:52:57.312466 (+   441us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:57.632515 (+320049us) spinlock_profiling.cc:243] Waited 6.12 ms on lock 0x7b18004a2360. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4453a6ac 00007f5d4186fe3b 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d41889e05 00007f5d4498b440 00007f5d4498bfdb 00007f5d44996a2b
0808 01:52:57.632666 (+   151us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:57.632693 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:57.688260 (+ 55567us) log.cc:844] Serialized 26 byte log entry
0808 01:52:57.988496 (+300236us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:52:58.019626 (+ 31130us) log.cc:844] Serialized 14 byte log entry
0808 01:52:58.026024 (+  6398us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:59.440793 (+1414769us) spinlock_profiling.cc:243] Waited 51 ms on lock 0x7b540029e0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:52:59.440872 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:59.440928 (+    56us) raft_consensus.cc:1685] finished
0808 01:52:59.440944 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:59.441110 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31430,"prepare.queue_time_us":27754,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"raft.queue_time_us":17879,"raft.run_cpu_time_us":368,"raft.run_wall_time_us":25285,"spinlock_wait_cycles":131379200,"thread_start_us":215239,"threads_started":2,"wal-append.queue_time_us":312454,"wal-append.run_cpu_time_us":2327,"wal-append.run_wall_time_us":1004386}
I20260808 01:52:59.437753 17700 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:59.471496 17703 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:59.429606 17770 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:52:59.473767 17779 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.503602 17756 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:59.504601 17639 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.515301  9479 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:52:59.516338  9521 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:59.517308  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:59.518043  9555 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
W20260808 01:52:59.492731 17755 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.183s	user 0.002s	sys 0.000s
I20260808 01:52:59.494627  9556 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:52:59.495043 17701 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:52:59.496488 17776 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.131s	user 0.002s	sys 0.000s
I20260808 01:52:59.496764  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:59.498319 17700 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:52:59.498661 17457 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:59.499657 17663 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:59.427201 17723 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:52:59.494454 17549 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.501s	user 0.009s	sys 0.004s
I20260808 01:52:59.562494 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:52:59.528888 17409 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:52:59.566552 17774 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:59.578670 17653 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:59.587635 17285 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:59.618825 17778 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.154s	user 0.002s	sys 0.000s
I20260808 01:52:59.585783 17549 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:59.628162 17549 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:59.601337 17756 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:52:59.632740  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 247674us
I20260808 01:52:59.639545  9630 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20260808 01:52:59.640196  9630 pending_rounds.cc:85] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 11
I20260808 01:52:59.640496  9630 pending_rounds.cc:107] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 21.12
I20260808 01:52:59.640780  9630 raft_consensus.cc:2889] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:59.649708  9523 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:52:59.688946 17708 raft_consensus.cc:3055] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:52:59.689693 17708 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:52:59.692059 17639 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:52:59.697551 17772 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.337s	user 0.002s	sys 0.000s
I20260808 01:52:59.646984 17671 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:59.715524 17779 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:59.719151 17671 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.719748 17775 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:59.720410 17646 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:59.720422 17775 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.721201 17646 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:59.724298 17775 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:59.725147 17802 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:59.725822 17802 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.602037 17774 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.727909 17646 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:52:59.682654 17358 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:52:59.728979  9476 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: []
I20260808 01:52:59.730168  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:59.731746  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:59.732498  9476 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:52:59.733330  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:59.733805  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:59.734369  9480 raft_consensus.cc:2410] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 24 because replica has last-logged OpId of term: 23 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260808 01:52:59.735476  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:59.743616  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:59.749819  9476 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: []
W20260808 01:52:59.756649  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 108154us
I20260808 01:52:59.758765  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:59.759778  9473 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:52:59.648229 17756 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.795989  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:59.797035  9479 raft_consensus.cc:2410] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 21 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260808 01:52:59.808959  9473 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: []
I20260808 01:52:59.811383  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:59.812299  9557 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:59.742405 17774 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:52:59.758008 17358 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:52:59.819846  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:52:59.819942  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:59.824487  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:59.796859 17639 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:59.827153  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:59.832185  9555 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:52:59.748319 17775 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.069s
I20260808 01:52:59.827839  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:59.830389  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:59.796782 17548 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:52:59.854053  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3498) took 1011 ms. Trace:
I20260808 01:52:59.854560  9475 rpcz_store.cc:276] 0808 01:52:58.842906 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:58.843047 (+   141us) service_pool.cc:224] Handling call
0808 01:52:58.843409 (+   362us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:59.033180 (+189771us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:59.033619 (+   439us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:59.048211 (+ 14592us) log.cc:844] Serialized 26 byte log entry
0808 01:52:59.228768 (+180557us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:59.228827 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:59.228932 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:59.697143 (+468211us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b2400013b08. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41713594 00007f5d4171169b 00007f5d418cd6d9 00007f5d418cd365 00007f5d418cc9ad 00007f5d445162fe 00007f5d4452da7c 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411
0808 01:52:59.853636 (+156493us) raft_consensus.cc:1685] finished
0808 01:52:59.853659 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:59.853955 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":139991,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":2712,"spinlock_wait_cycles":36739456,"thread_start_us":343400,"threads_started":2,"wal-append.queue_time_us":257374}
I20260808 01:52:59.872028  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:52:59.875594  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 55869.4us
I20260808 01:52:59.828075 17592 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:59.843639 17646 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:59.886152  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:52:59.831262 17761 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.024s
I20260808 01:52:59.885119  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3059) took 2139 ms. Trace:
I20260808 01:52:59.754016 17708 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:52:59.873355 17752 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.677s	user 0.003s	sys 0.000s
I20260808 01:52:59.895706  9631 rpcz_store.cc:276] 0808 01:52:57.733309 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:57.740057 (+  6748us) service_pool.cc:224] Handling call
0808 01:52:57.740595 (+   538us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:57.757669 (+ 17074us) raft_consensus.cc:1534] Early marking committed up to index 16
0808 01:52:57.803833 (+ 46164us) log.cc:844] Serialized 14 byte log entry
0808 01:52:57.941981 (+138148us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:57.975977 (+ 33996us) log.cc:844] Serialized 26 byte log entry
0808 01:52:57.976185 (+   208us) raft_consensus.cc:1645] Marking committed up to 17
0808 01:52:59.591008 (+1614823us) log.cc:844] Serialized 14 byte log entry
0808 01:52:59.872552 (+281544us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:59.872819 (+   267us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:52:59.872867 (+    48us) raft_consensus.cc:1685] finished
0808 01:52:59.872879 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:52:59.873020 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1631880,"prepare.queue_time_us":1703796,"prepare.run_cpu_time_us":140,"prepare.run_wall_time_us":3969,"raft.queue_time_us":16975,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":280,"thread_start_us":355480,"threads_started":2,"wal-append.queue_time_us":182511,"wal-append.run_cpu_time_us":2274,"wal-append.run_wall_time_us":1003975}
I20260808 01:52:59.910753  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:59.911721  9555 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 15.
I20260808 01:52:59.912021  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:52:59.876760 17556 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:52:59.895905  9554 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.020s	sys 0.032s
W20260808 01:52:59.917366  9554 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.020s	sys 0.032s
I20260808 01:52:59.861907 17358 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:59.898633  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:59.888013 17790 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:59.876418 17653 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:52:59.891839  9557 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:52:59.844425 17639 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:59.831404 17774 consensus_queue.cc:1059] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.053s
I20260808 01:52:59.898355 17700 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:52:59.870090 17548 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:59.910766 17763 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:59.915858 17710 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:52:59.893682 17765 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.041s
I20260808 01:52:59.950157  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:52:59.951243  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 187272us
I20260808 01:52:59.924701 17693 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:52:59.955631  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:52:59.903214 17646 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:59.973202  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:52:59.977356  9557 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:52:59.987038  9478 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21. Ops: []
I20260808 01:52:59.898613 17592 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:52:59.962216 17700 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:52:59.939922 17796 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:52:59.936949 17358 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:00.011261  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:00.017545 17805 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
I20260808 01:52:59.938614 17790 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:52:59.989987 17745 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:00.033257  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:00.035373  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:00.071722  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 980) took 3316 ms. Trace:
I20260808 01:53:00.005144 17616 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:00.072585  9636 rpcz_store.cc:276] 0808 01:52:56.755436 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:57.503604 (+748168us) service_pool.cc:224] Handling call
0808 01:52:57.530377 (+ 26773us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:57.583012 (+ 52635us) raft_consensus.cc:1534] Early marking committed up to index 17
0808 01:52:57.583523 (+   511us) log.cc:844] Serialized 14 byte log entry
0808 01:52:57.932691 (+349168us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:57.970525 (+ 37834us) log.cc:844] Serialized 26 byte log entry
0808 01:52:57.970931 (+   406us) raft_consensus.cc:1645] Marking committed up to 18
0808 01:52:59.590788 (+1619857us) log.cc:844] Serialized 14 byte log entry
0808 01:53:00.045904 (+455116us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:00.046045 (+   141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:00.071378 (+ 25333us) raft_consensus.cc:1685] finished
0808 01:53:00.071432 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:00.071640 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1675215,"prepare.queue_time_us":1632628,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":93,"raft.queue_time_us":245805,"raft.run_cpu_time_us":504,"raft.run_wall_time_us":19592,"thread_start_us":675705,"threads_started":3,"wal-append.queue_time_us":645329,"wal-append.run_cpu_time_us":2888,"wal-append.run_wall_time_us":1068568}
I20260808 01:53:00.109838  9634 pending_rounds.cc:85] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 12
I20260808 01:53:00.110246  9634 pending_rounds.cc:107] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 20.13
I20260808 01:53:00.110531  9634 raft_consensus.cc:2889] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:00.076074  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:52:59.963038 17763 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:00.019762 17802 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:00.124004  9636 pending_rounds.cc:85] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 12
I20260808 01:53:00.124478  9636 pending_rounds.cc:107] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 23.13
I20260808 01:53:00.124753  9636 raft_consensus.cc:2889] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:52:59.998534 17800 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:00.075132 17752 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260808 01:53:00.029764 17798 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:52:59.985375 17710 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:00.134982  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3317) took 1012 ms. Trace:
I20260808 01:53:00.032310 17548 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:00.105273 17726 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:00.151366  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:00.011456 17770 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:00.119438 17616 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:00.163895  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:00.164127  9599 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:00.170099  9476 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260808 01:53:00.172309  9557 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:00.182372  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:00.139086 17618 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:00.195091 17822 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
W20260808 01:53:00.196416 17809 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
W20260808 01:53:00.196882 17806 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.120s	user 0.000s	sys 0.000s
W20260808 01:53:00.200138 17817 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
W20260808 01:53:00.201745 17819 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.242s	user 0.000s	sys 0.001s
I20260808 01:53:00.163434 17700 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:00.219524  9477 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 18. Current term is 19. Ops: []
I20260808 01:53:00.020434 17796 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:00.205241  9633 pending_rounds.cc:85] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 13
W20260808 01:53:00.223922  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 59881.8us
I20260808 01:53:00.227281  9633 pending_rounds.cc:107] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260808 01:53:00.227713  9633 raft_consensus.cc:2889] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:00.233399 10367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 897) took 2048 ms. Trace:
I20260808 01:53:00.233973 10367 rpcz_store.cc:276] 0808 01:52:58.171212 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:58.171380 (+   168us) service_pool.cc:224] Handling call
0808 01:52:58.171792 (+   412us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:58.233931 (+ 62139us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:58.233966 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:52:58.234018 (+    52us) raft_consensus.cc:1645] Marking committed up to 17
0808 01:52:58.234054 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:00.219464 (+1985410us) spinlock_profiling.cc:243] Waited 61.9 ms on lock 0x7b5400690354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:00.219515 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:00.219686 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":142344448}
I20260808 01:53:00.234889 17790 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:00.227703  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:00.240475  9629 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260808 01:53:00.241291  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3328) took 1037 ms. Trace:
I20260808 01:53:00.241703  9629 rpcz_store.cc:276] 0808 01:52:59.203668 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:00.239697 (+1036029us) service_pool.cc:224] Handling call
0808 01:53:00.240084 (+   387us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:00.240957 (+   873us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:00.241212 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:53:00.244391  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:37234 (request call id 16) took 2695 ms. Trace:
I20260808 01:53:00.245042  9472 rpcz_store.cc:276] 0808 01:52:57.530049 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:57.638255 (+108206us) service_pool.cc:224] Handling call
0808 01:53:00.224644 (+2586389us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400392bd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44999b15 00007f5d4998bb39 00007f5d42ed8334 00007f5d42ed82c2 00007f5d42ed81f1 00007f5d42ed816c 00007f5d42ed7422 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:00.224877 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3018056960}
I20260808 01:53:00.078771 17745 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:00.192545 17799 raft_consensus.cc:3055] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:53:00.221539 17756 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:00.267333 17661 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.845s	user 0.010s	sys 0.005s
I20260808 01:53:00.160167 17798 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:00.191866 17548 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:53:00.281448  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 53845.9us
I20260808 01:53:00.304829  9558 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:00.306133  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:00.307117  9472 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 25.
I20260808 01:53:00.308847 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:00.325050  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:00.326462  9557 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:53:00.271587 17799 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:00.273022 17547 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:00.280113 17777 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:00.203815 17618 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:00.163677 17800 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:00.352380 17821 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:53:00.352484 10365 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:53:00.353477 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:00.353564 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:00.368193 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:00.304376 17639 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:00.351358 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:00.320688 17756 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:00.395484  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 991) took 2143 ms. Trace:
I20260808 01:53:00.461550 17710 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:00.323604 17672 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:00.413156  9550 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:00.317432 17548 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:00.425230  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:00.373191 17646 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:00.397294 17592 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:00.414227 17828 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:00.554028 17763 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:00.367010 17547 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:00.424441 17765 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:00.367884 17777 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:00.440759 17774 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:00.540987 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:00.547132 17832 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:00.317009 17746 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:00.563794 17672 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:00.557165  9632 rpcz_store.cc:276] 0808 01:52:58.251812 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:58.262116 (+ 10304us) service_pool.cc:224] Handling call
0808 01:52:58.296399 (+ 34283us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:58.297006 (+   607us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:58.297053 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:58.297722 (+   669us) log.cc:844] Serialized 26 byte log entry
0808 01:52:58.431164 (+133442us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:52:59.796068 (+1364904us) log.cc:844] Serialized 14 byte log entry
0808 01:53:00.395156 (+599088us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:00.395233 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:00.395284 (+    51us) raft_consensus.cc:1685] finished
0808 01:53:00.395294 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:00.395420 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1263760,"prepare.queue_time_us":1215932,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":239593,"raft.run_cpu_time_us":394,"raft.run_wall_time_us":39502,"thread_start_us":817318,"threads_started":4,"wal-append.queue_time_us":900441,"wal-append.run_cpu_time_us":3036,"wal-append.run_wall_time_us":1021870}
I20260808 01:53:00.574018 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:00.432185 17723 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.003s	sys 0.027s
I20260808 01:53:00.551311 17358 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:00.556926 17756 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:00.558777 17770 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:00.589262  9558 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:53:00.604388 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:00.550321 17830 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:00.615890  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 190770us
I20260808 01:53:00.467001 17799 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:00.585549 17674 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:00.583860 17828 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:00.616451 17763 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:00.646133  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3330) took 1441 ms. Trace:
W20260808 01:53:00.619810 17723 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.004s	sys 0.028s
I20260808 01:53:00.651705  9601 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:00.656188  9601 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:00.621007 17672 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:00.623718 17821 raft_consensus.cc:697] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:00.627794 17820 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:00.628317 17756 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:00.653695  9632 rpcz_store.cc:276] 0808 01:52:59.204233 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:00.626950 (+1422717us) service_pool.cc:224] Handling call
0808 01:53:00.645328 (+ 18378us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:00.645607 (+   279us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:00.645645 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:00.645711 (+    66us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:53:00.645757 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:00.645861 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:00.646031 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:53:00.684502  9430 thread.cc:641] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.016s	sys 0.016s
W20260808 01:53:00.667636  9479 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.632s	user 0.025s	sys 0.070s
W20260808 01:53:00.716697  9479 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.025s	sys 0.075s
I20260808 01:53:00.612600 17774 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:00.668749 17839 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:00.603431 17765 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:00.627358 17798 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:00.647895 17548 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:00.649678 17800 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:00.627981 17639 consensus_queue.cc:1059] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.074s
I20260808 01:53:00.618021 17832 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:00.721256 10369 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: []
I20260808 01:53:00.662601 17646 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260808 01:53:00.752099 17836 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260808 01:53:00.755892 17547 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:00.620553 17746 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:00.758255 17821 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:00.760654 17739 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 2.201s	user 0.006s	sys 0.000s
I20260808 01:53:00.794118 17677 raft_consensus.cc:3055] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:53:00.794636 17677 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:00.795636 17677 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:00.796973 17677 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:00.799203 10366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 895) took 2629 ms. Trace:
I20260808 01:53:00.800323 17774 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:00.802243 17816 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.783s	user 0.000s	sys 0.003s
I20260808 01:53:00.766883 17618 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:00.802632 17592 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:00.769963 17828 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:00.728217 17833 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:00.659469 17674 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:00.807826 17848 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:00.808547 17674 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:00.808502 17848 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:00.808848 17833 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:00.780635 17693 consensus_meta.cc:287] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.794s	user 0.010s	sys 0.010s
I20260808 01:53:00.780884  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:00.781738  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:00.781986  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:00.728116 17820 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:00.766883 17834 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:00.789947 17765 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:00.790118 17703 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:53:00.790279 17746 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:00.790743  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:00.799670 10366 rpcz_store.cc:276] 0808 01:52:58.170295 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:58.173072 (+  2777us) service_pool.cc:224] Handling call
0808 01:52:58.173557 (+   485us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:58.174057 (+   500us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:58.174107 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:58.174785 (+   678us) log.cc:844] Serialized 26 byte log entry
0808 01:52:58.295530 (+120745us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:52:58.295593 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:58.295697 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:00.717802 (+2422105us) spinlock_profiling.cc:243] Waited 32.5 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:00.794586 (+ 76784us) raft_consensus.cc:1685] finished
0808 01:53:00.794610 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:00.799107 (+  4497us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3085,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":98,"spinlock_wait_cycles":74657024,"thread_start_us":120195,"threads_started":1,"wal-append.queue_time_us":383917}
I20260808 01:53:00.643860 17830 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:00.728670 17745 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:00.763110 17777 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:00.664228 17763 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:00.718773 17844 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260808 01:53:00.835026  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 476724us
W20260808 01:53:00.814456 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 210297us
I20260808 01:53:00.820601  9472 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 20: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:00.813210 17820 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:00.812108 17674 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:00.830260  9473 raft_consensus.cc:2410] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 23 because replica has last-logged OpId of term: 22 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260808 01:53:00.821122 17693 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260808 01:53:00.716650  9430 thread.cc:608] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.016s	sys 0.016s
I20260808 01:53:00.827252 17703 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:00.841511  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:00.810458 17592 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:00.857208  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:00.857668  9559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3822) took 1570 ms. Trace:
I20260808 01:53:00.858371  9559 rpcz_store.cc:276] 0808 01:52:59.281530 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:59.373114 (+ 91584us) service_pool.cc:224] Handling call
0808 01:52:59.391621 (+ 18507us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:59.636042 (+244421us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:59.636094 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:59.645466 (+  9372us) log.cc:844] Serialized 26 byte log entry
0808 01:52:59.918224 (+272758us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:59.918259 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:52:59.918403 (+   144us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:00.801455 (+883052us) spinlock_profiling.cc:243] Waited 1.88 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:00.842013 (+ 40558us) raft_consensus.cc:1685] finished
0808 01:53:00.842038 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:00.852266 (+ 10228us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2870,"raft.queue_time_us":28632,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":4436,"spinlock_wait_cycles":4330368,"thread_start_us":495681,"threads_started":2,"wal-append.queue_time_us":356331}
I20260808 01:53:00.835582 17844 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:00.872252 10366 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 21 index: 18. (index mismatch)
I20260808 01:53:00.875293  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:00.827463 17834 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:00.878469 17548 consensus_queue.cc:1059] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:00.879693 17763 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:00.879923 17834 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:00.880163 17548 raft_consensus.cc:3055] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:53:00.880698 17854 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:00.880831 17548 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:00.881691 17820 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:00.883440 17592 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:00.870882 17833 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:00.884838 17746 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:00.885391 17830 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:00.887455 17768 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:53:00.888438 17768 consensus_queue.cc:1243] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 21.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260808 01:53:00.890910 17767 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:00.891355 17767 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:00.881350 17854 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:00.892310 17767 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:00.874220 17844 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:53:00.893829 17767 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:00.894990 17703 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:00.896252 17703 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:00.899998 17746 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:00.887570  9524 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:00.903448 17852 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260808 01:53:00.851999 17674 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:00.908164  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:00.881836 17548 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:00.911129 17548 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:00.911163  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:00.912439  9473 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 21: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:00.913486 17768 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:00.915812  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:00.916345 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 45108.4us
I20260808 01:53:00.917013  9480 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:53:00.917557  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:00.918244  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:00.919401  9480 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
W20260808 01:53:00.920794  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:00.914345 17796 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:00.925930  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:00.927971  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:00.931052  9473 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 21: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:00.927063 17674 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:00.884064 17636 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:00.931501 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 125629us
I20260808 01:53:00.914453 17768 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.021s
I20260808 01:53:00.923645 17854 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:00.974013  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:00.982806  9475 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
W20260808 01:53:01.005815 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 77945.8us
I20260808 01:53:00.981065 17714 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:53:01.002957 17725 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.806s	user 0.006s	sys 0.027s
I20260808 01:53:01.017624  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:01.031044 17725 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.834s	user 0.006s	sys 0.027s
I20260808 01:53:01.037112  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:01.039873 17869 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:01.043191 17869 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:01.043571 17869 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:01.041193 17849 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:01.047458 17849 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.052754  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3331) took 1848 ms. Trace:
I20260808 01:53:01.025624 17672 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:01.055090 10365 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 26: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:01.050657 17849 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:01.057024 17672 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:01.053388  9632 rpcz_store.cc:276] 0808 01:52:59.204487 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:00.747169 (+1542682us) service_pool.cc:224] Handling call
0808 01:53:00.779988 (+ 32819us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:00.787408 (+  7420us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:00.787456 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:00.792236 (+  4780us) spinlock_profiling.cc:243] Waited 3.16 ms on lock 0x7b2c000b67f8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:53:00.792715 (+   479us) log.cc:844] Serialized 26 byte log entry
0808 01:53:00.934007 (+141292us) raft_consensus.cc:1645] Marking committed up to 10
0808 01:53:00.934043 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:00.934135 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:01.052348 (+118213us) raft_consensus.cc:1685] finished
0808 01:53:01.052373 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:01.052666 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6790,"raft.queue_time_us":7461,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"spinlock_wait_cycles":7273344,"thread_start_us":140899,"threads_started":1,"wal-append.queue_time_us":189713}
I20260808 01:53:01.060294  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 997) took 1831 ms. Trace:
I20260808 01:53:01.060678  9632 rpcz_store.cc:276] 0808 01:52:59.229183 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:01.059269 (+1830086us) service_pool.cc:224] Handling call
0808 01:53:01.059709 (+   440us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:01.059926 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:01.059955 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:01.060005 (+    50us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:53:01.060039 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:01.060100 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:01.060229 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:53:01.062428  9632 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260808 01:53:01.063720  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 998) took 1834 ms. Trace:
I20260808 01:53:01.064158  9632 rpcz_store.cc:276] 0808 01:52:59.229544 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:01.061659 (+1832115us) service_pool.cc:224] Handling call
0808 01:53:01.062056 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:01.063269 (+  1213us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:01.063498 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:53:01.065860  9632 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260808 01:53:01.066737  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3333) took 1836 ms. Trace:
I20260808 01:53:01.067018  9632 rpcz_store.cc:276] 0808 01:52:59.230420 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:01.065101 (+1834681us) service_pool.cc:224] Handling call
0808 01:53:01.065472 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:01.066435 (+   963us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:01.066651 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:53:01.013968 17714 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:01.069887 17547 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260808 01:53:01.070411 17616 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.867s	user 0.024s	sys 0.015s
I20260808 01:53:01.071106 17616 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:01.073727 17714 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:01.078994 17616 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:01.085601 17771 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 1.881s	user 0.006s	sys 0.015s
W20260808 01:53:01.086650 17841 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.362s	user 0.000s	sys 0.001s
I20260808 01:53:01.089509 17869 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:01.092123 17869 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:01.094343 10369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 992) took 1585 ms. Trace:
I20260808 01:53:01.095017 10369 rpcz_store.cc:276] 0808 01:52:59.509442 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:59.509718 (+   276us) service_pool.cc:224] Handling call
0808 01:52:59.515116 (+  5398us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:00.773308 (+1258192us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:01.094008 (+320700us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400594cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:01.094250 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2773333760}
I20260808 01:53:01.096248 17756 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.097474 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:01.098210 10369 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:01.070472 17547 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:01.099357 17547 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:01.105125 17843 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.348s	user 0.002s	sys 0.000s
I20260808 01:53:01.112922 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:01.113272 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:01.114114 10360 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:53:01.114800 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:01.115213 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:01.115731 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:01.117364 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:01.119860 17844 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.122588 17828 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:01.122982  9632 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:53:01.123494  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3517) took 1049 ms. Trace:
I20260808 01:53:01.123265 17828 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.124269  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3335) took 1862 ms. Trace:
I20260808 01:53:01.124610 17833 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:01.125659 17828 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:01.126293 17828 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:01.126883 17834 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:01.127269 17828 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:01.127446 17834 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.129576 17828 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:01.129940 17834 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:01.122634 17844 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:01.131309  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3296) took 3059 ms. Trace:
I20260808 01:53:01.131814  9630 rpcz_store.cc:276] 0808 01:52:58.071857 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:58.072245 (+   388us) service_pool.cc:224] Handling call
0808 01:52:58.072580 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:52:59.641405 (+1568825us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:01.130994 (+1489589us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b54004213d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:01.131238 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3602476928}
I20260808 01:53:01.133316  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:01.134083  9630 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:01.134580  9600 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:01.135958 17844 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:53:01.136512  9600 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:01.137611 17844 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:53:01.138594 17756 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:01.138535  9600 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:01.140434 17756 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:01.141023 17756 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:01.141532 17756 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:01.126210 17820 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:01.123901  9478 rpcz_store.cc:276] 0808 01:53:00.074635 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:00.074854 (+   219us) service_pool.cc:224] Handling call
0808 01:53:00.075204 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:00.158699 (+ 83495us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:00.158753 (+    54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:00.158815 (+    62us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:53:00.158862 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:01.123246 (+964384us) spinlock_profiling.cc:243] Waited 83.2 ms on lock 0x7b54002bb6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:01.123298 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:01.123426 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":191287040}
I20260808 01:53:01.125298 17833 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.151158  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:01.151890  9630 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:01.124843  9632 rpcz_store.cc:276] 0808 01:52:59.262140 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:01.067775 (+1805635us) service_pool.cc:224] Handling call
0808 01:53:01.068158 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:01.123417 (+ 55259us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:01.123993 (+   576us) spinlock_profiling.cc:243] Waited 54.6 ms on lock 0x7b540027ef54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:01.124206 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":125609856}
I20260808 01:53:01.153275 10369 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:00.152705  9635 rpcz_store.cc:276] 0808 01:52:59.122533 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:59.138951 (+ 16418us) service_pool.cc:224] Handling call
0808 01:52:59.139521 (+   570us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:59.265212 (+125691us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:59.265244 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:59.285645 (+ 20401us) log.cc:844] Serialized 52 byte log entry
0808 01:52:59.396983 (+111338us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:59.857297 (+460314us) log.cc:844] Serialized 14 byte log entry
0808 01:53:00.078215 (+220918us) log.cc:844] Serialized 14 byte log entry
0808 01:53:00.134421 (+ 56206us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:00.134539 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:00.134608 (+    69us) raft_consensus.cc:1685] finished
0808 01:53:00.134627 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:00.134877 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":389806,"prepare.queue_time_us":354970,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":6166,"raft.queue_time_us":244439,"raft.run_cpu_time_us":353,"raft.run_wall_time_us":81612,"thread_start_us":557687,"threads_started":4,"wal-append.queue_time_us":134890}
I20260808 01:53:01.156618  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:01.158315 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:01.160171  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:01.161306  9635 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 22: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:01.162201  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:01.151427 17837 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:01.165113 17760 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.990s	user 0.008s	sys 0.015s
I20260808 01:53:01.165001 17837 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.167490 17837 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:01.172750 17447 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:01.163741  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:01.163820  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:01.157513 17672 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:01.150959 17820 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:01.182147  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:01.178565 17767 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:01.177166  9630 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:53:01.184391  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:01.185070  9630 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:53:01.192996  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:01.196151 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:01.200906  9550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2922) took 2017 ms (client timeout 1854 ms). Trace:
W20260808 01:53:01.201532  9550 rpcz_store.cc:269] 0808 01:52:59.183194 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:59.183795 (+   601us) service_pool.cc:224] Handling call
0808 01:53:01.200569 (+2016774us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540029f4d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:01.200762 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2543167872}
I20260808 01:53:01.183178 17672 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:01.180274 17758 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:01.182375 17798 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:01.301313 17372 raft_consensus.cc:3055] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:53:01.301870 17372 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:01.283586  9630 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:53:01.302893 17372 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.303534 17798 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.304338  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:01.305047  9630 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:01.306341 17708 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:01.306805 17798 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:01.311810 17547 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:01.299643  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:01.313695  9635 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:01.320755 17833 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:01.187525 17820 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:01.188812 17767 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:01.325363  9630 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:01.327512  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:01.331532  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1352) took 1876 ms. Trace:
I20260808 01:53:01.331892 17756 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:01.184942 17793 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:01.335469 17797 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:01.335276 17793 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.336275 17797 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:01.282518 17676 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:01.339401 17676 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.326181  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:01.345148  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:01.326076 17767 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:01.344452  9632 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:53:01.332000  9477 rpcz_store.cc:276] 0808 01:52:59.454831 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:59.455346 (+   515us) service_pool.cc:224] Handling call
0808 01:52:59.494858 (+ 39512us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:00.220176 (+725318us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:01.331231 (+1111055us) spinlock_profiling.cc:243] Waited 724 ms on lock 0x7b54002bacd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:01.331442 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1666046336}
W20260808 01:53:01.350652 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50809us
I20260808 01:53:01.318995  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:01.352000  9635 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:01.342482 17676 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:01.357491  9599 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.194:36095 timed out after 1.638s (SENT) [suppressed 19 similar messages]
I20260808 01:53:01.370918  9475 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:01.371714  9475 raft_consensus.cc:606] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:01.372085  9475 raft_consensus.cc:614] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:01.380843  9635 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260808 01:53:01.338620 17793 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:01.382206  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3337) took 1768 ms. Trace:
W20260808 01:53:01.368335 17700 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.026s	sys 0.028s
I20260808 01:53:01.382983  9635 rpcz_store.cc:276] 0808 01:52:59.614032 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:01.352701 (+1738669us) service_pool.cc:224] Handling call
0808 01:53:01.380313 (+ 27612us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:01.381868 (+  1555us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:01.382129 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:53:01.389261 17769 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.075s	user 0.012s	sys 0.002s
I20260808 01:53:01.389475 17756 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:01.389735 17769 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.183s	user 0.013s	sys 0.002s
I20260808 01:53:01.390782 17672 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:01.387518 17796 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:01.391860  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 2973a134fcef4b8da54331bcd12f1173 (from 6e7856bed1be43acaa821290d506ee97 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:01.396584 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:01.401211 10365 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:01.379739 17761 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:01.393678 17700 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.026s	sys 0.028s
I20260808 01:53:01.422367  9635 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260808 01:53:01.422710  9475 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3759209338e14437aff5f92a2fff36da"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:01.423298  9475 raft_consensus.cc:606] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:01.423344  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3339) took 1708 ms. Trace:
I20260808 01:53:01.399962 17547 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:01.424373  9635 rpcz_store.cc:276] 0808 01:52:59.715452 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:01.398972 (+1683520us) service_pool.cc:224] Handling call
0808 01:53:01.421684 (+ 22712us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:01.423051 (+  1367us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:01.423285 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:53:01.460891  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:01.430742 17671 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:01.434774  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 460e1059f5504707b891c4b0e0ac6a09: Illegal state: Replica 6e7856bed1be43acaa821290d506ee97 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 22 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:01.440488 10362 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:01.344787  9366 catalog_manager.cc:5697] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 17 to 19, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 19 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:01.449764 10365 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:01.455056 10369 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
mode: GRACEFUL
new_leader_uuid: "6e7856bed1be43acaa821290d506ee97"
 from {username='slave'} at 127.0.0.1:57156
I20260808 01:53:01.466388 10369 raft_consensus.cc:606] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Received request to transfer leadership to TS 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:01.394146 17796 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.469249  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:01.468107  9600 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:01.469911  9635 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:01.471489  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:01.479210  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:01.407053 17761 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:01.465325 17764 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.049s	user 0.049s	sys 0.035s
I20260808 01:53:01.486063  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:01.484767  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 127300us
I20260808 01:53:01.478302 17758 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:53:01.487918  9635 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:01.497735  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:01.489578 17764 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.076s	user 0.049s	sys 0.035s
I20260808 01:53:01.498509  9635 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:01.499356 17758 raft_consensus.cc:697] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:01.505208 10363 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260808 01:53:01.506376  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3322) took 2304 ms. Trace:
W20260808 01:53:01.506769  9558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2936) took 1645 ms (client timeout 1539 ms). Trace:
I20260808 01:53:01.509526 17796 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:01.513053  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:01.513299 17832 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:01.496794 17765 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:01.517552  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:01.517974 17765 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:01.507299  9558 rpcz_store.cc:269] 0808 01:52:59.861228 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:59.873747 (+ 12519us) service_pool.cc:224] Handling call
0808 01:53:01.506512 (+1632765us) spinlock_profiling.cc:243] Waited 379 ms on lock 0x7b54003147d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:01.506653 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":872779136}
I20260808 01:53:01.522794 17547 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:01.523737 17635 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:01.524660 17635 consensus_queue.cc:1243] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 17.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260808 01:53:01.507092  9638 rpcz_store.cc:276] 0808 01:52:59.201775 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:59.499379 (+297604us) service_pool.cc:224] Handling call
0808 01:52:59.499756 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:52:59.564351 (+ 64595us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:59.564397 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:52:59.573612 (+  9215us) log.cc:844] Serialized 26 byte log entry
0808 01:52:59.887575 (+313963us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:52:59.933750 (+ 46175us) log.cc:844] Serialized 14 byte log entry
0808 01:53:00.020444 (+ 86694us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:01.286996 (+1266552us) spinlock_profiling.cc:243] Waited 4.98 ms on lock 0x7b48002379d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:53:01.505906 (+218910us) spinlock_profiling.cc:243] Waited 64.1 ms on lock 0x7b54000dfa54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:01.505990 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:01.506060 (+    70us) raft_consensus.cc:1685] finished
0808 01:53:01.506080 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:01.506279 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49236,"prepare.queue_time_us":52072,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":204726,"raft.run_cpu_time_us":7290,"raft.run_wall_time_us":1206471,"spinlock_wait_cycles":158882560,"thread_start_us":368886,"threads_started":2,"wal-append.queue_time_us":396964,"wal-append.run_cpu_time_us":2935,"wal-append.run_wall_time_us":1198927}
I20260808 01:53:01.526983  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.527823  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.527992  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.481283 17671 consensus_queue.cc:1059] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.021s
I20260808 01:53:01.529287  9599 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:01.531005  9558 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:53:01.532912  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:01.532898  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.536870  9560 raft_consensus.cc:2410] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 21 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260808 01:53:01.535931  9638 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:53:01.528913 17661 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:01.539000 17877 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:01.539858 17877 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:01.541025 17860 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.513s	user 0.007s	sys 0.000s
I20260808 01:53:01.523622 17547 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.543942  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.544919  9559 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:53:01.545310 17547 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:01.546888 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:01.557471 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:01.558108  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.558257 10365 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
W20260808 01:53:01.559247 17838 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.806s	user 0.005s	sys 0.007s
I20260808 01:53:01.560067 17797 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:01.557806 17758 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:01.539065 17661 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:01.564860 17661 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:01.535257 17796 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:01.567173 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:01.496925 17829 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:01.574800 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:01.575052  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.576519 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:01.577997  9558 raft_consensus.cc:2410] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 24 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260808 01:53:01.581189 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:01.582818 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:01.583179  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.535511 17764 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:01.586476 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:01.588865 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:01.591310 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 908) took 1132 ms. Trace:
I20260808 01:53:01.591768 10361 rpcz_store.cc:276] 0808 01:53:00.456408 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:00.550004 (+ 93596us) service_pool.cc:224] Handling call
0808 01:53:00.565419 (+ 15415us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:00.624514 (+ 59095us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:00.624565 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:00.624630 (+    65us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:53:00.624676 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:01.588238 (+963562us) spinlock_profiling.cc:243] Waited 58.7 ms on lock 0x7b54005122d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:01.588296 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:01.588428 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":135074560}
I20260808 01:53:01.597525  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.598533  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.596913 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:01.601703  9560 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260808 01:53:01.603838  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.603904  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:01.605541  9560 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 10. (term mismatch)
I20260808 01:53:01.606312  9560 pending_rounds.cc:85] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 9
I20260808 01:53:01.606798  9560 pending_rounds.cc:107] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 16.10
I20260808 01:53:01.606593  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.605541  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.607545  9560 raft_consensus.cc:2889] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:01.582094 17700 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:01.613487  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.573197 17796 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.622105  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.592046 17547 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:01.623653  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:01.581609 17829 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.685839  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.687793 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:01.678887  9635 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:53:01.683282  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.703928  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:01.704983  9556 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:01.706435 17877 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:01.684285 17547 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.709571 17547 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:01.704030  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.588531 17764 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.711287  9557 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:01.713033  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.715773  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.716796  9560 raft_consensus.cc:2410] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260808 01:53:01.722494  9638 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:53:01.723701  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:01.723865 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:01.726614  9560 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:01.724380  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3340) took 1962 ms. Trace:
I20260808 01:53:01.728137  9638 rpcz_store.cc:276] 0808 01:52:59.762481 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:01.723576 (+1961095us) service_pool.cc:224] Handling call
0808 01:53:01.723849 (+   273us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:01.724062 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:01.724086 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:01.724130 (+    44us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:53:01.724157 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:01.724209 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:01.724325 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:53:01.732283  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.624507 17700 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.733245  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:01.745329  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:01.746050  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:01.745160  9556 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:01.748191 17700 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:01.751677  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.749575  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:01.753286  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:01.755093  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:01.755528  9558 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 19. Current term is 20. Ops: []
I20260808 01:53:01.756688  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.759021  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:01.759389  9556 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:53:01.774340  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.775342  9559 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:53:01.774628  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:01.778445  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.780251  9555 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 21: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:01.750653 17790 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:01.786388  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:01.790115  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:01.757184 17877 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:01.792519  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:01.787580 17845 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.138s	user 0.004s	sys 0.007s
I20260808 01:53:01.773612 17677 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:01.804435 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:01.782648  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:01.776245 17703 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:53:01.810220  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:01.811340  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.816506  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:01.818107  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:01.820375  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.819501 10332 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:01.834642  9523 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:01.839387  9523 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:01.840014 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:01.796785 17877 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:01.845199 10332 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:01.826056  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.848505 10361 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:01.853446  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:01.856063  9523 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:01.858479 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:01.860507  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:01.851439 17710 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:53:01.851655 17695 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:01.851675 17866 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:53:01.855479 17323 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:01.850613 17877 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:01.895464  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:01.819325 17677 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:01.822233 17880 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:53:01.856879 17881 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:01.854995 17708 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:01.902648  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:01.914978 17764 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:01.864784 17319 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:53:01.916476 17319 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:01.917032 17319 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:01.891929 17700 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:01.918085 17661 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:01.851651 17859 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.047s
I20260808 01:53:01.918777 17661 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.919879 17859 raft_consensus.cc:3055] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:01.920594 17859 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:01.921461 17661 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:01.903111  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:01.925012  9555 raft_consensus.cc:2376] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20. Re-sending same reply.
I20260808 01:53:01.895267 17695 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:01.918900 17700 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:01.926723 17695 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:01.927678 17700 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.929759 17677 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:01.863287 17797 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:53:01.905771 17829 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:01.912770 17881 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:01.931869 17372 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:01.932376 17881 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:01.915755 17764 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:01.944711  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:01.943187 17764 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.862313 17865 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.084s
I20260808 01:53:01.949584  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3325) took 2746 ms. Trace:
I20260808 01:53:01.950286 17865 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:53:01.950708 17865 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:01.950841 17769 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:01.951748 17865 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:01.953198 17865 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:01.957168 17880 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:01.950022  9634 rpcz_store.cc:276] 0808 01:52:59.202696 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:00.108957 (+906261us) service_pool.cc:224] Handling call
0808 01:53:00.109423 (+   466us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:00.138600 (+ 29177us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:00.138639 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:00.144999 (+  6360us) log.cc:844] Serialized 52 byte log entry
0808 01:53:00.452406 (+307407us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:53:01.085804 (+633398us) spinlock_profiling.cc:243] Waited 4.31 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:01.948520 (+862716us) log.cc:844] Serialized 14 byte log entry
0808 01:53:01.948946 (+   426us) log.cc:844] Serialized 14 byte log entry
0808 01:53:01.949123 (+   177us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:01.949204 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:01.949346 (+   142us) raft_consensus.cc:1685] finished
0808 01:53:01.949361 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:01.949505 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1536806,"prepare.queue_time_us":1508784,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":164562,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":18702,"spinlock_wait_cycles":9908992,"thread_start_us":248579,"threads_started":1,"wal-append.queue_time_us":457585}
I20260808 01:53:01.958390 17880 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:01.959723  9634 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260808 01:53:01.961664  9634 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
W20260808 01:53:01.965265  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:01.959631 17765 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:01.980543  9552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3818) took 2706 ms (client timeout 1638 ms). Trace:
W20260808 01:53:01.981143  9552 rpcz_store.cc:269] 0808 01:52:59.274028 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:59.278084 (+  4056us) service_pool.cc:224] Handling call
0808 01:53:01.980283 (+2702199us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b5400382a54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:01.980441 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3291261184}
I20260808 01:53:01.985914  9634 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260808 01:53:01.921772 17859 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:01.992331 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 131050us
W20260808 01:53:01.998301  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 215727us
I20260808 01:53:01.999321  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:02.001111  9634 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 21. Ops: []
I20260808 01:53:01.975559 17764 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:02.012041  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:02.013707  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:02.029351  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:02.050940  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:02.054507  9634 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260808 01:53:02.024449 17447 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:02.056803 17447 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:02.057358 17447 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:02.058430  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:02.060384  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:02.067917 17701 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:02.068609 17701 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:02.069173 17701 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:02.062145  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:02.077793  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:02.072702  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:02.081171  9474 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:53:02.080192  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:02.053908 17838 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260808 01:53:02.094091  9479 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 13. (term mismatch)
I20260808 01:53:02.094848  9479 pending_rounds.cc:85] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 12
I20260808 01:53:02.026589 17677 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:02.094437  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:02.095388  9479 pending_rounds.cc:107] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 19.13
I20260808 01:53:02.116178  9479 raft_consensus.cc:2889] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:02.023869 17859 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:02.111459 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 114678us
I20260808 01:53:02.071959 17764 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260808 01:53:02.115195 17845 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260808 01:53:02.086903 17765 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:53:02.115854 17677 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:02.123338 17886 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:53:02.085489 17790 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:02.126231 17764 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:02.186230 17677 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:02.197566 17764 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
W20260808 01:53:02.221285  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 151384us
I20260808 01:53:02.194700 17886 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:02.199599 17891 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260808 01:53:02.229004  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 148931us
I20260808 01:53:02.232311  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:02.238447  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:02.211444 17892 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:02.249779  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:02.234198 17891 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:02.231272 17886 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:53:02.265511 17650 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:02.266043 17650 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:02.267086 17650 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.268594 17650 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:02.256518  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:02.257678 17892 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:02.259200 17891 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:02.305578 17895 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
W20260808 01:53:02.291179 17887 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.169s	user 0.000s	sys 0.001s
W20260808 01:53:02.295452 17893 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
W20260808 01:53:02.304055  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:02.282649 17447 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.320278 17695 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:02.285645 17905 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:53:02.321895 17848 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:02.322767  9478 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:53:02.315467 17892 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:53:02.324888 17319 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:02.322152 17764 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.329042 17701 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.335371 17761 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:02.336124 17493 raft_consensus.cc:3055] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:53:02.340759  9557 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:02.316565 17790 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.375607 17447 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:02.376483 17319 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:02.385524  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 119601us
I20260808 01:53:02.388614  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:02.394532 17854 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:02.396175 17701 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:02.398029 17695 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:02.399027  9480 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:02.401026 17764 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:02.324229 17910 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:53:02.404682  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 148274us
I20260808 01:53:02.372241 17700 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:53:02.407608  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:02.381492 17493 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:02.418324  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:02.373975 17877 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.422876 10369 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:57156
I20260808 01:53:02.423697 10369 raft_consensus.cc:606] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:02.427971  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:02.419754 17447 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:02.434954  9476 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 24: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:02.421254 17700 raft_consensus.cc:697] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:02.440676  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:02.442263  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:02.430394 17708 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:02.436707 17547 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260808 01:53:02.454483 17919 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:02.461956 17877 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:02.460237 17493 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:02.464192 17881 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.531s	user 0.016s	sys 0.000s
I20260808 01:53:02.465939 17891 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.468426 17891 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:02.466387 17881 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:02.472617 17911 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:53:02.469760 17826 raft_consensus.cc:3055] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:02.472631 17832 raft_consensus.cc:3055] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Stepping down as leader of term 25
W20260808 01:53:02.475369  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47471.6us
I20260808 01:53:02.476727  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:02.475478  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:02.483950  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:02.485867  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:02.486337  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:02.488794  9556 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:53:02.489079  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:02.489835  9552 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260808 01:53:02.490196  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:02.491014  9551 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:53:02.491375  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:02.494699  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:02.497365  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:02.499415  9474 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:02.500490  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:02.501214  9550 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 22: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:02.502779 10369 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:57156
I20260808 01:53:02.503369 10369 raft_consensus.cc:606] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:02.503692  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:02.502498  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:02.474896 17832 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:53:02.489838 17909 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.178s	user 0.001s	sys 0.000s
I20260808 01:53:02.447813 17914 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.056s
I20260808 01:53:02.504273  9550 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:02.493911 17547 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260808 01:53:02.520828 17886 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:02.521397 17832 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:02.514174  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:02.522541 17832 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:02.523376  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:02.524355 17827 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:02.524760  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:02.525195 17829 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:02.524938 17677 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:02.526173  9481 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 20. Current term is 21. Ops: []
I20260808 01:53:02.524982 17827 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.504280 17919 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:02.474506 17826 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:02.525810 17829 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.527108  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:02.523298  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3326) took 3320 ms. Trace:
I20260808 01:53:02.504802  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:02.528162  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:02.528751  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3336) took 3131 ms. Trace:
I20260808 01:53:02.491020 17911 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:53:02.536724  9632 rpcz_store.cc:276] 0808 01:52:59.397333 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:01.358195 (+1960862us) service_pool.cc:224] Handling call
0808 01:53:01.382053 (+ 23858us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:01.392149 (+ 10096us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:01.392180 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:01.392688 (+   508us) log.cc:844] Serialized 26 byte log entry
0808 01:53:01.529378 (+136690us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:53:02.026077 (+496699us) log.cc:844] Serialized 14 byte log entry
0808 01:53:02.528375 (+502298us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:02.528473 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:02.528525 (+    52us) raft_consensus.cc:1685] finished
0808 01:53:02.528539 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:02.528682 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":983074,"prepare.queue_time_us":545705,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":515825,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":114,"thread_start_us":111661,"threads_started":1,"wal-append.queue_time_us":301346}
I20260808 01:53:02.536012  9636 rpcz_store.cc:276] 0808 01:52:59.203045 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:00.109514 (+906469us) service_pool.cc:224] Handling call
0808 01:53:00.123696 (+ 14182us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:00.150345 (+ 26649us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:00.150408 (+    63us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:00.151261 (+   853us) log.cc:844] Serialized 52 byte log entry
0808 01:53:00.553847 (+402586us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:53:01.621646 (+1067799us) spinlock_profiling.cc:243] Waited 2.09 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:01.963832 (+342186us) log.cc:844] Serialized 14 byte log entry
0808 01:53:01.964386 (+   554us) log.cc:844] Serialized 14 byte log entry
0808 01:53:02.522655 (+558269us) spinlock_profiling.cc:243] Waited 4.72 ms on lock 0x7b18004a0c80. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667 00007f5d449ae5e8
0808 01:53:02.522913 (+   258us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:02.523007 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:02.523062 (+    55us) raft_consensus.cc:1685] finished
0808 01:53:02.523074 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:02.523236 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2051631,"prepare.queue_time_us":1577044,"prepare.run_cpu_time_us":196,"prepare.run_wall_time_us":5381,"raft.queue_time_us":740683,"raft.run_cpu_time_us":441,"raft.run_wall_time_us":12638,"spinlock_wait_cycles":15650432,"thread_start_us":315461,"threads_started":1,"wal-append.queue_time_us":457156}
I20260808 01:53:02.502113 17695 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:02.547565  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:02.528740 17615 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:02.590653  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 47803.2us
I20260808 01:53:02.592805  9550 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:02.595276 10361 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: []
I20260808 01:53:02.597271 10369 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:02.539621 16835 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:53:02.540534 17793 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:02.612159  9559 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: []
I20260808 01:53:02.608994  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:02.599929  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:02.608889 16835 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:02.617015 10365 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260808 01:53:02.617416 16835 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.618398 17920 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:02.528108 17919 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.551054 17700 consensus_queue.cc:237] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.618623 16835 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:02.626895 17881 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:02.613238 17838 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260808 01:53:02.640254 17799 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:02.611397 17793 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:02.641203 17793 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:02.641795 17829 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:02.643586 17821 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:02.644207 17821 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:02.605103 17615 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.646932 17532 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:02.647544 17532 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.647861 17615 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:02.648487 17615 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:02.649075 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:02.648993 17615 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.649741 17532 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:02.598843 17827 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:02.655120 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:02.655383 16835 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:02.655891 16835 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.657205 17925 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:02.657791 17925 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.658661 17708 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:02.659366 17827 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:53:02.659263 17708 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.659868 17827 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:02.660367 17827 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:02.660593 17925 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:02.667443  9599 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:02.658190 17797 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:02.639956 17714 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:02.685842 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:02.686841 10365 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:02.685732 17797 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.688639 17672 consensus_queue.cc:1059] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260808 01:53:02.689191 17877 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:02.689424 17797 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:02.689796 17672 raft_consensus.cc:3055] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:02.689754 17877 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.690330 17672 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:02.680675 17881 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:02.703269 17881 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:02.705531 17881 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:02.705884 17886 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:02.707976  9630 raft_consensus.cc:2410] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 20 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260808 01:53:02.692227 17877 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:02.692965 17872 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:02.680653 17674 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:02.686663 17714 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.650377 17532 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.031s
I20260808 01:53:02.686812 17671 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:02.699121 17708 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:02.700659  9481 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 15. Current term is 16. Ops: []
I20260808 01:53:02.646446 17821 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:02.703126 17672 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:02.687372 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:02.712767  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:02.622772 17919 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:02.625241 17796 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:02.715591 17615 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:02.715823 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:02.717070 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:02.780293 10369 raft_consensus.cc:2393] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 21: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:02.716782 17793 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:02.721185  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:02.727226 17493 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:53:02.781334  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 68671.4us
I20260808 01:53:02.782950  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3074) took 1652 ms. Trace:
W20260808 01:53:02.786052  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:02.782579  9599 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:02.788961  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3329) took 3585 ms. Trace:
I20260808 01:53:02.783408  9635 rpcz_store.cc:276] 0808 01:53:01.130165 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:01.732202 (+602037us) service_pool.cc:224] Handling call
0808 01:53:01.744551 (+ 12349us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:01.745047 (+   496us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:01.745088 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:01.745762 (+   674us) log.cc:844] Serialized 52 byte log entry
0808 01:53:01.860079 (+114317us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:53:02.215539 (+355460us) log.cc:844] Serialized 14 byte log entry
0808 01:53:02.696647 (+481108us) spinlock_profiling.cc:243] Waited 34.6 ms on lock 0x7b48002379d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:53:02.741074 (+ 44427us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:02.741187 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:02.777402 (+ 36215us) raft_consensus.cc:1685] finished
0808 01:53:02.782621 (+  5219us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:02.782820 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":524423,"prepare.queue_time_us":226461,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":3536,"raft.queue_time_us":325768,"raft.run_cpu_time_us":2527,"raft.run_wall_time_us":160045,"spinlock_wait_cycles":79667072,"thread_start_us":436705,"threads_started":3,"wal-append.queue_time_us":260737}
I20260808 01:53:02.792060  9599 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:02.792531 10365 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
W20260808 01:53:02.794405  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:02.794168  9599 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:02.772238 17547 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:02.800450  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 992) took 4529 ms. Trace:
W20260808 01:53:02.803925  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:02.805284  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:02.808928 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 219131us
I20260808 01:53:02.811314  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3076) took 1680 ms. Trace:
I20260808 01:53:02.802145  9637 rpcz_store.cc:276] 0808 01:52:58.259710 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:58.259861 (+   151us) service_pool.cc:224] Handling call
0808 01:52:58.260275 (+   414us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:58.260771 (+   496us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:58.260803 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:58.261524 (+   721us) log.cc:844] Serialized 52 byte log entry
0808 01:52:58.940545 (+679021us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:52:59.914930 (+974385us) log.cc:844] Serialized 14 byte log entry
0808 01:53:01.942266 (+2027336us) log.cc:844] Serialized 14 byte log entry
0808 01:53:02.788699 (+846433us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:02.788802 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:02.788866 (+    64us) raft_consensus.cc:1685] finished
0808 01:53:02.788879 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:02.789053 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2743940,"prepare.queue_time_us":2329764,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":96,"raft.queue_time_us":650347,"raft.run_cpu_time_us":551,"raft.run_wall_time_us":33386,"thread_start_us":1840951,"threads_started":5,"wal-append.queue_time_us":1061955,"wal-append.run_cpu_time_us":5569,"wal-append.run_wall_time_us":1957627}
I20260808 01:53:02.679291 17826 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:02.817795  9599 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:02.786873 17671 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260808 01:53:02.818742 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:02.811931  9638 rpcz_store.cc:276] 0808 01:53:01.130870 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:01.732771 (+601901us) service_pool.cc:224] Handling call
0808 01:53:01.734588 (+  1817us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:01.734871 (+   283us) raft_consensus.cc:1534] Early marking committed up to index 14
0808 01:53:02.255647 (+520776us) log.cc:844] Serialized 14 byte log entry
0808 01:53:02.810649 (+555002us) spinlock_profiling.cc:243] Waited 1.12 ms on lock 0x7b1800352120. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4453a6ac 00007f5d4186fe3b 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667
0808 01:53:02.810818 (+   169us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:02.810889 (+    71us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:53:02.810960 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:02.811061 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:02.811232 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":558751,"prepare.queue_time_us":342244,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":313045,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":11846,"spinlock_wait_cycles":2576128,"thread_start_us":272660,"threads_started":3,"wal-append.queue_time_us":70644}
I20260808 01:53:02.814747 17672 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:02.745500 17674 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:02.826949  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260808 01:53:02.833305  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:02.825698  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:02.828099  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:02.834393  9632 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 21: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:02.834458  9552 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:02.791320 17756 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:02.831162  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:02.831844  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:02.831760  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:02.813910 17769 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:02.799944 17799 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:02.847361  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:02.848230  9559 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:02.829279  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:02.851099  9551 raft_consensus.cc:2410] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 21 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260808 01:53:02.835425  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:02.796548 17779 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:02.799748 17615 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:02.837970  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:02.842197  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:02.821477 17821 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:53:02.833336  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:02.850804  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:02.835088  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:02.858606  9638 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:53:02.858788  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:02.859664  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:02.781376 17725 consensus_queue.cc:1059] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260808 01:53:02.854164  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:02.847755 17799 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:02.853160  9637 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:53:02.860580  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:02.851787  9634 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:02.862004  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:02.815166 17547 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.841310 17925 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:02.867883  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:02.870410  9476 raft_consensus.cc:2410] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 24 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260808 01:53:02.847378 17756 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:02.840664 17758 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:02.878377  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:02.879539  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:02.879794  9632 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:02.880656  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:02.881314  9634 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:02.883594  9637 raft_consensus.cc:2410] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 22 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260808 01:53:02.884552  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:02.885293  9638 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:53:02.867326  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3327) took 3664 ms. Trace:
W20260808 01:53:02.871757  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:02.872502  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:02.889119  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:02.868357 17799 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:02.890165  9476 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 22: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:02.873298  9481 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 24: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:02.892409  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:02.899910  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:02.901523  9560 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:02.900369  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3324) took 3698 ms. Trace:
I20260808 01:53:02.903049  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:02.904434  9634 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:02.869886 17615 raft_consensus.cc:697] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:02.878026 17925 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:02.876086 17821 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:02.961733  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:02.918148  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:02.893167  9633 rpcz_store.cc:276] 0808 01:52:59.203376 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:00.198501 (+995125us) service_pool.cc:224] Handling call
0808 01:53:00.205006 (+  6505us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:00.325317 (+120311us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:00.325352 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:00.325997 (+   645us) log.cc:844] Serialized 52 byte log entry
0808 01:53:00.722850 (+396853us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:01.982430 (+1259580us) log.cc:844] Serialized 14 byte log entry
0808 01:53:02.724522 (+742092us) spinlock_profiling.cc:243] Waited 4.48 ms on lock 0x7b48002379d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:53:02.799459 (+ 74937us) log.cc:844] Serialized 14 byte log entry
0808 01:53:02.837306 (+ 37847us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:02.837389 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:02.867051 (+ 29662us) raft_consensus.cc:1685] finished
0808 01:53:02.867081 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:02.867254 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1749538,"prepare.queue_time_us":1353663,"prepare.run_cpu_time_us":158,"prepare.run_wall_time_us":4597,"raft.queue_time_us":725662,"raft.run_cpu_time_us":1423,"raft.run_wall_time_us":197891,"spinlock_wait_cycles":10301696,"thread_start_us":723727,"threads_started":3,"wal-append.queue_time_us":691237,"wal-append.run_cpu_time_us":1906,"wal-append.run_wall_time_us":1011424}
I20260808 01:53:02.920122  9480 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 24: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:02.920668  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:02.924398 17920 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:53:02.941890 17629 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:02.928563 17756 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:02.933697 17830 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:02.944661 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:02.903167  9631 rpcz_store.cc:276] 0808 01:52:59.202376 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:59.930421 (+728045us) service_pool.cc:224] Handling call
0808 01:52:59.931062 (+   641us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:52:59.963123 (+ 32061us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:52:59.984826 (+ 21703us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:52:59.999032 (+ 14206us) log.cc:844] Serialized 52 byte log entry
0808 01:53:00.170446 (+171414us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:53:00.224958 (+ 54512us) log.cc:844] Serialized 14 byte log entry
0808 01:53:01.927766 (+1702808us) spinlock_profiling.cc:243] Waited 4.87 ms on lock 0x7b18004da8e0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:53:01.936912 (+  9146us) spinlock_profiling.cc:243] Waited 22.9 ms on lock 0x7b18004da8e0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d483aada7 00007f5d4499dd87 00007f5d449ad49f 00007f5d449ad418
0808 01:53:01.937167 (+   255us) log.cc:844] Serialized 14 byte log entry
0808 01:53:02.865291 (+928124us) spinlock_profiling.cc:243] Waited 4.44 ms on lock 0x7b48002379d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:53:02.899156 (+ 33865us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:02.900075 (+   919us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:02.900135 (+    60us) raft_consensus.cc:1685] finished
0808 01:53:02.900149 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:02.900304 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":577528,"prepare.queue_time_us":1924709,"prepare.run_cpu_time_us":137,"prepare.run_wall_time_us":3716,"raft.queue_time_us":767339,"raft.run_cpu_time_us":2322,"raft.run_wall_time_us":377534,"spinlock_wait_cycles":74005376,"thread_start_us":716732,"threads_started":3,"wal-append.queue_time_us":881098,"wal-append.run_cpu_time_us":5032,"wal-append.run_wall_time_us":1258346}
I20260808 01:53:02.955389  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:02.981319  9638 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:02.916955  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:02.896703  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 347613us
I20260808 01:53:02.979141  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:02.962864  9632 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:02.983611  9552 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:53:02.964159  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:02.984555  9634 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:02.864840 17779 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:02.925076 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:02.968336  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:02.970350  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:02.982038  9637 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:53:02.969112  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:02.971792  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:02.975445  9601 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:02.879140 17708 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.087s
I20260808 01:53:02.977449  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:02.949985 17673 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:53:02.960546  9476 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 20. Current term is 21. Ops: []
I20260808 01:53:02.964315  9478 raft_consensus.cc:2410] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260808 01:53:02.963747 17829 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:02.965823 16835 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:02.992589  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:02.994707  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:03.001369  9474 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:02.961738 17925 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:03.011364 17547 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:03.011531 17695 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:03.012295 17695 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:03.014289 17866 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:02.993831 17797 consensus_queue.cc:1059] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
I20260808 01:53:02.996579  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:03.016390  9481 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:03.017782  9476 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21. Ops: []
I20260808 01:53:02.985342  9552 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:02.883013 17758 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:02.987391 10368 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:53:03.019482 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:03.020910 17793 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:03.026368  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 3075) took 1895 ms. Trace:
I20260808 01:53:03.028162  9630 rpcz_store.cc:276] 0808 01:53:01.130539 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:01.327198 (+196659us) service_pool.cc:224] Handling call
0808 01:53:03.026141 (+1698943us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54000db1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:03.026302 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3169334528}
I20260808 01:53:03.028592  9552 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260808 01:53:03.022121 17674 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:03.021473 17615 consensus_queue.cc:237] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:03.038064  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:03.025696  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:02.986366  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:03.031633  9559 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 21. Current term is 22. Ops: []
W20260808 01:53:03.032259  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:03.036192  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:03.044305  9632 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:53:03.045854  9631 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:53:03.047266  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:02.997532 17673 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:03.048970  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:53:03.060935  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:03.061097  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:03.050671  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:03.054260  9481 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:03.061391  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:53:03.051046  9631 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:53:03.068399  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 196618us
I20260808 01:53:03.051862  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:03.072057  9634 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:03.077040  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:03.083278 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:03.032255 17714 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:03.097270  9478 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260808 01:53:03.019363 17829 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:03.137936 17695 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:03.139214 17848 raft_consensus.cc:993] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: : Instructing follower 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:53:03.139585 17829 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:03.139884 17848 raft_consensus.cc:1081] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Signalling peer 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:53:03.142205 17757 consensus_queue.cc:1059] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:03.144292 17757 raft_consensus.cc:3055] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:03.144784 17757 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:03.146162 17931 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:03.149689 17757 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:03.150364 17932 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:03.060197 17821 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:03.152168 17757 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:03.115538 17921 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260808 01:53:03.098376 17894 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:03.165480 17894 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:03.106393  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:03.137377  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:03.169548  9632 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:03.088759 17793 raft_consensus.cc:3055] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Stepping down as leader of term 18
W20260808 01:53:03.188314  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 368033us
I20260808 01:53:03.189455  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:03.191654  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:03.192725  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:03.193271  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:03.194069 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:03.133461  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 158251us
I20260808 01:53:03.182951  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:03.183219  9476 raft_consensus.cc:2410] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 21 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260808 01:53:03.199582  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:03.135715  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:03.202631  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:03.182717 17714 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:03.184494 17793 raft_consensus.cc:740] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:03.224263  9632 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:03.225175 17650 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:53:03.227172  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:03.195873 17932 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:03.198242 17880 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:03.198084 17700 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:03.225829 17829 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:03.240988  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:03.234220  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:03.295994  9632 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
W20260808 01:53:03.297724  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 302862us
I20260808 01:53:03.308806  9524 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:03.318168  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:03.237360 17714 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:53:03.288627  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 61538.1us
W20260808 01:53:03.297163  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 56281.5us
I20260808 01:53:03.339615  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:03.340876  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:03.341204  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:03.328540  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:03.332636  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:03.252661 17880 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:03.334430  9558 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:53:03.238104 17932 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:03.293531  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:03.190107 17673 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:03.346987  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:03.367748  9638 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:03.333765  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:03.351435  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:03.348466 17493 consensus_meta.cc:287] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.563s	user 0.011s	sys 0.002s
I20260808 01:53:03.337133 17796 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:03.297003 17700 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:03.336501 17920 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260808 01:53:03.367343 17827 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.706s	user 0.012s	sys 0.004s
I20260808 01:53:03.316354 17650 raft_consensus.cc:697] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:03.320180 17933 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:03.370381  9632 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:03.376231  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:03.391990  9637 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:03.324081 17925 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:03.363889  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:03.359351  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:03.377272 17677 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:03.394997  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:03.335960 17939 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
W20260808 01:53:03.383589  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:03.345445 17790 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:03.364073 17880 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:03.380380 17700 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:03.396338  9601 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:03.368157 17931 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260808 01:53:03.405838  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:03.382589 17920 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:03.390794 17673 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:03.426643  9601 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:03.429908  9636 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 24: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:03.430357 10361 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: []
I20260808 01:53:03.431730  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:03.408139 17927 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:03.356446 17793 consensus_queue.cc:260] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:03.376956 17881 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.015s
I20260808 01:53:03.437425  9599 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:03.399070 17829 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:53:03.443719  9601 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:03.432857 17936 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.089s	user 0.001s	sys 0.001s
I20260808 01:53:03.390650 17827 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:03.451790  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:03.443771 17920 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:03.486649  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:03.457139 17830 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260808 01:53:03.456429  9476 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 15. Current term is 16. Ops: []
I20260808 01:53:03.436189 17629 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:03.455672  9638 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:53:03.501717  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:03.503175  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:03.503786  9638 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:03.492384  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:03.507887  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:03.508592  9632 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:03.509572  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:03.425419 17790 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:03.495961  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52348.8us
I20260808 01:53:03.514230  9601 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:03.515718  9631 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:53:03.517086  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:03.518950  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:03.478236 17923 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:03.493517 17615 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:03.519932 17797 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:03.432579 17714 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.18, Last appended by leader: 18, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:03.550202  9601 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:03.483896 17832 consensus_meta.cc:287] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.961s	user 0.014s	sys 0.012s
I20260808 01:53:03.550220  9638 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:03.564018  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:03.565689 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:03.521422 17821 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260808 01:53:03.568516  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:03.570930 17829 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:03.572013 17829 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:03.565404  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:03.562638  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 76059.2us
I20260808 01:53:03.574937  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:03.576730  9599 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:03.579169 17821 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:03.580222 17821 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:03.580701 17821 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:03.581187 17821 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:03.583364 17775 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:03.584252 17775 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:03.588009 17775 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:03.570317  9601 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:03.593019  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 246136us
I20260808 01:53:03.593473  9638 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:03.595052 17677 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.001s
I20260808 01:53:03.561008  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:03.599668  9631 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:53:03.539196 17894 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:03.601523  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:03.601188  9601 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:03.604569 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:03.612651  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:03.547868 17708 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:03.605697  9638 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260808 01:53:03.481172 17650 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:03.627010 17672 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.789s	user 0.013s	sys 0.008s
W20260808 01:53:03.627653 17940 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260808 01:53:03.615367  9601 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:03.631116  9636 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260808 01:53:03.631836 17629 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:03.632695  9636 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 19. Current term is 20. Ops: []
I20260808 01:53:03.633505  9601 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:03.634377 17629 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:03.636415 17854 raft_consensus.cc:3055] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:53:03.636936 17854 raft_consensus.cc:740] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:03.637946  9601 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:03.620967  9638 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 20. Current term is 21. Ops: []
I20260808 01:53:03.615525 17927 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:03.623356 17703 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:03.624471 17677 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:03.562389 17615 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:03.561757 16835 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260808 01:53:03.647074 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:03.562706 17923 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:03.601871 17775 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:03.640998  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:03.530999 17777 consensus_queue.cc:1059] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.032s
I20260808 01:53:03.647673 10362 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:03.603700 17830 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:03.650532 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:03.657500 10365 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: [21.18-21.18]
I20260808 01:53:03.409034 17932 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:03.648903 17615 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:03.663826 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:03.641362 17703 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:03.657130  9601 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:03.604138 17797 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:03.645946 17629 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:03.618220 17925 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:03.645583 17677 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:03.656665 17923 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:03.679876  9601 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:03.677079  9636 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:03.660854 17775 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:03.661535 17830 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:03.642467 17708 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:03.700629 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:02.789534  9629 rpcz_store.cc:276] 0808 01:52:59.203966 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:00.245519 (+1041553us) service_pool.cc:224] Handling call
0808 01:53:00.295474 (+ 49955us) raft_consensus.cc:1471] Updating replica for 4 ops
0808 01:53:00.322351 (+ 26877us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:00.322398 (+    47us) raft_consensus.cc:1539] Triggering prepare for 4 ops
0808 01:53:00.356125 (+ 33727us) log.cc:844] Serialized 104 byte log entry
0808 01:53:00.739322 (+383197us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:01.998335 (+1259013us) log.cc:844] Serialized 14 byte log entry
0808 01:53:02.742350 (+744015us) log.cc:844] Serialized 14 byte log entry
0808 01:53:02.772519 (+ 30169us) log.cc:844] Serialized 14 byte log entry
0808 01:53:02.784236 (+ 11717us) log.cc:844] Serialized 14 byte log entry
0808 01:53:02.788552 (+  4316us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:02.788645 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:02.788712 (+    67us) raft_consensus.cc:1685] finished
0808 01:53:02.788724 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:02.788890 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1615600,"prepare.queue_time_us":1376367,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":127,"raft.queue_time_us":528752,"raft.run_cpu_time_us":371,"raft.run_wall_time_us":120639,"spinlock_wait_cycles":72153728,"thread_start_us":717763,"threads_started":3,"wal-append.queue_time_us":681505,"wal-append.run_cpu_time_us":2887,"wal-append.run_wall_time_us":1064896}
I20260808 01:53:03.702184  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:03.706321  9476 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
W20260808 01:53:03.707149 17935 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.221s	user 0.001s	sys 0.000s
I20260808 01:53:03.707402 17925 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260808 01:53:03.702248 17799 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.743s	user 0.012s	sys 0.003s
W20260808 01:53:03.708576 17938 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.273s	user 0.000s	sys 0.001s
I20260808 01:53:03.709161 17799 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:03.684350 17927 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:53:03.681448 17703 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:03.710868 10362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 920) took 1060 ms. Trace:
I20260808 01:53:03.714006 17797 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:03.727198 10369 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:53:03.732054 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:03.700106 17677 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:03.705646 17830 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:03.759034  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:03.759603  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:03.688814  9630 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 26.
I20260808 01:53:03.738722  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:03.731667 10362 rpcz_store.cc:276] 0808 01:53:02.662531 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:02.663230 (+   699us) service_pool.cc:224] Handling call
0808 01:53:03.710663 (+1047433us) spinlock_profiling.cc:243] Waited 857 ms on lock 0x7b5400694954. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:03.710801 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1971563776}
I20260808 01:53:03.706652 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:03.765661  9637 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:03.768004  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:03.768334 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 61805us
I20260808 01:53:03.770313  9637 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: [20.10-22.11]
I20260808 01:53:03.771553 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:03.726485 17775 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:03.773463 17933 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:03.774796 17827 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:03.779209 17672 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:53:03.823999 17790 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:03.851647 17946 log.cc:927] Time spent T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.235s	user 0.000s	sys 0.002s
W20260808 01:53:03.853152 17945 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.194s	user 0.000s	sys 0.002s
I20260808 01:53:03.866549 17799 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:03.873198 17796 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260808 01:53:03.878110 17950 log.cc:927] Time spent T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260808 01:53:03.801242 17775 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:03.892951 17826 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:03.896130 17923 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:03.896157 17880 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:03.900820 17880 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:03.901751 17923 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:03.901907 17854 consensus_queue.cc:260] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:03.903414 17854 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
W20260808 01:53:03.730005  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:03.885617 17708 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:03.950733  9445 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.000s	sys 0.010s
I20260808 01:53:03.958498  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:03.957383  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 2971) took 1040 ms. Trace:
I20260808 01:53:03.959507  9551 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 24: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:03.983392 17848 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:03.969702  9635 raft_consensus.cc:2368] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22.
W20260808 01:53:03.981137  9445 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.000s	sys 0.010s
I20260808 01:53:03.966044  9558 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
W20260808 01:53:03.964634  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 205857us
W20260808 01:53:03.995921  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 653823us
I20260808 01:53:03.998381  9552 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
W20260808 01:53:03.952404  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 339587us
I20260808 01:53:04.002077  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:03.984890  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:03.985369 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 213784us
I20260808 01:53:04.009485  9554 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 22
I20260808 01:53:04.015370 17894 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:03.963866  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:04.017103  9557 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:03.970024 17708 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:03.952498  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 452193us
I20260808 01:53:04.019480 17708 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:04.022584 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:03.969236  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:04.016927 17894 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:03.967175  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:03.952669  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:03.982620  9550 rpcz_store.cc:276] 0808 01:53:02.917116 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:02.977319 (+ 60203us) service_pool.cc:224] Handling call
0808 01:53:02.977660 (+   341us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:03.228968 (+251308us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:03.229016 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:03.245450 (+ 16434us) log.cc:844] Serialized 52 byte log entry
0808 01:53:03.678352 (+432902us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:53:03.691329 (+ 12977us) log.cc:844] Serialized 14 byte log entry
0808 01:53:03.691569 (+   240us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:03.691638 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:03.956412 (+264774us) raft_consensus.cc:1685] finished
0808 01:53:03.956527 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:03.957075 (+   548us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7991,"prepare.queue_time_us":12625,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":54686,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":257,"thread_start_us":550059,"threads_started":2,"wal-append.queue_time_us":525438}
I20260808 01:53:04.039476 17880 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:04.040711 17880 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:04.047819  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:04.049736  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:04.050940  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:04.052459  9480 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 24: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:04.054811 17708 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:04.055318  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:04.055840  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 1006) took 1085 ms. Trace:
I20260808 01:53:04.056632  9479 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:04.056512  9630 rpcz_store.cc:276] 0808 01:53:02.970494 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:03.293148 (+322654us) service_pool.cc:224] Handling call
0808 01:53:04.055275 (+762127us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b540025bcd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:04.055537 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":553493760}
I20260808 01:53:04.045329 17960 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.013s
I20260808 01:53:04.058706  9475 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260808 01:53:04.060679 17881 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:04.061214 17881 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:04.069257 10368 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 14. (term mismatch)
I20260808 01:53:04.070029 10368 pending_rounds.cc:85] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting all ops after (but not including) 13
I20260808 01:53:04.071877 17775 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:04.075819 17779 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:04.077105 17547 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:04.078145 17547 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:04.079324 17942 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:04.088871 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:04.096885 17830 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:04.098073 17793 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:53:04.100782 17880 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:04.139633  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:04.100939  9523 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:04.105207 17848 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:04.070523 10368 pending_rounds.cc:107] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting uncommitted NO_OP operation due to leader change: 19.14
I20260808 01:53:04.112357  9477 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:04.135841  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:04.152573 17829 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:04.113569  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:04.115820  9480 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:04.102077 17853 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.017s
I20260808 01:53:04.160198  9481 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 20: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:04.161938  9638 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 21: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:04.121284 17758 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:04.147229 17547 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:04.135174 17960 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:04.163342 17960 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:04.163163 17848 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:04.136405 17943 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.029s
I20260808 01:53:04.167615 17832 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260808 01:53:04.120265  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 118714us
I20260808 01:53:04.148092 10368 raft_consensus.cc:2889] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:53:04.170830  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 70241.7us
I20260808 01:53:04.171644  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:04.153622  9475 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:04.173287 17547 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:04.173620 17880 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:53:04.174077 17547 raft_consensus.cc:697] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:04.174396 17880 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:04.175302 17880 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:04.142561  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:04.163231 10363 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: []
I20260808 01:53:04.184695  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:04.187603 17954 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:04.188167 17954 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:53:04.118636 17821 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.537s	user 0.010s	sys 0.010s
I20260808 01:53:04.189301 17821 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:04.190248  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:04.192929 17821 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:04.063424 17881 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 21.18, Last appended by leader: 18, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:04.195048 17853 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:04.195381 17547 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:04.196425 17881 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:04.164049 17960 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:04.198933 17932 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:03.909623 17920 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:04.196063 17853 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:04.204823 17920 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:04.140476  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:04.188637 17954 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:04.196012  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:04.207270 17848 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:04.210058  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:04.211477 17756 raft_consensus.cc:3055] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:53:04.212085 17756 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:04.213467 17695 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:04.211691 17892 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:04.214320 17892 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:04.215026 17892 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:04.159934 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:04.216342  9554 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:04.217345 10362 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 20: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:04.220280 17894 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:04.214016 17695 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:04.195142  9629 raft_consensus.cc:2393] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 24: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:04.224407 17695 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:04.226084 17695 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:04.213968 17756 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:04.156852  9560 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:53:04.212036  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:04.228327  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:04.226435  9555 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:60552
I20260808 01:53:04.234119  9555 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting forced leader election (received explicit request)
W20260808 01:53:04.235652  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 58968.4us
I20260808 01:53:04.206317 17853 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:04.239027  9555 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
W20260808 01:53:04.240828  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 56387us
I20260808 01:53:04.239288  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:04.243942  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 48085.1us
I20260808 01:53:04.244092 17891 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:04.248247  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:04.252449  9550 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260808 01:53:04.244673  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:04.253911 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:04.254441 17891 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 24 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:04.256283  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:04.257081  9481 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:53:04.256139 17891 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:04.258816  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:04.258706 10334 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:04.250487 17827 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.002s
I20260808 01:53:04.260180 17777 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:04.260720 17777 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:04.261229 17777 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:04.261816 17827 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260808 01:53:04.263219  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 122910us
I20260808 01:53:04.264369  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:04.265305  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:04.266269  9474 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:53:04.267820 17758 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:53:04.268828 17758 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260808 01:53:04.272876  9635 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3342) took 1472 ms (client timeout 1787 ms). Trace:
W20260808 01:53:04.278995  9635 rpcz_store.cc:269] 0808 01:53:02.806710 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:02.811676 (+  4966us) service_pool.cc:224] Handling call
0808 01:53:04.272357 (+1460681us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540003b1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:04.272757 (+   400us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2554956160}
I20260808 01:53:04.282032 10334 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:04.281483  9366 catalog_manager.cc:5697] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 15 to 18, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 18 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:04.283339 17853 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:04.234896 17829 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.049s
I20260808 01:53:04.284304  9366 catalog_manager.cc:5697] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 17 to 19, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 19 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:04.283828 17853 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:04.311950  9475 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:04.246908 17756 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
W20260808 01:53:04.294574 17615 consensus_meta.cc:287] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.609s	user 0.007s	sys 0.015s
I20260808 01:53:04.322813  9557 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:04.286398  9366 catalog_manager.cc:5697] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 16 to 22. New cstate: current_term: 22 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 01:53:04.324361 17703 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.559s	user 0.012s	sys 0.014s
I20260808 01:53:04.325496 17703 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:04.326458  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:04.326851 17933 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:04.327795 17615 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:04.329770 17933 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:04.330652 17615 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:04.331343  9632 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 24: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:04.312635 17853 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:04.247193 17830 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:04.330380 17933 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:04.347679 17677 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.574s	user 0.017s	sys 0.005s
I20260808 01:53:04.348682 17677 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:04.350843  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:04.352632 17949 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:04.353365 17949 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:04.354714 17949 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:04.356222 17700 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.924s	user 0.017s	sys 0.008s
W20260808 01:53:04.356899 17673 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.911s	user 0.008s	sys 0.016s
I20260808 01:53:04.357663  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:04.356304  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:04.358798  9552 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
W20260808 01:53:04.364692  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 45078.5us
W20260808 01:53:04.366768  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 53361.5us
I20260808 01:53:04.368422  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:04.370921  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:04.370373  9521 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:04.378957  9555 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:04.382962  9557 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:04.366624 17615 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:04.396044  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:04.400379  9555 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:04.403774  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:04.400163  9599 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:04.408313  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:04.370306 17700 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:04.415004 17758 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:04.414805  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:04.416990 17673 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:04.471824 17673 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:04.404074 17960 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:04.428110 17952 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:53:04.476042 17892 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:04.477299 17960 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:04.477793 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: true dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:04.478547 10363 raft_consensus.cc:3055] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:53:04.478878 10363 raft_consensus.cc:740] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:04.479509  9474 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:04.479840 10363 consensus_queue.cc:260] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:04.481220 10363 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
W20260808 01:53:04.486052  9369 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50064 (request call id 239) took 83911 ms (client timeout 15000 ms). Trace:
I20260808 01:53:04.432915 17703 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:04.441915  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3348) took 1618 ms. Trace:
I20260808 01:53:04.443254  9631 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:04.444358  9551 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 20. Current term is 21. Ops: []
I20260808 01:53:04.447734  9481 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: []
I20260808 01:53:04.451632  9472 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:04.452051  9366 catalog_manager.cc:5697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 16 to 19, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 19 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:04.403676 17615 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:04.459223  9478 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260808 01:53:04.461086 17777 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:04.463568  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:04.465025  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:04.465744  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 143317us
I20260808 01:53:04.466883  9473 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:04.467486  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:04.454253 17700 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:04.468740  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:04.471129 17677 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:04.486810 17673 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:04.488188 17954 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:04.560019  9558 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 22.
I20260808 01:53:04.489488  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:04.561232  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:04.561503  9474 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:53:04.562033  9481 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:53:04.562834  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:04.563143  9559 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 22.
I20260808 01:53:04.563683  9474 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:04.559333  9366 catalog_manager.cc:5697] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 9 to 11, term changed from 15 to 19, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 19 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:04.565531  9554 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 20. Current term is 21. Ops: []
I20260808 01:53:04.492900 17853 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:04.495730  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:04.496785  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:04.498035  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:04.479009 17892 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:04.431030  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:04.502861 17942 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:04.505234 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:04.506647  9557 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:04.507754 17547 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:53:04.508337 17854 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.604s	user 0.009s	sys 0.009s
I20260808 01:53:04.509929 17894 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:53:04.519553  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 104894us
I20260808 01:53:04.521365 17927 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260808 01:53:04.523936 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:04.518078  9632 rpcz_store.cc:276] 0808 01:53:02.823219 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:03.509287 (+686068us) service_pool.cc:224] Handling call
0808 01:53:04.424037 (+914750us) spinlock_profiling.cc:243] Waited 821 ms on lock 0x7b5400307fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:04.441789 (+ 17752us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1887376640}
I20260808 01:53:04.528324  9552 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:04.530934 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:04.543493  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:04.550819  9556 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:53:04.561888  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:04.595846  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:04.493682  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:04.597187  9636 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
W20260808 01:53:04.517721  9369 rpcz_store.cc:269] 0808 01:51:40.758140 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:40.873638 (+115498us) service_pool.cc:224] Handling call
0808 01:53:04.485688 (+83612050us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:52:52.971862 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:52:52.972651 (+   789us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:52:52.972715 (+    64us) write_op.cc:435] Acquired schema lock
0808 01:52:52.972737 (+    22us) tablet.cc:662] Decoding operations
0808 01:52:53.015879 (+ 43142us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:52:53.015997 (+   118us) write_op.cc:641] Partition lock acquired for write op
0808 01:52:53.016025 (+    28us) tablet.cc:685] Acquiring locks for 19 operations
0808 01:52:53.083164 (+ 67139us) tablet.cc:701] Row locks acquired
0808 01:52:53.083188 (+    24us) write_op.cc:260] PREPARE: finished
0808 01:52:53.083422 (+   234us) write_op.cc:270] Start()
0808 01:52:53.083614 (+   192us) write_op.cc:276] Timestamp: P: 1786153973083345 usec, L: 0
0808 01:52:53.083651 (+    37us) op_driver.cc:348] REPLICATION: starting
0808 01:52:53.084514 (+   863us) log.cc:844] Serialized 7522 byte log entry
0808 01:52:53.401732 (+317218us) op_driver.cc:464] REPLICATION: finished
0808 01:52:53.403130 (+  1398us) write_op.cc:301] APPLY: starting
0808 01:52:53.403286 (+   156us) tablet.cc:1367] starting BulkCheckPresence
0808 01:52:54.289345 (+886059us) tablet.cc:1370] finished BulkCheckPresence
0808 01:52:54.289525 (+   180us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:52:54.574131 (+284606us) spinlock_profiling.cc:243] Waited 3.65 ms on lock 0x7b1800314020. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188ce62 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:52:54.893353 (+319222us) spinlock_profiling.cc:243] Waited 4.68 ms on lock 0x7b1800314020. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4185ac56 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d446d171c 00007f5d446d4127
0808 01:52:54.895877 (+  2524us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b1800314020. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d446d171c 00007f5d446d4127 00007f5d446d55e0 00007f5d4840b24b 00007f5d48455454 00007f5d482fd2ba 00007f5d482ffdab 00007f5d482ff9b3 00007f5d483fe814
0808 01:52:55.684178 (+788301us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:52:55.684632 (+   454us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=38,key_file_lookups=38,delta_file_lookups=38,mrs_lookups=0
0808 01:52:55.684713 (+    81us) write_op.cc:312] APPLY: finished
0808 01:52:55.685363 (+   650us) log.cc:844] Serialized 172 byte log entry
0808 01:52:56.225824 (+540461us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:52:56.301669 (+ 75845us) write_op.cc:454] Released schema lock
0808 01:52:56.301761 (+    92us) write_op.cc:341] FINISH: Updating metrics
0808 01:52:56.302410 (+   649us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:52:56.302448 (+    38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6766,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":1105,"apply.run_cpu_time_us":79887,"apply.run_wall_time_us":2899175,"cfile_cache_hit":78,"cfile_cache_hit_bytes":83661,"num_ops":19,"prepare.queue_time_us":472193,"prepare.run_cpu_time_us":25436,"prepare.run_wall_time_us":304810,"raft.queue_time_us":38113,"raft.run_cpu_time_us":964,"raft.run_wall_time_us":2468,"replication_time_us":317939,"spinlock_wait_cycles":56692864,"thread_start_us":1071512,"threads_started":5,"wal-append.queue_time_us":879047,"wal-append.run_cpu_time_us":11407,"wal-append.run_wall_time_us":2169372}]]}
I20260808 01:53:04.494748  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:04.598387  9634 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:53:04.563195  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:04.600764  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:04.601394  9481 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 23
I20260808 01:53:04.559341  9476 raft_consensus.cc:2393] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 21: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:04.603214 10362 raft_consensus.cc:2368] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23.
I20260808 01:53:04.578225  9629 raft_consensus.cc:2376] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 26. Re-sending same reply.
I20260808 01:53:04.582860  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:04.583050 17933 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:04.587201  9366 catalog_manager.cc:5697] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 21 to 23, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 23 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:04.593969 10334 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:04.607962 10363 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:53:04.575160 17880 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:04.582860 17853 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:04.612143 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:04.612913  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 188696us
I20260808 01:53:04.618484  9599 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:04.625403  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:04.626166  9552 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:04.630511  9474 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:53:04.634341  9477 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: []
I20260808 01:53:04.634795  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:04.615278 17892 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:04.632944  9635 raft_consensus.cc:2376] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23. Re-sending same reply.
I20260808 01:53:04.637290  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:04.640978  9473 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:04.606388 17894 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:04.653129  9476 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 21 index: 18. Preceding OpId from leader: term: 22 index: 19. (index mismatch)
I20260808 01:53:04.652642  9523 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:04.613595 17942 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:04.639395 17927 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:04.656937 17920 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:04.667161 17777 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:04.650909  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:04.669153  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 50435.3us
I20260808 01:53:04.671355 17880 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:04.671790 17615 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:04.697512  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 62862.7us
I20260808 01:53:04.677446 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:04.678283 17853 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:04.679323 17793 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:53:04.639348 17892 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:04.679819 17933 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:04.681504  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3097) took 1803 ms. Trace:
I20260808 01:53:04.703600 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:04.680929  9366 catalog_manager.cc:5697] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 13 to 19. New cstate: current_term: 19 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:04.631690 17547 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:04.683692  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:04.684523  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:04.706709  9472 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 23
I20260808 01:53:04.687211  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:04.691857  9445 leader_election.cc:400] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23.
I20260808 01:53:04.692411 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:04.707182  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:04.696063 10368 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:53:04.700933  9481 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:53:04.653443 17830 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:04.707604  9445 leader_election.cc:403] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:53:04.704473 10362 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:53:04.707904  9555 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:04.704236  9631 rpcz_store.cc:276] 0808 01:53:02.877836 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:03.604204 (+726368us) service_pool.cc:224] Handling call
0808 01:53:04.442753 (+838549us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:04.538577 (+ 95824us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:04.681250 (+142673us) spinlock_profiling.cc:243] Waited 836 ms on lock 0x7b5400307fd0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:04.681415 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1923590400}
I20260808 01:53:04.689604 17942 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:04.718035  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:04.713580 10334 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:04.721787  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 83192.7us
I20260808 01:53:04.713550  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:04.696619 17920 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:04.707108 17892 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:53:04.728081  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 72637.7us
I20260808 01:53:04.730397  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:04.731046  9523 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:04.710477 17927 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:04.744539 17932 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:04.726702 17976 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:04.747336  9552 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:53:04.748601 17547 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:04.750753 17826 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:04.752816 17894 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:04.723748 17673 raft_consensus.cc:3055] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:53:04.738356  9445 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:04.768262  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:04.733031 17777 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:04.710093 17615 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.027s
I20260808 01:53:04.765328  9521 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:04.728938 17920 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
W20260808 01:53:04.784678  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 50349.8us
I20260808 01:53:04.761407 17932 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:53:04.796240  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 84188.6us
I20260808 01:53:04.765182 17695 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:53:04.788805  9630 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:53:04.803154  9599 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:04.776676 17673 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:04.787329 17793 raft_consensus.cc:697] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:04.791280 17881 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:53:04.792098 17848 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:04.809507  9633 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:53:04.821915  9472 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:53:04.799351 17932 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
W20260808 01:53:04.823510  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 187701us
I20260808 01:53:04.824860 10334 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:04.801931 17695 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:04.835821 17695 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:53:04.844308 17826 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:04.843696 17673 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:04.845098 17826 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:04.847188 17927 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:04.855333 17848 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:04.856173 17848 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
W20260808 01:53:04.861524  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:04.869100  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:04.812103 17942 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260808 01:53:04.871922  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 140923us
I20260808 01:53:04.838315 17954 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:04.808295 17777 raft_consensus.cc:697] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:04.846303 17673 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:04.809940 17830 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:53:04.880265  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:04.880350  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:04.882845  9555 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:53:04.884243  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:04.828059 17880 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23.
W20260808 01:53:04.885239  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 56237.2us
I20260808 01:53:04.892355  9481 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:04.899241  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:04.883206 17881 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:04.906152 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 98988.2us
I20260808 01:53:04.902637 17872 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:04.903855  9551 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
I20260808 01:53:04.902881 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:04.896064 17615 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election won for term 22
W20260808 01:53:04.914189  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 111238us
I20260808 01:53:04.908249 17872 raft_consensus.cc:697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:04.914948 17756 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:04.915704  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 179499us
I20260808 01:53:04.911136 17987 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.019s
I20260808 01:53:04.901365 17830 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:04.921437 10332 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:04.932736  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:04.927690  9601 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:04.927719 17793 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:04.815924  9366 catalog_manager.cc:5697] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:04.930426  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:04.940486  9521 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:04.942044  9599 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:04.945868 17769 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:04.946414 17769 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:04.946822 17769 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:04.950934 17827 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:53:04.951797 17827 raft_consensus.cc:697] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:04.953747 17827 consensus_queue.cc:237] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:04.940425 17960 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:04.963948 17960 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:04.944432  9601 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:04.936669 17615 raft_consensus.cc:697] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:04.940414 17854 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:04.971846  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:04.974619 17933 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260808 01:53:04.956996 17756 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:04.935410 17942 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.035s
W20260808 01:53:04.986763  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:04.965914 17960 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:04.969668 17758 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:04.989698 17756 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:04.943984 17777 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:04.996667  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:04.997438 17949 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:04.970261 17954 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260808 01:53:05.005374  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 75600.1us
I20260808 01:53:04.984855  9366 catalog_manager.cc:5697] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 16 to 19. New cstate: current_term: 19 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:05.013808 17756 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:05.018780 17954 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:05.034737 17872 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:05.075572 17769 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.019694 17758 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:05.079737 17758 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.081641 17695 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:05.032102 17677 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:05.083319 17677 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 24 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:05.084887 17695 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:05.084514 17677 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.086148  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:05.086581 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:05.086922  9479 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:05.088608 17952 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:05.089100 17952 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:53:05.031502  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 63970.7us
I20260808 01:53:05.044210  9521 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:05.044489 17987 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:05.045310 17756 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.049481  9559 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: []
I20260808 01:53:05.052150  9637 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
W20260808 01:53:05.052145  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:05.054297 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 145680us
I20260808 01:53:05.101796 10332 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:05.054884  9551 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:05.058919  9631 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260808 01:53:05.063025 17880 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:05.064015 17830 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.066932 17615 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.067946  9366 catalog_manager.cc:5697] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 9 to 18. New cstate: current_term: 18 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:05.006793 17881 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:05.122972  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:05.124028 17952 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.124686  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:05.125744 17952 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:05.070811 17954 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
W20260808 01:53:05.072692 17986 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.138s	user 0.002s	sys 0.003s
I20260808 01:53:05.020195 17949 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:05.008785 17892 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.109603  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:05.145303  9480 raft_consensus.cc:2393] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 21: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:53:05.120378  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 76393us
I20260808 01:53:05.128088  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:05.122308 17756 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:05.143875  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:05.111101 17695 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:05.145977  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:05.155069  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:05.161990 10369 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
I20260808 01:53:05.127731 17942 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:05.167222  9559 raft_consensus.cc:2368] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23.
I20260808 01:53:05.128139 17769 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:05.169338  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 88369.7us
I20260808 01:53:05.164374  9637 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
W20260808 01:53:05.167724  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 124832us
I20260808 01:53:05.143209 17673 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:05.177997  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:05.178013  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:05.183795  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:05.184742  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:05.163210 17695 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:05.192333  9633 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:05.193533  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:05.200258  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 108333us
I20260808 01:53:05.202380  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:05.203547  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:05.179391 17673 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:05.209616  9637 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:05.213698  9560 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:05.216384  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:05.209772  9523 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:05.176188 17826 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:05.217469  9638 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:05.224220  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:05.226289  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:05.228125  9634 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:05.206493 17673 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:05.233604  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:05.235589  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:05.236697 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 135002us
I20260808 01:53:05.238183  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:05.235271  9636 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:05.251300 10332 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:05.229175 17942 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:53:05.261675  9557 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:53:05.258496  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:05.268818  9629 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:05.223760 17848 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:05.272413  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:05.273139  9556 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 22
I20260808 01:53:05.284237 17920 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:05.296329 17995 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:05.296756  9555 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:05.296919 17995 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:05.299343 17994 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260808 01:53:05.300076 17920 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:05.300820 17920 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260808 01:53:05.295949 18002 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260808 01:53:05.311568 17826 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:05.252206 17989 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.040s
I20260808 01:53:05.277240 17872 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:53:05.231364 17949 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.316939  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:05.325469  9560 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:53:05.328261  9634 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
W20260808 01:53:05.329033 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 91582.6us
I20260808 01:53:05.306143  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:05.249287 17695 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.304113 17547 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:05.346379  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:05.337952  9556 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:05.233306  9366 catalog_manager.cc:5697] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 12 to 15. New cstate: current_term: 15 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:05.354125  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:05.345912  9634 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260808 01:53:05.355800 10332 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:05.330765 17892 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:05.347222 17986 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260808 01:53:05.328742 17872 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:05.363515 18003 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:05.364300 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:05.366833  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:53:05.367022 17854 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:05.446151  9638 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:53:05.446581  9633 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:05.375041 17695 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:05.376555 17848 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:05.377393 17932 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.377503  9366 catalog_manager.cc:5697] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:05.346649 17547 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:05.379091 17960 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.355763 18004 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:05.355515 17991 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:05.357834 17793 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:05.387884 17995 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:05.388890 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 52758.6us
I20260808 01:53:05.389792  9632 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:05.411609  9479 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:05.449049  9599 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:05.450816  9636 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:05.451713  9637 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
W20260808 01:53:05.453094  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 217822us
I20260808 01:53:05.505681  9524 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:05.508049  9631 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260808 01:53:05.508706 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:05.466697 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:05.494776  9629 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:53:05.475560  9634 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260808 01:53:05.445223 17872 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
W20260808 01:53:05.453845  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 226524us
W20260808 01:53:05.508234  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 178277us
I20260808 01:53:05.506014 17991 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.467617 17854 raft_consensus.cc:697] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:53:05.525151  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 76226us
I20260808 01:53:05.526711  9521 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:05.501153 17793 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:05.527416  9599 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:05.527416  9601 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:05.459035 17942 raft_consensus.cc:697] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:05.537544  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:05.534314  9444 connection.cc:256] RPC call timeout handler was delayed by -1.1705240879964549s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1545s for a response
W20260808 01:53:05.554054  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:05.554328  9444 connection.cc:256] RPC call timeout handler was delayed by -1.0742240879917517s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1652s for a response
W20260808 01:53:05.554791  9444 connection.cc:256] RPC call timeout handler was delayed by -1.025624088011682s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1706s for a response
I20260808 01:53:05.527915 17848 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:53:05.528548 17996 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260808 01:53:05.564213 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:05.565804  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:05.568871 10369 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21. Ops: []
I20260808 01:53:05.571336  9481 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260808 01:53:05.459522 18003 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:05.575033 17952 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:05.576372  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:05.578977  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:05.579524 18016 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260808 01:53:05.580487  9524 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:05.547523 17995 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:05.598384  9632 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:53:05.603551 18003 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:05.603322 17547 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.604203  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:05.604928 17547 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:05.563345 17758 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:05.606964 17673 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:05.606805 17952 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.609697 17673 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:05.610021 17952 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:05.610805  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:53:05.611137  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 93194.1us
I20260808 01:53:05.611285 17976 raft_consensus.cc:3055] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:05.611742 17976 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:05.613189 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 111191us
I20260808 01:53:05.616146 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:05.562089 18014 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:05.634929 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:05.635233 17942 consensus_queue.cc:237] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:05.581530 17989 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:05.632333 17756 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:05.525452 18004 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:05.643944  9521 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:05.596136 17615 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:05.652225 17615 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.649770 10362 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:05.638401 18014 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:53:05.658149  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:05.630539 17769 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.026s
I20260808 01:53:05.663415 17769 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:53:05.663851 18005 raft_consensus.cc:2764] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260808 01:53:05.664821 18013 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.069s
I20260808 01:53:05.627410 17952 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.029s
W20260808 01:53:05.676846  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 104319us
I20260808 01:53:05.714145  9479 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260808 01:53:05.650835  9366 catalog_manager.cc:5697] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 9 to 13. New cstate: current_term: 13 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:05.717106  9366 catalog_manager.cc:5697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 12 to 16. New cstate: current_term: 16 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:05.719130  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:05.649372 17854 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:05.656648 17881 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:05.728598  9600 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:05.736615 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 122627us
I20260808 01:53:05.736842 18012 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:53:05.645462 17996 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.034s
I20260808 01:53:05.738236 17995 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:05.739387 17932 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:05.741060 18013 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:53:05.757982 18013 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:05.758613 18013 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
W20260808 01:53:05.762380 17954 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.600s	user 0.009s	sys 0.001s
I20260808 01:53:05.741266 17991 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:05.742358 17995 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:05.767338 17996 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:05.771085 17995 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:05.770299  9480 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260808 01:53:05.663851 17769 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:05.667939 18014 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:05.674137 17826 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader election won for term 23
W20260808 01:53:05.777709  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 197299us
I20260808 01:53:05.771391 17996 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:05.791287  9481 raft_consensus.cc:3055] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:53:05.793630  9481 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:05.641804 17793 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.742520 10332 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:05.786929 17758 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:05.801769  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:05.804610  9556 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:05.789386 18012 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:05.791970 17960 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:05.810232  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:05.774012 17881 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:05.777168 17954 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.807850  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:05.803682  9559 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260808 01:53:05.790768 17673 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:05.810003 17848 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:53:05.815835 17960 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:05.824766 17848 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:05.826242 17999 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:05.809094 18012 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:05.816097  9444 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:05.826958 17999 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:05.827598 18012 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.828853 18004 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:05.829208  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:05.829568 18012 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:05.831012 18008 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:53:05.831318 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:05.831465 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:05.832839 18017 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260808 01:53:05.803123 17952 consensus_queue.cc:1059] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.030s
I20260808 01:53:05.826642 17848 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:05.826256 17769 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:05.840529 17769 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:05.841701  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:05.818660  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:05.842690 17700 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:53:05.843371  9472 raft_consensus.cc:2393] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 16: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:05.843717 18008 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:05.844643  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:05.844964 18008 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.845710  9480 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:05.846138  9480 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:05.819262  9475 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:05.792120 17826 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:05.848959 17826 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:05.773203 17976 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.850379 17952 raft_consensus.cc:3055] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:53:05.819700  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:05.851524 17952 raft_consensus.cc:740] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:05.852640 17952 consensus_queue.cc:260] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.835331  9474 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:05.826586  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:05.834564 18011 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260808 01:53:05.850195 17987 raft_consensus.cc:3055] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Stepping down as leader of term 20
W20260808 01:53:05.816607  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:05.804760 17932 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:53:05.859138 17987 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:05.843266 17758 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.859354 17932 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:05.851028 17976 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:05.808326  9481 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.817445 18024 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260808 01:53:05.825475  9556 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260808 01:53:05.860033  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:05.922613  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:05.923377  9472 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:05.861675  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:05.861711 17673 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:05.864043  9552 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:05.863770 17954 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:05.865662 17920 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:05.860998  9481 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:05.860350 17987 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.867599  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:05.943835  9478 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:05.944057 17987 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:05.945221  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:05.829818 17999 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:05.902849  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 61238.6us
I20260808 01:53:05.903893 17952 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:05.907902  9473 raft_consensus.cc:3055] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:05.908159 17848 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:05.909056 18019 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.312s	user 0.002s	sys 0.000s
I20260808 01:53:05.909541  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:05.910578 17615 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:05.910722 17881 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:05.911231  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:05.902966  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:05.913323  9480 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.915774 17777 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:05.917883 17996 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:05.918488 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:05.922482  9556 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260808 01:53:05.923496  9557 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:53:05.860453 17932 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.923981 18000 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:05.925024  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:05.927384 17854 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260808 01:53:05.929898  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:05.937119 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 194680us
I20260808 01:53:05.939919  9477 raft_consensus.cc:2393] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 16: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:05.972805  9473 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:06.005237  9550 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
W20260808 01:53:06.031520  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 101693us
W20260808 01:53:06.032289  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 129453us
W20260808 01:53:06.027796  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 89842.3us
I20260808 01:53:06.033603  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:06.040007  9480 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:06.041495  9475 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260808 01:53:06.043037  9599 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:06.043959  9472 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:06.044551  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:05.999914 17933 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:05.970952 17927 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:06.027011 18000 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:06.071483  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 44089.1us
I20260808 01:53:06.048295 17960 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:06.117785  9481 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 19 index: 10. (term mismatch)
I20260808 01:53:06.118245  9481 pending_rounds.cc:85] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 9
I20260808 01:53:06.118469  9481 pending_rounds.cc:107] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 20.10
I20260808 01:53:05.972353 17920 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:06.041563 18004 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:53:06.073698 17673 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:06.075083 18007 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:06.079680 18009 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:06.080651  9473 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:05.979698 18013 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:06.046693 17832 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:06.023957 17989 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:06.042987 17996 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260808 01:53:06.086520 17756 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:53:06.088919 17848 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:06.089857 18022 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.041s
I20260808 01:53:06.068813 18024 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:06.162348  9551 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
W20260808 01:53:06.091579 17872 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.561s	user 0.011s	sys 0.002s
I20260808 01:53:06.070228 18003 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:06.069981 17615 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.026s
I20260808 01:53:06.032572 17777 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.118770  9481 raft_consensus.cc:2889] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:53:06.168960  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 525054us
W20260808 01:53:06.119325 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:06.124468 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 97207.8us
W20260808 01:53:06.110263 17827 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.817s	user 0.020s	sys 0.024s
W20260808 01:53:06.128031 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 177041us
I20260808 01:53:06.134402  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:06.132923 18007 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:06.164069  9473 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:06.091581 18028 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.075s
I20260808 01:53:06.138865 18009 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.176599  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:06.178787 17827 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.892s	user 0.021s	sys 0.024s
I20260808 01:53:06.122639 17933 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:06.187585  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:06.187901 10369 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:06.125531 17927 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:06.189527  9481 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 22.
I20260808 01:53:06.190428  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:06.192797  9552 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260808 01:53:06.182343  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:06.193823 10362 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260808 01:53:06.195745  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:06.188727 17832 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:06.203394 17827 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:06.184219 18007 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:06.191568 17848 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:06.165691 17989 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.207115 17832 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.218307 17832 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:06.214468 17989 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:06.220613 18000 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:06.220960 10332 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:06.223374 17920 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:06.193475 17987 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:06.212069 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:06.189435 18003 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:06.173748 17872 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.224999 17995 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.198755 18011 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:06.230185  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:06.231542  9630 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:06.233515  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 111078us
W20260808 01:53:06.235314 18018 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.661s	user 0.000s	sys 0.004s
I20260808 01:53:06.235664 10366 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:06.236148 17933 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:06.236651  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:06.238018 17927 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:06.229584  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:06.243486  9631 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:06.210829 17673 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:06.251639 17673 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.228427 17848 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.254043 18013 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:06.257293  9631 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 22.
I20260808 01:53:06.247324  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:06.259285  9634 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:06.247828 18044 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:06.262935 18044 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:06.263399 18044 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:53:06.263778  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:06.264281  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:06.268261  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:06.268949  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:06.269898 17920 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:06.270608 17920 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:06.228226 17987 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:06.271391  9472 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:06.271766 17920 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:06.272621  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:06.273860  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:06.274554 17700 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:53:06.274894 17989 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:06.280586 10365 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:06.271793 17987 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:06.282454 17987 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:06.283115  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:06.285540 18009 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:06.250659 17995 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:06.287038  9599 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:06.287671  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:06.289577  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:06.248636 17872 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:06.243848 18003 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.249137 10363 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:53:06.249962 17793 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:06.250461 17777 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:06.204136 17827 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:06.239466 18042 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:53:06.246565 18011 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 24 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:06.294507  9632 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:53:06.275723 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:06.299777  9478 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:06.311379  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3887) took 1092 ms. Trace:
W20260808 01:53:06.300331 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 79470.1us
W20260808 01:53:06.288198  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48509.1us
I20260808 01:53:06.304379  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:06.232075 17996 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:06.307564  9473 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 13. (term mismatch)
I20260808 01:53:06.300360  9476 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260808 01:53:06.311178 10366 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:06.323194  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:06.324486  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:06.302538 17933 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:53:06.326749 17933 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:06.327203 17933 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:06.327534 17827 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:06.329071 18011 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:06.326777 18028 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.005s
I20260808 01:53:06.332492 17848 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:06.332975  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:06.318991 17881 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.015s
I20260808 01:53:06.311867  9560 rpcz_store.cc:276] 0808 01:53:05.219268 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:05.352020 (+132752us) service_pool.cc:224] Handling call
0808 01:53:05.352478 (+   458us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:05.466071 (+113593us) raft_consensus.cc:1534] Early marking committed up to index 12
0808 01:53:05.640820 (+174749us) log.cc:844] Serialized 14 byte log entry
0808 01:53:06.029167 (+388347us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:06.029850 (+   683us) log.cc:844] Serialized 52 byte log entry
0808 01:53:06.030013 (+   163us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:53:06.030248 (+   235us) log.cc:844] Serialized 14 byte log entry
0808 01:53:06.032564 (+  2316us) log.cc:844] Serialized 14 byte log entry
0808 01:53:06.032882 (+   318us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:06.032980 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:06.266191 (+233211us) spinlock_profiling.cc:243] Waited 30.4 ms on lock 0x7b180040ff60. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188ce62 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:53:06.311132 (+ 44941us) raft_consensus.cc:1685] finished
0808 01:53:06.311158 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:06.311302 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21753,"prepare.queue_time_us":334373,"prepare.run_cpu_time_us":156,"prepare.run_wall_time_us":140,"raft.queue_time_us":71457,"raft.run_cpu_time_us":404,"raft.run_wall_time_us":25815,"spinlock_wait_cycles":163798784,"thread_start_us":518320,"threads_started":2,"wal-append.queue_time_us":494119}
I20260808 01:53:06.320012 17995 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.024s
I20260808 01:53:06.319615 18042 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:06.327711 17673 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:06.335219 18003 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:06.335258 18042 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:53:06.343966  9473 pending_rounds.cc:85] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 12
I20260808 01:53:06.344394  9473 pending_rounds.cc:107] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260808 01:53:06.344795  9473 raft_consensus.cc:2889] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:06.320744  9634 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:53:06.331522  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:06.286401 18014 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:06.355051  9480 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 13. (term mismatch)
I20260808 01:53:06.359330  9480 pending_rounds.cc:85] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 12
I20260808 01:53:06.359885  9480 pending_rounds.cc:107] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 23.13
I20260808 01:53:06.360385  9480 raft_consensus.cc:2889] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:53:06.362707  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:06.350147 17927 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.013s
I20260808 01:53:06.307583 17700 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:06.342864 17777 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
W20260808 01:53:06.384382  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 52946.9us
I20260808 01:53:06.386080  9521 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:06.310091 17952 consensus_queue.cc:1059] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.068s
I20260808 01:53:06.348681  9475 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:06.425967  9475 raft_consensus.cc:606] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:06.426452  9475 raft_consensus.cc:614] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:53:06.387352  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 54597.1us
I20260808 01:53:06.387100 17700 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:06.387768 17932 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:06.377650 18024 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:06.387163 17777 raft_consensus.cc:3055] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:06.395355  9559 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21. Ops: []
I20260808 01:53:06.396584 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:06.442107  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:06.442200 10365 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:53:06.442890  9558 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 19.
I20260808 01:53:06.377163 17989 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:06.397550 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:06.397536 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:06.398654 18014 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:06.405550  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:06.405560 10362 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260808 01:53:06.406396 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:06.394173  9600 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:06.419863  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:06.420205  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:06.420737  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:06.420910  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:06.456121  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:06.456549  9551 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:06.457090  9550 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 21: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:06.429026  9630 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 22.
I20260808 01:53:06.456168  9556 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:06.446936 10361 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:06.429356  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:06.432447  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:06.433758  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47821.3us
W20260808 01:53:06.429401  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 107909us
W20260808 01:53:06.465270  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:06.440339  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:06.396860 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:06.378268 17949 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:06.455790 17830 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260808 01:53:06.455833  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 61793.5us
I20260808 01:53:06.469187  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:06.472894  9599 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:06.463900  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:06.484364  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:06.438385 18004 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:06.495280  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:06.488226  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:06.499024  9476 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 21: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:06.441219 17777 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:06.377302 17996 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.502018  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:06.458483 18009 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:06.512981 18046 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20260808 01:53:06.438220 17976 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.056s
I20260808 01:53:06.519210 17677 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Stepping down as leader of term 21
W20260808 01:53:06.520843  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 103b913d6c014692b6dea1780af2aa20 (from 6e7856bed1be43acaa821290d506ee97 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:06.478221 18012 raft_consensus.cc:3055] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:06.472103 17830 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:06.543953  9523 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:06.526503  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 56143.6us
I20260808 01:53:06.427214 10332 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:06.552773 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 192747us
I20260808 01:53:06.553547 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:06.554664 10332 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:06.556025 10332 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:06.515528 18044 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:06.525444  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 49613.8us
I20260808 01:53:06.550165  9445 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:06.529129  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 56384.4us
I20260808 01:53:06.530774  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:06.461630 17927 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.066s
I20260808 01:53:06.560370  9600 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:06.564492  9600 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:06.545356 18056 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:06.545775 17695 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:06.444936 18024 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.524747 17793 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:06.560634  9599 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:06.565521  9550 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:53:06.566650 10332 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:06.569000  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 105225us
I20260808 01:53:06.569792 10369 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260808 01:53:06.570796 18008 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:06.571523 17976 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:06.571612  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:06.572034 17854 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:53:06.550601 17830 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:06.572328 17999 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:53:06.575513 18044 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:06.587457 17848 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:06.536120 18009 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:06.543238 18012 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:06.508816 18004 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.543836 17949 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260808 01:53:06.580893 10361 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 22 index: 19. (index mismatch)
I20260808 01:53:06.494608 17952 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:06.533128 17677 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:06.542758 17942 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:06.588254  9551 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:06.586313 18056 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:06.564543 18047 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:06.569415 17673 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:53:06.627811  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:06.605547  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 134747us
I20260808 01:53:06.600626 17826 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260808 01:53:06.633797  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 83703.5us
I20260808 01:53:06.530292 18007 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.581523 17769 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:06.547483 18013 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.051s
I20260808 01:53:06.578097  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:06.563990 17989 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.064s
I20260808 01:53:06.558797 18028 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:06.656702  9559 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:06.658291  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:06.600881 17976 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:06.653172 18018 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260808 01:53:06.679565 18008 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:53:06.686842 17976 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:06.688064 18045 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:53:06.688640 18050 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260808 01:53:06.690790 18028 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:06.691399 18028 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
W20260808 01:53:06.695017  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:06.706229  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 145667us
I20260808 01:53:06.713883  9599 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:06.659065 17756 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:06.737178  9599 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:06.734398 17832 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260808 01:53:06.733503  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:06.742168  9472 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260808 01:53:06.743036 18029 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260808 01:53:06.745301 18004 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:06.745615  9472 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
W20260808 01:53:06.750890  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 121724us
I20260808 01:53:06.675851 17769 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:06.657313 17952 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:06.764231  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 59944.9us
I20260808 01:53:06.635334 17777 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.769889  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:06.773922  9629 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:06.768162  9599 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:06.691733 17677 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.783124 18006 consensus_queue.cc:1059] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:06.742583 17832 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:06.784373 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:06.792886  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:06.757050 17700 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:06.803368  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:06.765364 17769 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:06.778832 17848 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:06.746933 18012 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:06.776053 17991 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:06.770625 18047 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:06.768991 17854 raft_consensus.cc:697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:53:06.823778 18063 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
W20260808 01:53:06.825062 18057 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260808 01:53:06.824970 17848 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.802042 17832 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:06.832628 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:06.755009 17756 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.763213 18060 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.070s
W20260808 01:53:06.859016  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:06.873422 10369 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 21: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:53:06.873502  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:06.832671 17695 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:06.819118 17677 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:06.813658 17673 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:06.839481 17996 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:06.827384 17777 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Advancing to term 26
W20260808 01:53:06.906016  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 125447us
W20260808 01:53:06.853140 17933 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.525s	user 0.012s	sys 0.003s
I20260808 01:53:06.887400 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:06.902141 10365 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:57156
I20260808 01:53:06.765693 18056 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.855351 18012 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:06.855351 17999 raft_consensus.cc:3055] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Stepping down as leader of term 20
W20260808 01:53:06.917055  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 124285us
I20260808 01:53:06.927145  9472 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260808 01:53:06.927335 10365 raft_consensus.cc:606] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:06.931576 10365 raft_consensus.cc:614] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:06.921458 18024 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:06.936293 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 48978.6us
I20260808 01:53:06.939332  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:06.939697  9559 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260808 01:53:06.941349 17756 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:06.942163  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:06.942373  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:06.943286 18039 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.021s
I20260808 01:53:06.976661 18065 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.081s
I20260808 01:53:06.944941 18074 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:06.978893 18056 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:06.979143 18074 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:06.979692 18074 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:06.980494 18071 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:06.946967 18072 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election won for term 24
W20260808 01:53:06.949460  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:06.981189 18071 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:06.982237  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45598.8us
I20260808 01:53:06.950119 17854 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.983481 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:06.932932 17999 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:06.985697 17999 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:06.987160 17999 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:06.951323 18047 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.953181 17952 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:06.996064 18076 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:06.997136 18076 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:07.004177 18071 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:06.953701 17673 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:06.954655 18007 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:06.955161 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:06.962991 18062 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260808 01:53:06.963421 18067 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:06.935709 18009 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:06.964202  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:07.024308  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:06.964620  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:06.965839  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:06.966908  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:07.029510  9629 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 21: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:06.969098 17872 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:53:06.973186  9550 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 26: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:06.973124 10332 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:06.921960 17933 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:06.976948  9555 raft_consensus.cc:2410] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 24 because replica has last-logged OpId of term: 22 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260808 01:53:06.987449  9638 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260808 01:53:07.011185  9444 leader_election.cc:400] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23.
I20260808 01:53:07.016806 18072 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:07.027861 10369 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 18: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:07.029644  9476 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:53:07.023290 17952 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:07.046237 17952 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:07.046242 17793 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:07.051160 17793 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:07.052104  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:07.055022 17793 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:07.055805 17952 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:07.060331 17933 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:07.061739 10361 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:07.067831 10334 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:07.044555  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:07.069542 17949 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:53:07.070215 17949 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:07.070080 18072 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:07.071445 17949 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:07.078413 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 105384us
W20260808 01:53:07.081463 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:07.082436 10332 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:07.085451  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:07.086140 17826 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:07.092754  9555 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
W20260808 01:53:07.096217 18042 consensus_meta.cc:287] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.760s	user 0.015s	sys 0.000s
I20260808 01:53:07.096086 17769 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:07.096575 17695 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:07.099169 17830 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:07.101879 17976 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:07.104326 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:07.104926 17769 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:07.110392 18065 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:53:07.115559  9633 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: [23.12-23.12]
I20260808 01:53:07.116463  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: true dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:07.123711  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:07.130309  9630 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:07.123802  9554 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260808 01:53:07.119154  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:07.131575  9634 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:07.123384  9635 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:07.123126  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:07.123425  9637 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:07.132884  9556 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:07.123955  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:07.134874  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 11c99c174e234329801ee9b1ceae68d9 (from 7201605bb2f54c949a4ebe6e9b3b4b2a to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:07.137995  9558 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 21: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:07.140033  9631 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 26: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:07.141386  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:07.123871 17700 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.025s
I20260808 01:53:07.129366 18065 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:07.149942 18065 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:53:07.152025  9636 raft_consensus.cc:2410] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 21 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260808 01:53:07.153604  9475 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:07.157224  9634 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:07.123445 18080 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:07.166227  9551 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:07.123644 18066 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:07.134506 17933 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:07.096850 18042 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:07.174177  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:07.120021 18028 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:07.179234  9556 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:53:07.183123  9630 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:53:07.187638 10369 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:07.122391 17777 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.043s
W20260808 01:53:07.194089  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:07.152635 18076 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:07.192807  9600 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:07.193277  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:07.198879  9635 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 22: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:07.199911  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:07.200893  9556 raft_consensus.cc:1217] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Deduplicated request from leader. Original: 17.11->[19.12-21.13]   Dedup: 19.12->[21.13-21.13]
I20260808 01:53:07.201697  9556 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: [19.12-21.13]
I20260808 01:53:07.195138  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:07.129827 17848 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.051s
I20260808 01:53:07.196580  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:07.166846 17960 raft_consensus.cc:3055] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:53:07.203226  9634 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 21: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:07.197165  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:07.206696  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:07.210287 10368 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
I20260808 01:53:07.210232  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:07.211189 10365 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:57156
I20260808 01:53:07.211795  9634 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:53:07.211891 10365 raft_consensus.cc:606] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:07.213687  9632 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 21: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:07.193928 17995 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:07.165215 17999 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.043s
I20260808 01:53:07.226289 18073 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260808 01:53:07.229899  9551 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260808 01:53:07.207389 17960 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:07.211632 18009 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:53:07.214444 18076 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:07.237574  9476 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:07.254771  9476 raft_consensus.cc:606] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:07.255374  9476 raft_consensus.cc:614] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:53:07.221195 18045 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.532s	user 0.010s	sys 0.004s
I20260808 01:53:07.195981 17933 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:07.261878  9551 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260808 01:53:07.220758 17700 consensus_queue.cc:1243] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 22.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
W20260808 01:53:07.242728 18086 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260808 01:53:07.244112 18042 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:07.180946 18080 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:07.195850 18066 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:07.251471 18083 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260808 01:53:07.252038 17976 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:07.304934 18066 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:07.305642 18066 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:07.264889 18045 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:07.253607  9635 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:07.310941  9629 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
W20260808 01:53:07.240530 17932 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.800s	user 0.013s	sys 0.002s
I20260808 01:53:07.310391  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:07.288313  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:07.326735  9551 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:53:07.289324  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:07.328058  9634 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:07.300350 17756 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:07.254352 18076 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:07.306136 18066 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:07.280762 18009 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:07.310427 18080 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:07.336795  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:07.337393 18073 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260808 01:53:07.339047  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 99283.7us
I20260808 01:53:07.340302  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:07.339259  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:53:07.341574  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet e3814b146ccd472f9304a0be09ad3738 (from 6e7856bed1be43acaa821290d506ee97 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:07.343995  9521 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:07.266691 18092 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.073s
I20260808 01:53:07.357709  9634 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:53:07.358958  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:07.359711  9634 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
W20260808 01:53:07.360879  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:07.365656  9479 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260808 01:53:07.368934  9521 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:07.370272  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:07.373234  9481 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:07.345566 18008 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:07.378043  9634 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:07.292750 18011 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.062s
I20260808 01:53:07.378633  9551 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:53:07.400903  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:07.401996  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:07.402143  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:07.403458  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:07.355549 17932 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:07.411216  9551 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:53:07.413352  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:07.336133 17756 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:07.364938 18024 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:07.371268 17942 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:07.415683  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:07.415783  9554 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:07.417598 10365 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fca874793205463dbb922993f57bf7f5"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:57156
I20260808 01:53:07.417119  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:07.415328 17793 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.012s
I20260808 01:53:07.472361  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:07.427073  9551 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260808 01:53:07.428054 17942 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:07.483312 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:53:07.429984 17991 consensus_meta.cc:287] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.577s	user 0.012s	sys 0.000s
I20260808 01:53:07.425426 18024 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:07.411691 18008 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:07.482213 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:07.331665 17960 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:07.431118  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:07.467682  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:07.588414  9633 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:53:07.476531  9634 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:07.418963 18012 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:07.451123 18009 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:07.473747 17677 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.575s	user 0.010s	sys 0.001s
I20260808 01:53:07.475963 17756 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:07.453102 18028 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:07.472070 18066 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:07.479830  9473 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 24: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:53:07.496436  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:07.580567 10365 raft_consensus.cc:606] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:07.586228 10363 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:07.496105  9481 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
W20260808 01:53:07.430335 17832 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.535s	user 0.011s	sys 0.000s
I20260808 01:53:07.452598 17933 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:07.599380  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:07.635104  9637 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 21: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:07.577263 17920 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:07.623911  9633 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: []
I20260808 01:53:07.638741  9634 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:07.586858 17991 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:07.631593  9635 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:07.419634 18065 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:07.641225  9364 maintenance_manager.cc:419] P 3cd8e8730323483db1c5f22e90bb080c: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033397
I20260808 01:53:07.619592 17932 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:07.624809 17960 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:53:07.647759  9637 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260808 01:53:07.632050 17756 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:07.711961  9637 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
W20260808 01:53:07.695928 18054 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260808 01:53:07.732076  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3602) took 1172 ms. Trace:
I20260808 01:53:07.639580 17832 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:07.732589  9475 rpcz_store.cc:276] 0808 01:53:06.549188 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:06.549308 (+   120us) service_pool.cc:224] Handling call
0808 01:53:06.573671 (+ 24363us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:07.154039 (+580368us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:07.720974 (+566935us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b5400399754. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:07.721269 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1333267712}
I20260808 01:53:07.737716  9633 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
W20260808 01:53:07.741133  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 329495us
I20260808 01:53:07.743296  9638 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:53:07.644555 17920 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:07.745244 10363 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:07.746404  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:07.750573  9600 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:07.753750  9630 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 25: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:07.726260 18047 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:07.732275 18045 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:07.758035 10367 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260808 01:53:07.776445 18024 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:07.777493 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:07.779444  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 377377us
I20260808 01:53:07.781177  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:07.782616 18092 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:07.783062 18092 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:07.783423 18092 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:07.784812 17976 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:07.785403 17976 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:07.787104 17677 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:07.734321 18066 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:07.787842 18066 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:07.787705 17677 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:07.790304 18066 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:07.790752 17677 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:07.792728  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:07.722376 17756 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:07.793238  9636 raft_consensus.cc:1240] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:07.794250  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:07.786440 17976 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:07.796691  9478 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260808 01:53:07.771706 17942 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:07.761093 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:07.798332  9472 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:07.794965  9474 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:07.764092 18047 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:07.799829 18090 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:07.800364 18090 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:07.801524 18047 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:07.801326 18090 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:07.803261  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:07.804603 17920 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:07.806282 17756 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:07.808368 17832 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:07.808678 17830 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:07.809950  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 1034) took 1057 ms. Trace:
I20260808 01:53:07.810385  9632 rpcz_store.cc:276] 0808 01:53:06.752980 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:06.753162 (+   182us) service_pool.cc:224] Handling call
0808 01:53:07.809707 (+1056545us) spinlock_profiling.cc:243] Waited 443 ms on lock 0x7b540032c654. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:07.809881 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1018145152}
I20260808 01:53:07.812505 17942 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:07.813156 17942 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:07.815382  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:07.816313  9551 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 20: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:07.816586 18008 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:07.817243 18008 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:07.817788  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:07.817715 18008 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:07.818766  9633 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:07.823144  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:07.827152  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:07.829433  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:07.758889 17793 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:53:07.830659  9480 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:53:07.831302  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:07.832388 18056 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
W20260808 01:53:07.833001 18074 consensus_meta.cc:287] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.853s	user 0.011s	sys 0.005s
I20260808 01:53:07.833446 18074 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:07.833715  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:07.834975 18101 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:07.835623 18101 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 24 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:07.835857 10362 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 21: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:07.764890 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:07.839005  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:07.840080 10368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 937) took 1633 ms. Trace:
I20260808 01:53:07.829967 17942 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:07.840549 10368 rpcz_store.cc:276] 0808 01:53:06.203457 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:06.209469 (+  6012us) service_pool.cc:224] Handling call
0808 01:53:06.232696 (+ 23227us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:07.210826 (+978130us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:07.836694 (+625868us) spinlock_profiling.cc:243] Waited 977 ms on lock 0x7b54005122d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:07.836878 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2247786240}
I20260808 01:53:07.844625  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:07.802516 18090 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:07.760149 17991 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:07.846205  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:07.865418  9556 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 24: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:53:07.861958  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:07.873332  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:07.874997  9557 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:07.876268  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:07.877326  9478 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:53:07.879896 10364 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:53:07.870393 17989 consensus_queue.cc:1059] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.010s
I20260808 01:53:07.881492  9552 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260808 01:53:07.883616 10365 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:53:07.836575 18045 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:07.893956 18045 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:07.896651 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:07.901966  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:07.911536  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:07.913501  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:07.922629  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:07.929602 10368 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:07.932019 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:07.940593 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 1255) took 1154 ms. Trace:
I20260808 01:53:07.936535  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:07.969997 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:07.964764 10361 rpcz_store.cc:276] 0808 01:53:06.761721 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:06.761937 (+   216us) service_pool.cc:224] Handling call
0808 01:53:07.916196 (+1154259us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b5400512cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:07.916404 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":607384448}
W20260808 01:53:07.944159 18107 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.058s	user 0.000s	sys 0.002s
I20260808 01:53:08.038722  9600 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:07.908574 17991 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:07.941045 18011 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:08.039719  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:07.928064 17880 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:53:07.968029 17793 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:08.042323 17949 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:07.976797 17677 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:53:08.059369 17793 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:08.055122 17991 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:08.034372 18024 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
W20260808 01:53:08.064184  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:08.066241  9556 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:08.067353 10332 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:08.067806  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:08.076258  9634 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:53:08.069134  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:08.060328  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 01:53:08.070573  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:08.080435  9630 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:08.070155  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:08.077584  9629 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:53:07.942000 17952 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.19, Next index: 20, Last known committed idx: 17, Time since last communication: 0.125s
I20260808 01:53:08.083423  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:08.092677  9632 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:08.084062  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:08.086880  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:08.082279  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:53:07.897691 18101 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:08.095093  9634 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:08.062861 17695 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:08.063107 17942 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:08.106142  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:08.107720  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:08.110297  9477 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 16. (term mismatch)
I20260808 01:53:08.110576  9474 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260808 01:53:08.111503  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:08.072928 18065 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:08.110785  9477 pending_rounds.cc:85] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 15
I20260808 01:53:08.074769 18045 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:08.113747  9477 pending_rounds.cc:107] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 21.16
I20260808 01:53:08.114269  9477 raft_consensus.cc:2889] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:08.069284 17793 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:08.079720 17920 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:08.123502  9556 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260808 01:53:08.101501 17872 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election won for term 16
W20260808 01:53:08.113547 18060 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.033s	sys 0.061s
I20260808 01:53:08.112599 18074 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:08.113834 17952 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:08.150039  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:08.070771 17949 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:08.117444 18045 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:53:08.145866 18060 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.033s	sys 0.062s
I20260808 01:53:08.085065 18092 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:08.162590  9478 raft_consensus.cc:3055] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:53:08.122416 17920 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:08.113258 17695 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:08.176158 17793 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:08.154074 17952 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:08.179476 17949 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:08.182525  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:08.187426  9631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 1033) took 1760 ms (client timeout 2235 ms). Trace:
W20260808 01:53:08.170626 18102 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.530s	user 0.002s	sys 0.011s
W20260808 01:53:08.197045  9631 rpcz_store.cc:269] 0808 01:53:06.426921 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:06.429065 (+  2144us) service_pool.cc:224] Handling call
0808 01:53:08.187197 (+1758132us) spinlock_profiling.cc:243] Waited 677 ms on lock 0x7b54000a9bd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:08.187348 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1556709760}
I20260808 01:53:08.192742  9478 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:08.084769 18076 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:08.201521  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 2991) took 1049 ms. Trace:
I20260808 01:53:08.205781  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:08.205705  9631 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260808 01:53:08.116370 17942 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:08.157155 17872 raft_consensus.cc:697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:08.204131  9554 rpcz_store.cc:276] 0808 01:53:07.131871 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:07.151950 (+ 20079us) service_pool.cc:224] Handling call
0808 01:53:08.181372 (+1029422us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b54003142d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:08.181537 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":414417664}
I20260808 01:53:08.206019  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:08.208967  9478 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:08.161516 18045 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:08.216933  9481 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:08.219991  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:08.221805  9633 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260808 01:53:08.180172 17758 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:08.227499 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:08.226075  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 441603us
I20260808 01:53:08.228852  9634 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260808 01:53:08.201915 17920 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:08.235539  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:53:08.237358 17758 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:08.242419  9599 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:08.244501  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3371) took 1202 ms. Trace:
I20260808 01:53:08.245671  9637 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260808 01:53:08.240376 17832 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260808 01:53:08.211484 18076 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:08.265995  9638 rpcz_store.cc:276] 0808 01:53:07.041834 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:07.051727 (+  9893us) service_pool.cc:224] Handling call
0808 01:53:08.244213 (+1192486us) spinlock_profiling.cc:243] Waited 662 ms on lock 0x7b540027ea54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:08.244402 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1522978688}
I20260808 01:53:08.273214 10362 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260808 01:53:08.291955  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3388) took 1040 ms. Trace:
I20260808 01:53:08.282821 17952 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 22.19, Last appended by leader: 19, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:08.294545  9637 rpcz_store.cc:276] 0808 01:53:07.251719 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:07.712900 (+461181us) service_pool.cc:224] Handling call
0808 01:53:08.245157 (+532257us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:08.266757 (+ 21600us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:08.291655 (+ 24898us) spinlock_profiling.cc:243] Waited 532 ms on lock 0x7b540027ea50. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:08.291867 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1223122816}
W20260808 01:53:08.296603  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 63836.8us
W20260808 01:53:08.306351  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 64055.7us
I20260808 01:53:08.239883 17695 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:08.282995 18108 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.205s	user 0.001s	sys 0.000s
I20260808 01:53:08.227239 17769 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:08.210110 17949 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:08.305747 17989 raft_consensus.cc:3055] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:08.289721 18092 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:08.277829 17758 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:08.281759 18056 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:08.277242 17942 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:08.212291 18078 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.099s
I20260808 01:53:08.232327 17976 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.079s
I20260808 01:53:08.206732 17673 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.107s
I20260808 01:53:08.242877 18008 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:08.319176 17880 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:08.321345 18011 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:08.311635 17999 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election won for term 20
W20260808 01:53:08.364290  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 324647us
I20260808 01:53:08.372900 10362 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260808 01:53:08.302186 18029 consensus_queue.cc:1059] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260808 01:53:08.324362 17952 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:53:08.390437  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 420999us
I20260808 01:53:08.377878  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:53:08.393179 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 325890us
I20260808 01:53:08.327149 17989 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:53:08.403609  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 77186.8us
W20260808 01:53:08.414300  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:08.468384  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:08.458624  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:08.469087  9601 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:08.398428 18092 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:08.392395 17677 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:08.477718  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 99916.9us
I20260808 01:53:08.374400 18056 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:08.482786  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:08.499552  9554 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:08.484407 17695 raft_consensus.cc:3055] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:53:08.372211 17949 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:08.486819  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:08.490299  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:08.493068  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:08.498202  9478 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:08.476929  9524 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:08.485090  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:08.485188 17932 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:08.459410 18008 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:08.501629 17695 raft_consensus.cc:740] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:08.518985 18008 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:08.519701 17695 consensus_queue.cc:260] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:08.521392 18008 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:08.485771 17989 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:08.508101  9558 raft_consensus.cc:2410] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 22 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260808 01:53:08.504824  9524 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:08.556974  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 71983.5us
I20260808 01:53:08.513005 17832 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:08.513635  9560 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:08.374975 17872 consensus_queue.cc:237] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:08.521047 17695 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:53:08.517189 17920 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:08.388847 17758 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.079s
I20260808 01:53:08.525214  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:08.525683  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 56698.1us
I20260808 01:53:08.571512  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:08.453006 18069 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 13, Time since last communication: 0.049s
I20260808 01:53:08.551921  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:08.558327  9554 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
W20260808 01:53:08.514806 18109 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260808 01:53:08.507678 17615 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260808 01:53:08.568801  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 64073.5us
I20260808 01:53:08.552414 17942 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:08.601576 17942 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:08.602121 17942 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:08.602260 17677 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:08.605641 18123 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:53:08.496021 18092 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:08.578402 18007 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:08.579438 17920 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:08.555640 17756 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:08.555299 18111 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:53:08.575632 17673 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.015s
I20260808 01:53:08.576416 18066 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:08.586155  9599 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:53:08.632565  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46509.3us
W20260808 01:53:08.588891 18012 consensus_meta.cc:287] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.970s	user 0.009s	sys 0.004s
I20260808 01:53:08.590181 17880 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:08.591037 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:53:08.592168 17960 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.878s	user 0.011s	sys 0.002s
W20260808 01:53:08.594239 18090 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.722s	user 0.011s	sys 0.000s
I20260808 01:53:08.594741 17976 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:08.596315 18114 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
I20260808 01:53:08.501065 18011 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:08.568887 17832 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:08.581317 18008 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:08.577461 17999 raft_consensus.cc:697] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:08.630860 18123 raft_consensus.cc:697] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:08.573761 18009 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.744s	user 0.000s	sys 0.022s
W20260808 01:53:08.627156  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:08.576711 17830 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:08.606652 17615 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:08.702955  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:08.642123  9599 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:08.616390 17989 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:53:08.645071  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:08.622298 17952 consensus_queue.cc:1059] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260808 01:53:08.653380  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:08.655368  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:08.569507 18121 consensus_queue.cc:1059] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.081s
I20260808 01:53:08.658208  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:08.658499  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:08.575147  9557 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:08.693861  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:08.647312 18012 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:53:08.650054 18069 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:08.650565  9476 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
W20260808 01:53:08.709065  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 67043.4us
I20260808 01:53:08.662650 18008 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:08.718308  9473 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:53:08.655974 18124 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:08.709239  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:08.741600 10366 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.671s	user 0.010s	sys 0.017s
W20260808 01:53:08.742134 10366 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.010s	sys 0.017s
I20260808 01:53:08.743055 10366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 1032) took 1264 ms. Trace:
I20260808 01:53:08.745937  9559 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:08.646486 18066 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:08.708563 17615 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:08.618494 18101 consensus_queue.cc:1059] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.058s
I20260808 01:53:08.635195 17756 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:08.708062 18107 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
W20260808 01:53:08.714948  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.036s	sys 0.048s
W20260808 01:53:08.769603  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.036s	sys 0.048s
I20260808 01:53:08.724663 18118 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:08.745118  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:08.661746 17976 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:08.758282  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 48391.6us
I20260808 01:53:08.724692 18072 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:08.708513 18126 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:08.762193  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:08.763796  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:08.724236 17793 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:08.724246 18069 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:08.739967 18008 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:08.705689 18009 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.876s	user 0.000s	sys 0.023s
I20260808 01:53:08.731731 18090 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:08.715283 17830 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:08.781679 18129 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:08.780840  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 209416us
I20260808 01:53:08.745291 18029 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:08.772584 10366 rpcz_store.cc:276] 0808 01:53:07.478572 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:07.483001 (+  4429us) service_pool.cc:224] Handling call
0808 01:53:07.483494 (+   493us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:07.483944 (+   450us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:07.483975 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:07.484636 (+   661us) log.cc:844] Serialized 52 byte log entry
0808 01:53:07.731253 (+246617us) raft_consensus.cc:1645] Marking committed up to 19
0808 01:53:08.061886 (+330633us) log.cc:844] Serialized 14 byte log entry
0808 01:53:08.063113 (+  1227us) log.cc:844] Serialized 14 byte log entry
0808 01:53:08.742571 (+679458us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:08.742699 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:08.742763 (+    64us) raft_consensus.cc:1685] finished
0808 01:53:08.742781 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:08.742978 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4541,"prepare.queue_time_us":710639,"prepare.run_cpu_time_us":112,"prepare.run_wall_time_us":16069,"raft.queue_time_us":15866,"raft.run_cpu_time_us":491,"raft.run_wall_time_us":24741,"spinlock_wait_cycles":3328,"thread_start_us":1229927,"threads_started":3,"wal-append.queue_time_us":491960}
I20260808 01:53:08.765057 17949 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:08.822623 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:08.866612 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 397281us
W20260808 01:53:08.822564  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 264676us
W20260808 01:53:08.919893  9476 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 3570) took 1480 ms (client timeout 1865 ms). Trace:
I20260808 01:53:08.950210  9478 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 16. (term mismatch)
I20260808 01:53:09.008181  9478 pending_rounds.cc:85] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 15
I20260808 01:53:09.008728  9478 pending_rounds.cc:107] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 19.16
I20260808 01:53:08.810470 18121 raft_consensus.cc:3055] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:08.818879 18069 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:09.015878  9478 raft_consensus.cc:2889] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:53:09.017541  9476 rpcz_store.cc:269] 0808 01:53:07.334171 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:07.334621 (+   450us) service_pool.cc:224] Handling call
0808 01:53:08.813835 (+1479214us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b540021e154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:08.813998 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3011724544}
I20260808 01:53:08.930852 17758 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:08.917613 18007 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:09.041181  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:09.087404  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:09.089818  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 54521.5us
I20260808 01:53:08.941887 17952 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:09.006574 18024 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260808 01:53:08.818495 17793 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:08.805291 17999 consensus_queue.cc:237] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.050140 18074 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:09.115967  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:09.112533  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.025993 17949 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:09.042341 18045 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.794s	user 0.014s	sys 0.001s
I20260808 01:53:09.124044  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.119709  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.019827 18111 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:09.092671 18120 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:09.126727  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:09.089036 18092 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:09.082726 17758 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:08.815104 18126 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:08.942479 18090 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:09.081828 18121 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:09.145277  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:08.798327 18118 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.166218  9521 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:09.009676 18029 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.040885 18131 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:08.752677 18124 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:08.937242 18123 consensus_queue.cc:237] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:09.189388  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 49405.6us
I20260808 01:53:09.161304  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.164179  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.165757  9559 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:09.116683 18101 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:09.008006 18129 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:09.150568  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.116323 18132 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:09.149577 18024 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:09.190135  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:09.120832 17952 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:09.116466 18007 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:09.143554 17949 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.160427 17758 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:09.161213 18092 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:09.171711 18072 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:09.241575 18101 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.134573 18074 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:09.166540 18060 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.005s	sys 0.001s
W20260808 01:53:09.247051  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 80909.5us
I20260808 01:53:09.292296  9521 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:09.149452 18045 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:09.242995 18066 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:09.227137 17832 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:09.155911 18111 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:09.244640 18132 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.304304 17756 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:09.305100 18129 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:09.305595 18132 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:09.308917  9476 raft_consensus.cc:2410] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 24 because replica has last-logged OpId of term: 23 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
W20260808 01:53:09.313227  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 123219us
I20260808 01:53:09.314656  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:09.291116 18060 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.715s	user 0.005s	sys 0.002s
I20260808 01:53:09.160676 18120 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:09.249737 18024 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.306835 17976 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:09.305895 18129 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:09.196245 18131 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.279876 18092 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:09.346575  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 0b4331dfdd5b43448a400f2eb7b1be70: Illegal state: Replica 6e7856bed1be43acaa821290d506ee97 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 21 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:53:09.347688 17695 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.772s	user 0.014s	sys 0.000s
I20260808 01:53:09.348438 18092 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:09.313637 18066 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:09.330753 17830 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:09.357314 18045 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:09.357782 18140 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.165s	user 0.000s	sys 0.001s
I20260808 01:53:09.358749 18177 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:09.359450 18029 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:09.359716 18111 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:09.360302 18029 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:53:09.360850 18124 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:09.352250 18142 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260808 01:53:09.361368 18101 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:09.362779 18118 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:09.363189 18090 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:09.359894 18150 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260808 01:53:09.365151 18120 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:09.366714 18007 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:09.367134 18072 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.369082 18131 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:09.352528 18127 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.490s	user 0.000s	sys 0.001s
W20260808 01:53:09.371739 17989 consensus_meta.cc:287] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.650s	user 0.008s	sys 0.006s
W20260808 01:53:09.372049 18163 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
W20260808 01:53:09.373934 18076 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.070s	user 0.015s	sys 0.000s
I20260808 01:53:09.352905 18166 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260808 01:53:09.376520  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 62000.1us
I20260808 01:53:09.380012  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:09.325580 18132 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.026s
I20260808 01:53:09.396095 18177 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:53:09.396786 18177 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:09.397364 17960 raft_consensus.cc:3055] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:53:09.397933 17960 raft_consensus.cc:740] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:09.399415 17793 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:09.363421 18165 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:09.401332  9559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3933) took 1581 ms (client timeout 2090 ms). Trace:
W20260808 01:53:09.402292 18153 log.cc:927] Time spent T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.154s	user 0.000s	sys 0.001s
I20260808 01:53:09.400961 18165 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:09.401856  9559 rpcz_store.cc:269] 0808 01:53:07.820390 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:07.822796 (+  2406us) service_pool.cc:224] Handling call
0808 01:53:09.401109 (+1578313us) spinlock_profiling.cc:243] Waited 922 ms on lock 0x7b540008a7d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:09.401226 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2120336640}
I20260808 01:53:09.358803 18126 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:09.374804 17848 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:09.377997 17695 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:53:09.301323 18121 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:09.397269 18177 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:53:09.402972  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:09.446843  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66992.6us
I20260808 01:53:09.377532 18045 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:09.457646  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.377249 18159 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:09.402518 18145 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:09.460606  9476 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:53:09.377758 18129 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:09.441888 17832 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260808 01:53:09.463865  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:09.463781  9474 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
W20260808 01:53:09.464588  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:09.462141  9601 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:09.475769  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:09.465744 17942 consensus_meta.cc:287] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.863s	user 0.012s	sys 0.003s
I20260808 01:53:09.476547  9637 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
W20260808 01:53:09.469838  9363 tablet_replica.cc:1406] Time spent applying in-flights took a long time: real 0.701s	user 0.000s	sys 0.000s
I20260808 01:53:09.463739 17695 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:09.489327  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:09.489739  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:09.490489  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:09.491333  9559 raft_consensus.cc:2410] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260808 01:53:09.485654  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:09.465409 18006 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:09.460704 17848 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:09.488423  9631 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260808 01:53:09.473591 18121 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
W20260808 01:53:09.468032  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 175804us
I20260808 01:53:09.492082  9638 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
I20260808 01:53:09.356002 18066 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.494716  9632 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260808 01:53:09.494953 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:09.499387  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:09.501101  9524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.001s	sys 0.010s
I20260808 01:53:09.504197  9555 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 24. Current term is 25. Ops: []
W20260808 01:53:09.503043  9524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.001s	sys 0.010s
W20260808 01:53:09.505342  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 743235us
I20260808 01:53:09.500968  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:09.505594 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:09.504338  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:09.491569  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:09.509331  9559 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260808 01:53:09.468319 18045 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.487144 17949 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:09.486605 17769 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:09.511000 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:09.513885  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:09.521838 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:09.532512 10365 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 16. (term mismatch)
I20260808 01:53:09.533048 10365 pending_rounds.cc:85] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting all ops after (but not including) 15
I20260808 01:53:09.533351 10365 pending_rounds.cc:107] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting uncommitted NO_OP operation due to leader change: 23.16
I20260808 01:53:09.533682 10365 raft_consensus.cc:2889] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:09.521883 10369 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260808 01:53:09.538314  9632 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260808 01:53:09.538452 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:09.401890 18069 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.472589 18145 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:09.501943 17677 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.896s	user 0.016s	sys 0.028s
W20260808 01:53:09.721313 17677 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.115s	user 0.016s	sys 0.028s
I20260808 01:53:09.487653 17942 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.523547 18124 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:09.722450 18124 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:09.723150 18124 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:09.723837 17942 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:09.617877  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:09.619920 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:09.725569  9550 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 25: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:09.503283 18160 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:09.620546 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:09.726658  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:09.727604  9550 raft_consensus.cc:2410] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 24 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260808 01:53:09.729110  9550 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260808 01:53:09.618155 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:09.626979 18162 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.142s
I20260808 01:53:09.730818 18162 consensus_queue.cc:1243] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 17.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260808 01:53:09.623413  9555 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: [19.11-23.13]
I20260808 01:53:09.621520  9559 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:09.627249 17989 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260808 01:53:09.628502 18174 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.135s	user 0.000s	sys 0.000s
W20260808 01:53:09.705919  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 82443.5us
I20260808 01:53:09.629034 18173 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:09.626928 17872 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:53:09.499281 18006 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:09.735809  9599 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.194:36095 timed out after 2.090s (SENT) [suppressed 11 similar messages]
I20260808 01:53:09.737741 18006 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:09.632156  9554 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 15. Current term is 16. Ops: []
I20260808 01:53:09.630604 18078 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.015s
I20260808 01:53:09.471405 18159 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.541047 18158 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:09.631978 18146 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:09.527261 18074 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:09.636214  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.648118  9634 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: [19.11-23.13]
I20260808 01:53:09.740391 18012 raft_consensus.cc:3055] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:09.740907 18012 raft_consensus.cc:740] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:53:09.645339 18008 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.706s	user 0.011s	sys 0.001s
I20260808 01:53:09.741482  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.741663  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:09.741766 18012 consensus_queue.cc:260] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.650899  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:09.747531  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 75538.6us
I20260808 01:53:09.747695 18159 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:09.748522  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:09.742156 18008 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.630833 18147 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:09.750744 18147 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.751691  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:09.751871 18008 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:09.617775 18132 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:09.653228  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:09.541410 17932 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:09.655354  9552 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260808 01:53:09.753908  9558 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
W20260808 01:53:09.654938 18175 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260808 01:53:09.659485  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:09.661973  9636 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260808 01:53:09.755584 18154 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260808 01:53:09.756879  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3949) took 1019 ms. Trace:
I20260808 01:53:09.757318  9560 rpcz_store.cc:276] 0808 01:53:08.737425 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:08.737555 (+   130us) service_pool.cc:224] Handling call
0808 01:53:08.762116 (+ 24561us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:09.167314 (+405198us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:09.167371 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:09.192013 (+ 24642us) log.cc:844] Serialized 26 byte log entry
0808 01:53:09.461349 (+269336us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:09.461391 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:09.756453 (+295062us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b540002c1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:09.756516 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:09.756568 (+    52us) raft_consensus.cc:1685] finished
0808 01:53:09.756588 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:09.756813 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7227,"raft.queue_time_us":184185,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":3863,"spinlock_wait_cycles":889216256,"thread_start_us":198130,"threads_started":1,"wal-append.queue_time_us":262310}
I20260808 01:53:09.753147 18147 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:09.664019 18165 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:09.668831  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet ed168d59c43d4bfaa07d104032ad8977: Illegal state: Replica 259bcb3f0d254e8b9effac3277e654e0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 21 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:53:09.672787 18133 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.197s	user 0.000s	sys 0.001s
I20260808 01:53:09.674976  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:53:09.764917 18056 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:09.630985  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:09.642583  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:09.526669 18143 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:09.536561 18007 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:09.768385 18007 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.771200 18007 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:09.681737 18092 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:09.772439 18122 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:09.772310 18092 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.773164 18122 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.682284 18126 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:09.775054 18126 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:09.777400 18126 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:09.683104 18167 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:09.777832 18007 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:09.683241 18192 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:53:09.630923 18139 consensus_queue.cc:1059] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.162s
I20260808 01:53:09.686290  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:09.703624 17793 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
W20260808 01:53:09.706327 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52869.7us
I20260808 01:53:09.700345  9363 maintenance_manager.cc:643] P 3cd8e8730323483db1c5f22e90bb080c: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 1.850s	user 0.031s	sys 0.010s Metrics: {"bytes_written":4183,"delete_count":0,"lbm_write_time_us":94803,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":3,"reinsert_count":0,"update_count":159}
W20260808 01:53:09.778520 18149 log.cc:927] Time spent T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.586s	user 0.000s	sys 0.001s
I20260808 01:53:09.781016 18139 raft_consensus.cc:3055] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:09.781502 18139 raft_consensus.cc:740] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:53:09.709589 17615 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.935s	user 0.014s	sys 0.003s
I20260808 01:53:09.465880 17960 consensus_queue.cc:260] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:09.723664 17952 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:09.726899 18160 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.783694  9630 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:09.783991 18047 raft_consensus.cc:993] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: : Instructing follower 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:53:09.783838 17960 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:53:09.784794 18047 raft_consensus.cc:1081] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Signalling peer 52ccf4dcca1843f7bb29e9172e2d4ab4 to start an election
I20260808 01:53:09.629861 18069 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:09.785352 18160 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:09.782650 18139 consensus_queue.cc:260] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:09.788705 18078 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:09.789438 18078 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:09.782970 17615 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.792227 18078 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:09.792820 18074 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:09.793640 17615 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:09.742291  9480 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:53:09.794885  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:09.795047  9637 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 22: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:09.796074  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.796713 17848 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:09.797338  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.797900  9364 maintenance_manager.cc:419] P 3cd8e8730323483db1c5f22e90bb080c: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.212924
I20260808 01:53:09.798211  9480 raft_consensus.cc:2393] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 22: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:09.706760 10332 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:09.618216 18129 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.790733 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:09.799153 10363 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
W20260808 01:53:09.799268 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 94009.3us
I20260808 01:53:09.778323 18167 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.742430  9634 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:09.801977 18167 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:09.631444 18045 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:09.802299 18143 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:53:09.649991  9638 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:09.803128 18143 consensus_queue.cc:1059] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260808 01:53:09.743220  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.804144 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:09.804749 18143 raft_consensus.cc:3055] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:09.805023  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.805191 18143 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:09.740378 18146 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.806612  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.808068  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.808444 18146 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:09.808704  9478 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
W20260808 01:53:09.809621  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:09.797376  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:09.811568  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 125363us
I20260808 01:53:09.812552  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:09.813372  9551 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 26.
I20260808 01:53:09.813844 17976 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:09.754014 17932 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:09.758324 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:09.749245  9631 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:09.663030 18024 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:09.659184 17695 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.816439 18146 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260808 01:53:09.753252  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:09.817920  9633 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:09.817965 18146 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:09.782939  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.824172  9472 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:09.811653 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:09.793396 18074 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:09.785773 18192 consensus_queue.cc:1243] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 18.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260808 01:53:09.662070 18066 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:09.825829  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3582) took 1021 ms. Trace:
I20260808 01:53:09.825894 18192 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:09.826468 18074 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:09.806747  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:09.826979 18192 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:09.778426 18007 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.829694  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3585) took 1018 ms. Trace:
I20260808 01:53:09.829834 18192 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:09.808463  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3624) took 1157 ms. Trace:
I20260808 01:53:09.831193 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:09.826398  9472 rpcz_store.cc:276] 0808 01:53:08.803954 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:09.824919 (+1020965us) service_pool.cc:224] Handling call
0808 01:53:09.825311 (+   392us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:09.825499 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:09.825527 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:09.825572 (+    45us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:53:09.825606 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:09.825665 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:09.825768 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:53:09.753402 18132 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:09.834151  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:09.834524 18132 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:09.830214  9479 rpcz_store.cc:276] 0808 01:53:08.811258 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:09.828527 (+1017269us) service_pool.cc:224] Handling call
0808 01:53:09.829015 (+   488us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:09.829253 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:09.829290 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:09.829351 (+    61us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:53:09.829398 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:09.829468 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:09.829619 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:53:09.746085 18137 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:09.820351 17932 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:09.820868 18056 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 8, Time since last communication: 0.000s
I20260808 01:53:09.836371 18056 raft_consensus.cc:3055] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:53:09.836849 18056 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:09.837855 18056 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.821818 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:09.843225  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:09.843657  9550 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260808 01:53:09.844049 18056 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:09.845299 18078 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260808 01:53:09.806381 18143 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:09.848564 18143 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:09.814286  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:53:09.771248  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:09.853425 18132 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:09.854254 18147 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260808 01:53:09.854971  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 120547us
I20260808 01:53:09.855814 18132 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:09.856552  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:09.856930 18132 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:09.858348 18024 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:09.825606 10334 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:09.867534  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1438) took 1128 ms. Trace:
I20260808 01:53:09.867952 18047 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:53:09.821609  9551 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fca874793205463dbb922993f57bf7f5"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:60552
I20260808 01:53:09.869227  9551 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:53:09.827230 18074 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:09.830962  9475 rpcz_store.cc:276] 0808 01:53:08.651023 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:08.653958 (+  2935us) service_pool.cc:224] Handling call
0808 01:53:08.654358 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:09.034381 (+380023us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:09.034424 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:09.082411 (+ 47987us) log.cc:844] Serialized 26 byte log entry
0808 01:53:09.236076 (+153665us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:53:09.236131 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:09.808036 (+571905us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b540009a1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:09.808088 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:09.808137 (+    49us) raft_consensus.cc:1685] finished
0808 01:53:09.808151 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:09.808385 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":191890,"raft.queue_time_us":186718,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":14725,"spinlock_wait_cycles":398534400,"thread_start_us":87230,"threads_started":1,"wal-append.queue_time_us":242777}
I20260808 01:53:09.868038  9477 rpcz_store.cc:276] 0808 01:53:08.739167 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:08.741111 (+  1944us) service_pool.cc:224] Handling call
0808 01:53:08.741606 (+   495us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:09.192756 (+451150us) raft_consensus.cc:1534] Early marking committed up to index 18
0808 01:53:09.241087 (+ 48331us) log.cc:844] Serialized 14 byte log entry
0808 01:53:09.468252 (+227165us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:09.474478 (+  6226us) raft_consensus.cc:1645] Marking committed up to 18
0808 01:53:09.474536 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:09.867269 (+392733us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x7b54000217d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:09.867326 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:09.867467 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19648,"prepare.queue_time_us":48177,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":67,"raft.queue_time_us":91113,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":100,"spinlock_wait_cycles":1036944768,"thread_start_us":225121,"threads_started":2,"wal-append.queue_time_us":254728}
I20260808 01:53:09.870914  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.777354 18092 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:09.871758  9475 raft_consensus.cc:2393] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:09.864305  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:09.839375 18128 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.905s	user 0.002s	sys 0.002s
I20260808 01:53:09.872965 18122 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:09.841724 10365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 1262) took 1951 ms. Trace:
I20260808 01:53:09.873759 10365 rpcz_store.cc:276] 0808 01:53:07.890016 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:07.892497 (+  2481us) service_pool.cc:224] Handling call
0808 01:53:07.892975 (+   478us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:09.534359 (+1641384us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:09.841448 (+307089us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b5400690fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:09.841661 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3759685888}
I20260808 01:53:09.846585  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:09.839375 18162 consensus_queue.cc:1175] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6e7856bed1be43acaa821290d506ee97" responder_term: 26 status { last_received { term: 26 index: 18 } last_committed_idx: 18 last_received_current_leader { term: 26 index: 18 } }
I20260808 01:53:09.831429 18007 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:09.867823  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:09.869653  9551 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:09.882977 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:09.883302 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:09.886245 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:09.887138 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:09.887851 10365 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:09.868645 18047 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:09.889106 18047 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:53:09.889569  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:09.889626  9630 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:09.893138 18184 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260808 01:53:09.895144 17758 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.613s	user 0.013s	sys 0.000s
I20260808 01:53:09.846321  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:09.809803  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.896041 18012 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:09.890414  9635 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 21: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:09.810786  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.883914  9556 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:09.886538 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:09.789355 18139 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:09.905469 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:09.908627 17872 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:09.917688  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3581) took 1114 ms. Trace:
I20260808 01:53:09.919432  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:09.920327  9550 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:09.858795  9601 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:09.922592 10361 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:09.950505  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:09.929128  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:53:09.936587  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 47687us
I20260808 01:53:09.859197  9521 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:09.860497  9524 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:09.983028  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 122599us
I20260808 01:53:09.984464  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:09.861474  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:09.985366  9555 raft_consensus.cc:2376] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21. Re-sending same reply.
I20260808 01:53:09.947474  9523 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:09.865018  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:09.865317  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:09.987884  9552 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:09.866087  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:09.866251  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:09.988737  9550 raft_consensus.cc:2393] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:09.989248  9559 raft_consensus.cc:2410] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 24 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
W20260808 01:53:09.933629  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 50878us
I20260808 01:53:09.850453  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:09.864614  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:09.990602  9560 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
I20260808 01:53:09.993830  9558 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 26: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:09.994298 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:09.995467 10361 raft_consensus.cc:2393] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 17: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:09.918804  9480 rpcz_store.cc:276] 0808 01:53:08.803648 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:09.823706 (+1020058us) service_pool.cc:224] Handling call
0808 01:53:09.824040 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:09.824270 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:53:09.826533 (+  2263us) log.cc:844] Serialized 14 byte log entry
0808 01:53:09.892767 (+ 66234us) log.cc:844] Serialized 14 byte log entry
0808 01:53:09.904788 (+ 12021us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:09.904924 (+   136us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:53:09.905084 (+   160us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:09.905223 (+   139us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:09.917533 (+ 12310us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11507,"prepare.queue_time_us":1818,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":85,"raft.queue_time_us":14696,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":264,"thread_start_us":65057,"threads_started":1,"wal-append.queue_time_us":105837}
W20260808 01:53:09.996143  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 145760us
I20260808 01:53:09.988193  9445 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:10.079119  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:10.114776  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 124186us
I20260808 01:53:10.071587  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:10.063458  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 116092us
I20260808 01:53:10.075531 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:10.085589  9552 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:10.161957 10366 raft_consensus.cc:2393] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 23: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:53:10.062991 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 116678us
I20260808 01:53:10.162042  9473 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
W20260808 01:53:10.162613 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:10.123979  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 265259us
I20260808 01:53:10.160884  9600 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:10.165560 18009 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:10.165869  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:10.166124 18009 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:53:10.063814  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 78766.2us
I20260808 01:53:10.166172  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:09.996272  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 137135us
W20260808 01:53:09.938789 18206 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260808 01:53:10.171964  9521 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:09.997654  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:10.170215  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:10.180353  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:10.164162  9601 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:10.181969  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:10.183293  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 195172us
I20260808 01:53:10.184772  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:10.186069  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:09.937846 18192 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:10.097625 18121 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.593s	user 0.011s	sys 0.000s
I20260808 01:53:09.939687 18145 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:10.183295  9551 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:10.062539 18167 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:10.195446  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:10.075023 17952 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:10.064579 17932 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:53:09.993818 18184 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:10.063598 17989 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election won for term 24
W20260808 01:53:10.238943  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 43537.1us
W20260808 01:53:10.247967  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52626us
I20260808 01:53:10.128975 17615 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260808 01:53:10.236517  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66412.7us
I20260808 01:53:10.250248  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:09.935482 17758 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:10.237174  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 49785.6us
I20260808 01:53:10.249002  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:09.717636 18177 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:10.194626 18024 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:10.255410  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:10.276651  9558 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:10.258912  9521 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:10.251410  9551 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:10.194796 18146 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.066s
I20260808 01:53:10.192183 18126 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:10.168191 18092 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.071s
I20260808 01:53:10.129272 18072 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.111s
I20260808 01:53:10.192464 18011 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:10.192896 18158 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:10.234875 17756 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:10.307065 17756 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:10.309598 17756 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:10.233024 18147 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:10.259097 18193 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:10.310921 18147 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:10.238572 18176 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:10.312250 18176 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:10.312697 18176 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:10.313279 18147 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:10.283150 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: true dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:10.315507 10366 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Stepping down as leader of term 22
W20260808 01:53:10.315749  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66856.3us
I20260808 01:53:10.316519 18028 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:10.284860  9475 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:10.318181 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:10.317538 17960 consensus_meta.cc:287] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.533s	user 0.010s	sys 0.003s
I20260808 01:53:10.241740 17932 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:10.320924 17932 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:10.322218 18012 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:10.322889 18158 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:10.323099 18012 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:10.325273 18158 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:10.316845  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:10.331102 17756 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:53:10.331748 17756 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:10.332298 17756 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:53:10.333039  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:10.333505 18066 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:10.334120 18066 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:10.334856 18012 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:10.336427 18160 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:10.336668 18066 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:10.337067 18160 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:10.337616 18160 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:10.340507 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:10.296593 18011 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:10.347955 18011 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:10.351068 18206 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260808 01:53:10.352545 17932 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:10.289222  9636 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:10.190665 18124 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:10.293474  9637 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:53:10.267590 18168 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:10.373955 18168 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:10.374513 18168 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:10.375344  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3590) took 1342 ms. Trace:
I20260808 01:53:10.376761 18177 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:10.262348 18060 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:10.378846 17758 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:10.379154 18060 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:10.379699 18060 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:10.380772 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:10.381309 18124 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:10.375893  9479 rpcz_store.cc:276] 0808 01:53:09.033000 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:09.835704 (+802704us) service_pool.cc:224] Handling call
0808 01:53:09.836029 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:10.234883 (+398854us) raft_consensus.cc:1534] Early marking committed up to index 18
0808 01:53:10.234933 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:10.235000 (+    67us) raft_consensus.cc:1645] Marking committed up to 18
0808 01:53:10.235056 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:10.374988 (+139932us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b540038d1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:10.375067 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:10.375263 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41206,"raft.queue_time_us":41402,"spinlock_wait_cycles":840407552}
I20260808 01:53:10.383090 10365 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: []
W20260808 01:53:10.302580  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:10.386014 18177 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:10.386329  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:10.388531 18182 consensus_queue.cc:1059] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:10.389711 18182 raft_consensus.cc:3055] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:10.390137 18182 raft_consensus.cc:740] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:10.298038  9521 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:10.304476 18165 raft_consensus.cc:2764] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260808 01:53:10.304606 18155 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:10.305016 17989 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:10.315857 10366 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:10.353037  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3589) took 1320 ms. Trace:
I20260808 01:53:10.237250 17952 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:10.359557 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:10.250540 18047 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:10.167142 18009 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:10.278937 18024 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:10.443356  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:10.386816 18177 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:10.394954  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 97159.5us
W20260808 01:53:10.386901  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:10.395982  9550 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260808 01:53:10.393080 17976 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:10.453591  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:10.397790  9481 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260808 01:53:10.379225  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1441) took 1639 ms. Trace:
I20260808 01:53:10.408154  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:10.409386  9552 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260808 01:53:10.457105  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:10.410756  9551 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: []
I20260808 01:53:10.407649  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:10.416394  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47963.3us
I20260808 01:53:10.415506  9472 rpcz_store.cc:276] 0808 01:53:09.032641 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:09.832828 (+800187us) service_pool.cc:224] Handling call
0808 01:53:09.833282 (+   454us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:09.833803 (+   521us) raft_consensus.cc:1534] Early marking committed up to index 16
0808 01:53:09.834309 (+   506us) log.cc:844] Serialized 14 byte log entry
0808 01:53:09.914630 (+ 80321us) log.cc:844] Serialized 14 byte log entry
0808 01:53:09.915310 (+   680us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:09.916478 (+  1168us) log.cc:844] Serialized 26 byte log entry
0808 01:53:09.916882 (+   404us) raft_consensus.cc:1645] Marking committed up to 16
0808 01:53:09.916958 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:09.917198 (+   240us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:10.352805 (+435607us) raft_consensus.cc:1685] finished
0808 01:53:10.352831 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:10.352973 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":489,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":78,"raft.queue_time_us":1772,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":271,"thread_start_us":78898,"threads_started":1,"wal-append.queue_time_us":27509}
I20260808 01:53:10.359831  9478 pending_rounds.cc:85] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 14
I20260808 01:53:10.460852  9478 pending_rounds.cc:107] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 17.15
I20260808 01:53:10.432622 10332 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:10.463712  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 55700.3us
I20260808 01:53:10.465977  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:10.252612  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 67878.2us
I20260808 01:53:10.473889  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:10.474977  9636 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:10.455485  9474 rpcz_store.cc:276] 0808 01:53:08.740240 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:09.470067 (+729827us) service_pool.cc:224] Handling call
0808 01:53:09.470465 (+   398us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:09.490462 (+ 19997us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:09.490493 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:09.491354 (+   861us) log.cc:844] Serialized 26 byte log entry
0808 01:53:09.746019 (+254665us) spinlock_profiling.cc:243] Waited 2.82 ms on lock 0x7b180032f9e0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:53:10.133139 (+387120us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:10.133201 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:10.133297 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:10.378806 (+245509us) raft_consensus.cc:1685] finished
0808 01:53:10.378836 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:10.379140 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19520,"raft.queue_time_us":20126,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":4960,"spinlock_wait_cycles":6489856,"thread_start_us":386766,"threads_started":1,"wal-append.queue_time_us":791245}
I20260808 01:53:10.415089 18155 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:10.428658 18158 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:10.532140  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 50301.9us
I20260808 01:53:10.490775  9599 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:10.414552 18012 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:10.534873  9521 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:10.499642 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:10.489176  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:10.461324  9478 raft_consensus.cc:2889] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:10.450438 10366 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:10.409142 17991 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.700s	user 0.025s	sys 0.014s
I20260808 01:53:10.228417 18192 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:10.441574 18056 consensus_meta.cc:287] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.597s	user 0.010s	sys 0.002s
I20260808 01:53:10.531873 10360 raft_consensus.cc:2368] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 18. Current term is 19.
I20260808 01:53:10.480978  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:10.545924  9557 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:10.437623 18122 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:10.547878  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:10.572431 10366 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:10.428795 17952 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:10.548205  9521 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:10.548959  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:10.547796 18007 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:10.477762 17756 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:10.552263  9552 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:10.552856 18193 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:53:10.553550  9475 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:10.546999 18165 consensus_queue.cc:1059] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.009s
I20260808 01:53:10.533535 18124 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:53:10.559777 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 60288.2us
W20260808 01:53:10.560204 18139 consensus_meta.cc:287] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.657s	user 0.011s	sys 0.001s
I20260808 01:53:10.560901 17989 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:10.446604 18024 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:10.563642  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:10.565805 17932 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:10.534801 18163 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260808 01:53:10.567588  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:10.550801 17991 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.859s	user 0.025s	sys 0.015s
I20260808 01:53:10.595459  9475 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:10.596208  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:10.658561  9550 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:10.598923  9473 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:53:10.556075 18177 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:10.599768  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:10.600063  9476 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.003s	sys 0.023s
W20260808 01:53:10.660526  9476 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.003s	sys 0.023s
I20260808 01:53:10.478580 17830 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:53:10.602428 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:10.665369 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 63037.8us
I20260808 01:53:10.666249 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:10.602217 18045 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.011s	sys 0.014s
I20260808 01:53:10.532706 18126 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:10.604203  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:10.603909  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:10.583026 17793 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:10.633373  9638 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17. Ops: []
I20260808 01:53:10.641957  9552 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:10.646240 10361 raft_consensus.cc:2410] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 28 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260808 01:53:10.588271 18007 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:10.660574  9634 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:10.602458 17695 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.010s	sys 0.015s
I20260808 01:53:10.532071 18155 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:10.658856  9475 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:10.688051  9475 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:10.676198  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:10.689296  9550 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 21: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:10.571197 18158 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:10.687009 17793 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:53:10.672313 18045 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.011s	sys 0.015s
I20260808 01:53:10.597095 18160 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:10.588316 18122 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:10.695681  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:10.674898 18120 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:10.700881  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:10.678923 17848 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:53:10.693975 17695 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.011s	sys 0.015s
I20260808 01:53:10.704895  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:10.714054  9550 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:53:10.479538 18182 consensus_queue.cc:260] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:10.671451 18139 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:53:10.684548 18012 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:10.715292  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:10.693522 18007 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:10.733711  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:10.700269 17793 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:10.735428  9551 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:10.692534 17960 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:10.751515 17960 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:10.753890 17960 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:10.709642 18120 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:10.759817  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:10.762871  9475 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 14. (term mismatch)
I20260808 01:53:10.763604  9475 pending_rounds.cc:85] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 13
I20260808 01:53:10.764050  9475 pending_rounds.cc:107] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260808 01:53:10.764511  9475 raft_consensus.cc:2889] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:10.734305 18012 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:10.735561 17695 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:10.740012  9472 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:53:10.704417 18009 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:53:10.768327 18182 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:10.779129  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:10.779294  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:10.779587 18202 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260808 01:53:10.783461  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:10.784842  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:10.789575  9475 raft_consensus.cc:2410] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 28 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
W20260808 01:53:10.786108  9558 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.502s	user 0.010s	sys 0.001s
W20260808 01:53:10.790161  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:10.744704  9634 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 22.
I20260808 01:53:10.763362 18120 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:10.791126  9558 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:10.790449  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:10.788885  9550 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:10.793813  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:10.797253  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:10.750516 18126 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.19, Last appended by leader: 19, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:10.797509 18129 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:10.798435 18129 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:10.798931 18129 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:10.800947  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:10.801903  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:10.802309  9551 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:53:10.797333 18219 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.006s
I20260808 01:53:10.803700 18219 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:10.803843 18122 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:10.804150 18219 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:10.804692 18219 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:53:10.805071 18203 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260808 01:53:10.806582  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:10.792323 17960 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:53:10.807843  9558 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:53:10.808141 17960 raft_consensus.cc:3055] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:10.808563 17960 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:10.809609 17960 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:10.808595  9552 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:10.810231  9558 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 24. Current term is 25. Ops: []
W20260808 01:53:10.812201 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:10.812212  9552 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260808 01:53:10.815646  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:10.829316  9472 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:53:10.832171 10366 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:10.785881 18047 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:10.806721 18119 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.035s
W20260808 01:53:10.836884 18218 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260808 01:53:10.837087 18209 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260808 01:53:10.838910 18215 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260808 01:53:10.848160  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1440) took 2108 ms. Trace:
I20260808 01:53:10.848729  9476 rpcz_store.cc:276] 0808 01:53:08.739889 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:09.468752 (+728863us) service_pool.cc:224] Handling call
0808 01:53:09.487901 (+ 19149us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:09.493979 (+  6078us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:09.494014 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:09.499783 (+  5769us) log.cc:844] Serialized 26 byte log entry
0808 01:53:10.091538 (+591755us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b180036ad80. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:53:10.661096 (+569558us) raft_consensus.cc:1645] Marking committed up to 17
0808 01:53:10.661161 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:10.661265 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:10.847747 (+186482us) raft_consensus.cc:1685] finished
0808 01:53:10.847780 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:10.848073 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9704,"raft.queue_time_us":13013,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":27046,"spinlock_wait_cycles":33947520,"thread_start_us":569265,"threads_started":1,"wal-append.queue_time_us":1165732}
I20260808 01:53:10.835459 17756 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:10.826047 17960 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:10.860019 18158 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:10.862318  9474 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260808 01:53:10.864429 18060 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:10.772542 18056 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.046s
I20260808 01:53:10.866911  9477 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:10.868705  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:10.874992  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 3004) took 1233 ms. Trace:
I20260808 01:53:10.880738  9557 rpcz_store.cc:276] 0808 01:53:09.642022 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:09.642263 (+   241us) service_pool.cc:224] Handling call
0808 01:53:10.874766 (+1232503us) spinlock_profiling.cc:243] Waited 534 ms on lock 0x7b540029e5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:10.874922 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1229186432}
I20260808 01:53:10.795552 17695 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:10.866616 18184 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:53:10.886513 18168 consensus_meta.cc:287] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.511s	user 0.012s	sys 0.000s
I20260808 01:53:10.823828 18009 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:10.875801 18160 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:10.899405  9632 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.004s	sys 0.018s
W20260808 01:53:10.912490  9632 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.005s	sys 0.018s
W20260808 01:53:10.899801 10362 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.008s	sys 0.015s
W20260808 01:53:10.913215 10362 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.008s	sys 0.015s
I20260808 01:53:10.919620  9472 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:53:10.920845  9550 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260808 01:53:10.925988  9523 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:10.927250 18122 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:10.927829 18122 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:10.928812 18122 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:10.932962 10334 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:10.934284 18146 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:10.934782 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:10.934995 18146 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:10.935504 10360 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:10.936602 18078 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:53:10.937232 18078 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:10.937794 18146 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:10.938239 18078 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:10.943313  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:10.950637  9523 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:10.952081 18193 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:10.952670 18193 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:10.953114 18193 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:10.892449 18192 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:10.969846  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3143) took 1429 ms. Trace:
I20260808 01:53:10.973942  9558 raft_consensus.cc:1217] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Deduplicated request from leader. Original: 16.10->[17.11-21.13]   Dedup: 17.11->[19.12-21.13]
I20260808 01:53:10.902848 18184 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Advancing to term 27
W20260808 01:53:10.968899 18011 consensus_meta.cc:287] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.619s	user 0.013s	sys 0.000s
I20260808 01:53:10.974493  9558 pending_rounds.cc:85] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 11
I20260808 01:53:10.994284  9558 pending_rounds.cc:107] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 18.12
I20260808 01:53:10.994617  9558 raft_consensus.cc:2889] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:10.972124  9632 rpcz_store.cc:276] 0808 01:53:09.540026 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:09.540132 (+   106us) service_pool.cc:224] Handling call
0808 01:53:09.540482 (+   350us) raft_consensus.cc:1471] Updating replica for 3 ops
0808 01:53:09.773430 (+232948us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:09.773465 (+    35us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0808 01:53:09.774375 (+   910us) log.cc:844] Serialized 78 byte log entry
0808 01:53:10.250555 (+476180us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:53:10.967978 (+717423us) log.cc:844] Serialized 14 byte log entry
0808 01:53:10.969333 (+  1355us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:10.969467 (+   134us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:10.969531 (+    64us) raft_consensus.cc:1685] finished
0808 01:53:10.969551 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:10.969770 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":190537,"prepare.queue_time_us":450666,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":5582,"raft.queue_time_us":195146,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":118,"thread_start_us":1103170,"threads_started":2,"wal-append.queue_time_us":889778}
W20260808 01:53:10.998591  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:10.970415  9559 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.009s	sys 0.036s
I20260808 01:53:10.969521 18178 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:53:11.000533  9559 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.009s	sys 0.037s
I20260808 01:53:11.001519 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:53:11.002132 18176 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.689s	user 0.010s	sys 0.004s
I20260808 01:53:10.967496 18168 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:11.022192  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:11.027877  9636 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 22. Current term is 23. Ops: [22.18-22.18]
I20260808 01:53:11.036190  9600 leader_election.cc:400] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 18. Current term is 19.
I20260808 01:53:11.037999  9600 leader_election.cc:403] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260808 01:53:10.999841 17793 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:11.002895 18011 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:11.044417  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:11.047530  9632 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:11.044673  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:11.049293  9638 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:11.010491 18178 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:11.054090  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:11.064052  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:11.045871 18066 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:53:11.027820 18176 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:11.023059 18009 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:11.092226  9635 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260808 01:53:11.100183 18011 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:11.077267 18167 consensus_meta.cc:287] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.828s	user 0.009s	sys 0.005s
I20260808 01:53:11.107195 18167 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260808 01:53:11.107457 18224 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260808 01:53:11.109697 17793 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:11.115329  9480 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:11.121049 18176 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:11.122819 18193 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:11.127403 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:11.128083 10360 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:11.114042 18129 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:11.136080  9474 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:11.138411 18129 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:11.139137  9638 raft_consensus.cc:2368] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22.
I20260808 01:53:11.140657 18177 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:11.141541 18177 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:11.141708 18219 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:11.144465 18177 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:11.151743 18168 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:11.156064 10360 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:11.157759  9523 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:11.160672  9634 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:11.173177  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:11.175352  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:11.141208 18056 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:11.178918  9475 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:11.181515  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:11.179337  9634 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20. Ops: []
I20260808 01:53:11.169135 17793 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:11.160099 18193 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:11.215301  9445 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.943s	user 0.061s	sys 0.064s
I20260808 01:53:11.217378  9638 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 24. Current term is 25. Ops: []
W20260808 01:53:11.217370  9445 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.945s	user 0.061s	sys 0.064s
W20260808 01:53:11.191478 18124 consensus_meta.cc:287] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.586s	user 0.010s	sys 0.002s
I20260808 01:53:11.196404  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1444) took 1573 ms. Trace:
I20260808 01:53:11.223466  9480 rpcz_store.cc:276] 0808 01:53:09.623212 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:10.235643 (+612431us) service_pool.cc:224] Handling call
0808 01:53:10.236085 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:11.115887 (+879802us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:11.196085 (+ 80198us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b54002c1354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:11.196324 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2021556992}
I20260808 01:53:11.196959 17695 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:11.188158 18056 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:11.200703 18007 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:11.194357 18178 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:11.204666  9555 raft_consensus.cc:2410] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 22 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260808 01:53:11.208873  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:11.257302  9630 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
W20260808 01:53:11.191097 18024 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.523s	user 0.013s	sys 0.000s
W20260808 01:53:11.222728  9631 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.008s	sys 0.035s
I20260808 01:53:11.233234 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:11.258877  9631 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.008s	sys 0.036s
I20260808 01:53:11.259356 10369 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:11.241199  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:11.243921  9638 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260808 01:53:11.365758 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:11.367498  9601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.770s	user 0.016s	sys 0.055s
I20260808 01:53:11.368350 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:53:11.367898  9601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.770s	user 0.016s	sys 0.055s
I20260808 01:53:11.368914  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:11.369786  9473 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:11.370636  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:11.378023 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:11.378417  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:11.232933  9524 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:11.379060  9472 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
W20260808 01:53:11.379349  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 146559us
W20260808 01:53:11.380409  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:11.369495  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 794497us
I20260808 01:53:11.387828  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:11.385533 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:11.391569  9480 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260808 01:53:11.421204 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:11.427083  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:11.431100 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:11.445533  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:11.231103 18124 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:11.451843  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:11.446362  9476 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:53:11.242818 17793 raft_consensus.cc:697] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:11.377664 18165 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:11.421109 18158 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:11.454540  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:11.430474 17848 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:11.239252 18143 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.224s
I20260808 01:53:11.520967  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:11.525136  9474 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
W20260808 01:53:11.489820  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:11.499694  9551 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.694s	user 0.039s	sys 0.089s
I20260808 01:53:11.513800  9638 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:11.514008  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3631) took 2006 ms. Trace:
W20260808 01:53:11.489439  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:11.509066  9472 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:53:11.379019 18024 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:11.627645  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:11.579190  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:11.626212  9599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.026s	user 0.035s	sys 0.038s
I20260808 01:53:11.500293  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1449) took 1709 ms. Trace:
I20260808 01:53:11.514185  9524 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:11.634930  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 121044us
W20260808 01:53:11.636574  9599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.037s	user 0.035s	sys 0.038s
W20260808 01:53:11.637456  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 1.14673e+06us
W20260808 01:53:11.501865 18155 consensus_meta.cc:287] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.792s	user 0.008s	sys 0.005s
I20260808 01:53:11.583181  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:11.644856  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 61794.7us
W20260808 01:53:11.536994 18045 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.782s	user 0.033s	sys 0.030s
I20260808 01:53:11.633366  9481 rpcz_store.cc:276] 0808 01:53:09.791332 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:10.267262 (+475930us) service_pool.cc:224] Handling call
0808 01:53:10.291112 (+ 23850us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:10.455096 (+163984us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:11.499993 (+1044897us) spinlock_profiling.cc:243] Waited 65.6 ms on lock 0x7b540005fd54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:11.500220 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":150780800}
I20260808 01:53:11.514243 18158 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:53:11.455295 18126 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.006s	sys 0.045s
I20260808 01:53:11.506862 18230 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:11.517413 18177 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:11.653471  9551 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.847s	user 0.042s	sys 0.095s
W20260808 01:53:11.642094  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.848s	user 0.047s	sys 0.029s
W20260808 01:53:11.669094  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.875s	user 0.047s	sys 0.029s
I20260808 01:53:11.533655  9473 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:11.659005  9552 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: [23.12-23.12]
I20260808 01:53:11.535440 18176 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:11.535848 18111 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:11.584410  9629 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260808 01:53:11.661762  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:11.638499  9599 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:11.605216 17848 raft_consensus.cc:697] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:11.517694 17830 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:11.626765 18193 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:11.520262 18124 consensus_queue.cc:1059] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.140s
W20260808 01:53:11.670027  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 876256us
I20260808 01:53:11.680238  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:11.624981  9477 rpcz_store.cc:276] 0808 01:53:09.508034 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:09.871999 (+363965us) service_pool.cc:224] Handling call
0808 01:53:09.872318 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:10.880025 (+1007707us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:11.512911 (+632886us) spinlock_profiling.cc:243] Waited 994 ms on lock 0x7b54004047d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:11.513942 (+  1031us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2286823680}
W20260808 01:53:11.654484 18045 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.920s	user 0.033s	sys 0.042s
W20260808 01:53:11.659700 18126 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.859s	user 0.006s	sys 0.046s
I20260808 01:53:11.684113  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:11.700251 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 262833us
I20260808 01:53:11.659255 18182 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:11.654042 18120 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.827s	user 0.013s	sys 0.000s
W20260808 01:53:11.660681 18047 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.802s	user 0.015s	sys 0.014s
I20260808 01:53:11.680315  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:11.714900  9481 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 22
I20260808 01:53:11.540057 18165 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:11.684226  9637 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.693s	user 0.032s	sys 0.028s
I20260808 01:53:11.685281  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:11.676175  9599 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:11.688725  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:11.736281  9477 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 24
I20260808 01:53:11.736846  9473 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:11.665069 18231 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:11.661019 18158 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
W20260808 01:53:11.684655  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.111s	user 0.016s	sys 0.064s
W20260808 01:53:11.750826  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.177s	user 0.017s	sys 0.064s
W20260808 01:53:11.751543  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 1.20334e+06us
I20260808 01:53:11.752453  9521 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:11.669687 18162 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.691s	user 0.009s	sys 0.049s
I20260808 01:53:11.701422  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:11.759203  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 57888us
I20260808 01:53:11.673417  9474 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
W20260808 01:53:11.708807 10362 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.734s	user 0.009s	sys 0.046s
W20260808 01:53:11.763497 10362 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.788s	user 0.009s	sys 0.046s
I20260808 01:53:11.764564  9474 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:11.709760  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:11.767448  9476 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:11.667145 17756 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:11.655050 18155 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:11.655449 17793 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:11.737597  9637 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.747s	user 0.032s	sys 0.028s
I20260808 01:53:11.660769 17991 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.045s
I20260808 01:53:11.707918 18219 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:11.705561 18167 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:11.737247 18045 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:11.772804  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:11.859725  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 87037.4us
W20260808 01:53:11.735221 18047 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.856s	user 0.015s	sys 0.015s
I20260808 01:53:11.679996 18177 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:11.782269  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 106218us
I20260808 01:53:11.863973  9599 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:11.865065 18177 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:11.865784 18177 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:11.762502  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:11.866529  9599 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:11.867549 18177 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:11.681538 18176 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:11.870954 18176 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:11.871161  9557 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260808 01:53:11.755713  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:11.781239 17695 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:11.810215 18228 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:11.699608 18193 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:11.702246 17830 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:11.825187  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 1233) took 1391 ms. Trace:
W20260808 01:53:11.825835 18184 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.822s	user 0.012s	sys 0.001s
I20260808 01:53:11.736567 18168 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.096s
I20260808 01:53:11.715843 18220 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.122s
I20260808 01:53:11.831266 17848 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:11.834443 18225 log.cc:927] Time spent T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260808 01:53:11.841002 18165 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:11.677933 18230 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:11.904744  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 142364us
I20260808 01:53:11.882436 18177 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:11.849709 18155 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:11.850630 18143 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:11.868022  9599 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:11.868264 18233 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election won for term 25
W20260808 01:53:11.735809 18078 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.787s	user 0.025s	sys 0.017s
I20260808 01:53:11.679020 18111 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:11.872432  9473 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:11.937353  9473 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:11.938226  9473 raft_consensus.cc:606] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:11.878598  9481 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:11.741307 18126 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:11.758539 18162 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.782s	user 0.009s	sys 0.050s
W20260808 01:53:11.882642  9632 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.789s	user 0.013s	sys 0.017s
W20260808 01:53:11.939846  9632 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.846s	user 0.014s	sys 0.017s
W20260808 01:53:11.827508 18222 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.176s	user 0.000s	sys 0.000s
I20260808 01:53:11.750250 18231 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:11.883677  9551 rpcz_store.cc:276] 0808 01:53:10.433868 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:10.803322 (+369454us) service_pool.cc:224] Handling call
0808 01:53:10.804342 (+  1020us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:10.805065 (+   723us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:10.805246 (+   181us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:10.805772 (+   526us) log.cc:844] Serialized 26 byte log entry
0808 01:53:11.670902 (+865130us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:53:11.670966 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:11.671615 (+   649us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:11.824888 (+153273us) raft_consensus.cc:1685] finished
0808 01:53:11.824917 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:11.825104 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":135,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"thread_start_us":864662,"threads_started":1,"wal-append.queue_time_us":972444}
I20260808 01:53:11.719656 18182 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:11.717460 18120 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:11.896773  9474 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20. Ops: []
I20260808 01:53:11.899930 18056 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:11.885648 18122 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s	user 0.009s	sys 0.024s
I20260808 01:53:11.846952 18007 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:11.782810 18028 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.014s	sys 0.009s
I20260808 01:53:11.906038  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:11.879901 18167 raft_consensus.cc:697] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:11.917200 18129 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260808 01:53:11.921733 18012 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.132s	user 0.013s	sys 0.000s
I20260808 01:53:11.885686 17695 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:11.898170 18184 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:11.912658 18006 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:11.947505  9476 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
W20260808 01:53:11.950441  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 82499.4us
W20260808 01:53:11.949693 18078 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.001s	user 0.026s	sys 0.017s
I20260808 01:53:12.048470 18006 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:12.049464 18184 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:11.953118 18220 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:12.050895 18220 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:12.052114 18220 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:12.053951 18220 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:12.054837 10364 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 26. Current term is 27. Ops: []
W20260808 01:53:11.879792 18076 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.887s	user 0.037s	sys 0.060s
W20260808 01:53:11.916942 17952 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.319s	user 0.015s	sys 0.000s
W20260808 01:53:12.057363 18076 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.065s	user 0.038s	sys 0.061s
I20260808 01:53:12.057545 17952 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:12.060134 17952 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:12.060024  9599 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:11.919255 18165 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:12.060415 17756 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:53:12.061050 18165 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:12.061612  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:12.063050  9560 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:12.062889 18232 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:12.063859 18176 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:12.064509 18232 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:12.065369 18232 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:12.065838 18165 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:12.018234  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 112335us
I20260808 01:53:11.953181 18233 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:12.066253 18193 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:11.960383 18122 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.828s	user 0.010s	sys 0.024s
I20260808 01:53:11.954861 18228 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:12.026337 18056 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260808 01:53:12.030122 18243 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260808 01:53:11.951239 18143 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:12.039870  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1448) took 2248 ms. Trace:
W20260808 01:53:11.966946 18028 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.014s	sys 0.012s
I20260808 01:53:11.953876 18231 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:12.064647 18176 raft_consensus.cc:3055] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:53:12.067181 18233 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:12.070693  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:12.024621 10361 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.011s	sys 0.040s
I20260808 01:53:12.080070  9444 leader_election.cc:400] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22.
I20260808 01:53:12.141251  9477 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:12.141686  9444 leader_election.cc:403] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:53:12.017706 18024 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:12.084898  9479 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.008s	sys 0.061s
W20260808 01:53:12.145889  9479 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.008s	sys 0.062s
W20260808 01:53:12.080873  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 431161us
I20260808 01:53:12.062533 17756 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.052s
I20260808 01:53:12.086270  9478 rpcz_store.cc:276] 0808 01:53:09.791027 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:10.265900 (+474873us) service_pool.cc:224] Handling call
0808 01:53:10.266520 (+   620us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:10.554235 (+287715us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:10.554265 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:10.555245 (+   980us) log.cc:844] Serialized 52 byte log entry
0808 01:53:10.913590 (+358345us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:53:10.913658 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:11.512044 (+598386us) spinlock_profiling.cc:243] Waited 92.9 ms on lock 0x7b54002bacd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:11.512102 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:12.039487 (+527385us) raft_consensus.cc:1685] finished
0808 01:53:12.039514 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:12.039792 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15828,"raft.queue_time_us":16524,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":95,"spinlock_wait_cycles":213782272,"thread_start_us":357895,"threads_started":1,"wal-append.queue_time_us":1111695}
I20260808 01:53:12.067339 18239 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260808 01:53:12.027099 18129 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:53:12.024443  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:12.168370  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:12.093530 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:12.091495 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:12.065402 18230 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:11.956665 18126 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:12.180768  9478 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
W20260808 01:53:12.085707 18066 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.926s	user 0.014s	sys 0.000s
W20260808 01:53:12.136945  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 66382us
W20260808 01:53:12.096068 10363 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.033s	sys 0.063s
W20260808 01:53:12.183990 10363 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.034s	sys 0.064s
W20260808 01:53:12.143056 10361 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.011s	sys 0.060s
I20260808 01:53:12.087019 18056 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:12.030519 18120 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:12.082007 18147 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.063s
I20260808 01:53:12.147835  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:12.157186  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 1056) took 1582 ms. Trace:
I20260808 01:53:12.206900  9637 rpcz_store.cc:276] 0808 01:53:10.574614 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:10.583514 (+  8900us) service_pool.cc:224] Handling call
0808 01:53:10.584009 (+   495us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:10.893449 (+309440us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:10.893483 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:10.898566 (+  5083us) log.cc:844] Serialized 26 byte log entry
0808 01:53:11.779255 (+880689us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:11.779322 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:11.779427 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:12.156899 (+377472us) raft_consensus.cc:1685] finished
0808 01:53:12.156926 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:12.157110 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15023,"raft.queue_time_us":89565,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":8842,"thread_start_us":1081386,"threads_started":2,"wal-append.queue_time_us":1045414}
I20260808 01:53:12.093050 18176 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:12.209916 18176 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:12.211362 18176 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:12.157900 18167 consensus_queue.cc:237] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:12.212883 18160 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:12.160115 18182 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:12.213459 18160 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:12.161911 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:12.162230 18158 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:12.214443 10366 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:12.215764 18160 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:12.216470 18158 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:12.092319 18184 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:12.045812 18012 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:12.170003 10365 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:53:12.219630  9556 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 25. Ops: [23.12-23.12]
I20260808 01:53:12.217607 18184 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:12.196374  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:12.146176 18228 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:12.205950 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:12.223760  9559 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:12.223876 18228 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:12.225011 18228 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:12.228813 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:12.231067  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:12.233223  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 430026us
W20260808 01:53:12.297641  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 68557.9us
I20260808 01:53:12.259327 18060 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:12.287139 10362 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.000s	sys 0.017s
I20260808 01:53:12.256697 18124 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.004s
W20260808 01:53:12.255272  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 107525us
I20260808 01:53:12.323927  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:12.066565 18232 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:12.276566  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:12.171873 18129 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:12.252146  9559 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 27.
I20260808 01:53:12.206305 18056 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:12.306986 18255 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260808 01:53:12.331440 18251 log.cc:927] Time spent T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260808 01:53:12.332305  9472 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260808 01:53:12.332958 18160 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260808 01:53:12.249380 18158 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:12.205636 18066 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 18. Current term is 19.
I20260808 01:53:12.232646  9523 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:12.339927  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 107348us
W20260808 01:53:12.298548  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 218632us
W20260808 01:53:12.305122 10362 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.001s	sys 0.017s
W20260808 01:53:12.319890  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:12.266441 18121 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22.
I20260808 01:53:12.335577  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 1053) took 1764 ms. Trace:
I20260808 01:53:12.295529 18111 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:12.344967  9480 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:12.347689 10362 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 1281) took 2007 ms. Trace:
I20260808 01:53:12.343641  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:12.344977  9631 rpcz_store.cc:276] 0808 01:53:10.573414 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:10.573742 (+   328us) service_pool.cc:224] Handling call
0808 01:53:10.574154 (+   412us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:11.259602 (+685448us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:11.259653 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:11.379569 (+119916us) log.cc:844] Serialized 52 byte log entry
0808 01:53:11.905594 (+526025us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:53:11.905633 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:11.905712 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:12.335250 (+429538us) raft_consensus.cc:1685] finished
0808 01:53:12.335282 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:12.335494 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25142,"raft.queue_time_us":114607,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":9925,"thread_start_us":1058712,"threads_started":2,"wal-append.queue_time_us":753874}
I20260808 01:53:12.342870  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:12.357692 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:12.359866  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:12.304363 18162 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.040s
I20260808 01:53:12.342872 18158 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:12.321525 17615 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.044s
I20260808 01:53:12.341977 18145 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:12.348373 10362 rpcz_store.cc:276] 0808 01:53:10.340417 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:10.340579 (+   162us) service_pool.cc:224] Handling call
0808 01:53:10.340944 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:10.341368 (+   424us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:10.341401 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:10.341929 (+   528us) log.cc:844] Serialized 26 byte log entry
0808 01:53:10.913768 (+571839us) raft_consensus.cc:1645] Marking committed up to 16
0808 01:53:11.764101 (+850333us) log.cc:844] Serialized 14 byte log entry
0808 01:53:12.341265 (+577164us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:12.341557 (+   292us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:12.341602 (+    45us) raft_consensus.cc:1685] finished
0808 01:53:12.341615 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:12.347414 (+  5799us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":821893,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":20894,"raft.queue_time_us":283996,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":339,"thread_start_us":1937185,"threads_started":3,"wal-append.queue_time_us":1179612}
I20260808 01:53:12.362408  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:12.409704  9550 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 21: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:12.376019 18230 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:12.410542 18230 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:12.410993 18230 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:12.407946 18121 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.007s
I20260808 01:53:12.411726 18256 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:12.412299 18256 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:12.407990 18193 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:12.412961 18193 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:12.413275 18256 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:12.414224 18126 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:12.415489 18193 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:12.416023 17615 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:12.416548 17615 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:12.418888 18238 raft_consensus.cc:3055] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:12.419396 18238 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:53:12.420570 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 86010.9us
I20260808 01:53:12.420374 18238 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:12.350771 18182 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:12.422020 18238 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:12.422201 18182 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:12.422884 18182 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:12.368040  9631 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 27: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:12.374831 18255 consensus_queue.cc:1059] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.030s
W20260808 01:53:12.403789  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet fa45ab4daad84ee78514f98cb3738f23: Illegal state: Leader has not yet committed an operation in its own term
W20260808 01:53:12.403875  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:12.427461 18160 raft_consensus.cc:3055] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:12.427877 18160 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:12.428692 18160 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:12.431360 18047 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:12.433444 18047 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:53:12.372545 17695 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:12.435199 17695 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:12.386922  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:12.437801  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:12.430024 18230 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:12.405449 18158 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:12.459682 18182 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:12.461161  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3147) took 2811 ms. Trace:
I20260808 01:53:12.461553  9629 rpcz_store.cc:276] 0808 01:53:09.649758 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:09.649883 (+   125us) service_pool.cc:224] Handling call
0808 01:53:09.650200 (+   317us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:11.671446 (+2021246us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:12.460825 (+789379us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b54000a9bd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:12.461059 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4447337344}
I20260808 01:53:12.356309 17793 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:12.356611 18024 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:12.440966 17695 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:12.444315 18223 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.926s	user 0.003s	sys 0.000s
I20260808 01:53:12.430109 18160 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:12.450492 18177 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.514s	user 0.007s	sys 0.007s
I20260808 01:53:12.422971 18143 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:12.404945  9554 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:12.364709  9601 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:12.483701  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 119202us
I20260808 01:53:12.361828 18060 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:12.376186  9638 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260808 01:53:12.493615  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:12.497620  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:12.499127  9636 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 23: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:12.499786  9635 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260808 01:53:12.498461  9633 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:12.500586 18060 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:12.501322  9631 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260808 01:53:12.502653 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 1289) took 1120 ms. Trace:
I20260808 01:53:12.505586 10361 rpcz_store.cc:276] 0808 01:53:11.381851 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:11.382323 (+   472us) service_pool.cc:224] Handling call
0808 01:53:11.397556 (+ 15233us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:11.397856 (+   300us) raft_consensus.cc:1534] Early marking committed up to index 19
0808 01:53:12.184591 (+786735us) log.cc:844] Serialized 14 byte log entry
0808 01:53:12.502229 (+317638us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:12.502287 (+    58us) raft_consensus.cc:1645] Marking committed up to 19
0808 01:53:12.502334 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:12.502417 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:12.502572 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":430234,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"thread_start_us":1043830,"threads_started":2}
I20260808 01:53:12.501485 18143 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:12.519269 18143 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:12.521548 18143 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:12.507504 18028 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:12.523512 18028 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:12.524119  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:12.524786 18028 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:12.525774  9551 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:12.479879 18178 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:12.527467 18178 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:12.527966 18249 raft_consensus.cc:3055] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:53:12.528409 18249 raft_consensus.cc:740] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:12.529330 18249 consensus_queue.cc:260] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:12.531062 17615 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:12.533627 18182 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:12.533872  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3026) took 1777 ms. Trace:
I20260808 01:53:12.534227  9552 rpcz_store.cc:276] 0808 01:53:10.756303 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:10.813871 (+ 57568us) service_pool.cc:224] Handling call
0808 01:53:10.814285 (+   414us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:11.670743 (+856458us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:12.533555 (+862812us) spinlock_profiling.cc:243] Waited 844 ms on lock 0x7b54003255d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:12.533779 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1941908736}
I20260808 01:53:12.535358  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 3968) took 2147 ms. Trace:
I20260808 01:53:12.535137 18056 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:12.535844  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1454) took 1753 ms. Trace:
I20260808 01:53:12.536515  9478 raft_consensus.cc:606] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:12.536966  9478 raft_consensus.cc:614] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:12.536267  9475 rpcz_store.cc:276] 0808 01:53:10.782470 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:10.842605 (+ 60135us) service_pool.cc:224] Handling call
0808 01:53:10.845296 (+  2691us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:11.181954 (+336658us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:12.535573 (+1353619us) spinlock_profiling.cc:243] Waited 333 ms on lock 0x7b54002bc5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:12.535782 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":766457600}
I20260808 01:53:12.535765  9560 rpcz_store.cc:276] 0808 01:53:10.388010 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:10.393634 (+  5624us) service_pool.cc:224] Handling call
0808 01:53:10.428413 (+ 34779us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:12.063554 (+1635141us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:12.535087 (+471533us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b540002c1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:12.535264 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3758848384}
I20260808 01:53:12.539954  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:12.510466 18060 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:12.499687  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:12.537942  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3646) took 1169 ms. Trace:
I20260808 01:53:12.551605 10363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 1290) took 1111 ms. Trace:
I20260808 01:53:12.559163  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:12.560336  9472 pending_rounds.cc:85] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 18
I20260808 01:53:12.560822  9472 pending_rounds.cc:107] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 25.19
I20260808 01:53:12.561124  9472 raft_consensus.cc:2889] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:12.560376  9552 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 21: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:12.548349  9479 rpcz_store.cc:276] 0808 01:53:11.368056 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:11.372180 (+  4124us) service_pool.cc:224] Handling call
0808 01:53:11.445279 (+ 73099us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:11.445526 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 13
0808 01:53:12.146461 (+700935us) log.cc:844] Serialized 14 byte log entry
0808 01:53:12.537520 (+391059us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:12.537589 (+    69us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:53:12.537640 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:12.537774 (+   134us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:12.537883 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":806233,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":53,"raft.queue_time_us":3710,"thread_start_us":1027200,"threads_started":2}
I20260808 01:53:12.574625  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:12.583873  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:12.584852  9629 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:53:12.586735  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:12.587652  9629 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:12.481910 17793 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:12.594441 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:12.600080 10361 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:12.595541  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:12.541275 18119 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:12.607025  9633 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:12.609877  9551 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:12.609270  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:12.612380  9554 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:12.612416  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:12.521579 18006 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:12.482223 18024 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:12.620985  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 90465c2f160641a7901ab0815e723ce1: Illegal state: Replica 259bcb3f0d254e8b9effac3277e654e0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 22 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:12.620909  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:12.623340  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:12.624490  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:12.625061  9636 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
W20260808 01:53:12.617098  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:12.617885  9599 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:12.484956 18177 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:12.629199  9444 connection.cc:256] RPC call timeout handler was delayed by -1.3372802670055535s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20310000000000003s for a response
I20260808 01:53:12.616832  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:12.631721  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:12.632267  9631 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:12.551366 18168 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:12.628821  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:12.596427 18219 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.010s	sys 0.008s
I20260808 01:53:12.634846  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:12.637524 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:12.640085  9552 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:53:12.643450  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:12.611193 17932 raft_consensus.cc:3055] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Stepping down as leader of term 21
W20260808 01:53:12.678224  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 43555.5us
I20260808 01:53:12.678881  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:12.679124  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:12.614259 18249 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:53:12.687471  9550 raft_consensus.cc:2410] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 17 because replica has last-logged OpId of term: 16 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 14.
I20260808 01:53:12.691483  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:12.627269 18176 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.022s
I20260808 01:53:12.610623 18266 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:12.692951  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:12.703446  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:12.704195  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:12.710503  9551 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
W20260808 01:53:12.647513 18219 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.011s	sys 0.008s
I20260808 01:53:12.714569  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:12.716077 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:12.716284  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:12.716607  9599 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:12.692205  9635 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:12.717617  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:12.718254  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:12.719125  9635 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:12.720006  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:12.721092  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:12.722084  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:12.716085  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:12.729707  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:12.713693 18024 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:12.733608 18121 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:12.735087 18006 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:12.735612 18006 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:12.736652 18006 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:12.738121 18006 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:12.740087 17793 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:12.741348 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:12.742982  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:12.746316 18009 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:12.747006 18009 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:12.747248  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:12.748319 17615 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260808 01:53:12.747426 18009 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:12.748759  9599 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:12.750295  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:12.750444 18056 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:12.758097 10362 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260808 01:53:12.760423  9636 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:53:12.760614  9631 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
W20260808 01:53:12.763116 18231 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.627s	user 0.010s	sys 0.004s
I20260808 01:53:12.763696 18231 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:12.766271 18231 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:12.768164 18261 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260808 01:53:12.770401 18121 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:12.769395 10361 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:53:12.767593  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:12.772475  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:12.773211 18158 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:12.775034 18269 log.cc:927] Time spent T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260808 01:53:12.775303  9635 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28. Ops: [27.15-27.15]
I20260808 01:53:12.775509 18220 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:12.703194 18168 raft_consensus.cc:697] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:12.776481 18007 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:12.776940 18266 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:12.807184  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:12.807901  9479 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:53:12.807552 18168 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:12.778277 18119 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:12.808842 18119 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:12.779132 18177 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:12.691228 18147 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:12.768971 18230 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:12.718338  9560 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:12.760428 17793 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.003s
I20260808 01:53:12.677601 17932 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:12.801759  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:53:12.797158 18167 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.012s	sys 0.016s
I20260808 01:53:12.799249  9523 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:12.817009  9638 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:12.681097  9629 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
W20260808 01:53:12.819890 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 461389us
I20260808 01:53:12.825456  9635 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260808 01:53:12.780611  9554 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:12.836202  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:12.830636 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 408931us
W20260808 01:53:12.820998 18076 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.763s	user 0.008s	sys 0.025s
I20260808 01:53:12.810941 18056 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:12.846253  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:12.833429 18167 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.012s	sys 0.016s
I20260808 01:53:12.811499 18220 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:12.848763  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:12.849566  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:12.856209  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:12.867895  9560 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:12.842116 18231 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election won for term 22
W20260808 01:53:12.848536 18076 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.790s	user 0.009s	sys 0.025s
I20260808 01:53:12.911270  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:12.899988  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:12.859527 17848 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.046s
I20260808 01:53:12.845700 18147 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:12.843789 18121 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:12.838845 18009 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:12.908957  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:12.914086  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:12.915246  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 66572.9us
I20260808 01:53:12.924368  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:53:12.920225  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:12.861793 18220 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:12.922225  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:12.930223  9363 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.016s	sys 0.062s
I20260808 01:53:12.848917 18056 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:12.934518  9363 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.016s	sys 0.062s
W20260808 01:53:12.917239 18129 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.576s	user 0.013s	sys 0.000s
I20260808 01:53:12.867287 18162 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:12.898893 18266 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:12.942402  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:12.953521  9560 raft_consensus.cc:2410] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 18 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
W20260808 01:53:12.957243  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:12.922115 18121 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:12.927456 18231 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:12.937589 18056 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:12.980285  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:12.920678 18126 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:53:12.980921  9633 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:12.944945 18147 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:12.941046 18129 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:12.944280 18143 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.045s
I20260808 01:53:12.979410 18121 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:12.979528 17615 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:12.965883 18178 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:12.983793 18126 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:13.008580 18126 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:53:12.988206 18147 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:13.008772 18231 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:12.941394 17932 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:12.992971 18266 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:13.039013 18272 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.232s	user 0.001s	sys 0.001s
I20260808 01:53:12.988503 18024 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:13.041070  9559 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21. Ops: [20.10-20.10]
I20260808 01:53:13.042254  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:13.042444  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:13.043318 18276 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
I20260808 01:53:13.048156  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3030) took 2272 ms. Trace:
I20260808 01:53:12.995800 17793 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:12.995540 18262 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:13.053047  9363 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.727s	user 0.018s	sys 0.062s
I20260808 01:53:12.966692 18162 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:13.086898  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:13.061884 18275 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
W20260808 01:53:13.091872  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:13.062220 18285 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260808 01:53:13.090925  9556 rpcz_store.cc:276] 0808 01:53:10.775707 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:10.890484 (+114777us) service_pool.cc:224] Handling call
0808 01:53:10.891047 (+   563us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:12.220184 (+1329137us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:13.047741 (+827557us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54003264d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:13.048065 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3054365056}
I20260808 01:53:13.062409 18147 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:13.057663 18145 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:13.107033  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:13.096542 18024 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:13.111455 18262 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:13.116863 18288 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260808 01:53:13.125157  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:13.127139  9629 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:53:13.082783 18158 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:13.134595  9633 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
W20260808 01:53:13.141856  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 217587us
I20260808 01:53:13.108096 18009 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:13.115056 18047 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:13.154727  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:13.154747  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:13.156567  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:13.156991  9629 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 28: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:13.148959  9444 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:13.158192  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:13.162914  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:13.162791  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:13.164252  9629 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 25: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:13.165709  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:13.090394 18266 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:13.107721 17793 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:13.199029  9634 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260808 01:53:13.199415 18262 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:13.204440 18060 consensus_meta.cc:287] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.659s	user 0.009s	sys 0.002s
I20260808 01:53:13.205109 18060 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:13.167140 18277 raft_consensus.cc:2764] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260808 01:53:13.161227 17989 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:13.189554  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:13.142192 18145 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:13.166114 18047 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:13.208148 18290 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260808 01:53:13.248044 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:53:13.251955  9555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3973) took 2861 ms (client timeout 2829 ms). Trace:
W20260808 01:53:13.252755  9555 rpcz_store.cc:269] 0808 01:53:10.389959 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:10.396553 (+  6594us) service_pool.cc:224] Handling call
0808 01:53:13.251349 (+2854796us) spinlock_profiling.cc:243] Waited 760 ms on lock 0x7b540002ec54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:13.251767 (+   418us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1747777280}
I20260808 01:53:12.552140 10363 rpcz_store.cc:276] 0808 01:53:11.440701 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:11.440907 (+   206us) service_pool.cc:224] Handling call
0808 01:53:11.441367 (+   460us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:11.441917 (+   550us) raft_consensus.cc:1534] Early marking committed up to index 17
0808 01:53:12.199876 (+757959us) log.cc:844] Serialized 14 byte log entry
0808 01:53:12.544974 (+345098us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:12.545057 (+    83us) raft_consensus.cc:1645] Marking committed up to 17
0808 01:53:12.551244 (+  6187us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:12.551354 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:12.551532 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14809,"prepare.queue_time_us":386420,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":179,"thread_start_us":1016476,"threads_started":2}
W20260808 01:53:13.234220  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:13.239845 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:13.234985  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 1059) took 1721 ms. Trace:
I20260808 01:53:13.294240  9630 rpcz_store.cc:276] 0808 01:53:11.513795 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:11.514207 (+   412us) service_pool.cc:224] Handling call
0808 01:53:11.604016 (+ 89809us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:12.362956 (+758940us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:12.362990 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:12.363034 (+    44us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:12.363065 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:13.234707 (+871642us) spinlock_profiling.cc:243] Waited 747 ms on lock 0x7b540003c0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:13.234766 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:13.234920 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22352,"raft.queue_time_us":11889,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":4310,"spinlock_wait_cycles":1718237568}
I20260808 01:53:13.179772 18271 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:13.173101 18129 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:13.267822  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:13.306603  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:13.307533 18012 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:13.307967  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:13.268085  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:13.208581 18266 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:13.209446 18277 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.075s
I20260808 01:53:13.216065 17989 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:13.218863 18267 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
I20260808 01:53:13.286062 17793 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:13.286728 17848 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:13.287436 18047 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:13.287653  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:13.221139 18145 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:13.226481 18056 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:13.226794 18302 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260808 01:53:13.228255 18024 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:13.233130 18289 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
W20260808 01:53:13.289952 18246 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.033s	user 0.000s	sys 0.003s
I20260808 01:53:13.259614  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3029) took 2484 ms. Trace:
I20260808 01:53:13.351997  9557 rpcz_store.cc:276] 0808 01:53:10.775410 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:10.889286 (+113876us) service_pool.cc:224] Handling call
0808 01:53:10.889773 (+   487us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:11.871835 (+982062us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:13.259146 (+1387311us) spinlock_profiling.cc:243] Waited 981 ms on lock 0x7b54001acf54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:13.259502 (+   356us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2256066432}
I20260808 01:53:13.301340 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:13.354058 10363 raft_consensus.cc:2410] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 27 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 19.
I20260808 01:53:13.302970  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:13.355896  9629 raft_consensus.cc:2410] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 27 because replica has last-logged OpId of term: 26 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 14.
W20260808 01:53:13.356353  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:13.312549  9633 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:13.323835  9634 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 22. Current term is 23. Ops: [22.18-22.18]
I20260808 01:53:13.325526 18060 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:13.326968 18121 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:13.308950 18241 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 1.066s	user 0.006s	sys 0.000s
I20260808 01:53:13.332207  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:13.363986  9638 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:53:13.286103 18076 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:13.365342 18076 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:13.366863 18056 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:13.366550 18076 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:13.351336 18024 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:13.368278 18024 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:13.336018 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:13.334064  9445 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:13.373055 10366 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:53:13.373802  9445 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:13.241668 18162 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:13.312280 18271 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:13.225953 18220 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:13.232950 18119 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:13.332345 17848 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:13.244901 18132 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.100s
I20260808 01:53:13.417501  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:13.368240 18076 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:13.423237  9550 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:13.419766  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:13.421164  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:13.426548  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:13.421758  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:13.428910  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:13.430008  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:13.430994  9473 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 25: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:13.433339  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:13.434185  9550 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 28: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:13.430106  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:13.434978  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:13.432418  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:13.435753  9473 raft_consensus.cc:2376] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23. Re-sending same reply.
I20260808 01:53:13.437351  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:13.438390  9473 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 22: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:13.441547  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:13.395514 18056 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:13.379820 18266 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:13.449311  9473 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:13.459103  9477 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:13.444375 18024 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:13.384087 18147 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:13.463558 18147 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:13.465549 18126 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:13.395107 18271 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:13.470196 18271 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:13.470993  9557 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:13.471433  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3165) took 2412 ms. Trace:
W20260808 01:53:13.434546 18302 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260808 01:53:13.471854  9632 rpcz_store.cc:276] 0808 01:53:11.058970 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:11.059936 (+   966us) service_pool.cc:224] Handling call
0808 01:53:11.060282 (+   346us) raft_consensus.cc:1471] Updating replica for 3 ops
0808 01:53:11.060712 (+   430us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:11.060730 (+    18us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0808 01:53:11.065467 (+  4737us) log.cc:844] Serialized 78 byte log entry
0808 01:53:11.940433 (+874966us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:53:11.967615 (+ 27182us) log.cc:844] Serialized 14 byte log entry
0808 01:53:12.042280 (+ 74665us) log.cc:844] Serialized 14 byte log entry
0808 01:53:12.045547 (+  3267us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:12.045635 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:13.289236 (+1243601us) spinlock_profiling.cc:243] Waited 3.29 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:13.471164 (+181928us) raft_consensus.cc:1685] finished
0808 01:53:13.471190 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:13.471370 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":198658,"prepare.run_cpu_time_us":161,"prepare.run_wall_time_us":50442,"raft.queue_time_us":8443,"raft.run_cpu_time_us":368,"raft.run_wall_time_us":80790,"spinlock_wait_cycles":74819200,"thread_start_us":846574,"threads_started":1,"wal-append.queue_time_us":1077512}
I20260808 01:53:13.476944  9479 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260808 01:53:13.477560  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:13.474252  9472 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:13.478327  9480 raft_consensus.cc:2376] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24. Re-sending same reply.
I20260808 01:53:13.492311  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:13.493158  9479 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 16.
W20260808 01:53:13.501945  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:13.454406 18056 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:13.507157  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:53:13.508337  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 339722us
I20260808 01:53:13.509763  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:13.455161 18012 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260808 01:53:13.513051  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:53:13.512943  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 120141us
I20260808 01:53:13.514285  9479 raft_consensus.cc:2376] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20. Re-sending same reply.
I20260808 01:53:13.516345 18129 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 01:53:13.517246 18129 consensus_queue.cc:1243] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 20.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260808 01:53:13.516535  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:13.517788 18297 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:13.518193 18297 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:13.518718 18129 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:13.519294 18129 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:13.519177 18297 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:13.520576 18297 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:13.520289 18129 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:13.521477 18220 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:13.504796 18056 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:13.524392  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:13.530742  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 184693us
I20260808 01:53:13.532558  9552 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: []
I20260808 01:53:13.532603  9632 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: [23.13-23.13]
I20260808 01:53:13.533893  9559 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260808 01:53:13.536717  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:13.534305  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:13.535073  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:13.537453  9637 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260808 01:53:13.533710  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:13.541646  9634 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 17: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:13.542207  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:13.542937  9559 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:13.544164  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:13.547102  9559 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260808 01:53:13.548003  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:13.549930  9636 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:13.550933  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:13.551731  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:13.515014 18266 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.031s
I20260808 01:53:13.556113  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:13.538378  9363 maintenance_manager.cc:643] P 3cd8e8730323483db1c5f22e90bb080c: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 3.735s	user 0.141s	sys 0.099s Metrics: {"cfile_cache_miss":8,"cfile_cache_miss_bytes":25402,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":67613,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":23,"lbm_write_time_us":88575,"lbm_writes_1-10_ms":2,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":6,"peak_mem_usage":259429,"reinsert_count":0,"thread_start_us":703972,"threads_started":3,"update_count":171,"wal-append.queue_time_us":663529}
I20260808 01:53:13.564054  9636 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:13.566030  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:13.491539 17848 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:13.526922 18147 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260808 01:53:13.571219  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:13.568503  9555 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:13.568737  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:13.568145  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:13.572019  9479 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:13.570315  9364 maintenance_manager.cc:419] P 3cd8e8730323483db1c5f22e90bb080c: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260808 01:53:13.574115  9363 maintenance_manager.cc:643] P 3cd8e8730323483db1c5f22e90bb080c: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {}
I20260808 01:53:13.574110  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:13.572587  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:13.575456  9552 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:13.576169  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:13.579289  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:13.555753 17989 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:13.582823  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:13.583068  9636 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: []
I20260808 01:53:13.584846 18147 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:13.585479 18147 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:13.586633 18147 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:13.588362 18220 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:13.588462 18147 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:13.589015 18220 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:13.599022  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:13.599843  9474 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:53:13.602028 18312 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260808 01:53:13.602919 18119 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:13.603456 18024 raft_consensus.cc:3055] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:13.603906 18024 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:13.604856 18024 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:13.611913  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:13.613029  9472 raft_consensus.cc:2410] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 26 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260808 01:53:13.590080 18220 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:13.614758 18220 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
W20260808 01:53:13.617513 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:13.623225  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:13.624056  9480 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:13.625433  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:13.626178  9550 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:53:13.628441  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:13.632099  9445 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:13.636541  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:13.667946 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:13.668421  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:13.668843 10366 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
W20260808 01:53:13.645622 18315 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260808 01:53:13.568231 18121 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:13.652189 18316 consensus_queue.cc:1059] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260808 01:53:13.688112  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 56107.7us
I20260808 01:53:13.655412 17848 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:13.661793  9552 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:53:13.671867  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:13.676119  9480 raft_consensus.cc:2410] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 25 because replica has last-logged OpId of term: 23 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 11.
I20260808 01:53:13.700094 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:13.701507  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:13.705704 10365 raft_consensus.cc:2393] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 17: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:13.628899 18239 raft_consensus.cc:3055] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:13.688769 18271 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:13.684557 18060 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:13.707435  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:13.687947 18119 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:13.727037  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:13.731585  9480 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 25.
I20260808 01:53:13.703691 18309 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:53:13.705487 18277 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:13.731978  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:13.729519  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:13.737639  9472 raft_consensus.cc:2410] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 18 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260808 01:53:13.710503 18092 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260808 01:53:13.723629 18239 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:13.748987  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:13.752382 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:13.755203 18184 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:13.732748 18060 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:13.760278 18219 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:13.760960 18219 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:13.722942 18167 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.022s
I20260808 01:53:13.763098  9445 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:13.733222 18119 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:13.765025 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:13.766876 18139 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:13.767323 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:13.767454 18139 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:13.768880 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:13.769652 18139 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:13.770249 17793 consensus_queue.cc:1059] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260808 01:53:13.772387 10365 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260808 01:53:13.774459 18167 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:13.775162 18167 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:13.775632 18167 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:13.778728 18024 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:13.770524 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:13.732412 18121 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:13.766300 18119 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:13.790206  9550 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260808 01:53:13.786707  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:13.734177 18309 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:13.780491 10362 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:13.790753 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:13.733919 18168 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.902s	user 0.038s	sys 0.074s
I20260808 01:53:13.795826  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3025) took 3071 ms. Trace:
I20260808 01:53:13.797416  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:13.807051 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:13.807682  9555 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:13.796618 10364 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 17.
I20260808 01:53:13.808385 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:13.798745  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:13.789740 18121 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:13.782927 18184 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:13.731035 18271 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:13.795893 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:13.813134 10334 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:13.802861 18257 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:13.747045 18266 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:13.807402  9558 rpcz_store.cc:276] 0808 01:53:10.724616 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:10.827906 (+103290us) service_pool.cc:224] Handling call
0808 01:53:10.828432 (+   526us) raft_consensus.cc:1471] Updating replica for 3 ops
0808 01:53:10.995363 (+166931us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:53:11.010945 (+ 15582us) log.cc:844] Serialized 14 byte log entry
0808 01:53:12.995173 (+1984228us) spinlock_profiling.cc:243] Waited 6.89 ms on lock 0x7b18002c9c20. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d4453242a 00007f5d4499de0a
0808 01:53:13.130517 (+135344us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:13.131241 (+   724us) log.cc:844] Serialized 52 byte log entry
0808 01:53:13.131394 (+   153us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:53:13.131427 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:13.795362 (+663935us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54003147d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:13.795449 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:13.795519 (+    70us) raft_consensus.cc:1685] finished
0808 01:53:13.795540 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:13.795745 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":15560,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":59,"raft.queue_time_us":1437,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":7208,"spinlock_wait_cycles":349874688,"thread_start_us":134012,"threads_started":1,"wal-append.queue_time_us":2109511}
W20260808 01:53:13.810254  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 177079us
I20260808 01:53:13.792665 18060 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:13.799999 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:13.812870 18168 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.988s	user 0.038s	sys 0.074s
I20260808 01:53:13.851091 10362 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
W20260808 01:53:13.871230  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 141822us
I20260808 01:53:13.798964 18309 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:53:13.820438  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:13.809741 18124 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:13.822875  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:13.885161  9558 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 25.
I20260808 01:53:13.812346 18121 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:13.877754  9523 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:13.815407 18257 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:13.891009  9555 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:13.843334  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:13.883796  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 63430.2us
I20260808 01:53:13.889183  9631 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:13.887614  9523 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:13.892382  9476 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:13.892892  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:13.805497 18056 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:13.894297  9524 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:13.898028  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:13.818536 18076 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:13.891295  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:13.818404 18162 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:53:13.867065 18219 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:13.859599 18266 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:13.902006  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 125105us
W20260808 01:53:13.909052  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:13.890080 17695 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:53:13.859617 18122 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.038s
I20260808 01:53:13.860620 18312 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.037s
I20260808 01:53:13.893124 18012 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:13.915782  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:13.817358 18184 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:13.919101  9475 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 27.
I20260808 01:53:13.876878 17793 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.028s
I20260808 01:53:13.926116  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 1246) took 1087 ms. Trace:
I20260808 01:53:13.897346 17989 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:13.884814 18092 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260808 01:53:13.902498 18076 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:13.930862  9556 rpcz_store.cc:276] 0808 01:53:12.838869 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:13.123268 (+284399us) service_pool.cc:224] Handling call
0808 01:53:13.283391 (+160123us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:13.283700 (+   309us) raft_consensus.cc:1534] Early marking committed up to index 15
0808 01:53:13.506157 (+222457us) log.cc:844] Serialized 14 byte log entry
0808 01:53:13.687531 (+181374us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:13.687602 (+    71us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:13.687664 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:13.687745 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:13.925786 (+238041us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b540029e0d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:13.926001 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16972,"prepare.queue_time_us":177838,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":27570,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":9405,"spinlock_wait_cycles":367191808,"thread_start_us":381215,"threads_started":2,"wal-append.queue_time_us":249775}
I20260808 01:53:13.900100 18314 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:53:13.913311 17695 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:13.814976 18239 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:13.908775 18060 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:13.891165 18124 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:13.931989 18324 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:53:13.940606 18262 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260808 01:53:13.937007  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45190 (request call id 5) took 1758 ms. Trace:
I20260808 01:53:13.902470 18250 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.041s
I20260808 01:53:13.965737 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:13.919579 18006 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:13.931417 18229 consensus_queue.cc:1059] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20260808 01:53:13.975200  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:13.924049 18012 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:13.945115 18076 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:13.985621  9478 rpcz_store.cc:276] 0808 01:53:12.169287 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:12.180544 (+ 11257us) service_pool.cc:224] Handling call
0808 01:53:13.927903 (+1747359us) spinlock_profiling.cc:243] Waited 355 ms on lock 0x7b5400392bd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4498dc33 00007f5d4998d41a 00007f5d42ee1934 00007f5d42ee18c2 00007f5d42ee17f1 00007f5d42ee176c 00007f5d42ee0a22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:13.928033 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":815954176}
I20260808 01:53:13.946970 18314 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:13.988924 18168 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:13.990617  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 74965.4us
I20260808 01:53:13.973466  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3414) took 1336 ms. Trace:
I20260808 01:53:14.009649  9631 rpcz_store.cc:276] 0808 01:53:12.637148 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:12.762494 (+125346us) service_pool.cc:224] Handling call
0808 01:53:12.763132 (+   638us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:13.892250 (+1129118us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:13.973109 (+ 80859us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540027e054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:13.973376 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2588968704}
I20260808 01:53:13.968451 17695 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:53:13.962951 18271 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:13.995031 18239 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:13.992988 18324 raft_consensus.cc:697] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:14.022766  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:14.016841  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 117448us
I20260808 01:53:14.012528 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:14.002048 18012 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:13.964648 18231 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.032s
W20260808 01:53:14.014047  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 116129us
W20260808 01:53:14.028412  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:14.034090  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:14.038211 18056 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:14.039606 18056 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:14.040860 18056 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:14.041487 18056 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:14.044622  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:14.049639 18184 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:14.021104 18119 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:14.052284 18119 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:14.055344 17830 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:13.996990 18006 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:13.992734 18266 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:14.018294 18168 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:14.066020 18168 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:14.017238 18297 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260808 01:53:14.051568 18325 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260808 01:53:14.013906 18314 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:53:14.080670  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:14.083125  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:14.083216  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:14.083125  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:14.084270  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:14.017859 18124 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:14.089035  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 145157us
I20260808 01:53:14.090448  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:14.090548  9556 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:14.031937 18250 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:14.093005  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:14.096266  9555 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260808 01:53:14.096757  9636 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260808 01:53:14.097517  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:14.098696  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:14.098870 18335 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:14.099864 18335 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:14.100242 18335 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:14.101082 18335 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:14.101703 18092 consensus_queue.cc:1059] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:14.102351 18335 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:14.103001 18324 consensus_queue.cc:237] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:14.104193 18119 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:14.102782 18092 raft_consensus.cc:3055] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:53:14.104879 18092 raft_consensus.cc:740] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:14.103258 17830 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:53:14.081655 18220 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:14.106922 18220 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:14.107880 18220 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:14.109223 18220 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:14.112232 18120 raft_consensus.cc:3055] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:14.112708 18120 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:14.113564 18120 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:14.119786  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:14.122143 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 207702us
I20260808 01:53:14.123731  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:14.124740  9633 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 23: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:14.128022  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:14.137036  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:14.151121 10332 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:14.138556  9559 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
W20260808 01:53:14.167179 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:14.114637 18120 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:14.165535  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:14.096081 18250 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:14.171641  9555 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260808 01:53:14.152091 18162 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:14.171911 18006 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:14.098272 18168 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.074s
I20260808 01:53:14.172696 18060 raft_consensus.cc:697] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:14.196460 18060 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:14.196969 18167 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:14.197746 18331 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260808 01:53:14.182024 18147 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.593s	user 0.012s	sys 0.005s
I20260808 01:53:14.077519 18056 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:14.181636 18250 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:14.159123 17615 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.029s
I20260808 01:53:14.207388 18121 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:14.207856 18167 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:14.226468 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:14.209704 18297 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:53:14.189360 18162 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:14.229326 18139 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:14.236230 18277 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:14.232052 18168 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:14.241355 18336 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:14.286628 18336 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:14.306970 18314 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:14.308961 18309 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:14.309978 18314 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:14.311663 18309 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:14.353856  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:14.361138  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 195539us
I20260808 01:53:14.361457  9474 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17. Ops: []
W20260808 01:53:14.373777  9366 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55058 (request call id 132) took 29150 ms (client timeout 15000 ms). Trace:
W20260808 01:53:14.347231  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 460e1059f5504707b891c4b0e0ac6a09 (from 6e7856bed1be43acaa821290d506ee97 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:14.387128  9472 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:14.400344 17991 raft_consensus.cc:3055] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:53:14.530303 17991 raft_consensus.cc:740] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:14.532853 17991 consensus_queue.cc:260] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:14.402738  9477 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:14.403991 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:14.407622 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:14.410148 18024 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:14.536055 10360 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:53:14.536535 18024 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:14.539507 17991 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:14.539862 18024 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:14.429421 18121 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:14.431150  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:14.541477  9475 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:53:14.437067  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:14.441504  9481 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:14.442629 18162 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:14.444510  9476 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:14.443462 18168 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:14.446287 18277 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:14.503109  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:14.563024 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:14.564682 18349 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:14.563199  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 406796us
I20260808 01:53:14.314340 18336 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:14.580987  9474 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260808 01:53:14.565865 18355 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:14.605294 18277 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:14.605944 18277 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:14.607548 18277 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:14.609794 18277 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:14.611457  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:14.612251  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 228961us
I20260808 01:53:14.614251 18119 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260808 01:53:14.615446 18119 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:14.612365 18336 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:14.615249 18266 consensus_queue.cc:1059] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.002s
I20260808 01:53:14.616916 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:14.615170 18355 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:14.627882 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:14.631424 18168 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:14.574331 18122 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:14.632979 18122 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:14.633694 18323 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.736s	user 0.001s	sys 0.002s
W20260808 01:53:14.635084  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:14.640666 18316 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:14.641218  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:14.642180 18316 raft_consensus.cc:3055] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:53:14.642541  9474 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:14.649796  9474 raft_consensus.cc:606] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:14.638127 18324 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.005s
I20260808 01:53:14.643707 18328 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Stepping down as leader of term 27
I20260808 01:53:14.643925  9473 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260808 01:53:14.650173  9474 raft_consensus.cc:614] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:14.648442  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:53:14.650645  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:14.652804 10360 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:53:14.644841  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:14.660076  9476 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:14.654625 18328 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:14.672137 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 392651us
I20260808 01:53:14.676393 18132 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:14.677696 18132 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:14.678962  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 58109.6us
I20260808 01:53:14.680187  9599 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:14.683769 18355 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:14.686182  9479 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:14.688199 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 1063) took 1709 ms. Trace:
I20260808 01:53:14.689731  9552 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260808 01:53:14.688772 10361 rpcz_store.cc:276] 0808 01:53:12.978949 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:12.979150 (+   201us) service_pool.cc:224] Handling call
0808 01:53:12.993112 (+ 13962us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:13.053857 (+ 60745us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:13.053918 (+    61us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:13.060194 (+  6276us) log.cc:844] Serialized 52 byte log entry
0808 01:53:13.430998 (+370804us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:13.474191 (+ 43193us) log.cc:844] Serialized 14 byte log entry
0808 01:53:13.474446 (+   255us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:13.474525 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:14.635515 (+1160990us) spinlock_profiling.cc:243] Waited 4.43 ms on lock 0x7b1800301420. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:53:14.687576 (+ 52061us) raft_consensus.cc:1685] finished
0808 01:53:14.687691 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:14.687936 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40467,"prepare.queue_time_us":42093,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":65388,"raft.run_cpu_time_us":389,"raft.run_wall_time_us":22692,"spinlock_wait_cycles":10187136,"thread_start_us":322446,"threads_started":1,"wal-append.queue_time_us":410885}
I20260808 01:53:14.691586  9523 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:14.531811  9366 rpcz_store.cc:269] 0808 01:52:45.223090 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:45.223427 (+   337us) service_pool.cc:224] Handling call
0808 01:53:06.277696 (+21054269us) spinlock_profiling.cc:243] Waited 2.44 ms on lock 0x7b540005c228. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d49916114 00007f5d483e6dbc 00007f5d483a87eb 00007f5d483a85c4 00007f5d4a8c1ad1 00007f5d4a8c232f 00007f5d4a782f0b 00007f5d4a8894be 00007f5d48d424a4 00007f5d48d42432
0808 01:53:14.373382 (+8095686us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:53:06.514603 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:53:06.515463 (+   860us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:53:06.515528 (+    65us) write_op.cc:435] Acquired schema lock
0808 01:53:06.515552 (+    24us) tablet.cc:662] Decoding operations
0808 01:53:06.517258 (+  1706us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:53:06.517412 (+   154us) write_op.cc:641] Partition lock acquired for write op
0808 01:53:06.517439 (+    27us) tablet.cc:685] Acquiring locks for 15 operations
0808 01:53:06.690238 (+172799us) tablet.cc:701] Row locks acquired
0808 01:53:06.690262 (+    24us) write_op.cc:260] PREPARE: finished
0808 01:53:06.690509 (+   247us) write_op.cc:270] Start()
0808 01:53:06.690729 (+   220us) write_op.cc:276] Timestamp: P: 1786153986690412 usec, L: 0
0808 01:53:06.690775 (+    46us) op_driver.cc:348] REPLICATION: starting
0808 01:53:06.691800 (+  1025us) log.cc:844] Serialized 5962 byte log entry
0808 01:53:07.304475 (+612675us) op_driver.cc:464] REPLICATION: finished
0808 01:53:07.766757 (+462282us) write_op.cc:301] APPLY: starting
0808 01:53:07.766965 (+   208us) tablet.cc:1367] starting BulkCheckPresence
0808 01:53:08.055931 (+288966us) tablet.cc:1370] finished BulkCheckPresence
0808 01:53:08.056099 (+   168us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:53:09.065290 (+1009191us) spinlock_profiling.cc:243] Waited 3.83 ms on lock 0x7b1800263380. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188ce62 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:53:09.276747 (+211457us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:53:09.277389 (+   642us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=30,key_file_lookups=30,delta_file_lookups=49,mrs_lookups=0
0808 01:53:09.277498 (+   109us) write_op.cc:312] APPLY: finished
0808 01:53:09.278180 (+   682us) log.cc:844] Serialized 139 byte log entry
0808 01:53:09.485319 (+207139us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:53:09.497411 (+ 12092us) write_op.cc:454] Released schema lock
0808 01:53:09.509752 (+ 12341us) write_op.cc:341] FINISH: Updating metrics
0808 01:53:09.636583 (+126831us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:53:09.636629 (+    46us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2999579,"reported_tablets":25,"spinlock_wait_cycles":5616512,"child_traces":[["op",{"apply.queue_time_us":430153,"apply.run_cpu_time_us":69024,"apply.run_wall_time_us":1862750,"cfile_cache_hit":62,"cfile_cache_hit_bytes":66917,"num_ops":15,"prepare.queue_time_us":235796,"prepare.run_cpu_time_us":53276,"prepare.run_wall_time_us":407693,"raft.queue_time_us":299481,"raft.run_cpu_time_us":123409,"raft.run_wall_time_us":308202,"replication_time_us":613435,"spinlock_wait_cycles":8814336,"thread_start_us":1072044,"threads_started":5,"wal-append.queue_time_us":594969,"wal-append.run_cpu_time_us":25083,"wal-append.run_wall_time_us":2435210}]]}
I20260808 01:53:14.575346 18349 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:14.692700  9556 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
W20260808 01:53:14.704483  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:14.713122  9475 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:53:14.682499 18230 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:14.649446 18316 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:53:14.728232  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:14.728708  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3049) took 1171 ms. Trace:
W20260808 01:53:14.729647 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:14.729316 18316 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:14.730780 18239 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.702s	user 0.004s	sys 0.016s
I20260808 01:53:14.729131  9557 rpcz_store.cc:276] 0808 01:53:13.556697 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:13.556903 (+   206us) service_pool.cc:224] Handling call
0808 01:53:13.574949 (+ 18046us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:13.575411 (+   462us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:13.575447 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:13.594848 (+ 19401us) log.cc:844] Serialized 26 byte log entry
0808 01:53:13.731821 (+136973us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:53:13.731863 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:13.731930 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:14.632562 (+900632us) spinlock_profiling.cc:243] Waited 3.29 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:14.728073 (+ 95511us) raft_consensus.cc:1685] finished
0808 01:53:14.728105 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:14.728464 (+   359us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":117,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":71595,"spinlock_wait_cycles":7570816,"thread_start_us":136538,"threads_started":1,"wal-append.queue_time_us":222546}
I20260808 01:53:14.731720  9477 raft_consensus.cc:2410] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 23 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260808 01:53:14.732646 18349 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:14.733253 18122 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:14.734310  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:14.734936  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:14.735546  9552 raft_consensus.cc:2393] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 22: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:14.735831  9555 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:14.720433 18028 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:14.728247 18220 consensus_meta.cc:287] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.618s	user 0.018s	sys 0.009s
I20260808 01:53:14.737128 18028 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:14.737771  9472 raft_consensus.cc:2410] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 24 because replica has last-logged OpId of term: 23 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260808 01:53:14.740310 18028 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:14.727381 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:14.741501  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 50315.4us
I20260808 01:53:14.742213 18024 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:53:14.743146  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:14.743922  9555 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
I20260808 01:53:14.745162 18132 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:14.747222  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:14.750053  9552 raft_consensus.cc:1240] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21. Ops: [20.10-20.10]
W20260808 01:53:14.752952  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:14.753988  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:14.754918 18168 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:14.756119 18230 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:14.757781 18230 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:14.755857 18168 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:14.760901  9474 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:14.763365 18024 raft_consensus.cc:697] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:14.764914 18024 consensus_queue.cc:237] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:14.767840 18314 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:14.734495 18262 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:14.769879  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:14.705430  9476 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
W20260808 01:53:14.769862  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:14.773820 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:14.783463 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:14.767279 18168 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:14.771271 18262 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:14.774525 18316 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:14.793229 18028 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:14.796787 18076 consensus_meta.cc:287] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.778s	user 0.009s	sys 0.014s
I20260808 01:53:14.798146 18129 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:14.804855  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:14.865921 10366 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 17.
I20260808 01:53:14.871166  9557 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 25: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:14.869763 18366 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.009s
I20260808 01:53:14.852188 18076 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:14.890863  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 114535us
I20260808 01:53:14.861465 18028 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:14.893604  9599 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:14.907366 10362 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 26: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:14.903193  9445 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:14.914600  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:14.918745 18255 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:14.919607 18255 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:14.920986  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:14.921175  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:14.922012 18132 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Leader election won for term 17
I20260808 01:53:14.890357 18129 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:14.929167  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 229255us
I20260808 01:53:14.922725 18060 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260808 01:53:14.941504 17989 raft_consensus.cc:3055] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Stepping down as leader of term 21
W20260808 01:53:14.939003  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 56737.6us
I20260808 01:53:14.928527 18132 raft_consensus.cc:697] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:14.944064  9521 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.061s (SENT) [suppressed 20 similar messages]
W20260808 01:53:14.957051 18120 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.776s	user 0.007s	sys 0.021s
I20260808 01:53:14.958928 18028 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:14.947134 17989 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:53:14.960803 18369 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260808 01:53:14.961493 17989 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:14.962921 17989 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:14.966770 17695 consensus_meta.cc:287] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.847s	user 0.016s	sys 0.008s
I20260808 01:53:14.967409 17695 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:14.971892 18250 consensus_meta.cc:287] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.763s	user 0.019s	sys 0.002s
I20260808 01:53:14.972424 18126 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:14.972555 18250 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:14.974390 10364 raft_consensus.cc:2368] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28.
W20260808 01:53:14.976778  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 298186us
I20260808 01:53:14.977867  9445 leader_election.cc:400] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28.
I20260808 01:53:14.978415  9445 leader_election.cc:403] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
I20260808 01:53:14.978850  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:14.979617 18126 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:14.980119 18239 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260808 01:53:14.981624 18028 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:14.984434 18328 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:14.983305 18028 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:14.985757 18328 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:14.987278 18129 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:14.979722  9552 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 16. Current term is 17. Ops: []
I20260808 01:53:14.988830 18028 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:14.995103 18006 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:14.995720 18006 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:14.996181 18006 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:14.996219 18076 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:15.001098  9557 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:14.970319 17695 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:14.981130  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:14.997726  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:15.016258  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 5cf7e7eadc2347aa84c98413849ebfc9 (from 6e7856bed1be43acaa821290d506ee97 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
W20260808 01:53:15.032028 18012 consensus_meta.cc:287] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.961s	user 0.019s	sys 0.008s
I20260808 01:53:15.032681 18012 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:15.035761  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:15.037424  9472 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 26: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:15.039148 18132 consensus_queue.cc:237] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:15.040804 17991 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.501s	user 0.013s	sys 0.004s
I20260808 01:53:15.045336 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:14.919107 18262 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:15.046373 18126 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28.
I20260808 01:53:15.048074 18262 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:15.052385 18092 consensus_queue.cc:260] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:15.053427 17991 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260808 01:53:15.053587  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 104273us
I20260808 01:53:15.054169 17991 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:15.054855 17991 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:15.054745  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:15.056296 18346 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
W20260808 01:53:15.061785  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:15.054224 18092 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:15.021219 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:15.039361 18255 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:15.063797 18336 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:15.082090 18126 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:15.083235 18259 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:53:15.083334 18126 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:15.067525 18277 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260808 01:53:15.084638  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1465) took 1327 ms. Trace:
I20260808 01:53:15.085117 17932 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:15.086488 18126 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:15.082384 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 71909.1us
I20260808 01:53:15.093436  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:15.100013  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:15.100775  9557 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 26: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:15.097080 18327 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:15.110589 18193 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:15.111285 10362 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d2b2a88358244f93970835ea3120c380"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:57156
I20260808 01:53:15.112121 10362 raft_consensus.cc:606] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:15.112526 10362 raft_consensus.cc:614] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:15.085209  9480 rpcz_store.cc:276] 0808 01:53:13.733734 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:13.733919 (+   185us) service_pool.cc:224] Handling call
0808 01:53:13.734275 (+   356us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:13.862530 (+128255us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:13.862559 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:14.362831 (+500272us) spinlock_profiling.cc:243] Waited 1.64 ms on lock 0x7b2c00134ff8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:53:14.363986 (+  1155us) log.cc:844] Serialized 26 byte log entry
0808 01:53:14.517868 (+153882us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:14.517921 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:15.040904 (+522983us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b540021e154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:15.041000 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:15.050659 (+  9659us) spinlock_profiling.cc:243] Waited 6.83 ms on lock 0x7b5400129ba0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490919e 00007f5d44913d10 00007f5d44913cb8 00007f5d44913c38 00007f5d44913bf3 00007f5d44912e7c 00007f5d4a837996 00007f5d4a832148 00007f5d4494fbab 00007f5d449526a0
0808 01:53:15.057187 (+  6528us) raft_consensus.cc:1685] finished
0808 01:53:15.057212 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:15.061202 (+  3990us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4208,"raft.queue_time_us":7738,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":5561,"spinlock_wait_cycles":313516928,"thread_start_us":141745,"threads_started":1,"wal-append.queue_time_us":145071}
I20260808 01:53:15.106735 18327 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:15.111677 18193 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:15.116120  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:15.121291  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 140311us
I20260808 01:53:15.124788  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:15.125768  9634 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 27: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:15.125631 18193 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:15.133424  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:15.138573 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:15.139374  9472 raft_consensus.cc:2410] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 22 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260808 01:53:15.152017 17848 consensus_queue.cc:1059] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 16.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.018s
W20260808 01:53:15.156622  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 102087us
I20260808 01:53:15.146566 18250 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:15.166631  9473 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 11. (term mismatch)
I20260808 01:53:15.167299  9473 pending_rounds.cc:85] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 10
I20260808 01:53:15.168797  9473 pending_rounds.cc:107] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 21.11
I20260808 01:53:15.169075  9473 raft_consensus.cc:2889] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:15.169561  9475 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:15.170543  9472 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260808 01:53:15.172268  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:15.173158  9481 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 26: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:53:15.173607  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet d2b2a88358244f93970835ea3120c380 (from 7201605bb2f54c949a4ebe6e9b3b4b2a to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:15.158424 17848 raft_consensus.cc:3055] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:53:15.176239  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:15.161347 18257 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:15.176889  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:15.182380  9476 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:15.164403 18145 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:15.185915  9481 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 22: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:15.149394 18193 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:15.186218  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:15.189038 18193 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:15.188195  9599 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:15.177497 17848 raft_consensus.cc:740] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:53:15.195520  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:15.195848 18336 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260808 01:53:15.207399  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:15.208256  9554 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:15.210253  9559 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260808 01:53:15.211187  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:15.211763  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:15.212599 18012 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:15.213601 17848 consensus_queue.cc:260] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 16.15, Last appended by leader: 15, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:15.180658 18259 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.016s
I20260808 01:53:15.219002 18229 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:15.196866 18028 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:15.226820  9632 raft_consensus.cc:2410] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 26 because replica has last-logged OpId of term: 24 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 12.
W20260808 01:53:15.240983  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 95583.6us
I20260808 01:53:15.242486 18271 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:15.243155 18271 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:15.246106 18271 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:15.248380 18229 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:15.236029 18162 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.021s
I20260808 01:53:15.237918  9551 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
W20260808 01:53:15.258594  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:15.260038  9559 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:15.227636 18028 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:15.252156 18229 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:15.266328  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:15.198626 18145 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:15.262974  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:15.271214 18076 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:15.279084 17848 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:53:15.281493  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:15.282339  9636 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:53:15.274570 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:15.283744 18331 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260808 01:53:15.283800 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47351.6us
W20260808 01:53:15.285012  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:15.265936 18028 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:15.287133 18297 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:15.287678 18297 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:15.288070 18297 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:15.290488  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:15.300660 18145 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:15.301849 18076 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:15.302906 18193 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:15.304849 18076 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:15.306995 18257 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:15.320214  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:15.321144  9557 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:53:15.271724 18327 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:15.307569  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:15.324352 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:15.324481 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:15.324566 18336 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:15.325235 10364 raft_consensus.cc:2393] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 28: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:15.325614 18162 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:15.325990 18162 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:53:15.308382 18364 log.cc:927] Time spent T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.565s	user 0.005s	sys 0.002s
I20260808 01:53:15.326728 18230 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260808 01:53:15.327919 18385 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.264s	user 0.000s	sys 0.003s
W20260808 01:53:15.309417  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:15.315132  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:15.316509 18006 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:15.317548  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:15.325176 18336 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:15.326382 18162 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:15.307603  9473 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 28.
W20260808 01:53:15.339367  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 66772.8us
W20260808 01:53:15.336366 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:15.334400 10368 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:15.335245 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:15.326284  9599 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:53:15.341524 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:15.331660  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:15.340687 10334 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:15.345479  9476 raft_consensus.cc:606] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:15.345955  9476 raft_consensus.cc:614] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:15.345402  9557 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
W20260808 01:53:15.353407  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:15.353407  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:15.355239 18250 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader election won for term 28
I20260808 01:53:15.355935 18250 raft_consensus.cc:697] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:15.356094 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:15.357014 10361 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 26: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:53:15.339730 18342 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.011s	sys 0.007s
W20260808 01:53:15.358201 18342 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.011s	sys 0.007s
W20260808 01:53:15.330600 18381 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.197s	user 0.000s	sys 0.001s
I20260808 01:53:15.364424 10369 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260808 01:53:15.360008  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 97545.2us
I20260808 01:53:15.361092  9523 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:15.330938 18365 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.589s	user 0.001s	sys 0.005s
W20260808 01:53:15.351884 18387 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260808 01:53:15.365620  9599 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:15.374713 10365 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:15.375129  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet e3814b146ccd472f9304a0be09ad3738 (from 6e7856bed1be43acaa821290d506ee97 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:15.345324 18336 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:15.378046  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:15.379621 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:15.376881  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260808 01:53:15.390482  9551 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260808 01:53:15.390749 17991 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:15.392011 18006 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:15.398814 17991 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:15.367988 18255 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:15.400629  9521 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:15.412873 10365 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260808 01:53:15.394167 17830 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 15, Time since last communication: 0.026s
I20260808 01:53:15.383453 18316 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260808 01:53:15.426813 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:15.417462  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:53:15.416554 18366 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:15.404714 18255 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:15.401117 18327 raft_consensus.cc:2764] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election decision vote started in defunct term 17: won
I20260808 01:53:15.357105 18250 consensus_queue.cc:237] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:15.423384 17830 consensus_queue.cc:1243] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 22.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260808 01:53:15.440310  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3687) took 1055 ms. Trace:
I20260808 01:53:15.443648  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:15.432984 18366 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:15.432616 18255 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:15.430928 18193 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260808 01:53:15.492974  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 118599us
W20260808 01:53:15.471338  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 132239us
W20260808 01:53:15.505345  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 61866.3us
I20260808 01:53:15.507859  9476 raft_consensus.cc:1275] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260808 01:53:15.460417  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:15.471598 18366 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:15.491456 18324 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260808 01:53:15.519577  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 59294.5us
W20260808 01:53:15.527748  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:15.450188  9478 rpcz_store.cc:276] 0808 01:53:14.384939 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:14.387510 (+  2571us) service_pool.cc:224] Handling call
0808 01:53:14.388140 (+   630us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:14.389163 (+  1023us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:14.389226 (+    63us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:14.390328 (+  1102us) log.cc:844] Serialized 26 byte log entry
0808 01:53:14.571901 (+181573us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:14.571948 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:14.572033 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:15.219933 (+647900us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:15.437225 (+217292us) raft_consensus.cc:1685] finished
0808 01:53:15.437252 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:15.440060 (+  2808us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31522,"raft.queue_time_us":137579,"raft.run_cpu_time_us":486,"raft.run_wall_time_us":53649,"spinlock_wait_cycles":43242368,"thread_start_us":165496,"threads_started":1,"wal-append.queue_time_us":350236}
W20260808 01:53:15.541337 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 206673us
W20260808 01:53:15.560328  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 194783us
I20260808 01:53:15.555367  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:53:15.582741  9560 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:15.586776 18158 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:53:15.592988 18324 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 9, Time since last communication: 0.001s
I20260808 01:53:15.594108 18324 consensus_queue.cc:1243] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 25.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260808 01:53:15.594782 18324 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:15.595206 18324 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:15.596381 18324 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:15.597905 18324 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:15.603056  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:15.611431  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:15.612378 18271 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260808 01:53:15.622993 18028 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:15.639642  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 120780us
I20260808 01:53:15.646896  9560 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:15.571751 17830 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 15, Time since last communication: 0.064s
W20260808 01:53:15.664855  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:15.669523 10363 raft_consensus.cc:2410] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 24 because replica has last-logged OpId of term: 22 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 17.
W20260808 01:53:15.670876  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 87956us
I20260808 01:53:15.649062 18394 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:15.649910 17695 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:15.679917 17830 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:15.680667 18230 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:15.680800 17830 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:15.679800 18394 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:15.682971  9367 catalog_manager.cc:5697] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 10 to 17, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 17 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:15.679795 17695 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:15.685698 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:15.688907 17830 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:15.689167 18392 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260808 01:53:15.688140 17695 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:15.691864 18390 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260808 01:53:15.693387  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 121956us
I20260808 01:53:15.694643 18349 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:15.694720  9601 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:15.695451 18349 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:15.695799 17830 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:15.696465 10362 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260808 01:53:15.696867  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:15.697749 17830 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:15.698084 18349 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:15.700737  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 51841.4us
I20260808 01:53:15.701973 18327 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:15.650009 18271 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:15.645889 18297 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:15.703670  9521 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:15.705619 18297 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:15.659017 18229 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:15.707078 18193 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260808 01:53:15.700575  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:15.685485  9367 catalog_manager.cc:5697] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 13 to 16, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 16 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:15.708977  9480 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:53:15.709911  9367 catalog_manager.cc:5697] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 9 to 13, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 13 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:15.710903 18327 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:15.721958 18316 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:15.722997 18327 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:15.711512  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 100244us
I20260808 01:53:15.686439 10366 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:15.725028  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:15.725399  9600 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:15.726756 18230 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:15.726897 18297 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:15.727245 18230 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:15.727485 18297 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:15.727772 18230 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:15.727936 18297 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:15.715716 18342 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:15.731058 18342 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:15.731279  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:15.731853  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:15.732107 18342 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 23.20, Last appended by leader: 20, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:15.717602 18028 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:15.717381 18366 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:15.718039 17991 raft_consensus.cc:3055] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:15.718665  9601 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:15.740792 18394 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:15.741837  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:15.748360  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:15.756314  9559 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:15.756367  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:15.752991 10363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56266 (request call id 1300) took 1032 ms. Trace:
I20260808 01:53:15.748731 10334 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:15.762887  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:15.759109 10363 rpcz_store.cc:276] 0808 01:53:14.720942 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:14.726815 (+  5873us) service_pool.cc:224] Handling call
0808 01:53:15.750807 (+1023992us) spinlock_profiling.cc:243] Waited 928 ms on lock 0x7b5400690354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:15.752918 (+  2111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2134004480}
W20260808 01:53:15.769457  9560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 3997) took 2348 ms (client timeout 2472 ms). Trace:
I20260808 01:53:15.770143  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:15.770824  9478 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:15.770941 18366 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:15.771973 18259 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:15.772439 18259 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:15.772835 18259 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Advancing to term 24
W20260808 01:53:15.769936  9560 rpcz_store.cc:269] 0808 01:53:13.421384 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:13.421528 (+   144us) service_pool.cc:224] Handling call
0808 01:53:15.769187 (+2347659us) spinlock_profiling.cc:243] Waited 2.13 s on lock 0x7b540002d5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:15.769346 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4891034368}
W20260808 01:53:15.778602  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:15.782374 18297 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:15.784899 18297 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:15.787676 18162 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:15.791805  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:15.810391  9524 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:15.813460 18342 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:15.799190 10364 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:15.791095  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:15.755736  9556 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:15.811923  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:15.814734  9560 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 23
I20260808 01:53:15.855595 18028 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:15.856137 18028 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:15.856540 18028 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:15.853953 18012 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:15.863221 18012 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:15.863720 18012 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:15.749486 17991 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:53:15.794433 18328 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.808s	user 0.011s	sys 0.006s
W20260808 01:53:15.875074 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:15.758025 18092 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.119s
I20260808 01:53:15.840963 18193 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:15.894114  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:15.753053 17695 consensus_queue.cc:1048] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.118s
I20260808 01:53:15.799486 18277 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.097s
I20260808 01:53:15.791237 18259 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:15.812641 18255 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:15.920879  9560 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:53:15.872712 18162 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:15.935290 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:15.852996 18336 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:15.902601 18193 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:15.948486 18193 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:15.953742 18250 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:15.954607  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:15.963025  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:15.963313  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:15.964370  9555 raft_consensus.cc:2410] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 21 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260808 01:53:15.964624  9633 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 17: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:53:15.972417  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:15.976212  9559 raft_consensus.cc:2468] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:15.982987  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:15.983821  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:15.991520  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:15.992298  9634 raft_consensus.cc:1240] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 22. Ops: [20.12-20.12]
I20260808 01:53:15.998116  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:16.011802  9550 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 26: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:16.011459  9555 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260808 01:53:16.014040  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:15.970157 18028 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:15.954542 18250 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:16.015442  9632 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
W20260808 01:53:16.028234  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 261461us
I20260808 01:53:16.045696  9633 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 21: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:16.042491  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:16.090211  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:16.086277  9554 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:16.105681  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:16.106832  9559 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:16.109720  9560 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:16.112315  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:16.114131  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:16.115026  9638 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:16.115551  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:16.116040  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:16.116750  9638 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:16.117453 18259 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:16.116627  9635 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:16.120229  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:16.123088  9558 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:16.125012 18271 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:16.125772 18271 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:16.126261  9629 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:16.127506  9637 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28. Ops: [27.15-27.15]
I20260808 01:53:16.128196 18271 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:16.129607  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 394741us
W20260808 01:53:16.134285  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:16.131824  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:16.121276  9632 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260808 01:53:16.030548 17991 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:16.158824  9550 raft_consensus.cc:2393] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 23: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:16.152586  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:16.162446  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:16.167169 18271 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:16.167667  9634 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260808 01:53:16.167935 18271 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:16.168368  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:16.171444 18271 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:16.173063 18271 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:16.173754 18271 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:16.172595  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:16.177233 18092 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260808 01:53:16.177892  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:16.178885 18092 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260808 01:53:16.179564  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:16.180706 18271 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:16.190335 18092 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:16.191823 18271 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:16.192696 18271 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:16.193384 18399 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260808 01:53:16.196017  9637 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
W20260808 01:53:16.198482  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:16.143298 18316 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:16.209453  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:16.228811  9476 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260808 01:53:16.230275  9636 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
W20260808 01:53:16.230950  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:16.204584 17830 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:16.239082  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:16.243811  9474 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:16.035914  9367 catalog_manager.cc:5697] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 12 to 16. New cstate: current_term: 16 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
W20260808 01:53:16.233866  9521 connection.cc:256] RPC call timeout handler was delayed by -1.0724087450071238s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15280000000000002s for a response
I20260808 01:53:16.248136  9630 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
W20260808 01:53:16.255800  9521 connection.cc:256] RPC call timeout handler was delayed by -1.0535087450116407s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1549s for a response
W20260808 01:53:16.257607  9521 connection.cc:256] RPC call timeout handler was delayed by -1.0058087450161111s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1602s for a response
I20260808 01:53:16.224484 18403 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:16.263316  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 564697us
W20260808 01:53:16.263686  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 54369.7us
W20260808 01:53:16.244275 18230 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.510s	user 0.009s	sys 0.006s
I20260808 01:53:16.271785  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:16.247027 17991 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:16.150074 18250 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:16.252769 18255 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:16.311136 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:16.269716 18277 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260808 01:53:16.269409 18271 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:16.320202  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:16.273902 18336 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:16.273072 18229 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:16.245543 18076 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.067s
W20260808 01:53:16.325006  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:16.343879  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:16.329159  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:16.245314 18316 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:16.342924  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:16.268278 17830 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:16.363191 18342 consensus_meta.cc:287] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.518s	user 0.010s	sys 0.003s
I20260808 01:53:16.320488  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:16.277174 18230 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:16.270638 18403 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:16.363618  9630 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:16.340605 18028 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:16.378392  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:16.411000 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:16.411809 18407 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.069s
W20260808 01:53:16.412214 18324 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.814s	user 0.012s	sys 0.005s
I20260808 01:53:16.413138 18324 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:16.413534 18012 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.549s	user 0.006s	sys 0.009s
I20260808 01:53:16.413856  9555 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 14. (term mismatch)
I20260808 01:53:16.413650 18324 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:16.414266  9555 pending_rounds.cc:85] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 13
I20260808 01:53:16.414710  9555 pending_rounds.cc:107] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260808 01:53:16.414973  9555 raft_consensus.cc:2889] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:16.417712 18230 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:16.418128 18316 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:16.379801 18331 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:16.418778 18396 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.023s	sys 0.027s
W20260808 01:53:16.419214 18396 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.023s	sys 0.027s
W20260808 01:53:16.419982  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 112563us
I20260808 01:53:16.420651 18403 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:16.421571  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:16.422847 17830 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:16.423565 17830 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:16.427181  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:16.428706 18403 consensus_queue.cc:1059] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:16.400192 18229 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:16.426299  9601 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:16.405676  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:16.412154 18408 log.cc:927] Time spent T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260808 01:53:16.430224 18403 raft_consensus.cc:3055] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:16.447767 18403 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:16.447623 18331 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:16.449270 18403 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:16.450816 18331 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:16.433090 18092 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:53:16.430142  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:16.466138  9557 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.567s	user 0.011s	sys 0.002s
I20260808 01:53:16.466747  9557 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:16.452450 18230 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260808 01:53:16.452481 18401 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.312s	user 0.000s	sys 0.003s
I20260808 01:53:16.454982 18328 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:16.455261  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:16.467356  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:16.470448  9474 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 24.
I20260808 01:53:16.414119 18012 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:16.410223 18404 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260808 01:53:16.477654 18381 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260808 01:53:16.469274 18092 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:16.480356  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:16.536892  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 56659us
I20260808 01:53:16.537882  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:16.545184 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:16.546056 10366 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Advancing to term 26
W20260808 01:53:16.546903  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 91703.1us
I20260808 01:53:16.478736 18407 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:16.578168 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:16.479178 18331 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:16.540087 18342 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:16.538354 18092 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:16.431408 18229 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:16.546319 18271 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:16.596850  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:16.614986  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66654.4us
I20260808 01:53:16.616489  9601 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:16.587390 10334 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:16.625734  9599 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:16.627092 18076 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:16.628113 18076 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:16.629541 18076 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:16.633018 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 45720.6us
I20260808 01:53:16.582728 18407 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:16.583425 18403 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:16.646422  9521 leader_election.cc:336] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.528s (SENT)
I20260808 01:53:16.592228 17848 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:16.615425 18230 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:16.409507  9367 catalog_manager.cc:5697] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 10 to 15, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 15 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:53:16.614969 18421 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:16.647655  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:16.593917 18342 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:16.616339 18271 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:16.581333 18328 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:16.690424  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 74034.6us
I20260808 01:53:16.651310 18407 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:16.598421 18331 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:16.704741 10366 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:16.715106  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:16.716084  9476 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 24: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:16.717106  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:16.719112  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:16.719949  9478 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 22: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:16.720968  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:16.721799  9479 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:16.723090  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:16.724335 18411 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:16.724433  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:16.725445  9477 raft_consensus.cc:2410] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 22 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260808 01:53:16.725100 18411 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:16.727172  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:16.727489 18411 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:16.728065 17830 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:16.728974  9481 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:16.728790 17830 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:16.728969 18411 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:16.730271 18411 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:16.731529  9472 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 13. (term mismatch)
I20260808 01:53:16.736084  9472 pending_rounds.cc:85] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 12
I20260808 01:53:16.736362  9472 pending_rounds.cc:107] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 23.13
I20260808 01:53:16.736619  9472 raft_consensus.cc:2889] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:53:16.743204  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 123092us
I20260808 01:53:16.743688 18336 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:16.744275  9521 leader_election.cc:336] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.549s (SENT)
I20260808 01:53:16.744596 18336 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:16.744891  9521 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:16.746143 18425 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260808 01:53:16.697167 18271 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:16.704164 18394 raft_consensus.cc:2804] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:16.689206 18158 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:16.696967 18342 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:16.698846 17848 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:16.698195 18421 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:16.746623  9521 leader_election.cc:336] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.602s (SENT)
W20260808 01:53:16.652730 18417 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.174s	user 0.000s	sys 0.001s
I20260808 01:53:16.756094  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:16.757210  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:16.784909  9475 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:16.758540  9477 raft_consensus.cc:1240] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:16.785157 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:16.689571 18406 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:16.759594  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:16.762136  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:16.752358  9479 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:16.778995  9600 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:16.649992 18229 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260808 01:53:16.780356  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:16.762137 17848 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:16.783488  9521 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:16.791005  9473 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 21: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:16.755012  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:53:16.779253  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:16.715142 18420 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.246s	user 0.000s	sys 0.001s
I20260808 01:53:16.793385  9478 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:16.765120 18328 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:16.794764 18255 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:16.795506 18255 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:53:16.796748 18193 consensus_meta.cc:287] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.847s	user 0.015s	sys 0.000s
I20260808 01:53:16.796666 18255 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:16.797260 18193 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:16.751681 18411 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:16.802088  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:16.803547  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:16.751861 18336 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:16.805876  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:16.806249 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:16.807696 18421 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:16.809682 18331 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:16.810559 18277 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:16.811175 18324 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:16.748128 18394 raft_consensus.cc:697] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:16.715204 17991 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.102s
I20260808 01:53:16.749354 18158 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:16.819815  9550 raft_consensus.cc:2368] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24.
W20260808 01:53:16.837121  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 299342us
I20260808 01:53:16.839673  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:16.843904  9480 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
W20260808 01:53:16.844951  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:16.839289  9473 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:16.836690 18277 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:16.879177  9475 raft_consensus.cc:2410] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 22 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260808 01:53:16.893783 18012 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:16.835192 18430 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:16.903064 18394 consensus_queue.cc:237] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:16.906898  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 160318us
W20260808 01:53:16.907671  9521 leader_election.cc:336] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.753s (SENT)
I20260808 01:53:16.908051  9521 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:16.915287 18193 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:16.916888 18342 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:16.920220 18328 consensus_queue.cc:1059] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:16.885728 18277 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:16.924101 18342 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:16.889674 10334 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:16.917218  9367 catalog_manager.cc:5697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 11 to 16. New cstate: current_term: 16 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:16.936628  9559 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:16.949193 18158 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:16.953423 18271 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:16.955679 17830 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:16.935925 18092 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:16.958528 18324 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260808 01:53:16.959610 18092 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:16.960436 18092 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.000s
I20260808 01:53:16.936303 18250 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:53:16.924995 18342 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.038s
I20260808 01:53:16.963136 18250 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:16.963586 18250 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:53:16.969408  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:16.969482  9481 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 19. Current term is 20. Ops: []
W20260808 01:53:16.975277  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 67646.3us
W20260808 01:53:16.976393  9521 leader_election.cc:336] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.792s (SENT)
I20260808 01:53:16.954047 17932 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:53:16.977623  9521 leader_election.cc:336] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.842s (SENT)
I20260808 01:53:16.978106  9521 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:16.906010 18407 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:16.993850 18407 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:16.995172 18441 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:16.996387 18158 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260808 01:53:16.996903 18435 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260808 01:53:16.962282 18092 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.035s
I20260808 01:53:17.002190 18230 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:17.001075 18331 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.008s
I20260808 01:53:17.020607 17830 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:17.021510 17830 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:17.023250  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.023664 18230 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:17.026204 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:17.027073  9633 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:17.008033 18271 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:17.046373  9521 leader_election.cc:336] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.856s (SENT)
W20260808 01:53:17.020555 10332 leader_election.cc:336] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.727s (SENT)
I20260808 01:53:17.027823 10363 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:17.047636  9559 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
W20260808 01:53:17.050617  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 427934us
I20260808 01:53:17.051626  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:17.051774  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:17.052597  9475 raft_consensus.cc:2410] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 25 because replica has last-logged OpId of term: 23 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 19.
I20260808 01:53:17.055496  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:17.047628  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:17.058750  9599 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:17.058027  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:17.057965  9476 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
W20260808 01:53:17.062590 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:17.060115  9556 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:53:17.063568  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.063850  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:17.061695  9600 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:17.064579  9476 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:53:17.046329 18092 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:17.028714 17830 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:17.100672  9599 leader_election.cc:400] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24.
I20260808 01:53:17.103416  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.108510 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:17.119560 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:17.063215 18396 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:16.963805  9367 catalog_manager.cc:5697] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 13 to 17, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 17 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:17.122468  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:17.067329  9479 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: []
I20260808 01:53:17.157713  9476 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:17.162143  9633 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 27.
I20260808 01:53:17.110095 18406 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:17.124308  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:17.162590  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 98904.8us
I20260808 01:53:17.169107  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.171663 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:17.174697 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 154249us
I20260808 01:53:17.110999 18324 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:17.180255  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:17.180563  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:17.192699  9559 raft_consensus.cc:3060] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:17.195251  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:17.195541  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:17.196146  9479 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:17.196695  9552 raft_consensus.cc:2410] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 21 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260808 01:53:17.201136 10332 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:17.202893  9550 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:17.164675 18259 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:17.205999  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 283969us
W20260808 01:53:17.209623  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 163296us
I20260808 01:53:17.112098 18092 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:17.210971 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:17.217234  9633 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:17.218508  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:17.219039  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:17.219344  9550 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:17.220500  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:17.221216  9560 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:17.222488  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:17.223409  9560 raft_consensus.cc:2393] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 25: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:17.208266  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:17.228070  9521 leader_election.cc:336] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.866s (SENT)
I20260808 01:53:17.228781  9521 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:17.175522 18406 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:17.235844  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:17.238201  9555 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:17.236977  9524 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:17.195581 18076 raft_consensus.cc:3055] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:53:17.248109  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:17.252475  9560 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:17.248158  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:17.254137  9631 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:17.255730  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:17.255893  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:17.264066  9636 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260808 01:53:17.265460  9551 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: [23.20-23.20]
I20260808 01:53:17.224970 18259 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:17.271740  9559 raft_consensus.cc:2468] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:17.272290  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:17.215998 18396 raft_consensus.cc:697] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:17.244796 18328 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:17.273787 18421 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:17.277282 18443 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260808 01:53:17.277769 18012 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:17.205137 17848 consensus_queue.cc:1059] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.062s
I20260808 01:53:17.237321 18440 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.071s
W20260808 01:53:17.278785  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:17.288118  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 60097.2us
I20260808 01:53:17.289642  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:17.240926 18449 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.059s
I20260808 01:53:17.277093  9599 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:17.279248  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.235028 18193 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:17.248857 18076 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:17.286073 18259 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:17.275068 18425 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:17.271215 18257 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260808 01:53:17.286134 18433 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.165s	user 0.001s	sys 0.000s
I20260808 01:53:17.242583 18406 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:17.277094  9524 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:17.302120  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:17.298770 18396 consensus_queue.cc:237] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:17.284534 18439 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.173s	user 0.000s	sys 0.000s
I20260808 01:53:17.296416 18277 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:17.329985  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52952.2us
I20260808 01:53:17.315126  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:53:17.332710  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.315526  9633 raft_consensus.cc:2368] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 27. Current term is 28.
I20260808 01:53:17.316011  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.317281  9636 raft_consensus.cc:2393] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 22: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:17.319509  9476 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:17.313403  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.303532 18328 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:17.322629 18443 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:17.304054 18421 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:17.323082  9560 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:17.324407  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:17.305025 18428 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:17.305562 18092 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:17.326960  9559 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:17.328876  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:17.371901  9550 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 22: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:53:17.330230 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 129190us
I20260808 01:53:17.316543 18429 raft_consensus.cc:3055] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:17.313779  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:17.336001  9637 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:17.347615  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.321395  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:17.381729  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 68065us
I20260808 01:53:17.351176  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:17.383399  9524 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:17.353315  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.389392  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:17.338814 18012 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:17.396106  9638 raft_consensus.cc:2393] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 22: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:17.389860  9599 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:17.357879  9631 raft_consensus.cc:2410] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 26 because replica has last-logged OpId of term: 24 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 12.
I20260808 01:53:17.352490 18328 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:17.322305 18193 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:17.429877 18193 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:17.398651  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:17.431155  9550 raft_consensus.cc:2393] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 23: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:17.399621  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:17.432950  9559 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:17.430158  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:17.401248  9636 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260808 01:53:17.401599  9556 raft_consensus.cc:2393] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 17: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:17.366043 18443 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:17.440694  9560 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
W20260808 01:53:17.426651  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:17.446358 10364 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260808 01:53:17.400321 18012 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:17.305274  9367 catalog_manager.cc:5697] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 12 to 16, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 16 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:17.368635 18421 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:17.411444 18452 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260808 01:53:17.418635  9479 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260808 01:53:17.419798  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:17.419981 18277 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:17.420657 18342 raft_consensus.cc:3055] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:17.421883  9599 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:17.399816 18324 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:17.381039 18429 raft_consensus.cc:740] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:17.396734 18432 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:17.364516 17932 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.063s
I20260808 01:53:17.433817  9632 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:17.449249  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:17.450521  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:17.452039  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:17.512203  9638 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:17.406114  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.462496  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.513214  9633 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 23: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:17.403730 18394 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:53:17.464053 18012 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:17.467705  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:17.437662 18454 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:17.474956  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 53163.6us
I20260808 01:53:17.477551  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:17.525707  9560 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 26.
I20260808 01:53:17.439302 18455 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:17.481029  9474 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 26.
I20260808 01:53:17.440487 18443 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:17.464903 18250 consensus_meta.cc:287] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.501s	user 0.014s	sys 0.001s
I20260808 01:53:17.399223 18428 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:17.538283 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 164247us
I20260808 01:53:17.466962 18336 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:17.522913 18427 log.cc:927] Time spent T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260808 01:53:17.479070 18342 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:17.551369  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:17.587035  9633 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 23: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:17.510521 18277 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:17.484257  9555 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:17.513315 18432 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:17.571316  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.596796  9635 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:53:17.482314 18448 raft_consensus.cc:2804] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:17.483096 18092 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.061s
W20260808 01:53:17.603500 18459 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
W20260808 01:53:17.605762  9521 leader_election.cc:336] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.939s (SENT)
I20260808 01:53:17.445035 18076 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:17.607642 18342 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:17.599768 18475 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.091s	user 0.002s	sys 0.000s
I20260808 01:53:17.587321 18336 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:17.560216 18250 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:17.618933 10363 raft_consensus.cc:2368] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18.
I20260808 01:53:17.619865 18448 raft_consensus.cc:697] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:17.621344 18448 consensus_queue.cc:237] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:17.586499 18454 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:17.624331 18428 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:17.625284 18428 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:17.626070 18454 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:17.625898 18428 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:17.561399 18455 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:17.628230  9480 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:17.600337 18429 consensus_queue.cc:260] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:17.615653 18458 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
W20260808 01:53:17.633749 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:17.654072  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:17.640164  9632 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:17.655234  9635 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:17.588424 18331 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:17.641876  9480 raft_consensus.cc:606] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:17.659349  9480 raft_consensus.cc:614] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:17.618350 18250 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:17.669852  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.670997  9632 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 27.
I20260808 01:53:17.593832  9559 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:17.631034  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.672845  9633 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 22: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:17.606420  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:17.663389 18455 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:17.664513  9555 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:17.595436  9367 catalog_manager.cc:5697] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 8 to 14, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 14 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:17.601243 18434 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:17.679226 18434 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:17.665179 18429 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:17.599579 18277 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:17.666358 18336 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:17.599334 18432 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:17.667152  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.667387 18331 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:17.685654  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.692431  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.697548  9633 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:53:17.704388  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:17.705372  9632 raft_consensus.cc:2368] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 25. Current term is 26.
I20260808 01:53:17.706439  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:17.716576  9638 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:53:17.718102  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.707285  9632 raft_consensus.cc:2376] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22. Re-sending same reply.
W20260808 01:53:17.729156  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 145c5a9982a24c8fa4e86d0d99cbbdfb (from 6e7856bed1be43acaa821290d506ee97 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:17.729476  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:17.704290 10332 leader_election.cc:336] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.695s (SENT)
I20260808 01:53:17.730912 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:17.740893 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 140892us
I20260808 01:53:17.747081  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.748193  9633 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:17.752580  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.753731 18324 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:17.776690  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.777688  9638 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 28: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:17.782446  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:17.710559 18230 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:17.728830 18271 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:17.786980  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.689956 18328 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:17.793040  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:17.796168  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.797123  9632 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 22.
I20260808 01:53:17.797969  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:17.799855  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:17.799926  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:17.789461  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.800834  9474 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:17.801054  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.805823  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:17.777384 18449 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:17.807129  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.788491 18477 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:17.813853  9474 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:17.780221 18403 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260808 01:53:17.815656  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 209967us
I20260808 01:53:17.817401  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:17.818176  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:17.794075 18428 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:17.819752  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.821229  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:17.822212  9521 leader_election.cc:336] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.996s (SENT)
I20260808 01:53:17.822369  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.822731  9521 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:17.826799 18092 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:17.827009 18442 consensus_queue.cc:1059] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.007s
I20260808 01:53:17.820926 18472 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:53:17.828917 18012 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:17.830937 18482 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:17.831833 18482 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:17.832123 18012 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:17.831043  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:17.833827  9638 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 28: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:17.834264 18482 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:17.832648 18277 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:17.835098  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:17.837977 18193 raft_consensus.cc:3055] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:53:17.840396  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:17.807030 18230 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:17.683835  9367 catalog_manager.cc:5697] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 12 to 17. New cstate: current_term: 17 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
W20260808 01:53:17.842410 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 100444us
I20260808 01:53:17.843075 18250 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:53:17.844820 18403 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:17.797897 18271 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:17.863619  9638 raft_consensus.cc:2368] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 25. Current term is 26.
I20260808 01:53:17.847368  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:17.849920 18331 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:17.850826 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 146558us
I20260808 01:53:17.863965 17932 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:17.838061 18421 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:17.853258 18434 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:17.878994 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:17.854714  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:17.864892 18328 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:17.874833 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:17.863672  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.873224 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:17.852152 18324 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:17.888473  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:17.911692  9635 raft_consensus.cc:2410] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 25 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260808 01:53:17.895586  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:17.920677  9629 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
W20260808 01:53:17.898821  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 52322us
I20260808 01:53:17.915288 10361 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
W20260808 01:53:17.899402 10332 leader_election.cc:336] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.706s (SENT)
I20260808 01:53:17.875303  9630 raft_consensus.cc:2368] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21.
W20260808 01:53:17.901667  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 83569.9us
I20260808 01:53:17.883798  9474 raft_consensus.cc:606] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:17.924147  9474 raft_consensus.cc:614] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:17.921873  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:17.911424  9633 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:17.898940  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.896461  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:17.900933  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 84592.6us
W20260808 01:53:17.929566  9521 leader_election.cc:336] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.002s (SENT)
I20260808 01:53:17.929607  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:17.893293 18406 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.523s	user 0.011s	sys 0.000s
I20260808 01:53:17.898541 18277 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260808 01:53:17.917712 17932 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:17.935647  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.936102 18406 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:17.938110  9631 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 24: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:17.938530  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:17.939251  9480 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:53:17.920106  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:17.937100  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:17.865041 18193 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:17.872769 18230 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:17.941615 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 68473.1us
I20260808 01:53:17.877362 18403 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:17.944708  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:17.956187  9476 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:17.947769  9637 raft_consensus.cc:2368] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23.
I20260808 01:53:17.932637  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.949803  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.950284  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:17.944579  9445 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:17.950507  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:17.951581 18193 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:17.952023  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.900235 18428 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:17.864295 18342 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:17.901275 18488 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:53:17.864332 18485 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:17.965240  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:17.979946  9476 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 16 FOLLOWER]: Advancing to term 18
I20260808 01:53:17.980049  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:17.980939  9637 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:53:17.967772 18428 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:17.956756 18277 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:17.885962 18271 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:17.968578  9632 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 25: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:17.937175 17932 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:17.966650  9635 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:17.976212  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:18.002429  9638 raft_consensus.cc:2410] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 22 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260808 01:53:18.002475  9476 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:53:17.976979  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:17.854091 18477 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.004949 10334 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:18.011289 18483 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:18.020888 18496 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:18.021062 18483 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.021798 18496 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:18.022399 18496 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:18.024693 18483 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:18.028649 18257 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:53:18.029708 18257 raft_consensus.cc:697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:53:18.030938 18497 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260808 01:53:18.032850 18421 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:18.033267  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:18.035056  9550 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260808 01:53:18.036954  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:18.037976  9636 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:18.040738  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:18.045222  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 100860us
I20260808 01:53:17.987308  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:18.052942  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:18.053855  9476 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:53:17.963158 18472 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:17.990033  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 60520.9us
I20260808 01:53:17.985049 18428 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:18.054569  9550 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260808 01:53:17.990922 18193 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:17.998327 18277 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:17.872172  9367 catalog_manager.cc:5697] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 8 to 15, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 15 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:53:18.000607 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 101246us
I20260808 01:53:18.062131  9629 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 28.
I20260808 01:53:18.053789  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:17.988699  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 20 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:18.067416  9477 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:17.993036  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:18.054651  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:18.052852  9524 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:18.052126  9637 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
W20260808 01:53:18.061592  9521 leader_election.cc:336] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.020s (SENT)
I20260808 01:53:18.004788 18441 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260808 01:53:18.063126 10332 leader_election.cc:336] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.720s (SENT)
I20260808 01:53:18.071130 10332 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:17.955431 18328 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:18.078943  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:18.080672  9472 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260808 01:53:18.068733  9521 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:18.068712  9554 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:18.069063  9524 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:18.082123  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:18.083443  9635 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:18.060947 18277 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:18.097628 18277 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:18.103350 18429 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:18.105957 18432 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.110769  9633 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 24.
I20260808 01:53:18.112496  9633 raft_consensus.cc:1275] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260808 01:53:18.119160  9367 catalog_manager.cc:5697] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 14 to 17. New cstate: current_term: 17 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:18.120069  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:18.120743  9472 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:18.059103 18472 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.124832 18472 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:18.126557  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:18.129302 18406 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:18.130697 18489 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:18.131163 18489 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:18.124302 18432 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:18.088968  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:18.091230 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:18.096436  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:18.120134  9633 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 15. (term mismatch)
I20260808 01:53:18.134429  9633 pending_rounds.cc:85] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 14
I20260808 01:53:18.134727  9473 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:18.135087  9633 pending_rounds.cc:107] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 21.15
I20260808 01:53:18.086232  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:18.135548  9633 raft_consensus.cc:2889] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:18.136049  9476 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 26: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:53:18.132135  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 44705.5us
I20260808 01:53:18.143662  9635 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 23. Ops: []
I20260808 01:53:18.143791  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:18.144850 18396 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:18.145514  9635 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 20. Current term is 21. Ops: []
I20260808 01:53:18.075461 18421 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:18.145687  9473 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:18.078799 18328 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:18.172287  9554 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260808 01:53:18.121654  9367 catalog_manager.cc:5697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 12 to 16. New cstate: current_term: 16 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:18.131558 18489 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:18.129830 18496 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:18.185145  9472 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:18.188218  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:18.132050 18255 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.059s
I20260808 01:53:18.163502 18421 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:18.193780 18421 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:18.196412 18394 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:18.162817 18483 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:18.177141 18328 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:18.207118  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:18.217491  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:18.217726  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:18.219790  9638 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260808 01:53:18.214361  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:18.198076 18076 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:18.224486  9552 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:18.229449  9601 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:18.233423  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 177923us
W20260808 01:53:18.234097  9521 leader_election.cc:336] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.021s (SENT)
I20260808 01:53:18.210300 18432 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:18.238716  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:18.147904 18257 consensus_queue.cc:237] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 16.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:18.213442 18477 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:18.244184 18496 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:18.249828 18489 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.253403  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:18.254195  9474 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:18.255007  9632 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:18.255968 18489 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:18.262796  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 2973a134fcef4b8da54331bcd12f1173 (from 6e7856bed1be43acaa821290d506ee97 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:18.263825  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:18.264519  9552 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:53:18.267537  9473 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:18.268292  9473 raft_consensus.cc:606] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:18.268684  9473 raft_consensus.cc:614] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:18.270720  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:18.271610  9551 raft_consensus.cc:2393] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 24: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:53:18.272817  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 4bce6ea2a8244811bf007638181a417a (from 6e7856bed1be43acaa821290d506ee97 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:18.273280 18428 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:18.275657  9473 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:18.278295  9473 raft_consensus.cc:606] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:18.278972  9473 raft_consensus.cc:614] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:53:18.279951 18443 consensus_meta.cc:287] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.693s	user 0.008s	sys 0.006s
W20260808 01:53:18.280514  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet affdd24b03ae4ee795a90703613305f7 (from 6e7856bed1be43acaa821290d506ee97 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:18.280501 18443 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.281490  9630 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:18.281919  9552 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:53:18.282307  9634 raft_consensus.cc:2368] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 21. Current term is 22.
I20260808 01:53:18.283829  9367 catalog_manager.cc:5697] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 7 to 16, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 16 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:53:18.287125 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 51831.2us
I20260808 01:53:18.294611  9480 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:18.296417  9524 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:18.299360  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 60786.1us
I20260808 01:53:18.303390 18396 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:18.303644 18444 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:18.303423 18394 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:18.305060 18394 consensus_queue.cc:1243] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 24.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260808 01:53:18.305735 18396 raft_consensus.cc:3055] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:18.306000 18485 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:18.307120 18485 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:18.308498 18485 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.304477  9556 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:18.306118 18396 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:18.235495 18336 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:18.316468 18396 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.316555 18336 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:18.320501 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:18.327772 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:18.329459 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:18.329947 10369 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260808 01:53:18.330119  9445 leader_election.cc:400] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18.
I20260808 01:53:18.330402 10364 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 27: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:18.332665 18394 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:18.333382 18394 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.330530  9445 leader_election.cc:403] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:53:18.336407 18394 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:18.338162 18454 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18.
I20260808 01:53:18.342582 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:18.344123 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:18.344228  9558 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
W20260808 01:53:18.345196  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 111115us
I20260808 01:53:18.348968 18396 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:18.349519 18483 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:18.351938  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:18.356652  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:18.228353 18462 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.131s
I20260808 01:53:18.359673 18443 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:18.360365 18448 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:18.344628 18259 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:18.322118 18336 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:18.413290 18259 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:18.413954 18259 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:18.413868 18336 consensus_queue.cc:1059] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:18.360976 18461 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:18.301152 18432 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:18.286198  9367 catalog_manager.cc:5697] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 12 to 16, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 16 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:18.404043 18255 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.013s
I20260808 01:53:18.417567 18432 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.417671  9367 catalog_manager.cc:5697] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 10 to 15. New cstate: current_term: 15 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:18.419746  9367 catalog_manager.cc:5697] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 10 to 14, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 14 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:18.323067 18092 raft_consensus.cc:3055] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:18.367241 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:18.237914 18366 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:18.369417 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:18.320209  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:18.426318  9480 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:18.399536  9558 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
W20260808 01:53:18.405521 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 61494.3us
I20260808 01:53:18.406016  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:18.416245 18448 consensus_queue.cc:1059] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:18.421700  9523 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:18.421711 10364 raft_consensus.cc:2410] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 23 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260808 01:53:18.374279  9474 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
W20260808 01:53:18.438186  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:18.439906  9477 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:53:18.435192  9524 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:18.451220 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:18.403462 17991 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:18.450878  9524 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:18.409499 18462 consensus_queue.cc:1243] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 26.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260808 01:53:18.449186 18425 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:18.480829 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:18.425359 18092 raft_consensus.cc:740] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:18.443064 18428 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:18.463973 17932 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:18.503744 10364 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 27: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:53:18.454763  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:18.405522 18483 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.454489 18336 consensus_queue.cc:1243] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 24.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260808 01:53:18.473212 18257 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:53:18.514007  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.035s	sys 0.032s
W20260808 01:53:18.524127 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 94227.9us
I20260808 01:53:18.528429  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:18.529773  9558 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 27: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:18.531656 10364 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fca874793205463dbb922993f57bf7f5"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:57156
I20260808 01:53:18.532626 10364 raft_consensus.cc:606] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:18.533118 10364 raft_consensus.cc:614] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:53:18.535858  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.036s	sys 0.032s
W20260808 01:53:18.536770  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 586548us
I20260808 01:53:18.540855  9559 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:18.541868 18472 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.001s
I20260808 01:53:18.542649 18472 consensus_queue.cc:1243] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 22.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260808 01:53:18.545297  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3246) took 1173 ms. Trace:
I20260808 01:53:18.545820  9629 rpcz_store.cc:276] 0808 01:53:17.372131 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:18.075112 (+702981us) service_pool.cc:224] Handling call
0808 01:53:18.075867 (+   755us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:18.164060 (+ 88193us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:18.164113 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:18.181497 (+ 17384us) log.cc:844] Serialized 52 byte log entry
0808 01:53:18.378727 (+197230us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:53:18.378800 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:18.378915 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:18.544978 (+166063us) raft_consensus.cc:1685] finished
0808 01:53:18.545012 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:18.545219 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40444,"raft.queue_time_us":58380,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":5558,"thread_start_us":181695,"threads_started":1,"wal-append.queue_time_us":306693}
I20260808 01:53:18.482334  9367 catalog_manager.cc:5697] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 14 to 18. New cstate: current_term: 18 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:18.472061 17991 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:18.571977 18162 raft_consensus.cc:3055] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 LEADER]: Stepping down as leader of term 27
I20260808 01:53:18.550664  9367 catalog_manager.cc:5697] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 8 to 13, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 13 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:18.572525 18162 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:18.573629 18162 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:18.559222  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet fca874793205463dbb922993f57bf7f5 (from 7201605bb2f54c949a4ebe6e9b3b4b2a to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:18.568934 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:18.575731  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:53:18.571162  9600 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:18.578054  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:53:18.519234 18193 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:18.554708  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:18.581499  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
W20260808 01:53:18.551818 18403 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.580s	user 0.013s	sys 0.001s
W20260808 01:53:18.594201  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 244023us
W20260808 01:53:18.595127  9521 leader_election.cc:336] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.616s (SENT)
I20260808 01:53:18.498334  9480 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 27.
I20260808 01:53:18.563400 17991 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:18.604238  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 49628us
I20260808 01:53:18.605480  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:18.575192 17932 raft_consensus.cc:697] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:18.581117 18394 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:18.611169  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:18.591646 18487 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:18.619168  9635 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:18.620360  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:18.623057  9521 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:18.609401 18454 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:18.614355  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:18.625263  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:18.626011  9630 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:18.593534 18193 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:18.628005  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:18.629714  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:18.630520  9638 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 25
I20260808 01:53:18.609489 18407 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:18.632587  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:18.633034  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:18.633380  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:18.643429  9552 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:18.634117  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:18.644379  9629 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:18.634948  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:18.611572 18162 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:18.645146  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:18.646787  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:18.648473  9631 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:53:18.649842  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 247713us
I20260808 01:53:18.649976  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:18.551061 18092 consensus_queue.cc:260] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.650731  9601 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:18.641153  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:18.651508  9481 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20260808 01:53:18.600927 18403 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.647971  9558 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:18.652850  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:18.658254  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:18.625147 18472 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:18.661513  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:18.665715  9558 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:18.652704  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:18.629153 18454 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:18.668691  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:18.672439  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:18.612502 18477 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.042s
W20260808 01:53:18.675504  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 276862us
W20260808 01:53:18.678732  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 83606us
I20260808 01:53:18.680564  9601 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:18.680979  9521 leader_election.cc:336] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.057s (SENT)
I20260808 01:53:18.686527 18483 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:18.687801 18514 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260808 01:53:18.660570 18505 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:18.692178  9552 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:18.655282 18485 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:18.627864 18487 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.663512 18432 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260808 01:53:18.701225  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:18.704552  9631 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:18.673441 17991 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260808 01:53:18.700271 18277 consensus_meta.cc:287] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.602s	user 0.013s	sys 0.001s
I20260808 01:53:18.677088 18230 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260808 01:53:18.671986 18454 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:18.678472 18342 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260808 01:53:18.703316  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 3b8e0e8fda6744ad877514b11658b9d9: Illegal state: Leader has not yet committed an operation in its own term
I20260808 01:53:18.683492 18092 raft_consensus.cc:3060] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:18.683372 18472 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:18.701082 18406 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:18.723827  9557 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:18.704731 18441 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:18.683245 18394 consensus_queue.cc:1059] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.030s
I20260808 01:53:18.689738 18328 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.699337 17932 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.738042 18255 raft_consensus.cc:3055] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:18.739015 17830 consensus_queue.cc:1059] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:18.739332 18259 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:18.739702 18513 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
W20260808 01:53:18.740082  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:18.744165  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:18.757350  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 76393.5us
W20260808 01:53:18.781243  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:18.715049 18277 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:18.704692 18505 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:18.750721 18255 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:18.782969  9521 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:18.784972  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:18.710851 18193 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.831635 10360 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:18.832814  9477 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 16 index: 15. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260808 01:53:18.782275 18443 raft_consensus.cc:3055] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:18.745633 18230 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
I20260808 01:53:18.756615  9638 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:18.780913 18490 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:18.853263  9475 raft_consensus.cc:2368] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 24. Current term is 25.
I20260808 01:53:18.870277  9557 pending_rounds.cc:85] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 10
W20260808 01:53:18.880012  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:18.880272  9557 pending_rounds.cc:107] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20260808 01:53:18.887226  9557 raft_consensus.cc:2889] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:53:18.892175  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 109326us
I20260808 01:53:18.887763  9557 pending_rounds.cc:107] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 20.12
I20260808 01:53:18.893631  9557 raft_consensus.cc:2889] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:18.894042  9557 pending_rounds.cc:107] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 22.13
I20260808 01:53:18.894363  9557 raft_consensus.cc:2889] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:18.747434 18472 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.854050 18443 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:18.869050 18425 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260808 01:53:18.889487 18421 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.688s	user 0.007s	sys 0.007s
I20260808 01:53:18.909078  9521 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:18.869086 18519 raft_consensus.cc:3055] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Stepping down as leader of term 22
W20260808 01:53:18.916831  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:18.921066  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:18.877076 18526 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:53:18.924525  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 266329us
I20260808 01:53:18.889972 18403 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:18.927579  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:18.928336  9472 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 22: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:53:18.928524  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 224199us
I20260808 01:53:18.877992 18490 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.876288 18407 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:18.937891  9599 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:18.914713 18478 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:18.952261  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:18.953469  9477 raft_consensus.cc:2410] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 26 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260808 01:53:18.948050  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:18.930841 18519 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:18.897415 18434 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.042s
I20260808 01:53:18.962984  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:18.943996 18444 consensus_meta.cc:287] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.639s	user 0.009s	sys 0.006s
I20260808 01:53:18.961361  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:18.969823  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:18.970415  9551 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:18.966157  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:18.974465  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 278406us
I20260808 01:53:18.926127 18421 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.976785  9559 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
W20260808 01:53:18.978209  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:18.948689 18428 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:18.963581 17830 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:18.991948  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:18.967886 18442 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:53:18.994572  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet c99678e59ec845b28293f811d76f24e8: Illegal state: Replica 52ccf4dcca1843f7bb29e9172e2d4ab4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 21 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:18.997741  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:18.974494 18230 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:18.931730 18526 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:19.004926  9558 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:18.978597 18444 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:53:18.990286 18521 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.272s	user 0.001s	sys 0.001s
W20260808 01:53:19.056967 18524 log.cc:927] Time spent T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.209s	user 0.000s	sys 0.002s
I20260808 01:53:19.053294 18505 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:19.059948  9552 raft_consensus.cc:1240] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
W20260808 01:53:19.060861 18525 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.257s	user 0.000s	sys 0.002s
I20260808 01:53:19.062503  9560 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:19.062781 18526 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:18.965811 18478 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:19.005568  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:18.995956 17830 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:53:19.017199 18517 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.273s	user 0.001s	sys 0.000s
I20260808 01:53:19.014859 17991 raft_consensus.cc:3055] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:19.113075  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:19.018455  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 1507) took 1041 ms. Trace:
I20260808 01:53:19.020653 18421 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:19.031011 18407 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:18.998780 18529 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election won for term 21
W20260808 01:53:19.033419  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 124509us
I20260808 01:53:19.133308  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:19.034389 18328 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:19.035200 18277 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:19.142916 18529 raft_consensus.cc:697] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:19.144186 18529 consensus_queue.cc:237] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:19.035878 18519 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:19.122722 17991 raft_consensus.cc:740] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:19.154659 18421 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260808 01:53:19.040406 18490 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:18.971374 18443 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:19.046921 18259 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:19.155369 17991 consensus_queue.cc:260] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:19.009471  9554 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 16 index: 15. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
W20260808 01:53:19.072070  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 66649.5us
I20260808 01:53:19.157003 17991 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:19.154227  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:19.157637  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:19.158752  9521 leader_election.cc:400] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23.
I20260808 01:53:19.159183  9521 leader_election.cc:403] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:53:19.159341  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:19.159569 18528 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.181s	user 0.001s	sys 0.000s
I20260808 01:53:19.076303  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:19.080775  9560 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
W20260808 01:53:19.088886 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 245530us
I20260808 01:53:18.993870 18531 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260808 01:53:19.008960 18230 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:19.112210 10364 raft_consensus.cc:2410] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 29 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260808 01:53:19.122959  9475 rpcz_store.cc:276] 0808 01:53:17.977575 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:17.977904 (+   329us) service_pool.cc:224] Handling call
0808 01:53:19.018174 (+1040270us) spinlock_profiling.cc:243] Waited 785 ms on lock 0x7b5400399754. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:19.018377 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1806399872}
I20260808 01:53:19.160455 18328 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:19.161754 18230 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:19.162182 18328 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:19.162869 18530 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:19.161466 18490 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:19.164223 18490 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23.
I20260808 01:53:19.052278 18487 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:19.111441 18442 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election won for term 23
W20260808 01:53:19.068508  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:19.014808 18527 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260808 01:53:19.160058 18277 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.003s
I20260808 01:53:19.163426 18530 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:19.188182 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:19.189218 10363 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Advancing to term 25
W20260808 01:53:19.068336 18523 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.316s	user 0.002s	sys 0.000s
I20260808 01:53:19.139662  9472 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260808 01:53:19.165982 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:19.162884 18328 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:19.207898 10334 leader_election.cc:400] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 24. Current term is 25.
I20260808 01:53:19.208467 10334 leader_election.cc:403] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:53:19.167486  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:19.165774  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:19.168500  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 193234us
I20260808 01:53:19.111215 17830 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:19.175297  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:19.219147  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 53431us
I20260808 01:53:19.219939  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:19.154366 18454 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:19.167713  9477 raft_consensus.cc:2393] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 25: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:19.211771  9601 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:19.216104 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:19.223838  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:19.185600 18394 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:19.188498 18442 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:19.247984  9551 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:19.191092 18530 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:19.215878 18461 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:19.239022 18516 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election won for term 23
W20260808 01:53:19.271327  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 103910us
I20260808 01:53:19.286047 18478 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:19.287009 18442 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:19.254741 18425 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.034s
W20260808 01:53:19.294328 18535 log.cc:927] Time spent T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260808 01:53:19.252626 18434 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.031s
I20260808 01:53:19.296512 18432 raft_consensus.cc:3055] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:19.297086 18432 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:19.295486 18454 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:19.298622 18454 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:19.299361 18454 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:19.300843 18403 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260808 01:53:19.302222 18454 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:19.239208 18537 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.082s
W20260808 01:53:19.299568  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 57255.8us
W20260808 01:53:19.303973 18536 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260808 01:53:19.239238 18489 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:19.304659  9599 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:19.304979 18462 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:19.266955 18394 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:19.305949 18406 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:19.307224  9600 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:19.299474  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 60486.3us
I20260808 01:53:19.311065  9601 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:19.308274  9521 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:19.315793 10363 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:19.336613  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 4074) took 1114 ms. Trace:
W20260808 01:53:19.311198 18092 consensus_meta.cc:287] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.586s	user 0.009s	sys 0.004s
I20260808 01:53:19.274420 18461 raft_consensus.cc:697] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:53:19.361088  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:19.344164 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 110329us
I20260808 01:53:19.337134  9550 rpcz_store.cc:276] 0808 01:53:18.222080 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:18.222223 (+   143us) service_pool.cc:224] Handling call
0808 01:53:18.222539 (+   316us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:18.235713 (+ 13174us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:18.235768 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:18.289430 (+ 53662us) log.cc:844] Serialized 26 byte log entry
0808 01:53:18.317417 (+ 27987us) spinlock_profiling.cc:243] Waited 24.1 ms on lock 0x7b48000c03d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:53:18.593837 (+276420us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:53:18.593890 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:18.593980 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:19.319456 (+725476us) raft_consensus.cc:1685] finished
0808 01:53:19.319485 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:19.336510 (+ 17025us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5084,"raft.queue_time_us":14690,"raft.run_cpu_time_us":3093,"raft.run_wall_time_us":68836,"spinlock_wait_cycles":55341312,"thread_start_us":223531,"threads_started":1,"wal-append.queue_time_us":330766}
W20260808 01:53:19.394218  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 87157.7us
W20260808 01:53:19.395147  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 175225us
I20260808 01:53:19.312750 18516 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:19.320210 18366 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:19.298230 18432 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:19.405398 18516 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:19.406284 18546 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260808 01:53:19.341885 18489 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:53:19.416045  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 109664us
I20260808 01:53:19.409162 17830 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:19.411155 18461 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:19.392544 18162 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 24. Current term is 25.
W20260808 01:53:19.412441  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 104285us
I20260808 01:53:19.420061  9599 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:53:19.421584  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 201727us
I20260808 01:53:19.425384  9445 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:19.431913  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:19.432577  9632 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:19.434432 17830 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:19.440482  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 129552us
I20260808 01:53:19.441844  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:19.445425  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:19.455194  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:19.455274  9601 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:19.455931  9551 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:19.458633  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:19.436190 18396 raft_consensus.cc:2764] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260808 01:53:19.476483  9521 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:19.447073 18432 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:19.478190  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:19.479751  9632 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 29.
I20260808 01:53:19.443217 18478 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:19.443423 18092 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:19.494194 18538 raft_consensus.cc:2804] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:19.460346 18444 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260808 01:53:19.495059 18540 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
W20260808 01:53:19.496531 18544 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260808 01:53:19.496855 18394 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:19.504607  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:19.500311  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 72810.5us
W20260808 01:53:19.525336  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:19.512467  9551 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:19.510373  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:19.534374  9599 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:19.494772 18543 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:19.489816 18193 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:19.536914 18366 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:53:19.557032  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 98473.8us
I20260808 01:53:19.566960  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:19.538646 18444 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:19.571389 18366 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:19.565253 18543 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:19.574958  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 135063us
I20260808 01:53:19.575654 18092 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:19.576637 18092 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 23.20, Last appended by leader: 20, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:19.498169 18539 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260808 01:53:19.571691 18558 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260808 01:53:19.575110 18394 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260808 01:53:19.499104 18478 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:19.601357  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:19.597687 18552 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260808 01:53:19.584280  9521 leader_election.cc:336] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.685s (SENT)
I20260808 01:53:19.575445 18555 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:19.607556  9521 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:19.614517 10332 leader_election.cc:336] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.524s (SENT)
I20260808 01:53:19.614665  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:19.631273  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:19.588495 18543 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:19.607165 18540 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:19.627791 18277 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260808 01:53:19.680287 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 233311us
W20260808 01:53:19.683985 10332 leader_election.cc:336] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.572s (SENT)
W20260808 01:53:19.685895  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 175655us
I20260808 01:53:19.655467 18193 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:53:19.658752 18552 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:19.613175 18328 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:19.712221  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:19.713032  9631 raft_consensus.cc:2376] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 28. Re-sending same reply.
I20260808 01:53:19.668196 18539 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:19.670563 18540 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:19.786943 18406 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election won for term 22
W20260808 01:53:19.803272  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 167736us
W20260808 01:53:19.796864  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 212620us
I20260808 01:53:19.805538 18489 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:19.805899  9635 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:19.798051  9637 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260808 01:53:19.806468 18483 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:19.806118 18489 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:19.807101 18483 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:19.809281 18489 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:19.809850 18483 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:19.816150  9521 leader_election.cc:336] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.022s (SENT)
W20260808 01:53:19.817250 10332 leader_election.cc:336] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.726s (SENT)
I20260808 01:53:19.817902  9521 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:19.711426 18552 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:19.786650 17830 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:19.835582  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:19.802778 17991 consensus_meta.cc:287] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.645s	user 0.006s	sys 0.008s
I20260808 01:53:19.839206 18555 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:19.845850  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:19.846408 18563 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.255s	user 0.000s	sys 0.001s
I20260808 01:53:19.845777 17830 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:19.846899  9633 raft_consensus.cc:2376] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 24. Re-sending same reply.
I20260808 01:53:19.847085  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:19.837700 18406 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:19.847779  9637 raft_consensus.cc:2376] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 24. Re-sending same reply.
I20260808 01:53:19.847961  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:19.848672  9633 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:53:19.848819  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:19.849140 17830 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:19.849534  9637 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:19.848436 18406 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:19.832686 18483 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260808 01:53:19.886930 18564 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.277s	user 0.001s	sys 0.000s
I20260808 01:53:19.831488 18539 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:19.892576  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:19.892776  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:19.891827  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:19.892809  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 76850.2us
I20260808 01:53:19.894279  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:19.831292 18552 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:19.902761 10332 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:19.684829 18530 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:19.912791  9636 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 24: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:19.875180 18477 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:19.922966  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 383475us
I20260808 01:53:19.843695 18555 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:19.918800 18230 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.756s	user 0.012s	sys 0.002s
I20260808 01:53:19.953202  9634 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260808 01:53:19.954846  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:19.963223  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:19.964177  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:19.964861  9631 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
W20260808 01:53:19.965045  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:19.965811  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:19.966753  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:19.967551  9629 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:19.899181 18537 consensus_queue.cc:1048] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.043s
I20260808 01:53:19.980027  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:19.969902  9633 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:19.981268  9638 raft_consensus.cc:2368] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 25. Current term is 26.
I20260808 01:53:19.934732 18553 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:19.982622  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:19.913439 18257 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.043s
I20260808 01:53:19.996383  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:19.988654  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:53:19.997442  9633 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 28: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:19.895560 18540 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:19.907804 18539 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:20.000939  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:19.995025  9638 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:19.990597 18548 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.397s	user 0.001s	sys 0.001s
I20260808 01:53:20.008867  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:19.981016 18557 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:19.932287 18477 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:19.997463 18478 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:20.019068  9629 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:20.021590  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:20.019469  9521 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.601s (SENT) [suppressed 29 similar messages]
I20260808 01:53:19.987099 18584 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:19.998277 18572 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.225s	user 0.001s	sys 0.000s
I20260808 01:53:20.055809 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:20.012457 18454 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.011s	sys 0.022s
W20260808 01:53:20.058374 18551 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.102s	user 0.001s	sys 0.001s
I20260808 01:53:19.957340 18230 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:19.998988 18579 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:20.064361  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:20.064304 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:20.073844 10332 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:20.061672  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:20.064471 18454 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.011s	sys 0.022s
W20260808 01:53:20.069744 18550 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.579s	user 0.001s	sys 0.003s
I20260808 01:53:20.084375  9635 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260808 01:53:20.086078  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:20.058926 18569 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:20.098853  9637 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:53:20.100867  9521 leader_election.cc:400] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 25. Current term is 26.
I20260808 01:53:20.070668 18478 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:20.062453 18556 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:20.109723  9521 leader_election.cc:403] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260808 01:53:20.083623 18553 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:20.059149 18557 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:20.123172 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:20.122697 18441 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:20.133461  9638 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:53:20.134595  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:20.135416  9638 raft_consensus.cc:2368] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23.
I20260808 01:53:20.136415  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:20.136993  9638 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 26
I20260808 01:53:20.106036 18193 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:20.111851 18478 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:20.111047 18537 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:20.076419 18579 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:20.141110  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:20.121508 18257 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:20.072520 18584 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:20.156960  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:20.161432  9477 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
W20260808 01:53:20.163427 18570 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20260808 01:53:20.155174 18441 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:20.159308 18554 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:20.173339 18441 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:20.173224 18554 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:20.125481 18566 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:20.175218 18566 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:20.176237 18477 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:20.176275  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3100) took 1341 ms. Trace:
I20260808 01:53:20.129206 18574 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:20.177855 18566 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:20.176935  9557 rpcz_store.cc:276] 0808 01:53:18.834609 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:18.839060 (+  4451us) service_pool.cc:224] Handling call
0808 01:53:18.870002 (+ 30942us) raft_consensus.cc:1471] Updating replica for 4 ops
0808 01:53:18.895469 (+ 25467us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:18.895619 (+   150us) raft_consensus.cc:1539] Triggering prepare for 4 ops
0808 01:53:19.064712 (+169093us) spinlock_profiling.cc:243] Waited 2.08 ms on lock 0x7b2c000f57d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:53:19.065528 (+   816us) log.cc:844] Serialized 104 byte log entry
0808 01:53:19.322803 (+257275us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:53:19.322857 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:19.322954 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:20.175886 (+852932us) raft_consensus.cc:1685] finished
0808 01:53:20.175912 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:20.176201 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":778,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":10845,"spinlock_wait_cycles":4779904,"thread_start_us":256776,"threads_started":1,"wal-append.queue_time_us":360719}
W20260808 01:53:20.175227 18586 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.250s	user 0.000s	sys 0.001s
I20260808 01:53:20.178432 18574 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:20.179705 18230 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:20.180884 18574 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:20.181617 18575 log.cc:927] Time spent T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.181s	user 0.000s	sys 0.001s
I20260808 01:53:20.184886  9480 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260808 01:53:20.128898 18403 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:20.198789 18403 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:20.200650 18472 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:20.201208 18472 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:20.116277 18569 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:20.204547  9633 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:53:20.213740  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:20.214586  9550 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:20.185340 18328 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:20.186525 10334 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:20.171424 18555 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:20.188946 18584 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:20.142877 18478 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:20.128496 18489 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 25. Current term is 26.
I20260808 01:53:20.125599 18553 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:20.164847 18560 log.cc:927] Time spent T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.151s	user 0.000s	sys 0.001s
I20260808 01:53:20.257014  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:20.165838 18595 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
W20260808 01:53:20.243568 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 57109.6us
I20260808 01:53:20.245083  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:20.248193  9560 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:20.261832  9632 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:53:20.262239  9554 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:20.263168  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:20.264235  9632 raft_consensus.cc:2410] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 26 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260808 01:53:20.143927 18598 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:20.257956  9556 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
W20260808 01:53:20.258497  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 162851us
I20260808 01:53:20.265460  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:20.266865  9560 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260808 01:53:20.288808  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:20.293895  9559 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260808 01:53:20.294207  9552 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:20.244747 18193 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:20.289157  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:20.297071  9555 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:20.282496  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:20.298903  9554 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 23
I20260808 01:53:20.300251  9638 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:20.301774  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:20.303349  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:20.305040  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:20.305212  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:20.305794  9632 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 28.
I20260808 01:53:20.306023  9638 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:20.307745 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:20.282164  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:20.312342  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:20.312772  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:20.313140  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:20.314327  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:20.315039  9632 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:20.325847  9632 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:20.327781  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:20.326308  9521 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:20.202229 18472 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:20.328661  9559 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:20.329280  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:20.280386 18576 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:20.334614  9521 leader_election.cc:400] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23.
I20260808 01:53:20.336172  9521 leader_election.cc:403] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:53:20.295801 18259 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:20.332181  9555 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:20.337190  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:20.340301  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:20.339746  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:20.344682  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:20.285060 18598 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:20.280514 18530 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:20.308846 18489 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:20.285288 18574 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.062s
I20260808 01:53:20.349339  9521 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:20.349642  9560 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260808 01:53:20.350416  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:20.348219  9552 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 29.
I20260808 01:53:20.356829  9638 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:20.332996 18576 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:20.379031  9556 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:20.359875  9555 raft_consensus.cc:2393] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 22: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:20.371620  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:20.516162 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:20.516451  9560 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:53:20.377192  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:20.351261 18538 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.039s	sys 0.029s
I20260808 01:53:20.363509 18539 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:20.351262 18537 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:20.381429  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 3742) took 1325 ms. Trace:
I20260808 01:53:20.392349 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:20.393795 18579 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:20.394596 18569 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:20.342834 18403 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:20.397022 18432 consensus_meta.cc:287] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.905s	user 0.009s	sys 0.013s
I20260808 01:53:20.525359 18277 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:20.354651 18259 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:20.408845  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 103b913d6c014692b6dea1780af2aa20: Illegal state: Leader has not yet committed an operation in its own term
I20260808 01:53:20.361519 18472 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:20.487123  9554 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:20.510627 18574 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:20.363737 18598 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:20.351351 18531 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election won for term 29
I20260808 01:53:20.355899 18555 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:20.337221 17932 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:20.313138 18193 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:20.494828  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:20.505947  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:20.508056  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:20.509574  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:20.510452  9550 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:20.359938  9521 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:20.511773  9477 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:20.566708  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:20.567368  9477 raft_consensus.cc:606] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:20.567902  9477 raft_consensus.cc:614] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:20.513022 18530 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:20.570422  9473 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:20.517019  9476 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:20.515249  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:20.573051  9478 raft_consensus.cc:3060] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:20.518213  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:20.519238 18576 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:20.521997  9472 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:20.541093 18483 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23.
I20260808 01:53:20.493459 18518 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:20.628084  9477 raft_consensus.cc:1275] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20260808 01:53:20.524470 18604 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:20.524736  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:20.628793  9477 pending_rounds.cc:85] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 11
I20260808 01:53:20.629186  9477 pending_rounds.cc:107] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 20.12
I20260808 01:53:20.629433  9477 raft_consensus.cc:2889] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:20.522714  9481 rpcz_store.cc:276] 0808 01:53:19.055818 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:19.056138 (+   320us) service_pool.cc:224] Handling call
0808 01:53:20.381206 (+1325068us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x7b54002dc3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:20.381338 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":741349248}
W20260808 01:53:20.511538 18552 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.588s	user 0.018s	sys 0.007s
I20260808 01:53:20.351213 18328 consensus_queue.cc:1059] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260808 01:53:20.512814 18257 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260808 01:53:20.519658 18538 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.842s	user 0.039s	sys 0.034s
I20260808 01:53:20.543102  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:20.527304 18537 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:20.519629 18608 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:20.568553  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:20.580842  9635 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260808 01:53:20.572232  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:20.587899 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:20.522346  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:20.621861  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:20.617719  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:20.631831  9633 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 23: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:53:20.639930  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 0b4331dfdd5b43448a400f2eb7b1be70 (from 6e7856bed1be43acaa821290d506ee97 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:20.644131  9477 raft_consensus.cc:1275] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 12. (term mismatch)
I20260808 01:53:20.647571  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3260) took 1205 ms. Trace:
I20260808 01:53:20.573395 18531 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 29 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:20.615537 18554 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:20.634629 18548 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260808 01:53:20.588696 18555 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:20.554155 18574 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:20.587052 18530 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:20.762558  9633 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260808 01:53:20.738423  9481 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:20.800000 18193 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:20.725773 18407 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:20.738713 17932 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:20.803375 18555 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:20.739423 18539 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:20.807281 18539 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:20.811990  9638 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 22: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:20.725026  9477 pending_rounds.cc:85] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 11
W20260808 01:53:20.738627  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 355927us
I20260808 01:53:20.813094  9477 pending_rounds.cc:107] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 22.12
I20260808 01:53:20.797724 18608 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:20.821416 18608 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:20.821838  9638 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:20.813444  9477 raft_consensus.cc:2889] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:20.815088  9600 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:20.815475 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:20.724677  9630 rpcz_store.cc:276] 0808 01:53:19.427578 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:19.427814 (+   236us) service_pool.cc:224] Handling call
0808 01:53:19.428219 (+   405us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:19.428451 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 14
0808 01:53:19.447830 (+ 19379us) log.cc:844] Serialized 14 byte log entry
0808 01:53:19.699882 (+252052us) log.cc:844] Serialized 14 byte log entry
0808 01:53:19.796559 (+ 96677us) log.cc:844] Serialized 14 byte log entry
0808 01:53:19.796985 (+   426us) log.cc:844] Serialized 14 byte log entry
0808 01:53:20.197155 (+400170us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b180036e9e0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:53:20.391920 (+194765us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b180036e9e0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667 00007f5d449ae5e8
0808 01:53:20.632367 (+240447us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:20.632424 (+    57us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:53:20.632481 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:20.632553 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:20.632690 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":68387,"prepare.queue_time_us":605771,"prepare.run_cpu_time_us":156,"prepare.run_wall_time_us":8684,"raft.queue_time_us":382501,"raft.run_cpu_time_us":466,"raft.run_wall_time_us":90200,"spinlock_wait_cycles":27364864,"thread_start_us":415760,"threads_started":3,"wal-append.queue_time_us":372934}
I20260808 01:53:20.725545 18441 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:20.796217  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 436333us
I20260808 01:53:20.726760 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:20.738829  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 100330us
I20260808 01:53:20.834028  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:20.651824  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 108774us
I20260808 01:53:20.655807  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:20.837786  9472 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:20.839154  9479 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:20.782003  9474 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260808 01:53:20.843600  9521 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:20.844743  9478 raft_consensus.cc:2468] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:20.848064  9473 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 20: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:20.852800  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:20.729097 18605 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:20.728633 18328 consensus_queue.cc:1243] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 24.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260808 01:53:20.724823 18557 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:20.865254  9474 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3759209338e14437aff5f92a2fff36da"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:20.762315 18553 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:20.877277  9474 raft_consensus.cc:606] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:20.880707  9474 raft_consensus.cc:614] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:20.881127  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:20.728809 18566 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.135s
I20260808 01:53:20.650597 18538 raft_consensus.cc:697] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:20.833633 18489 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:20.762315 18567 consensus_queue.cc:1059] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.104s
I20260808 01:53:20.847407 18537 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:20.639402 18552 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:20.891312 18259 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:20.879887 18605 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:20.846516 18441 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:20.846334 18407 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.041s
I20260808 01:53:20.889992 18557 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260808 01:53:20.894313 18612 log.cc:927] Time spent T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Append to log took a long time: real 0.303s	user 0.001s	sys 0.004s
I20260808 01:53:20.895103 18617 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:20.896715 18342 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:20.897572 18328 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260808 01:53:20.898116 18483 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:20.899350  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:20.900938 18530 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:20.907270 18531 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:20.941254 18530 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:20.941953 18530 raft_consensus.cc:697] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:20.942171 18441 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:20.943002 18530 consensus_queue.cc:237] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:20.649554 18604 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:20.946915 18604 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:20.874363 18366 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:20.640473 18518 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:20.922389  9558 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 28.
I20260808 01:53:20.866219 18506 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260808 01:53:20.951683 18462 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:20.883134 18092 consensus_queue.cc:1059] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.046s
I20260808 01:53:20.952422 18462 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:20.888854 18403 consensus_queue.cc:1048] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.041s
I20260808 01:53:20.939909  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:20.943645 18619 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:20.954602 18259 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:20.955618 18619 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:20.957333 10363 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:20.944283 18483 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:20.968560 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:20.955842 18259 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:20.908838 18255 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:20.975891 18529 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:20.968333 10362 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 23 index: 20. Preceding OpId from leader: term: 25 index: 21. (index mismatch)
I20260808 01:53:20.935959 18342 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:20.976191 18255 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:20.934852 18539 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:20.988312  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:20.993176  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:20.967980 18518 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:21.023082  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:20.991824  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:21.042960  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:20.992554  9559 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260808 01:53:20.968779 18617 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:20.987764 18342 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:20.988454 18432 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:21.053798  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 208691us
I20260808 01:53:21.055310  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:21.063184  9560 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 22 index: 19. Preceding OpId from leader: term: 25 index: 21. (index mismatch)
I20260808 01:53:21.063472  9555 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:53:21.057778  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:21.059114  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 181721us
I20260808 01:53:21.061596  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:21.065610  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:21.066269  9556 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 26
I20260808 01:53:21.066855  9559 raft_consensus.cc:2410] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 27 because replica has last-logged OpId of term: 25 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260808 01:53:21.039171 18552 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:21.040145 18574 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:21.074512  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:21.042209 18076 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.024s
I20260808 01:53:21.082727  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:21.088034 10332 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:20.977190 18538 consensus_queue.cc:237] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.053596 18557 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:21.096961  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:21.097999  9559 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:21.101313  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:21.109369  9556 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
W20260808 01:53:21.110687  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:21.111021  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:21.112413  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:21.115465 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:21.080957 18518 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:21.077446 18605 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:21.113376  9473 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:21.051720 18529 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:21.080268 18555 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:21.140666  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 63975.5us
I20260808 01:53:21.095715 18574 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:21.192235  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:21.197329  9551 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:53:21.193791  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:21.119899 18566 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:21.077195 18432 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:21.068629 18617 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.104719 18553 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.104s
I20260808 01:53:21.105623 18557 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.107475 18613 consensus_queue.cc:1048] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.114s
I20260808 01:53:21.135871 18565 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.117s
I20260808 01:53:21.222704 18566 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:21.252496  9550 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:53:21.255000  9552 raft_consensus.cc:2410] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260808 01:53:21.105114 18483 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:21.263646  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 191021us
I20260808 01:53:21.228771 18605 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:21.262385 18566 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:21.267414 18619 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:21.218606 18555 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.277592 18632 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:21.307425  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 163484us
I20260808 01:53:21.311579 18632 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.311892  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:21.314096 18632 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:21.314241  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:21.320470 18632 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:21.321076 18632 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:21.321463 18632 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:21.276851 18604 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.031s
I20260808 01:53:21.278707 18506 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.031s
I20260808 01:53:21.304638 18259 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:21.353029  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 1105) took 1236 ms. Trace:
W20260808 01:53:21.344535 18621 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.286s	user 0.001s	sys 0.000s
I20260808 01:53:21.356047  9630 rpcz_store.cc:276] 0808 01:53:20.116872 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:20.825792 (+708920us) service_pool.cc:224] Handling call
0808 01:53:20.826273 (+   481us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:20.851925 (+ 25652us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:20.851974 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:20.852711 (+   737us) log.cc:844] Serialized 52 byte log entry
0808 01:53:21.115446 (+262735us) raft_consensus.cc:1645] Marking committed up to 19
0808 01:53:21.267363 (+151917us) log.cc:844] Serialized 14 byte log entry
0808 01:53:21.281521 (+ 14158us) log.cc:844] Serialized 14 byte log entry
0808 01:53:21.352540 (+ 71019us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:21.352659 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:21.352717 (+    58us) raft_consensus.cc:1685] finished
0808 01:53:21.352732 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:21.352941 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46497,"prepare.queue_time_us":188642,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":102832,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":48041,"spinlock_wait_cycles":8765056,"thread_start_us":435930,"threads_started":3,"wal-append.queue_time_us":245259}
I20260808 01:53:21.303025 18605 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.351612 18633 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:21.379981  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:21.383402  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:21.389428  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:21.390583  9630 raft_consensus.cc:2376] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23. Re-sending same reply.
I20260808 01:53:21.393227  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:21.394129  9630 raft_consensus.cc:2376] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23. Re-sending same reply.
I20260808 01:53:21.395632  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:21.396939  9630 raft_consensus.cc:2410] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 26 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260808 01:53:21.310400 18529 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.399324  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:21.400089  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:21.400823  9637 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:21.400965  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:21.402413  9637 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260808 01:53:21.398213  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:21.405401  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3267) took 1210 ms. Trace:
I20260808 01:53:21.405727  9637 rpcz_store.cc:276] 0808 01:53:20.194779 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:21.401690 (+1206911us) service_pool.cc:224] Handling call
0808 01:53:21.402051 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:21.405108 (+  3057us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:21.405334 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:53:21.406411  9630 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 17.
I20260808 01:53:21.385195 18633 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:21.388092 18615 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:21.415374 18622 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.203s	user 0.001s	sys 0.001s
I20260808 01:53:21.427935  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3274) took 1097 ms. Trace:
I20260808 01:53:21.433552  9630 rpcz_store.cc:276] 0808 01:53:20.330500 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:21.413394 (+1082894us) service_pool.cc:224] Handling call
0808 01:53:21.426956 (+ 13562us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:21.427458 (+   502us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:53:21.427513 (+    55us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:21.427578 (+    65us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:53:21.427623 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:21.427704 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:21.427851 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":135}
I20260808 01:53:21.436790  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3271) took 1125 ms. Trace:
I20260808 01:53:21.438238  9637 rpcz_store.cc:276] 0808 01:53:20.311244 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:21.416134 (+1104890us) service_pool.cc:224] Handling call
0808 01:53:21.436024 (+ 19890us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:21.436301 (+   277us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:21.436339 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:21.436402 (+    63us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:21.436449 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:21.436555 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:21.436707 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:53:21.443248  9473 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260808 01:53:21.413458 18619 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:21.473865  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:21.451009 18432 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:21.475054 18619 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:21.477206 18605 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:21.477903 18619 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:21.440208 18615 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:21.437785 18625 log.cc:927] Time spent T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.138s	user 0.000s	sys 0.001s
I20260808 01:53:21.425804 18633 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:53:21.470794 18576 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.841s	user 0.014s	sys 0.001s
I20260808 01:53:21.441699 18604 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:21.493978 18636 log.cc:927] Time spent T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260808 01:53:21.493116  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:21.496964 18615 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:21.498397 18642 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260808 01:53:21.482545 18462 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:21.507022 18617 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:21.512491  9445 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:21.513446 18506 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:21.515338 18506 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260808 01:53:21.522575  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:21.523976  9555 raft_consensus.cc:2410] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 24 because replica has last-logged OpId of term: 23 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260808 01:53:21.525257  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:21.525952  9555 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:21.534457  9637 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260808 01:53:21.547557  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:21.549050  9555 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:53:21.532533 18610 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
W20260808 01:53:21.537314 18255 consensus_meta.cc:287] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.520s	user 0.009s	sys 0.000s
W20260808 01:53:21.595986 18621 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260808 01:53:21.540856 18604 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.600737 18604 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:21.602902  9634 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:53:21.603765 18448 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:21.604463 18448 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.564083 18532 raft_consensus.cc:3055] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:21.564371  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 1104) took 1451 ms. Trace:
I20260808 01:53:21.606150 18532 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:21.567157  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:21.606562  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:21.606986  9637 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:21.607280 18532 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.608003  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:21.609508  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:21.606209  9638 rpcz_store.cc:276] 0808 01:53:20.113118 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:20.822899 (+709781us) service_pool.cc:224] Handling call
0808 01:53:20.823274 (+   375us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:20.823835 (+   561us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:20.823874 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:20.832461 (+  8587us) log.cc:844] Serialized 52 byte log entry
0808 01:53:21.099844 (+267383us) raft_consensus.cc:1645] Marking committed up to 19
0808 01:53:21.246738 (+146894us) log.cc:844] Serialized 14 byte log entry
0808 01:53:21.250643 (+  3905us) log.cc:844] Serialized 14 byte log entry
0808 01:53:21.252346 (+  1703us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:21.252553 (+   207us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:21.564048 (+311495us) raft_consensus.cc:1685] finished
0808 01:53:21.564075 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:21.564294 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32100,"prepare.queue_time_us":295927,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":108,"raft.queue_time_us":78462,"raft.run_cpu_time_us":432,"raft.run_wall_time_us":22856,"spinlock_wait_cycles":4803968,"thread_start_us":365804,"threads_started":2,"wal-append.queue_time_us":392251}
I20260808 01:53:21.610736  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:21.611130  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:21.611462  9638 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:21.612336  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:21.613091  9638 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260808 01:53:21.614002  9637 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260808 01:53:21.569690 18555 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:21.573269  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:21.540380 18576 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:21.579747 18608 consensus_meta.cc:287] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.758s	user 0.013s	sys 0.001s
I20260808 01:53:21.615793 10364 raft_consensus.cc:2368] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21.
I20260808 01:53:21.615551 18608 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.617470 18576 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:21.617802 18608 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:21.618927 18472 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.075s	user 0.015s	sys 0.000s
I20260808 01:53:21.619799 18561 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:21.620417 18561 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:21.621223  9638 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: []
W20260808 01:53:21.623564 18537 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.700s	user 0.014s	sys 0.000s
W20260808 01:53:21.624006 18641 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
I20260808 01:53:21.624141 18537 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:21.618871  9560 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260808 01:53:21.613987 10360 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 1328) took 1100 ms. Trace:
I20260808 01:53:21.593338 18519 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:53:21.592414 18342 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.523s	user 0.014s	sys 0.000s
I20260808 01:53:21.625514 10360 rpcz_store.cc:276] 0808 01:53:20.513307 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:20.513483 (+   176us) service_pool.cc:224] Handling call
0808 01:53:20.513803 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:20.859705 (+345902us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:20.859757 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:20.859819 (+    62us) raft_consensus.cc:1645] Marking committed up to 19
0808 01:53:20.859865 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:21.613764 (+753899us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b5400694954. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:21.613807 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:21.613919 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":794777344}
I20260808 01:53:21.624015  9638 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260808 01:53:21.630832  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:21.631829  9550 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:21.636065  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:21.637440  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:21.638125  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:21.635425  9638 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260808 01:53:21.646292  9637 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:21.648002  9550 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:21.650028 18532 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:53:21.641316 18478 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.361s	user 0.014s	sys 0.004s
I20260808 01:53:21.651409 18478 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:21.653429 18604 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.007s
I20260808 01:53:21.656049 18632 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.657833 18403 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:21.658416 18403 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.659096 18581 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:21.659680 18581 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.660804 18403 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:21.661290 18650 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:21.653718 18478 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:21.667347  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:21.665865  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:21.666124  9479 raft_consensus.cc:3055] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:21.669473  9479 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:21.670585  9479 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.672551  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:21.673111  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:21.673295  9480 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:53:21.661846 18581 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:21.675045  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:21.675527 18637 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:21.666256  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:21.675768  9477 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:21.679632  9550 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260808 01:53:21.674412  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:21.683531 10334 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:21.684173 18472 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:21.684818 18472 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:21.685622 18259 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:21.686090 18259 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:21.686461 18259 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:21.633513 18342 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.690884  9475 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26. Ops: []
W20260808 01:53:21.691342  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:21.692011  9550 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
W20260808 01:53:21.701983  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:21.708058  9600 leader_election.cc:400] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21.
I20260808 01:53:21.710968  9600 leader_election.cc:403] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:53:21.904695 18638 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 21
I20260808 01:53:21.721341  9477 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:53:21.697285 18604 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:21.907281 18604 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:21.908754  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 1301) took 1532 ms. Trace:
I20260808 01:53:21.699502 18328 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:21.909734 18328 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.909212  9558 rpcz_store.cc:276] 0808 01:53:20.376356 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:20.376948 (+   592us) service_pool.cc:224] Handling call
0808 01:53:21.908476 (+1531528us) spinlock_profiling.cc:243] Waited 405 ms on lock 0x7b540002cbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:21.908646 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":930704640}
I20260808 01:53:21.916435  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:21.917161  9556 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:21.917215  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:21.918074  9558 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:21.921692 18557 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:21.758811 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:21.661975 18650 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.700408 18530 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:21.704172 18605 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:21.705678 18342 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:21.773325 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:21.699703 18637 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:21.769753 18632 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:21.717170 18092 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.063s
I20260808 01:53:21.781551 18472 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:21.785804 18631 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:21.928081 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:21.929364 18162 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:21.930024 18631 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:21.930030 18162 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:21.930959 18162 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:21.931353 18530 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:21.806914 18640 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:21.933105 18640 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.803169 18537 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:21.931066 18605 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.930665 18631 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:21.940310 18611 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:21.940963 18611 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.743501 18651 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.078s
I20260808 01:53:21.871814  9480 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260808 01:53:21.874121 18597 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:21.943029 18597 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:21.852397 18627 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:21.947561 18627 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:21.893478 18574 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:21.960606 18574 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:21.962749 10361 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:21.874668 18643 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.135s
I20260808 01:53:21.902884  9555 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
W20260808 01:53:21.931869  9634 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3495) took 1981 ms (client timeout 1584 ms). Trace:
I20260808 01:53:21.948838  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:21.952649  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:21.978487  9638 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:21.980718 18277 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Stepping down as leader of term 23
W20260808 01:53:21.951475  9521 leader_election.cc:336] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.584s (SENT)
I20260808 01:53:21.964262 18533 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:21.981364 18277 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:21.981853 18533 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.976588 18643 consensus_queue.cc:1243] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 22.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260808 01:53:21.986814 18597 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.988759 18597 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:21.990134  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:21.968103 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:21.995736 18076 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260808 01:53:21.851393 18598 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.998s	user 0.049s	sys 0.023s
I20260808 01:53:21.996279 18076 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:22.008507 18076 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:22.008322 18576 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260808 01:53:22.009111 18643 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:22.009754 18643 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:22.001312 18277 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:21.986447 10334 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:22.017553  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 3759209338e14437aff5f92a2fff36da (from 6e7856bed1be43acaa821290d506ee97 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:22.020008  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 3273) took 1698 ms. Trace:
I20260808 01:53:22.020573  9632 rpcz_store.cc:276] 0808 01:53:20.321428 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:20.339456 (+ 18028us) service_pool.cc:224] Handling call
0808 01:53:22.019698 (+1680242us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540027ef54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:22.019919 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2988316032}
I20260808 01:53:22.026935  9479 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
W20260808 01:53:22.029791  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 83762.4us
W20260808 01:53:22.016611 18598 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.169s	user 0.049s	sys 0.023s
I20260808 01:53:22.031368 18537 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:53:22.039942  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:22.058106  9632 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Advancing to term 29
W20260808 01:53:22.041631  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 90193.5us
W20260808 01:53:22.064869  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 100169us
I20260808 01:53:22.079334  9523 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:22.081065 18448 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:22.147913  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 439956us
W20260808 01:53:21.976985  9634 rpcz_store.cc:269] 0808 01:53:19.950667 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:19.954474 (+  3807us) service_pool.cc:224] Handling call
0808 01:53:21.931605 (+1977131us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b540003c5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:21.931735 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3788358016}
I20260808 01:53:22.065724 18617 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:22.072422  9524 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:22.066063  9599 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:22.082723 17932 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.023s
I20260808 01:53:22.190840 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:22.214381  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 153604us
I20260808 01:53:22.088351 18540 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
W20260808 01:53:22.227977  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 161990us
I20260808 01:53:22.229485 18643 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:22.229954 18627 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:22.230400  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 82043e5428784a1eb53a98811ecbbbb4: Illegal state: Leader has not yet committed an operation in its own term
I20260808 01:53:22.230713 18255 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:22.231216 18255 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:22.231549  9632 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 29.
I20260808 01:53:22.231658 18255 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:22.232333 18561 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:22.235498  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:22.236286 18650 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:22.239046 18627 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:22.239899 18627 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:22.237524  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:22.249362  9558 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 24.
I20260808 01:53:22.159574 18443 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
W20260808 01:53:22.167954 18531 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.190s	user 0.019s	sys 0.019s
W20260808 01:53:22.221496  9521 leader_election.cc:336] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.648s (SENT)
I20260808 01:53:22.253885  9638 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 24.
I20260808 01:53:22.206377 18277 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:22.256219  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:22.256505  9556 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 29.
W20260808 01:53:22.266162  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 318115us
W20260808 01:53:22.268471  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.004s	sys 0.016s
W20260808 01:53:22.270707  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.004s	sys 0.016s
I20260808 01:53:22.205780 18599 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:53:22.278028  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:22.273175  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 505268us
W20260808 01:53:22.261085 18531 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.289s	user 0.020s	sys 0.020s
W20260808 01:53:22.292193 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 227368us
I20260808 01:53:22.300652 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:22.301357  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 222134us
I20260808 01:53:22.256140 18611 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:22.305255  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:22.266731 18628 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260808 01:53:22.306321  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:22.303409  9557 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.021s	sys 0.026s
I20260808 01:53:22.443548 18640 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:22.492281  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 228569us
I20260808 01:53:22.320181 18637 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:22.259821 18255 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:22.297008 18366 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260808 01:53:22.299388 18599 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:22.487304 18162 consensus_meta.cc:287] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.556s	user 0.010s	sys 0.012s
I20260808 01:53:22.382169 18533 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:22.315250 18627 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:22.384160 18604 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:22.442847 18532 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.792s	user 0.014s	sys 0.014s
I20260808 01:53:22.455683 17932 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:22.456753 18605 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:22.457432 18677 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:22.457660 18678 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:22.459767 18530 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:22.534214 18662 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.636s	user 0.003s	sys 0.009s
I20260808 01:53:22.446858 18569 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:22.461299 18574 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:22.462922 18658 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.574s	user 0.005s	sys 0.000s
I20260808 01:53:22.535046 18557 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:22.535763 18557 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:22.463096  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3134) took 1010 ms. Trace:
W20260808 01:53:22.466737 18661 log.cc:927] Time spent T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.559s	user 0.011s	sys 0.000s
W20260808 01:53:22.471695 18259 consensus_meta.cc:287] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.785s	user 0.007s	sys 0.013s
W20260808 01:53:22.472106 18519 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.839s	user 0.022s	sys 0.006s
W20260808 01:53:22.476171 18566 consensus_meta.cc:287] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.173s	user 0.024s	sys 0.013s
W20260808 01:53:22.478168  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:22.479775  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:22.513223 18531 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:22.564489 18255 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:22.564774 18531 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:22.444521  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 372121us
I20260808 01:53:22.565569  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:22.566836 18679 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 01:53:22.567200  9636 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:22.568408  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:22.568826 18574 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:22.568509 18566 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:22.569610  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:22.570555  9476 raft_consensus.cc:2410] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 25 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260808 01:53:22.486140 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:22.488359  9557 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.022s	sys 0.026s
W20260808 01:53:22.488219 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 466084us
I20260808 01:53:22.535070 18569 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:22.488832  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 182597us
I20260808 01:53:22.538434 18557 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:22.490532  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 185370us
I20260808 01:53:22.491377  9475 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:22.493248  9479 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260808 01:53:22.494211 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:22.494469  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:22.477044 18631 consensus_meta.cc:287] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.523s	user 0.010s	sys 0.009s
W20260808 01:53:22.497485  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:22.500579  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 220663us
I20260808 01:53:22.276669 18328 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:22.572312  9630 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 27: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:22.573037  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:22.498914 18638 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.594s	user 0.015s	sys 0.006s
I20260808 01:53:22.491967 18230 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:22.587116  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:22.539373  9554 rpcz_store.cc:276] 0808 01:53:21.452474 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:21.643948 (+191474us) service_pool.cc:224] Handling call
0808 01:53:21.644288 (+   340us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:21.648281 (+  3993us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:21.648313 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:21.668475 (+ 20162us) log.cc:844] Serialized 52 byte log entry
0808 01:53:22.091928 (+423453us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:22.091981 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:22.092082 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:22.462770 (+370688us) raft_consensus.cc:1685] finished
0808 01:53:22.462799 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:22.463004 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":83,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":16910,"thread_start_us":331137,"threads_started":1,"wal-append.queue_time_us":544453}
I20260808 01:53:22.602069 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:22.496395 18538 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:22.572036  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 4094) took 1185 ms. Trace:
I20260808 01:53:22.602046 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:22.607164  9480 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:22.592170  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:22.608731  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3135) took 1116 ms. Trace:
I20260808 01:53:22.630823  9557 rpcz_store.cc:276] 0808 01:53:21.386550 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:21.403801 (+ 17251us) service_pool.cc:224] Handling call
0808 01:53:21.404154 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:21.404406 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 13
0808 01:53:21.777034 (+372628us) log.cc:844] Serialized 14 byte log entry
0808 01:53:22.571673 (+794639us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:22.571730 (+    57us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:53:22.571777 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:22.571851 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:22.571974 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":200187,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":4732,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":94,"thread_start_us":1154361,"threads_started":2,"wal-append.queue_time_us":681305}
W20260808 01:53:22.582362  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 87990.1us
I20260808 01:53:22.584115  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:22.541503 18677 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:22.599000  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:22.634548  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 50532.7us
I20260808 01:53:22.635454  9472 raft_consensus.cc:2368] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23.
I20260808 01:53:22.637187  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:22.512274  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 4108) took 1002 ms. Trace:
I20260808 01:53:22.604331 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:22.581162 10362 raft_consensus.cc:2410] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 25 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260808 01:53:22.590602  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:22.579734  9475 raft_consensus.cc:606] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:22.639418  9475 raft_consensus.cc:614] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:53:22.573151  9636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3490) took 2624 ms (client timeout 1751 ms). Trace:
I20260808 01:53:22.633412  9521 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:22.602424 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:22.643474  9521 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:22.643687  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 21 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:22.645623  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:22.663182  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:22.664529  9632 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:22.575573 18569 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:22.632094  9552 rpcz_store.cc:276] 0808 01:53:21.492366 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:21.672550 (+180184us) service_pool.cc:224] Handling call
0808 01:53:21.672996 (+   446us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:21.673468 (+   472us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:21.673504 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:21.674285 (+   781us) log.cc:844] Serialized 52 byte log entry
0808 01:53:21.808012 (+133727us) raft_consensus.cc:1645] Marking committed up to 20
0808 01:53:21.858603 (+ 50591us) log.cc:844] Serialized 14 byte log entry
0808 01:53:21.858997 (+   394us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:21.859090 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:22.533442 (+674352us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:22.541167 (+  7725us) raft_consensus.cc:1685] finished
0808 01:53:22.608430 (+ 67263us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:22.608637 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":53610,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":5230,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":16690,"spinlock_wait_cycles":452271872,"thread_start_us":131539,"threads_started":1,"wal-append.queue_time_us":223736}
I20260808 01:53:22.648535  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:22.648535  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:22.668345  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:22.669493  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:22.669714  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:22.670179  9476 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:22.670342  9638 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:22.671649  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:22.673019  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:22.585443 18654 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.696s	user 0.001s	sys 0.002s
W20260808 01:53:22.611814 18591 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 24 current term: 24 is higher.
I20260808 01:53:22.653586  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:22.510304 18076 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:22.674351  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:22.638913  9550 rpcz_store.cc:276] 0808 01:53:21.510031 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:21.696552 (+186521us) service_pool.cc:224] Handling call
0808 01:53:21.697013 (+   461us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:21.697321 (+   308us) raft_consensus.cc:1534] Early marking committed up to index 12
0808 01:53:21.705498 (+  8177us) log.cc:844] Serialized 14 byte log entry
0808 01:53:22.511872 (+806374us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:22.511939 (+    67us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:53:22.511983 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:22.512051 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:22.512210 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":8544,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":250072,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"spinlock_wait_cycles":1200000,"thread_start_us":805722,"threads_started":2,"wal-append.queue_time_us":527495}
I20260808 01:53:22.586575 18676 raft_consensus.cc:2764] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election decision vote started in defunct term 23: won
W20260808 01:53:22.641177  9636 rpcz_store.cc:269] 0808 01:53:19.949062 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:19.979547 (+ 30485us) service_pool.cc:224] Handling call
0808 01:53:22.572916 (+2593369us) spinlock_profiling.cc:243] Waited 2.56 s on lock 0x7b540027e054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:22.573045 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5881557888}
I20260808 01:53:22.650728  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:22.513851 18162 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:22.579931 18659 log.cc:927] Time spent T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.710s	user 0.002s	sys 0.013s
I20260808 01:53:22.681057  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:22.658587  9556 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:22.706189 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:22.590592 18566 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:22.587270 18531 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:22.683647  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:22.712390  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:22.713112  9631 raft_consensus.cc:2410] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 24 because replica has last-logged OpId of term: 23 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260808 01:53:22.713801  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:22.714764 18539 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:22.715322 18539 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:22.719311  9478 raft_consensus.cc:2410] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 25 because replica has last-logged OpId of term: 23 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 17.
I20260808 01:53:22.626739 18638 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 20. Current term is 21.
I20260808 01:53:22.578663 18599 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:22.576731 18613 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:22.689325 18666 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:53:22.732395  9631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3496) took 2781 ms (client timeout 1648 ms). Trace:
I20260808 01:53:22.732388  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:22.689507  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 5cf7e7eadc2347aa84c98413849ebfc9 (from 6e7856bed1be43acaa821290d506ee97 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:22.559741 18092 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.123s
I20260808 01:53:22.737931  9635 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 12. (term mismatch)
I20260808 01:53:22.675148 18677 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:22.699649  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:22.596939 18533 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.104s
I20260808 01:53:22.703014  9634 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260808 01:53:22.703583 18617 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:22.597491 18532 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:22.604343 18259 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:22.704867 18162 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:22.592410 18605 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.113s
I20260808 01:53:22.582651 18631 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:22.705402 10361 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:22.707484  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3708) took 1107 ms. Trace:
I20260808 01:53:22.707737 18681 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:22.570644 18574 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:22.629153 18255 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.079s
W20260808 01:53:22.709141 18633 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.169s	user 0.020s	sys 0.012s
W20260808 01:53:22.763214 18540 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.512s	user 0.013s	sys 0.000s
I20260808 01:53:22.729252  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:22.797122  9556 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 23
I20260808 01:53:22.798220  9637 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:53:22.797145  9629 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:22.737017  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 3277) took 1170 ms. Trace:
I20260808 01:53:22.679567 18679 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:22.695081 18675 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:22.738849  9635 pending_rounds.cc:85] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 11
I20260808 01:53:22.739660  9479 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260808 01:53:22.800045  9635 pending_rounds.cc:107] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 23.12
I20260808 01:53:22.800475  9635 raft_consensus.cc:2889] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:22.742668  9554 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:22.800964  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:22.799697  9630 rpcz_store.cc:276] 0808 01:53:21.567051 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:21.572874 (+  5823us) service_pool.cc:224] Handling call
0808 01:53:22.736685 (+1163811us) spinlock_profiling.cc:243] Waited 957 ms on lock 0x7b540027e554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:22.736906 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2200358272}
I20260808 01:53:22.743782  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:22.802552  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:22.732949  9631 rpcz_store.cc:269] 0808 01:53:19.951331 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:19.990224 (+ 38893us) service_pool.cc:224] Handling call
0808 01:53:22.731690 (+2741466us) spinlock_profiling.cc:243] Waited 2.7 s on lock 0x7b54001b44d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:22.732295 (+   605us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6206148352}
I20260808 01:53:22.588081 18530 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.099s
I20260808 01:53:22.747148 10360 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:22.684516 17932 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:22.754959  9478 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260808 01:53:22.760826  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:22.763506  9480 raft_consensus.cc:2376] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22. Re-sending same reply.
I20260808 01:53:22.738763 18676 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:22.767068  9473 rpcz_store.cc:276] 0808 01:53:21.600260 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:21.601737 (+  1477us) service_pool.cc:224] Handling call
0808 01:53:21.602137 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:21.602702 (+   565us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:21.602737 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:21.603403 (+   666us) log.cc:844] Serialized 26 byte log entry
0808 01:53:21.826108 (+222705us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:53:21.853452 (+ 27344us) log.cc:844] Serialized 14 byte log entry
0808 01:53:21.853921 (+   469us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:21.854108 (+   187us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:22.579262 (+725154us) spinlock_profiling.cc:243] Waited 376 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:22.707167 (+127905us) raft_consensus.cc:1685] finished
0808 01:53:22.707273 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:22.707410 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":32558,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":58,"raft.queue_time_us":42192,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":201,"spinlock_wait_cycles":865496448,"thread_start_us":222161,"threads_started":1,"wal-append.queue_time_us":265462}
I20260808 01:53:22.769821  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:22.773164 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 472623us
I20260808 01:53:22.776861 10369 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:22.822404  9632 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:53:22.778744  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:22.812055  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:22.779511  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:22.780187 18633 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:22.780869 18574 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:22.783209  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:22.761862 18532 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:22.772311 18617 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:22.787415 18259 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:22.783845 18631 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:22.795769 18597 consensus_meta.cc:287] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.806s	user 0.012s	sys 0.004s
I20260808 01:53:22.795625  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:22.758625 18366 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.040s
I20260808 01:53:22.803470  9630 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:22.864219  9477 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260808 01:53:22.839246 10332 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:22.840647  9560 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:22.677214  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:22.737923 18613 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:22.841256  9638 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 21. Current term is 22. Ops: [20.13-21.14]
I20260808 01:53:22.848342  9481 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:22.850536  9473 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:22.851480  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:22.890259  9630 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260808 01:53:22.848907  9601 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:22.850996  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:22.892220  9632 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:22.832453  9476 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:22.811575  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:22.853852  9600 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:22.845813  9556 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
W20260808 01:53:22.816612  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:22.848851  9479 raft_consensus.cc:2410] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 27 because replica has last-logged OpId of term: 26 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260808 01:53:22.841120  9634 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:53:22.852097 18608 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.905s	user 0.007s	sys 0.027s
I20260808 01:53:22.866168  9474 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 22. Current term is 23. Ops: []
W20260808 01:53:22.856679 18537 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.758s	user 0.030s	sys 0.014s
I20260808 01:53:22.810989 17932 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:22.866050  9472 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:22.809765 18518 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:22.897311 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:22.920436 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:22.846755 18478 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.835s	user 0.019s	sys 0.030s
I20260808 01:53:22.900871  9476 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: [20.12-22.13]
W20260808 01:53:22.903589  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 54807.7us
W20260808 01:53:22.917593  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 63836.1us
I20260808 01:53:22.918524  9479 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 24 index: 18. (term mismatch)
I20260808 01:53:22.925889  9479 pending_rounds.cc:85] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 17
I20260808 01:53:22.926262  9479 pending_rounds.cc:107] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 23.18
I20260808 01:53:22.926582  9479 raft_consensus.cc:2889] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:53:22.921778  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 195559us
I20260808 01:53:22.927920  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:22.929996  9633 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
W20260808 01:53:22.954833  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:22.927281  9601 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:22.849357 18676 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:22.990569  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 1113) took 1084 ms. Trace:
I20260808 01:53:22.991160  9637 rpcz_store.cc:276] 0808 01:53:21.906419 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:21.948462 (+ 42043us) service_pool.cc:224] Handling call
0808 01:53:22.990371 (+1041909us) spinlock_profiling.cc:243] Waited 778 ms on lock 0x7b540003c0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:22.990495 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1790472064}
I20260808 01:53:22.992340  9524 leader_election.cc:400] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23.
I20260808 01:53:22.992915  9524 leader_election.cc:403] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:53:22.776729 18681 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:22.917464 18537 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.835s	user 0.030s	sys 0.019s
W20260808 01:53:22.930188 18478 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.912s	user 0.019s	sys 0.031s
I20260808 01:53:22.919888 18540 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:22.902798 18615 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:22.920686 18611 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:23.001803  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 74661.4us
I20260808 01:53:22.919708 17932 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:22.858704 18617 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:22.900386 18608 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.956s	user 0.008s	sys 0.028s
I20260808 01:53:22.922724 18443 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:22.993184  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:22.898424 18259 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:53:22.837566 18675 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:23.050834  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:22.995174 18597 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:22.997908 18579 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.989s	user 0.010s	sys 0.031s
W20260808 01:53:23.058063 18656 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 1.130s	user 0.020s	sys 0.037s
I20260808 01:53:23.091753  9550 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260808 01:53:23.010710 18637 raft_consensus.cc:2804] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:23.004376 18537 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:23.105571  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:22.933586 18518 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.064924 18628 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:23.134502  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:23.095274 18682 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:23.090490 18076 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:23.106732 18579 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.098s	user 0.011s	sys 0.031s
W20260808 01:53:23.107000 18656 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 1.220s	user 0.020s	sys 0.037s
I20260808 01:53:23.102692 18443 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:23.110944 18685 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader election won for term 29
I20260808 01:53:23.102339 18259 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:23.125870  9637 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:23.131596 18679 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23.
I20260808 01:53:23.068158 18615 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.129545 18692 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:23.094198 18230 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.042s
I20260808 01:53:23.143518 18599 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader election won for term 24
W20260808 01:53:23.195814  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 61471.9us
I20260808 01:53:23.107182 18597 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.086629 18611 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.209438  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:23.209815  9637 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260808 01:53:23.210325  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:23.210945  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:23.212710 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 292356us
I20260808 01:53:23.227494  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:23.228955  9554 raft_consensus.cc:2410] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 27 because replica has last-logged OpId of term: 26 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 12.
I20260808 01:53:23.240589  9555 raft_consensus.cc:1275] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260808 01:53:23.244163  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:23.245308  9555 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:23.170078 18443 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:23.280936  9558 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:23.286931  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:23.165570 18259 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:53:23.287791  9554 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:23.162626 18462 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.049s
I20260808 01:53:23.291955  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:23.292694  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:23.294418  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 84203.9us
I20260808 01:53:23.297547  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:23.142135 18628 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:23.302129  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:23.189210 18537 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:23.308454  9551 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260808 01:53:23.295928  9560 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:23.208388 18529 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election won for term 26
W20260808 01:53:23.310870  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:23.203256 18605 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:23.151818 18682 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.316916  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:23.215207 18532 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:23.210932 18685 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:23.325583  9481 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
I20260808 01:53:23.328421  9473 raft_consensus.cc:606] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:23.328864  9473 raft_consensus.cc:614] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:23.332093  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:23.333017  9555 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:23.308759 18597 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:23.285337 18633 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:23.291011 18574 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:23.337055  9554 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260808 01:53:23.376816  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:23.362954  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:23.363600  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:23.378633  9550 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 27
I20260808 01:53:23.379079  9560 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:23.372833  9634 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:23.385841  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:23.386797  9551 raft_consensus.cc:3060] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 24
I20260808 01:53:23.391088  9552 raft_consensus.cc:2410] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 26 because replica has last-logged OpId of term: 25 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 12.
I20260808 01:53:23.313153 18092 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:23.405082  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:23.458644  9634 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260808 01:53:23.406360  9555 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:23.471621 10361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 1335) took 1159 ms. Trace:
I20260808 01:53:23.472267 10361 rpcz_store.cc:276] 0808 01:53:22.312601 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:22.453517 (+140916us) service_pool.cc:224] Handling call
0808 01:53:22.453939 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:22.766668 (+312729us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:23.471209 (+704541us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b540041f0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:23.471512 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":577804032}
I20260808 01:53:23.474632  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3718) took 1550 ms. Trace:
I20260808 01:53:23.484510  9554 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260808 01:53:23.488009  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3712) took 1818 ms. Trace:
I20260808 01:53:23.488695  9474 rpcz_store.cc:276] 0808 01:53:21.664253 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:21.664435 (+   182us) service_pool.cc:224] Handling call
0808 01:53:21.664744 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:22.900698 (+1235954us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:23.482325 (+581627us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540021dc54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:23.482640 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2762518272}
I20260808 01:53:23.475265  9477 rpcz_store.cc:276] 0808 01:53:21.924092 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:21.924270 (+   178us) service_pool.cc:224] Handling call
0808 01:53:21.924622 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:22.864647 (+940025us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:23.474260 (+609613us) spinlock_profiling.cc:243] Waited 939 ms on lock 0x7b54002bbbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:23.474535 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2160056576}
I20260808 01:53:23.350062 18615 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:23.516593 18682 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:23.370867 18529 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:23.357785 18681 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:23.358251 18676 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:23.376379 18591 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:23.540391  9550 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
I20260808 01:53:23.361078 18613 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:23.546943  9560 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
W20260808 01:53:23.552107  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 487420us
I20260808 01:53:23.499841 18076 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:23.564396  9524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.001s	sys 0.017s
W20260808 01:53:23.522166 18677 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.764s	user 0.011s	sys 0.000s
W20260808 01:53:23.567770  9524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.001s	sys 0.017s
I20260808 01:53:23.365830 18605 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:23.541558 18684 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:23.580727  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:23.582659  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:23.561837 18695 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:23.568665  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 510593us
I20260808 01:53:23.561349 18617 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:23.587350  9551 raft_consensus.cc:2468] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:23.601377  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:23.565166 18693 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:23.622483  9524 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:23.626600  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.002s	sys 0.021s
W20260808 01:53:23.627497  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.003s	sys 0.021s
I20260808 01:53:23.630332  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:23.631336  9475 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
W20260808 01:53:23.633921  9599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.012s	sys 0.010s
W20260808 01:53:23.634371  9599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.013s	sys 0.010s
I20260808 01:53:23.573931 18677 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:23.637830  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:23.574453 18076 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:23.710611  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 21 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:23.712042 18637 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.004s	sys 0.020s
I20260808 01:53:23.715090  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:23.639350 18676 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:23.713014 18682 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.021s
I20260808 01:53:23.732450 18676 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:23.649116 18478 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.008s	sys 0.015s
I20260808 01:53:23.734643  9556 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:23.609391 18615 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.100s
W20260808 01:53:23.713701 18637 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.004s	sys 0.021s
I20260808 01:53:23.746971 18637 raft_consensus.cc:697] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:23.735662 18613 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:23.743517 18697 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
I20260808 01:53:23.732985 18676 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:23.628921 18681 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.092s
W20260808 01:53:23.735430 18478 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.009s	sys 0.015s
I20260808 01:53:23.781526  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:23.584494 18684 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:23.791718  9551 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:23.746980 18677 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:23.789150  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:23.633289 18693 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.741804 18611 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:23.799547  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:23.801096  9559 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:23.748363 18637 consensus_queue.cc:237] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.562069 18591 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.606105 18695 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.208982 18692 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:23.779271 18628 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:23.793695  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 70831.5us
I20260808 01:53:23.581952 18685 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:23.831748  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3295) took 1169 ms. Trace:
I20260808 01:53:23.784770 18613 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.842442  9559 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 22.
I20260808 01:53:23.848996  9631 rpcz_store.cc:276] 0808 01:53:22.662924 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:22.804247 (+141323us) service_pool.cc:224] Handling call
0808 01:53:22.850448 (+ 46201us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:22.850746 (+   298us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:53:22.850771 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:22.860175 (+  9404us) log.cc:844] Serialized 26 byte log entry
0808 01:53:23.211662 (+351487us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:53:23.211704 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:23.211780 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:23.831430 (+619650us) raft_consensus.cc:1685] finished
0808 01:53:23.831460 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:23.831656 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":291982,"threads_started":1,"wal-append.queue_time_us":590476}
W20260808 01:53:23.829795 18259 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.505s	user 0.013s	sys 0.001s
I20260808 01:53:23.859496  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:23.782044 18617 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.047s
I20260808 01:53:23.651363 18529 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.862452  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:23.850557  9551 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 22.
W20260808 01:53:23.828469  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:23.790462 18443 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.918907  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:23.890018 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 589097us
W20260808 01:53:23.896289  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.035s	sys 0.106s
W20260808 01:53:23.923239  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.038s	sys 0.109s
W20260808 01:53:23.923717  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 627303us
W20260808 01:53:23.865182 18599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.013s	sys 0.001s
W20260808 01:53:23.916918  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:23.829165 18677 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:23.863896 18628 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:23.927048 18599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.013s	sys 0.001s
I20260808 01:53:23.939586  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:23.950958 18076 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:23.952098  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3300) took 1207 ms. Trace:
I20260808 01:53:23.952560  9636 rpcz_store.cc:276] 0808 01:53:22.744820 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:22.890426 (+145606us) service_pool.cc:224] Handling call
0808 01:53:22.917799 (+ 27373us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:22.924545 (+  6746us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:22.924577 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:22.935603 (+ 11026us) log.cc:844] Serialized 52 byte log entry
0808 01:53:23.551591 (+615988us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:53:23.551649 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:23.551744 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:23.951812 (+400068us) raft_consensus.cc:1685] finished
0808 01:53:23.951835 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:23.952030 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12071,"raft.queue_time_us":293,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":12850,"thread_start_us":446822,"threads_started":1,"wal-append.queue_time_us":770427}
I20260808 01:53:23.947597  9632 raft_consensus.cc:2393] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 23: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:23.961323  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:23.962106  9555 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:23.954984 18530 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:23.964010  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:23.955744 18613 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:23.955524 18597 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260808 01:53:23.965624 18633 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:23.966341 18597 raft_consensus.cc:3055] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:23.966219 18633 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.966805 18597 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:23.967667 18597 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.968621 18633 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:23.968957 18597 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:23.969604  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:23.957482 18611 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:23.958112 18715 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:23.970955 18633 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:23.971231 18715 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.972040  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:23.973104  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:23.973685 18715 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:23.973948 18328 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:23.958508 18599 raft_consensus.cc:697] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:23.958915  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:23.959268 18366 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:23.975565 18599 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:23.948596  9475 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:23.974411 18328 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:23.977321 18328 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:23.980140 18611 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.975823 18366 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:23.982161  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3301) took 1237 ms. Trace:
I20260808 01:53:23.982527 18611 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:23.983734 18611 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:23.983889 18366 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:23.962037  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:23.984378 18611 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.986106 18692 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:23.974311 18715 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:23.986634 18633 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:23.942178 18677 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:23.991379  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:23.993726 18633 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:23.994415 18633 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:23.987129 18715 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.995222  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:23.927124 18615 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.060s
I20260808 01:53:23.996592 18591 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:23.996768 18633 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:23.997084 18615 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:23.997594 18615 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:23.954474  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:23.992549 18328 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.998641 18615 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:24.006444 18615 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:24.008366 18684 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:24.009490  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:24.004839  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:24.004276  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:24.007057 18443 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:24.012058 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:24.029000  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:24.030767 18628 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:24.033350 18628 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:24.035311 17932 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.942s	user 0.006s	sys 0.007s
I20260808 01:53:24.036000 17932 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:23.986460 18611 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:24.005470 18328 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:24.022513  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:23.982785  9630 rpcz_store.cc:276] 0808 01:53:22.745266 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:22.891114 (+145848us) service_pool.cc:224] Handling call
0808 01:53:22.891385 (+   271us) raft_consensus.cc:1471] Updating replica for 3 ops
0808 01:53:22.903873 (+ 12488us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:22.903908 (+    35us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0808 01:53:22.993679 (+ 89771us) log.cc:844] Serialized 78 byte log entry
0808 01:53:23.563098 (+569419us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:23.819736 (+256638us) log.cc:844] Serialized 14 byte log entry
0808 01:53:23.831722 (+ 11986us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:23.831810 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:23.981830 (+150020us) raft_consensus.cc:1685] finished
0808 01:53:23.981864 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:23.982084 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39732,"prepare.queue_time_us":394191,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":38977,"raft.run_cpu_time_us":401,"raft.run_wall_time_us":67926,"thread_start_us":720552,"threads_started":2,"wal-append.queue_time_us":967758}
I20260808 01:53:24.023619 18591 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260808 01:53:24.024523  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:24.047693  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:24.004808 18693 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:23.917315 18259 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:24.040850  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:24.042837 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:24.053546  9474 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:24.044497  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:24.054332 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:24.055123  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:24.030511  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:24.005266 18539 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:24.048141  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:23.991987 18677 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:24.060057  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:24.091567  9636 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:24.064188 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:23.996613 18715 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:24.055820  9630 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 15.
W20260808 01:53:24.047590 18537 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.015s	sys 0.026s
I20260808 01:53:24.057835 10332 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:24.049888 18591 consensus_queue.cc:1243] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 24.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260808 01:53:24.072613  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:24.075508 18688 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:24.077392  9474 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:24.078121 18531 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:24.052881 18530 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:24.078816 18710 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:24.071631 18259 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:24.106124 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48385.8us
W20260808 01:53:24.143788  9477 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.011s	sys 0.032s
W20260808 01:53:24.144843  9477 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.519s	user 0.011s	sys 0.032s
I20260808 01:53:24.144891 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:24.097326  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 44972.1us
I20260808 01:53:24.038587 17932 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:24.100288 18537 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.016s	sys 0.026s
I20260808 01:53:24.114382 18539 consensus_queue.cc:1059] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.043s
I20260808 01:53:24.157449 18723 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:53:24.158185 18723 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:24.158929 18723 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:53:24.158079 18539 raft_consensus.cc:3055] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:24.163283 18539 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:24.143826 18688 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:24.117410 18712 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:24.164803 18539 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:24.166091 18688 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:24.086090 18633 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:24.167678  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:24.172590  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:24.150894 18530 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:24.143998 18531 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:24.166821 18539 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:24.184206  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 319839us
W20260808 01:53:24.189524  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 118207us
I20260808 01:53:24.197726  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:24.180873 18530 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:24.192997 18531 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:24.225600 18597 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260808 01:53:24.204499 18328 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election won for term 23
W20260808 01:53:24.364308  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 139736us
I20260808 01:53:24.351495  9636 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
W20260808 01:53:24.357686  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 132484us
I20260808 01:53:24.369858  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:24.371105  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:24.371795  9636 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:24.372630  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:24.373208  9636 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:53:24.379709 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:24.379657  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:24.382658  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:24.384888 10332 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:24.365814 18637 consensus_queue.cc:1048] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.023s
I20260808 01:53:24.389106 18531 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:24.392954  9552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 4110) took 1184 ms. Trace:
I20260808 01:53:24.393461  9552 rpcz_store.cc:276] 0808 01:53:23.208957 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:23.209135 (+   178us) service_pool.cc:224] Handling call
0808 01:53:24.392695 (+1183560us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b54003255d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:24.392882 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":405638016}
I20260808 01:53:24.396231 18628 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:24.397907  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3310) took 1469 ms. Trace:
I20260808 01:53:24.398278  9637 rpcz_store.cc:276] 0808 01:53:22.928853 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:23.293043 (+364190us) service_pool.cc:224] Handling call
0808 01:53:23.325963 (+ 32920us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:23.606869 (+280906us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:23.606919 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:23.606974 (+    55us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:53:23.607025 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:24.397633 (+790608us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b5400420ed4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:24.397685 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:24.397844 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":645342976}
I20260808 01:53:24.399593  9521 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:24.401750  9521 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:24.403061 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:24.404574  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:24.405959 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:24.366456 18676 consensus_meta.cc:287] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.573s	user 0.013s	sys 0.008s
I20260808 01:53:24.351914 18566 raft_consensus.cc:3055] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:24.411621  9599 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:24.412451 18566 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:24.416728 18617 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:24.417906 18566 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:24.419116 18255 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:24.420149 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:24.421820  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:24.422528  9637 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 29.
I20260808 01:53:24.423620  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:24.424420  9637 raft_consensus.cc:2410] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260808 01:53:24.424460  9521 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:24.425421  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:24.425973  9637 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:24.426096  9521 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:24.426839 18715 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:24.426962 18682 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:53:24.427983 18715 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:24.428429  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:24.409452 18615 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:24.429711  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:24.429759 18615 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:24.432281 18615 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:24.434315 18715 raft_consensus.cc:697] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:24.435477 18715 consensus_queue.cc:237] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:24.438618  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:24.440587  9521 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:24.353137 18675 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:24.444499 18531 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:24.366354 18722 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:24.446899 18722 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:24.449538 18722 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:24.450663 18633 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:24.453661 18530 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:24.397756 18680 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:24.458154  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 4113) took 1244 ms. Trace:
I20260808 01:53:24.458608  9556 rpcz_store.cc:276] 0808 01:53:23.213764 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:23.213884 (+   120us) service_pool.cc:224] Handling call
0808 01:53:23.214207 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:23.735316 (+521109us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:24.457863 (+722547us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b540002c1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:24.458082 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1196248832}
I20260808 01:53:24.459726 18723 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:24.460433  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 4111) took 1250 ms. Trace:
I20260808 01:53:24.461077 10365 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: []
I20260808 01:53:24.462131 18723 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:24.415195 18579 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:24.415634 18628 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:24.464500 18628 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:24.413133  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:24.464957 18628 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
W20260808 01:53:24.465597  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 52548.4us
I20260808 01:53:24.466195  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:24.411978 18732 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:24.468147 18506 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:24.468757 18506 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:24.469715 18506 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:24.415719 18676 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:24.474436  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:24.475823  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:24.476908  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:24.413335  9599 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:24.413280 10332 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:24.460848  9558 rpcz_store.cc:276] 0808 01:53:23.210268 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:23.210623 (+   355us) service_pool.cc:224] Handling call
0808 01:53:24.460249 (+1249626us) spinlock_profiling.cc:243] Waited 63.7 ms on lock 0x7b540002f154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:24.460368 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":146456832}
I20260808 01:53:24.487365  9631 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
W20260808 01:53:24.505221  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:24.482321  9521 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:53:24.506819  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 93550us
I20260808 01:53:24.508317  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:24.513029  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:24.514057  9630 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:24.515436  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:24.517561 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 21 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:24.517611 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 104407us
I20260808 01:53:24.518611  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:24.511265  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:24.519644 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:53:24.484268 18666 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.705s	user 0.013s	sys 0.034s
I20260808 01:53:24.527631  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:24.530634  9599 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:24.533696  9472 raft_consensus.cc:2393] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 25: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:24.548878  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:24.549170  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 61645.8us
I20260808 01:53:24.553436  9558 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:24.556249  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:24.564277 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 44757.5us
I20260808 01:53:24.564556  9554 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:24.565428 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:24.558023  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:24.566103  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:24.560240  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:24.442765  9636 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:53:24.572485  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:24.573335  9636 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:24.573967  9637 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
I20260808 01:53:24.498494 18676 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:24.505762 18617 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:24.567957  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:24.587427  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:24.531790 17932 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:53:24.529045 18703 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260808 01:53:24.510619 18615 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260808 01:53:24.614174  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 50fd024764ea49c7a1f16e992e7599f3: Illegal state: Replica 52ccf4dcca1843f7bb29e9172e2d4ab4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 23 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:24.614578 18615 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:24.615231 18615 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:24.578405 10332 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:24.580014  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 61483.3us
I20260808 01:53:24.524366 18637 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:24.510135 18734 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260808 01:53:24.551590 18666 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.753s	user 0.014s	sys 0.034s
I20260808 01:53:24.582998  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:24.576380  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:24.550590 18539 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:24.582125  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:24.598750  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3507) took 4545 ms. Trace:
W20260808 01:53:24.601320 18742 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260808 01:53:24.601867 18738 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:53:24.575033 18599 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:24.584749 18611 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260808 01:53:24.608645 18443 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:24.524478 18733 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.067s
I20260808 01:53:24.590364 18531 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:24.582052 18688 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:24.611372 18706 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:24.611815 18709 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:24.575106 18591 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260808 01:53:24.581528 18720 consensus_queue.cc:1059] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.037s
I20260808 01:53:24.595623 17932 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:24.646809  9636 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:24.702813  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:24.703645  9636 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
W20260808 01:53:24.627393 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 49126.3us
I20260808 01:53:24.671113  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:24.671453  9444 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:24.671974  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 160814us
W20260808 01:53:24.672667  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 142166us
I20260808 01:53:24.707393  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:24.672724  9637 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:24.669198  9635 rpcz_store.cc:276] 0808 01:53:20.053096 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:20.812860 (+759764us) service_pool.cc:224] Handling call
0808 01:53:22.731835 (+1918975us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:22.844748 (+112913us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:24.598398 (+1753650us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b54001b44d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:24.598615 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4393685248}
I20260808 01:53:24.682837  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3730) took 1864 ms. Trace:
I20260808 01:53:24.685935  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:24.629676 18723 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:24.628746 18328 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:24.691140 18539 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:24.595276 18628 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:24.710752  9476 rpcz_store.cc:276] 0808 01:53:22.818325 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:22.924517 (+106192us) service_pool.cc:224] Handling call
0808 01:53:23.481510 (+556993us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:23.484437 (+  2927us) raft_consensus.cc:1534] Early marking committed up to index 9
0808 01:53:23.484469 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:23.499706 (+ 15237us) log.cc:844] Serialized 26 byte log entry
0808 01:53:23.841997 (+342291us) raft_consensus.cc:1645] Marking committed up to 9
0808 01:53:23.842056 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:23.842161 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:24.024875 (+182714us) raft_consensus.cc:1685] finished
0808 01:53:24.024904 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:24.669244 (+644340us) spinlock_profiling.cc:243] Waited 557 ms on lock 0x7b540039a150. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:24.682740 (+ 13496us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2436,"raft.queue_time_us":17842,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":37264,"spinlock_wait_cycles":1280106752,"thread_start_us":330947,"threads_started":1,"wal-append.queue_time_us":511063}
I20260808 01:53:24.708792 10332 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:24.664007 18637 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:24.691673 18599 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:53:24.731716  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 65044.4us
I20260808 01:53:24.825479  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:24.739101  9637 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:24.737864  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:24.706606 17932 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:24.750957  9557 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260808 01:53:24.707680 18688 raft_consensus.cc:697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:24.838143  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3294) took 2177 ms. Trace:
I20260808 01:53:24.692241 18443 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:24.844681  9638 rpcz_store.cc:276] 0808 01:53:22.661277 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:22.706597 (+ 45320us) service_pool.cc:224] Handling call
0808 01:53:22.840880 (+134283us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:22.889376 (+ 48496us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:24.837888 (+1948512us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b540008dc50. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:24.838077 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":307665152}
W20260808 01:53:24.848120  9629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3501) took 4843 ms (client timeout 1648 ms). Trace:
I20260808 01:53:24.709441 18706 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:53:24.848647  9629 rpcz_store.cc:269] 0808 01:53:20.000094 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:20.064096 (+ 64002us) service_pool.cc:224] Handling call
0808 01:53:24.842756 (+4778660us) spinlock_profiling.cc:243] Waited 2.73 s on lock 0x7b54001b3fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:24.842938 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6273444224}
W20260808 01:53:24.853541  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 122405us
I20260808 01:53:24.854604  9638 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27. Ops: [26.15-26.15]
I20260808 01:53:24.854806  9599 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:24.824517 18637 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:24.814646 18723 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:53:24.813598 18531 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:24.844605 18678 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:24.870541  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 152616us
I20260808 01:53:24.847415 18676 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:24.814231 18539 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:24.859110 18751 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:24.861689 18530 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:24.885021 18539 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:24.886178 18599 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:24.851593 18675 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.025s
I20260808 01:53:24.890455  9629 pending_rounds.cc:85] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 11
I20260808 01:53:24.897306  9629 pending_rounds.cc:107] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 24.12
I20260808 01:53:24.897636  9629 raft_consensus.cc:2889] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:24.862066 18733 consensus_queue.cc:1059] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.017s
I20260808 01:53:24.874735 18092 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:24.872653 18709 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.023s
I20260808 01:53:24.899168 18092 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:24.899722 18092 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:24.876334 18328 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:24.903286 18750 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:24.903877 18750 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:24.906194 18750 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:24.873708 18678 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:24.893425 18688 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:24.895836  9635 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 26. Current term is 27. Ops: [26.14-26.14]
I20260808 01:53:24.912159  9638 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
I20260808 01:53:24.878182 18676 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:24.912067 18443 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:24.920763 18761 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.003s
I20260808 01:53:24.924973 18676 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:24.927399 18709 raft_consensus.cc:3055] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:24.928457 18709 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:24.929947 18709 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:24.931840 18762 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:24.932554 18762 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:24.933826 18762 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:24.916913 18539 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:24.894495 18706 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:24.957261 18678 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:25.004388  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 136116us
I20260808 01:53:25.008298  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:53:25.016316  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 154599us
I20260808 01:53:25.017289  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3314) took 1001 ms. Trace:
I20260808 01:53:25.017906  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:25.019968  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 105554us
I20260808 01:53:25.025192  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:25.027479  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:25.031234  9557 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:25.022030  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:24.942973 18628 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:25.050304 18605 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:25.055790 18539 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:25.056598 18539 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:25.058387 18539 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:25.063328 18675 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:25.064726 18675 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:25.065451  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:25.067579 18328 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:25.068331 18328 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:25.068958 18328 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:25.071079 18443 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:25.072285 18443 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.017925  9637 rpcz_store.cc:276] 0808 01:53:24.015909 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:24.826616 (+810707us) service_pool.cc:224] Handling call
0808 01:53:24.827055 (+   439us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:24.841036 (+ 13981us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:24.841078 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:24.847886 (+  6808us) log.cc:844] Serialized 52 byte log entry
0808 01:53:24.915363 (+ 67477us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:53:24.915410 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:24.915483 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:25.016393 (+100910us) raft_consensus.cc:1685] finished
0808 01:53:25.016441 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:25.016903 (+   462us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":51752,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":4244,"thread_start_us":49299,"threads_started":2,"wal-append.queue_time_us":83736}
I20260808 01:53:25.074246 18751 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:25.075874 18751 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:25.076563 18751 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:25.071265 18732 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:25.085218  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3159) took 1123 ms. Trace:
I20260808 01:53:25.086948 18765 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:25.091116 18732 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:25.108969  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 1122) took 1235 ms. Trace:
I20260808 01:53:25.111718  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:25.111923  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:25.113471  9552 raft_consensus.cc:2410] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 30 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260808 01:53:25.113938  9635 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:25.114795  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:25.116458 18443 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:25.086041  9555 rpcz_store.cc:276] 0808 01:53:23.961200 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:23.964983 (+  3783us) service_pool.cc:224] Handling call
0808 01:53:24.394811 (+429828us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:24.395208 (+   397us) raft_consensus.cc:1534] Early marking committed up to index 13
0808 01:53:24.580630 (+185422us) log.cc:844] Serialized 14 byte log entry
0808 01:53:24.848234 (+267604us) log.cc:844] Serialized 14 byte log entry
0808 01:53:24.848417 (+   183us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:24.848467 (+    50us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:53:24.848512 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:24.848579 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:25.084674 (+236095us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b54003255d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:25.084949 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36666,"prepare.queue_time_us":52718,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":76,"raft.queue_time_us":36720,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":14164,"spinlock_wait_cycles":1009173120,"thread_start_us":412867,"threads_started":2,"wal-append.queue_time_us":285551}
W20260808 01:53:25.123127  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 115608us
I20260808 01:53:25.124266 18518 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:25.125452 18530 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:25.126971 18530 consensus_queue.cc:1243] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 20.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260808 01:53:25.128297 18732 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:25.129732  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 64665.5us
I20260808 01:53:25.132691 18675 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:25.148182  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:45190 (request call id 15) took 2380 ms. Trace:
I20260808 01:53:25.148408  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:25.148881  9473 rpcz_store.cc:276] 0808 01:53:22.767466 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:22.850389 (+ 82923us) service_pool.cc:224] Handling call
0808 01:53:25.147590 (+2297201us) spinlock_profiling.cc:243] Waited 438 ms on lock 0x7b5400399254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4498dc33 00007f5d4998d41a 00007f5d42ee1934 00007f5d42ee18c2 00007f5d42ee17f1 00007f5d42ee176c 00007f5d42ee0a22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:25.147859 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1008279296}
W20260808 01:53:25.152245  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 7b5ea129398a42fb9981fca652027cd6 (from 6e7856bed1be43acaa821290d506ee97 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:25.155447 18553 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:25.156183  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:25.158771 18638 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:25.109652  9631 rpcz_store.cc:276] 0808 01:53:23.873076 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:24.829192 (+956116us) service_pool.cc:224] Handling call
0808 01:53:24.829696 (+   504us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:24.863889 (+ 34193us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:24.863922 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:24.863968 (+    46us) raft_consensus.cc:1645] Marking committed up to 19
0808 01:53:24.863998 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:25.108199 (+244201us) spinlock_profiling.cc:243] Waited 33.9 ms on lock 0x7b540025bcd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:25.108297 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:25.108656 (+   359us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":78075264}
I20260808 01:53:25.159126  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:25.180078  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:25.159694 18638 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:25.191143  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:25.196679 18533 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:25.197841 18443 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:25.193526  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:25.202200  9555 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:25.202499 18738 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:25.215060 18738 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:25.215785 18738 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:25.191380  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:25.219344  9558 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:25.221504 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:25.232020  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 73171.8us
I20260808 01:53:25.239111 18255 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:25.244866 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:25.250227 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 67436us
I20260808 01:53:25.251610 18092 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:25.254588 18638 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.259220  9559 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:25.259481 10369 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fca874793205463dbb922993f57bf7f5"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:57156
I20260808 01:53:25.261065 10369 raft_consensus.cc:606] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:25.261721 10369 raft_consensus.cc:614] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:53:25.264634  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet fca874793205463dbb922993f57bf7f5 (from 7201605bb2f54c949a4ebe6e9b3b4b2a to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:25.265684  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260808 01:53:25.234104  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:25.279745 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:25.280879 18638 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:25.282451 18553 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:25.283152 18553 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:25.283706 18553 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
W20260808 01:53:25.264102  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:25.274253  9601 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:25.296615 18518 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election won for term 23
W20260808 01:53:25.298902 18615 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.683s	user 0.021s	sys 0.007s
I20260808 01:53:25.300097  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:25.299911 18615 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.301705  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:25.301867  9560 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:25.303119  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:25.304569  9552 raft_consensus.cc:2393] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 26: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:25.305984  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:25.313927  9552 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:25.314344  9555 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:25.312063 18518 raft_consensus.cc:697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:25.304459 18539 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.359901 10369 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: []
I20260808 01:53:25.368099  9558 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
W20260808 01:53:25.369760  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 136043us
W20260808 01:53:25.370872  9524 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.672s (SENT) [suppressed 3 similar messages]
I20260808 01:53:25.372244 18746 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260808 01:53:25.374194 17932 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.500s	user 0.020s	sys 0.007s
I20260808 01:53:25.374750 18752 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:25.374866 17932 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.375771 18752 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:25.361395 18328 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.372344  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:25.373440 18092 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:25.386910  9552 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
I20260808 01:53:25.384891  9557 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 30: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:25.379108 18747 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.027s
I20260808 01:53:25.363884 18615 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:25.378468 17932 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:25.422098  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 49952.2us
I20260808 01:53:25.425220  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:25.426184 10332 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:25.426695 18747 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:25.427533 18747 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:25.428197 18747 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
W20260808 01:53:25.431599 18637 consensus_meta.cc:287] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.554s	user 0.016s	sys 0.009s
I20260808 01:53:25.426853 18746 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260808 01:53:25.459486 18752 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:25.480899  9473 raft_consensus.cc:1275] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260808 01:53:25.489715  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3506) took 5436 ms. Trace:
I20260808 01:53:25.490414 18637 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.492681  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:25.495415 18539 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:25.462456 18483 consensus_queue.cc:1059] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:25.467175 18599 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:53:25.469627 18328 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:25.471836 18774 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
W20260808 01:53:25.476189  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:25.469770  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 160801us
W20260808 01:53:25.521749  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 159759us
I20260808 01:53:25.490248  9633 rpcz_store.cc:276] 0808 01:53:20.052813 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:20.799740 (+746927us) service_pool.cc:224] Handling call
0808 01:53:22.577729 (+1777989us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:22.930469 (+352740us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:24.461332 (+1530863us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b540027e054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:25.489285 (+1027953us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b540027e050. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:25.489476 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4897192064}
I20260808 01:53:25.496717 18738 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:25.511117 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:25.517686 18747 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.520306  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:25.493220 18637 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:25.517971 18518 consensus_queue.cc:237] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.542784  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3324) took 1018 ms. Trace:
I20260808 01:53:25.543351  9637 rpcz_store.cc:276] 0808 01:53:24.524485 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:25.076513 (+552028us) service_pool.cc:224] Handling call
0808 01:53:25.077370 (+   857us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:25.078190 (+   820us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:25.078217 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:25.106948 (+ 28731us) log.cc:844] Serialized 26 byte log entry
0808 01:53:25.181321 (+ 74373us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:53:25.181374 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:25.181451 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:25.542459 (+361008us) raft_consensus.cc:1685] finished
0808 01:53:25.542482 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:25.542716 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":401,"spinlock_wait_cycles":8802688,"thread_start_us":73626,"threads_started":1,"wal-append.queue_time_us":212310}
I20260808 01:53:25.530114 18483 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:25.557241  9556 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260808 01:53:25.557785 18483 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.560653  9472 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 26: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:53:25.562803  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:25.572712  9599 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:25.535720 18553 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:25.560652  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:25.589155 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 72093.6us
W20260808 01:53:25.591902 18751 consensus_meta.cc:287] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.514s	user 0.020s	sys 0.001s
I20260808 01:53:25.605170 18755 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:25.606799 18760 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:25.608349 18760 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:25.611229 18676 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:25.612651 18676 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:25.615089  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 3305) took 2671 ms. Trace:
I20260808 01:53:25.615595  9632 rpcz_store.cc:276] 0808 01:53:22.927499 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:22.927639 (+   140us) service_pool.cc:224] Handling call
0808 01:53:25.584027 (+2656388us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540003c5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:25.599202 (+ 15175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2343016192}
W20260808 01:53:25.619645  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 63311.5us
I20260808 01:53:25.621335  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:25.625224 18750 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:53:25.625407 18553 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:25.626788 18762 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:25.631350 18738 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:25.639154  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3160) took 1089 ms. Trace:
I20260808 01:53:25.639581  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:25.640872  9637 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260808 01:53:25.641567 18483 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:25.645738  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:25.646828  9631 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:53:25.647176 18738 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:25.648514 18738 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.652324 18755 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:25.653138 18755 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:25.654421  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:25.654650  9472 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:25.655681  9632 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260808 01:53:25.656983  9480 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:25.657213 18706 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:25.658144 18706 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:25.664136 10334 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:25.658586 18699 raft_consensus.cc:3055] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:25.667603 18699 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:25.639876  9560 rpcz_store.cc:276] 0808 01:53:24.550038 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:24.552567 (+  2529us) service_pool.cc:224] Handling call
0808 01:53:24.584703 (+ 32136us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:25.302477 (+717774us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:25.638481 (+336004us) spinlock_profiling.cc:243] Waited 717 ms on lock 0x7b54003264d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:25.639066 (+   585us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1648337792}
I20260808 01:53:25.671639 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:25.653538 18695 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:25.645820  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:25.672696 10369 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:25.679104  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:25.680171 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:25.670583 18699 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.681427 18699 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:53:25.690304  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:25.671376 18725 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 1.467s	user 0.007s	sys 0.011s
W20260808 01:53:25.686067  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 261144us
I20260808 01:53:25.698210  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:25.674118 18760 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:25.674440 18755 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:25.639391 18751 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:25.707248 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 108482us
I20260808 01:53:25.718395  9474 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260808 01:53:25.719895 18761 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:25.696817 18599 raft_consensus.cc:697] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:25.745422 18751 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:25.694129 18637 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.047s
W20260808 01:53:25.747120  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 126088us
I20260808 01:53:25.752344 18566 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:25.757560 18574 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:53:25.752350  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:25.759922 18755 consensus_queue.cc:1059] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:25.761070 18706 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:25.762183 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 98411.5us
I20260808 01:53:25.775414  9599 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:25.780608 18734 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:25.781564 18734 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:25.780253 18553 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 15, Time since last communication: 0.003s
I20260808 01:53:25.770741 18747 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:25.773310 17932 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:25.793161 18759 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:25.793838 18759 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:25.794360 18759 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:25.795217 18529 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:25.796030 18529 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:25.799070 18631 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:25.799664 18631 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:25.793300 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:25.800125 18599 consensus_queue.cc:237] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:25.802445  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:25.778254 18695 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:25.803478 18695 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:25.804736  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:25.805922  9473 raft_consensus.cc:2410] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260808 01:53:25.806301 18695 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:25.780376 18690 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:25.800200  9600 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:25.811776  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 128920us
I20260808 01:53:25.785276 18518 raft_consensus.cc:3055] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:25.776355 18483 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:25.825037 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:25.825817 18734 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:25.827309 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:25.827682 10363 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:25.829201 18690 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:25.829825 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:25.830216  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 1322) took 1913 ms. Trace:
I20260808 01:53:25.832197 18690 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:25.830940  9551 rpcz_store.cc:276] 0808 01:53:23.916282 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:23.917501 (+  1219us) service_pool.cc:224] Handling call
0808 01:53:23.918007 (+   506us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:23.918390 (+   383us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:23.918429 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:25.659299 (+1740870us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b2c000f57d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:53:25.660099 (+   800us) log.cc:844] Serialized 26 byte log entry
0808 01:53:25.707558 (+ 47459us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:25.707597 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:25.721899 (+ 14302us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:25.829582 (+107683us) raft_consensus.cc:1685] finished
0808 01:53:25.829618 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:25.829949 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5046016,"thread_start_us":46890,"threads_started":1,"wal-append.queue_time_us":110613}
I20260808 01:53:25.833256 18690 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:25.831302 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 51976.4us
I20260808 01:53:25.793241  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:53:25.840024  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 67281.4us
I20260808 01:53:25.841632 18483 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.842237 18569 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:25.843616  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:25.815446 18787 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:25.844142 18569 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.844847 18787 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:25.845526 18691 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260808 01:53:25.845906  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59474.3us
I20260808 01:53:25.847234  9631 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260808 01:53:25.847409  9521 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:25.848634 18699 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:25.849074 18699 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:25.849475 18699 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
W20260808 01:53:25.849735  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 57897.7us
I20260808 01:53:25.840117 18529 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:25.851442  9521 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:25.852686 18615 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:25.787076 18676 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:25.853734 18615 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:25.854359 18615 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:25.855786 18615 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.818712 18706 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:25.855628  9479 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:25.857621 18706 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:25.858145 18706 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:25.860145 18690 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:25.861470  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:25.862113 18776 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.351s	user 0.001s	sys 0.001s
I20260808 01:53:25.863947 18675 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:25.866732  9599 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:25.841683 18796 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:25.800151 18631 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:25.843258  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:25.839964 17932 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:25.873441 18738 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:25.881292  9632 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260808 01:53:25.878664  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:25.884258 18569 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:25.887302  9480 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:25.887483  9521 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:25.881659  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:25.880426  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:25.879944  9630 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: [20.12-22.13]
I20260808 01:53:25.895570 18566 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:25.895949 18532 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:25.887815 18611 raft_consensus.cc:3055] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Stepping down as leader of term 15
I20260808 01:53:25.897966 18611 raft_consensus.cc:740] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:25.899326 18611 consensus_queue.cc:260] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:25.900848 18611 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:53:25.901331 18796 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:25.904495 17932 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:25.905324 17932 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:25.905766 17932 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:25.906155 17932 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:25.908077 18617 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:25.908478 18690 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:25.908705  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:53:25.882117 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:25.823688 18518 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:25.789069 18751 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.088s
I20260808 01:53:25.846151 18787 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.919925 10365 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:25.882061 18483 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:25.925376 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 115592us
W20260808 01:53:25.925248  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 49647.4us
I20260808 01:53:25.917725 18699 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:25.928602  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52174.8us
I20260808 01:53:25.932422  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:25.929350  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:25.945909 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:25.947582  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:25.920980 18759 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.952088 18637 consensus_queue.cc:1048] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:25.953612 18759 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:25.956193  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:25.957099  9630 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 26: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:25.959288 10361 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 15. Current term is 16. Ops: []
I20260808 01:53:25.963836  9477 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:25.965443  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:25.967279 18628 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.005s
I20260808 01:53:25.970868 18617 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:25.975256  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:25.976826 18569 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:25.976989  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:25.977458 18569 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:25.975339 18699 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:25.983271  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:25.982942 18745 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.002s
I20260808 01:53:25.984717  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:25.985085 18745 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:25.985492  9634 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:53:25.985544 18745 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:25.987269 18745 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.984145  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:25.978564  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:25.993197  9637 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260808 01:53:25.996280  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:25.999085  9479 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:25.923856 18675 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.041s
I20260808 01:53:25.997133  9635 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:26.005328  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:26.007215  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:26.008548  9633 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:25.995600 18761 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.016s
I20260808 01:53:26.014276 18761 consensus_queue.cc:1243] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 6e7856bed1be43acaa821290d506ee97 log is divergent from this leader: its last log entry 24.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260808 01:53:25.933993 18690 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.073s
I20260808 01:53:25.992592 18695 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:26.016137  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:26.019480  9634 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 29.
I20260808 01:53:25.996060 18532 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:26.020707 18755 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election won for term 27
W20260808 01:53:26.029588  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:26.029923  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.021799  9638 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:26.035336  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:26.037950  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:26.038935 18608 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:26.039706 18608 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.978556 18569 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:26.047987  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:26.043340  9601 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:26.057425  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.057631  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:25.955415 18518 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:26.043673 18794 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.116s	user 0.001s	sys 0.001s
I20260808 01:53:26.063293 18738 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260808 01:53:26.044066 18745 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:26.076996  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.042667 18608 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:26.081302  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 46143us
I20260808 01:53:26.074648  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.061241 18755 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:26.083889 10334 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:26.096685  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:26.047202 18695 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:26.099952  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:26.099622 18553 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:26.106586  9472 raft_consensus.cc:2410] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 23 because replica has last-logged OpId of term: 22 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260808 01:53:26.107419  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.094372 18569 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:26.089740 18691 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 23.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.024s
I20260808 01:53:26.124640  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:26.124806  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 54764.6us
I20260808 01:53:26.126965  9481 raft_consensus.cc:2393] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 25: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:26.129056  9599 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:26.135279  9477 raft_consensus.cc:1240] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:26.098397 18631 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:26.139990  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 44661.9us
I20260808 01:53:26.111279 18553 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:26.147578 10369 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 17. Current term is 18. Ops: []
W20260808 01:53:26.149418  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 106202us
I20260808 01:53:26.151957  9601 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:26.137902 18759 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:26.131084 18747 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:26.157169  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.140862 18566 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:26.162808 18579 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:26.167788 18723 consensus_queue.cc:1059] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.010s
I20260808 01:53:26.169188 18723 raft_consensus.cc:3055] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 LEADER]: Stepping down as leader of term 27
I20260808 01:53:26.169595 18723 raft_consensus.cc:740] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:26.171288  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:26.163784 18539 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:26.207291 18539 raft_consensus.cc:697] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:26.208407 18539 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:26.209069  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.210492 18695 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:26.213162 18553 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:26.214416  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:26.216182  9630 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:26.217231  9477 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 23. Current term is 24. Ops: [22.15-23.16]
W20260808 01:53:26.218088  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:26.218714  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 89794.1us
I20260808 01:53:26.161330 18747 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:26.224298  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:26.225865  9552 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:26.226831  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:26.226770 18713 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.104s
I20260808 01:53:26.227483  9478 pending_rounds.cc:85] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 12
I20260808 01:53:26.227852  9478 pending_rounds.cc:107] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 24.13
I20260808 01:53:26.228016  9560 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 26: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:26.228152  9478 raft_consensus.cc:2889] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:26.228380 18713 raft_consensus.cc:3055] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:26.228798 18713 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:26.229844 18713 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:26.230769 18803 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260808 01:53:26.230839 18627 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:26.243160  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:26.243973  9558 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:26.248860  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:26.250793  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:26.251586  9556 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:26.253069  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:26.234413  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:26.254104  9550 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
W20260808 01:53:26.254735  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:26.235105  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:26.239390  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:26.253157 18519 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:26.235895 18553 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:26.249420 18709 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:53:26.259729  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:26.166507 18706 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:26.263424  9631 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260808 01:53:26.256083 18747 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:26.262856 18695 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:26.269055  9551 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:26.257742 18765 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:26.278069 18765 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:26.279315  9554 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260808 01:53:26.280491 18695 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:26.282780 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:26.215262 18723 consensus_queue.cc:260] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:26.288300 18695 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:26.289155 18723 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:26.259584 18762 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:26.272501  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 110380us
I20260808 01:53:26.275477  9636 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260808 01:53:26.275974  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:26.297036  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:26.301074  9599 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:26.307663  9551 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:26.310809  9550 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260808 01:53:26.318539  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.309000  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:26.324959  9558 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:26.334518  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:26.338477 18675 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260808 01:53:26.338974 18761 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260808 01:53:26.347582  9477 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:26.353475 18699 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:26.353919  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.354260 18699 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:26.356751 18637 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:26.356848 18699 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:26.357579 18637 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:26.360000 18637 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:26.323534 18605 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:26.361466 18605 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:26.361964 18605 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:26.257702 18579 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.054s
I20260808 01:53:26.371063  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:26.174360 18755 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:26.312331 18765 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:26.383461  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:26.390587  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:26.391261  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:26.295173 18762 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:26.307305 18553 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:26.337240 17932 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:26.422221 18760 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:26.423190 18760 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:26.427299 18760 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:26.427551 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:26.429678 18760 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:26.432143 18760 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:26.373440 18698 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.037s
I20260808 01:53:26.401785 18759 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260808 01:53:26.442476  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:26.444625  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:53:26.459496  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 82485us
I20260808 01:53:26.460471  9477 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:26.414789 18532 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.059s
I20260808 01:53:26.487676 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:26.434104 18809 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:26.466225 18698 consensus_queue.cc:1243] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 26.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260808 01:53:26.447445 18759 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:26.497146 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:26.493744  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:26.468101 18631 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:26.410113 18611 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.077s
I20260808 01:53:26.478994 18695 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:26.476938 18761 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:26.500319  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:26.498203 10361 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
W20260808 01:53:26.501426  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 100936us
I20260808 01:53:26.484723 18608 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:26.511198 18732 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:26.510950 18608 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:26.511258  9599 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:26.512321  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:26.511667 18732 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:26.501452 18759 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:26.506078 18695 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:26.532097  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 21 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.534358 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:26.496999 18617 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.029s
I20260808 01:53:26.508318 18765 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:26.527890 18529 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:26.545420 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:26.528369 18706 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:26.534687 18695 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:26.551127 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:26.533068 18732 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
W20260808 01:53:26.543164 18804 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.271s	user 0.000s	sys 0.002s
I20260808 01:53:26.555125  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.542644 18762 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:26.603160 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:26.604029 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:26.605865 18617 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:26.606429 18819 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:26.606376 18617 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:26.608897 18617 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
W20260808 01:53:26.611253 18806 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.193s	user 0.000s	sys 0.001s
W20260808 01:53:26.615971 18807 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260808 01:53:26.617885 18538 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:26.618222  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:26.618899 17932 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:26.619154  9476 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 25: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:26.619154 18608 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:26.620311  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:26.621246 18553 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:26.623870  9476 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: [20.12-22.13]
I20260808 01:53:26.626714  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.627562 18760 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:26.629009 18812 log.cc:927] Time spent T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260808 01:53:26.622481  9523 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:26.631230 18608 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:26.631654 18608 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:26.632038 18608 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:26.629873  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.635145 18819 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:26.636379 18819 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:26.637022 18518 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
W20260808 01:53:26.639195  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 85781.1us
I20260808 01:53:26.639968 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:26.640870 10360 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:26.649662 18631 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:26.559753 18529 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:26.561376 18809 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:26.630043 18627 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:53:26.558002 18765 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:25.910907  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:53:26.679405  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.680392  9478 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 24: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:26.683053  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:26.659608 18818 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:26.683709  9523 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:26.684348 18818 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:26.685204  9551 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260808 01:53:26.686228 17932 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260808 01:53:26.686867 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:26.689124 18617 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:26.691742 18617 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:26.693023 10360 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:26.688308 18765 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:26.694358  9600 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:26.686589 18818 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:26.697563 18745 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.616s	user 0.012s	sys 0.000s
I20260808 01:53:26.699198 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:26.700639 18747 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:26.702832 18713 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:26.704025  9634 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:26.669234 18738 consensus_queue.cc:1059] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.037s
W20260808 01:53:26.709204 18805 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.477s	user 0.002s	sys 0.000s
I20260808 01:53:26.711846 18801 raft_consensus.cc:3055] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:26.712347 18801 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:26.713393 18801 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:26.714885 18762 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:26.715508 18762 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:26.717423  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:26.717278 18605 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:26.717979 18762 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:26.719352 18529 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:26.721709  9474 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 14. (term mismatch)
I20260808 01:53:26.722187  9474 pending_rounds.cc:85] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 13
I20260808 01:53:26.722419  9474 pending_rounds.cc:107] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 21.14
I20260808 01:53:26.722645  9474 raft_consensus.cc:2889] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:26.712358 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:26.723294 18809 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:26.726995  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 214753us
I20260808 01:53:26.727690  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:26.737902  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 43637.7us
W20260808 01:53:26.732800  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 102740us
I20260808 01:53:26.740850  9476 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
I20260808 01:53:26.737304 18690 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:53:26.766875  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:26.734716  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:26.724905 18566 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:26.770131 18627 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:26.718619 18762 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:26.772733  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:26.730818 18801 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Advancing to term 24
W20260808 01:53:26.793733  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:26.802999  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.767155 18765 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election won for term 23
W20260808 01:53:26.779784  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 45206.2us
I20260808 01:53:26.812331 18824 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260808 01:53:26.839195  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 60342.1us
I20260808 01:53:26.769663 18690 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:26.775161 18698 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:26.771812 18818 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:26.799618 18605 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:26.866364 18698 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:26.867645 18695 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:26.871241  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.871789 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:26.829361 18822 raft_consensus.cc:3055] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Stepping down as leader of term 17
I20260808 01:53:26.873245  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.876122 18759 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:26.879035  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.879521 18698 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:26.780663 18566 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:26.880594  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.878297  9552 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 22. Current term is 23. Ops: []
W20260808 01:53:26.879338  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:26.873452 18822 raft_consensus.cc:740] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:26.881845  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:26.848846 18824 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:26.852002 18765 raft_consensus.cc:697] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:26.859423 18690 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:26.897375  9521 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:53:26.969846  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 72606.8us
I20260808 01:53:26.812902 18627 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:26.962325 18738 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:26.997742 18738 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:26.998200  9552 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:26.845000 18762 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:26.998974 18738 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:27.001540 18762 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:26.911021  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3756) took 1355 ms. Trace:
I20260808 01:53:27.002334  9475 rpcz_store.cc:276] 0808 01:53:25.549198 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:25.557964 (+  8766us) service_pool.cc:224] Handling call
0808 01:53:26.719879 (+1161915us) spinlock_profiling.cc:243] Waited 4.58 ms on lock 0x7b5800005498. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:53:26.720110 (+   231us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:26.720383 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:53:26.723142 (+  2759us) log.cc:844] Serialized 14 byte log entry
0808 01:53:26.883290 (+160148us) log.cc:844] Serialized 14 byte log entry
0808 01:53:26.903517 (+ 20227us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:26.903605 (+    88us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:53:26.904339 (+   734us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:26.904446 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:26.904633 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":144087,"prepare.queue_time_us":16301,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":92,"raft.queue_time_us":64097,"spinlock_wait_cycles":22405504,"thread_start_us":135553,"threads_started":1}
I20260808 01:53:27.004745  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.005829  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.006527  9475 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 27: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:26.924778  9473 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 24. Current term is 25. Ops: [24.18-24.18]
I20260808 01:53:27.007483  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:26.997124 18700 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:27.011929 18700 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:27.012362 18700 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:27.015422 18738 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:27.022208  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:26.926290 18569 consensus_meta.cc:287] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.811s	user 0.007s	sys 0.007s
I20260808 01:53:26.905179 18824 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:26.927255 18818 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:26.892910 18695 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:27.031795  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 21 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.898046 18759 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:26.934958  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.935309  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:26.936326  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:26.937472 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:27.043291  9476 raft_consensus.cc:2393] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 25: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:27.043383 10360 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 16: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:26.939278 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:53:26.940541  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:27.045686  9478 raft_consensus.cc:2393] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:26.940876 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:27.047233 10361 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:26.941205 18566 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:27.055119  9600 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:26.940873 18732 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:26.944109 18608 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:26.891772 18747 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:26.951484  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:26.951743  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:26.952528 18822 consensus_queue.cc:260] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:26.802314 18819 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.082s
I20260808 01:53:26.992765 18627 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:26.997030  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 21 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:27.033463 10332 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:27.099188  9557 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 25.
W20260808 01:53:27.099375  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 148054us
I20260808 01:53:27.100476  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:27.101843  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:27.034657  9474 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
W20260808 01:53:26.931308 18811 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.330s	user 0.001s	sys 0.000s
I20260808 01:53:27.107190 18530 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:27.035499 18695 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:27.108312 18695 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:27.108530 18691 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:27.035862 18611 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:27.111085 18611 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:27.111610 18765 consensus_queue.cc:237] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:27.052745  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:27.113637  9558 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:53:27.053521  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:27.060549  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 125348us
I20260808 01:53:27.115258  9557 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 24: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:27.062256  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:27.069512 18732 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:27.070837  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:27.071661  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:27.072346  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:27.074901 18366 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:27.030464 18483 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.045s
I20260808 01:53:27.060472  9599 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:27.087900  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:27.156913  9552 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
I20260808 01:53:27.087905  9631 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 26: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:27.132498  9554 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 27: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:53:27.136613  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:27.054862 18796 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:27.152976  9555 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 24: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:27.165453  9633 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:27.088766 18761 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260808 01:53:27.174614  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 114236us
I20260808 01:53:27.174924 18819 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:27.177392  9630 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:27.178696  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:27.179957  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:27.180761  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:27.181128  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.182797  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.184226  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.190393  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:27.190797  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:27.222735 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 53944.7us
I20260808 01:53:27.191134  9550 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:27.192314  9638 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260808 01:53:27.205951  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.209317 18695 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:27.212312 18700 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:27.054150 18762 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.080s
I20260808 01:53:27.116819 18518 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:27.218417  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:27.220230  9599 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:27.223443  9637 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:27.221712  9472 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:27.244920  9599 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:27.247067  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:27.246392 18611 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:27.250944 18675 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260808 01:53:27.252202 18688 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:27.252595 18688 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:27.253502 18688 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:27.257000 18836 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.335s	user 0.006s	sys 0.000s
I20260808 01:53:27.260715  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:27.267165  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.269080  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.272034  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:27.273247  9558 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 25: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:27.273411 18759 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:27.275075  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:27.275727  9637 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:27.164470 18366 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:27.276367 18824 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:27.277699  9556 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 24: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:27.241822 18574 consensus_queue.cc:1059] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.037s
I20260808 01:53:27.217455 18796 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:27.278990  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:27.279500 18688 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:27.243598 18518 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:27.280329 18518 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:27.280431  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:27.281163  9630 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:27.275522  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:27.285148 10332 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:27.285820  9554 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:27.270553  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.248433  9478 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
W20260808 01:53:27.295347  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45497.5us
I20260808 01:53:27.274434  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:27.299026  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:27.304039 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:27.296878  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:27.317553 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 67570.3us
I20260808 01:53:27.317843  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:27.339254  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.340204  9478 raft_consensus.cc:2393] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 24: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:27.248026 18819 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:27.352531  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:27.298228 18530 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:27.353663  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:27.354382  9552 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:27.357252  9636 raft_consensus.cc:1240] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260808 01:53:27.358844 18699 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:27.358963 18530 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:27.359354  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:27.362207  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:27.339646 18519 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260808 01:53:27.362836  9555 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:27.363106 18519 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:27.363669 18519 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 15 FOLLOWER]: Advancing to term 16
I20260808 01:53:27.364271  9555 raft_consensus.cc:3055] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:27.364635  9555 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:27.365458  9555 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:27.366425  9555 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:27.366919 18825 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:27.367337 18760 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:27.367805 18760 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:27.368778 18760 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:27.371295  9550 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 29. Current term is 30. Ops: []
I20260808 01:53:27.372783  9550 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: [23.13-23.13]
W20260808 01:53:27.378204 18830 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.449s	user 0.002s	sys 0.001s
I20260808 01:53:27.378206  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:27.380349  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:27.381516  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:27.382050 18700 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:27.384411  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.385201 18787 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:27.386584  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.385321  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 3776) took 1331 ms. Trace:
I20260808 01:53:27.390934  9479 rpcz_store.cc:276] 0808 01:53:26.053920 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:26.055120 (+  1200us) service_pool.cc:224] Handling call
0808 01:53:27.384229 (+1329109us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b5800005498. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:53:27.385219 (+   990us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8360704}
I20260808 01:53:27.315231 18827 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:27.392592 18827 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:27.382108  9633 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:53:27.393543  9475 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 15. (term mismatch)
I20260808 01:53:27.393980  9475 pending_rounds.cc:85] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 14
I20260808 01:53:27.394194  9475 pending_rounds.cc:107] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 26.15
I20260808 01:53:27.394528  9475 raft_consensus.cc:2889] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:27.395625  9637 raft_consensus.cc:2410] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 24 because replica has last-logged OpId of term: 23 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260808 01:53:27.397344  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:27.390214  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 44283us
I20260808 01:53:27.394181  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:27.406368  9634 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:27.399001  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:27.401126  9560 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27. Ops: [24.14-26.15]
W20260808 01:53:27.404572  9524 leader_election.cc:336] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.500s (SENT)
I20260808 01:53:27.375496  9550 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:27.418946  9632 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:27.402320  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 1133) took 1140 ms. Trace:
I20260808 01:53:27.424243  9629 rpcz_store.cc:276] 0808 01:53:26.262208 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:26.262391 (+   183us) service_pool.cc:224] Handling call
0808 01:53:26.947788 (+685397us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b5800231698. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d4185589e 000000000038d209 00007f5d49914940 00007f5d499dced4 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:53:26.948009 (+   221us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:27.063075 (+115066us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:27.063108 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:27.063174 (+    66us) raft_consensus.cc:1645] Marking committed up to 19
0808 01:53:27.063208 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:27.401854 (+338646us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b54000df054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:27.401926 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:27.402088 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":300752000}
I20260808 01:53:27.409548  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:27.426749  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:27.428436  9550 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: [23.12-23.12]
I20260808 01:53:27.428321  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:27.429634  9634 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:27.301874 18366 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:27.431384  9560 pending_rounds.cc:85] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 12
I20260808 01:53:27.431771  9560 pending_rounds.cc:107] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 20.13
I20260808 01:53:27.432086  9560 raft_consensus.cc:2889] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:27.344273 18162 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:27.459299  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.460924  9479 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 26: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:27.435675  9630 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:27.345510 18569 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.098s
I20260808 01:53:27.473748  9630 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:27.510840  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:27.485774  9480 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 23. Current term is 24. Ops: [23.13-23.13]
I20260808 01:53:27.485124  9629 raft_consensus.cc:1240] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:27.438997  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:53:27.472571 10365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 1019) took 1690 ms (client timeout 2043 ms). Trace:
W20260808 01:53:27.514515 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:27.516700  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3777) took 1462 ms. Trace:
I20260808 01:53:27.518229  9478 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 25: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:27.518615  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:27.520501  9552 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:27.517339  9480 rpcz_store.cc:276] 0808 01:53:26.054451 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:26.056714 (+  2263us) service_pool.cc:224] Handling call
0808 01:53:27.468419 (+1411705us) spinlock_profiling.cc:243] Waited 2.03 ms on lock 0x7b5800005498. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:53:27.485450 (+ 17031us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:27.513541 (+ 28091us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:27.516630 (+  3089us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4678528}
I20260808 01:53:27.438576 18162 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:27.526276  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:27.514588 10365 rpcz_store.cc:269] 0808 01:53:25.779258 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:25.784954 (+  5696us) service_pool.cc:224] Handling call
0808 01:53:27.465671 (+1680717us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b540046c154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:27.465830 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":232397568}
I20260808 01:53:27.468361 18700 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:53:27.468156 18605 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:27.530298  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:27.530493  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.533425  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:27.530436  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.438720 18738 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.084s
I20260808 01:53:27.543663  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:27.548071  9478 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:27.477734 18827 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:27.551425  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.551604  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.438911 18529 consensus_queue.cc:1059] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.087s
I20260808 01:53:27.552012  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:27.539337  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 460e1059f5504707b891c4b0e0ac6a09: Illegal state: Replica 6e7856bed1be43acaa821290d506ee97 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 23 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:27.591275  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:27.529982 18611 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:27.591769  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:27.527879 18801 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:27.523399 18791 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:27.593031  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:27.594412  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 189875us
I20260808 01:53:27.526160 18162 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:27.598574  9632 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 24. Current term is 25. Ops: [24.18-24.18]
I20260808 01:53:27.595270  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.598724  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:53:27.613458  9524 leader_election.cc:336] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.672s (SENT)
I20260808 01:53:27.610167 18700 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:27.625125 18846 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260808 01:53:27.630142  9554 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:27.635896  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 21 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.598773 18611 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:27.614874  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:27.647110  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 329366us
I20260808 01:53:27.618005  9480 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:27.544879 18608 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:27.582490 18695 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260808 01:53:27.614053  9524 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:27.549396 17932 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.067s
I20260808 01:53:27.617866  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:27.605273 18627 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:27.584389 18518 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:27.624265 18700 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:27.655357 18611 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:27.655809 18827 raft_consensus.cc:3055] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:27.545189 18819 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:27.616492 18801 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:27.696055  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.664871 18759 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:27.705034  9479 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 24: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:27.683787 18700 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:27.691757 18695 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:27.646446 18519 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:27.638504 18791 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:27.691042 18827 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:27.673611 18627 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:27.718259  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 104835us
I20260808 01:53:27.719089  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:53:27.734077  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:27.729296 18818 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:27.713632 18698 raft_consensus.cc:3055] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:27.742405 18162 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:27.743628 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:27.749270 10365 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:27.749436  9473 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 26: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:27.744571  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:27.747123  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:27.752113  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:27.752305  9599 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:27.752142  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:27.753242  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:27.746439  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:27.754145  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:27.755801  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:27.757803  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:27.759223  9555 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 12. (term mismatch)
I20260808 01:53:27.759546  9555 pending_rounds.cc:85] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 11
I20260808 01:53:27.759773  9555 pending_rounds.cc:107] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 20.12
I20260808 01:53:27.759958  9555 raft_consensus.cc:2889] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:27.760227  9555 pending_rounds.cc:107] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 22.13
I20260808 01:53:27.760391  9555 raft_consensus.cc:2889] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:53:27.760969  9524 leader_election.cc:336] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.752s (SENT)
I20260808 01:53:27.761950 18854 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:27.768851 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:27.768424  9560 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260808 01:53:27.771392 10367 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:27.771378  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:27.773339  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.772384  9523 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:27.774129  9551 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260808 01:53:27.709077 18759 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:27.782509  9560 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: []
W20260808 01:53:27.769836 18832 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.836s	user 0.000s	sys 0.003s
I20260808 01:53:27.793920 18831 consensus_queue.cc:1059] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:27.779781 18738 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.017s
I20260808 01:53:27.796211 18553 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260808 01:53:27.796882 18843 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:27.767086 18631 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.031s
I20260808 01:53:27.789512  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.753405 18698 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:27.791301  9552 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:27.785097  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:27.799722  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.770143 18539 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:27.773065 18818 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260808 01:53:27.807963  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46766.8us
I20260808 01:53:27.808454  9557 raft_consensus.cc:2410] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 28 because replica has last-logged OpId of term: 26 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260808 01:53:27.816140  9560 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20260808 01:53:27.823632 18688 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:27.824380 18688 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:27.827769 18574 consensus_queue.cc:1059] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260808 01:53:27.835685 18688 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:27.792886 17932 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260808 01:53:27.890403  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 105373us
I20260808 01:53:27.807989 18818 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:27.894351  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:27.891958  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:27.897961  9473 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:27.882229 17932 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:27.915375 18791 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:27.916910 18856 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260808 01:53:27.915745 18162 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:27.922963  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:27.887856 18787 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election won for term 25
W20260808 01:53:27.933607 18826 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.015s	user 0.004s	sys 0.001s
I20260808 01:53:27.937919  9629 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:27.938700  9630 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260808 01:53:27.939116 10367 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:27.895166 18827 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:27.948781 18862 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.146s	user 0.001s	sys 0.000s
I20260808 01:53:27.949071 18827 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:27.951103  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:27.954262 18162 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:27.955493 18162 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:27.938766 18700 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:27.959967 18822 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:53:27.961196 18700 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:27.960055 18162 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:27.965914  9551 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:27.884177 18539 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:53:27.967540  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 55616.5us
I20260808 01:53:27.967245  9635 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:27.942322  9629 pending_rounds.cc:85] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 14
I20260808 01:53:27.971855  9629 pending_rounds.cc:107] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 26.15
I20260808 01:53:27.972118  9629 raft_consensus.cc:2889] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:27.972716  9635 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 23. Current term is 24. Ops: []
W20260808 01:53:27.973104  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 220894us
I20260808 01:53:27.969648  9481 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
I20260808 01:53:27.974001  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:27.978170  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:27.981658  9634 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 28: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:27.897287 18818 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:27.986583  9630 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 29. Current term is 30. Ops: []
W20260808 01:53:27.998239  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 246192us
I20260808 01:53:28.001173  9601 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:27.968407 17932 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:28.018661  9635 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:27.994431 18738 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:27.975517 18529 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:27.996095 18690 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.985s	user 0.007s	sys 0.008s
I20260808 01:53:27.971261 18787 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:28.026358  9630 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260808 01:53:28.029284  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:28.029848  9478 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 25
I20260808 01:53:28.004664 18801 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:28.036314  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:27.984938 18539 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.030411 18845 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:28.027088 18738 raft_consensus.cc:697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 24 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:27.967334 18698 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.039111 18538 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:28.049789 18787 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.062333  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.029919 18529 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.043089 18819 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:28.028575 18690 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.075331 18868 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:28.078837 18864 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:28.044536 18627 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:28.062633 18845 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.084937 18824 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:28.089692 18759 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:28.090227 18865 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260808 01:53:28.091104 18698 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:28.091817 18539 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:28.092864  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 4184) took 1013 ms. Trace:
I20260808 01:53:28.115160  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.070813 18538 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:53:28.116215  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 139852us
W20260808 01:53:28.120334  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:28.135792  9557 rpcz_store.cc:276] 0808 01:53:27.079155 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:27.219972 (+140817us) service_pool.cc:224] Handling call
0808 01:53:28.092589 (+872617us) spinlock_profiling.cc:243] Waited 560 ms on lock 0x7b5400325ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:28.092785 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1288188672}
I20260808 01:53:28.121001  9552 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 29. Current term is 30. Ops: [29.16-29.16]
W20260808 01:53:28.133234 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:28.112931 18255 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:28.116524 18699 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:28.118870 18819 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.149015  9554 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260808 01:53:28.150389 18870 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260808 01:53:28.156034  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:28.155509  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 181600us
W20260808 01:53:28.160452  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:28.143188 18255 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:28.172487 18876 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:28.172626  9599 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:53:28.187220  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:28.122861 18868 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.181231 18255 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:28.162191 18699 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.123453 18864 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.155045 18872 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.032s
I20260808 01:53:28.199164 18824 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:28.207206 18819 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:28.207428 18824 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.208124 18872 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:28.210129 18824 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:28.211409  9480 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260808 01:53:28.211615 18699 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:28.199072 18801 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:28.213523 18617 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:28.214097 18617 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:28.160980 18738 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.181921 18611 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.217581 18611 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:28.213711 18801 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.224735 18801 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:28.208441 18878 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:28.201912 18690 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:28.210466  9481 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 3788) took 1824 ms (client timeout 1862 ms). Trace:
I20260808 01:53:28.233210 17932 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.229646  9478 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
W20260808 01:53:28.232515 18850 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.599s	user 0.002s	sys 0.002s
W20260808 01:53:28.233947  9481 rpcz_store.cc:269] 0808 01:53:26.386546 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:26.387700 (+  1154us) service_pool.cc:224] Handling call
0808 01:53:28.210236 (+1822536us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b54000938d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:28.210391 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3382690560}
I20260808 01:53:28.233786 18878 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:28.240665 18878 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:28.237581  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.244612 18818 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.246986 18819 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:28.248623 18864 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:28.249332  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:28.251965 18868 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:28.252601  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:28.253077 18877 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.053s
I20260808 01:53:28.254879 18569 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:28.255082  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:28.199455 18880 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:28.255463 18569 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.256036 18880 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.256578  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:28.257622 18569 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:28.257946  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:28.258632  9560 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:28.258997 18877 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:28.259616 18877 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.260738 18255 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.261947 18854 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:28.262132 18877 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:28.197846 18845 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:28.263381 18818 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:28.263216 18255 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:28.267805  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:28.267688  9599 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:28.285462  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:28.286891  9552 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 28: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:28.289582 17932 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:28.290409  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:28.291301  9559 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
W20260808 01:53:28.299710  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 92742.8us
I20260808 01:53:28.300479  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3344) took 1104 ms. Trace:
I20260808 01:53:28.300945  9635 rpcz_store.cc:276] 0808 01:53:27.195882 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:27.973935 (+778053us) service_pool.cc:224] Handling call
0808 01:53:28.006604 (+ 32669us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:28.019368 (+ 12764us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:28.300183 (+280815us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b540003b1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:28.300409 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26982656}
I20260808 01:53:28.303377  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 1149) took 1103 ms. Trace:
I20260808 01:53:28.303848  9635 rpcz_store.cc:276] 0808 01:53:27.200128 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:28.302154 (+1102026us) service_pool.cc:224] Handling call
0808 01:53:28.302636 (+   482us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:28.302917 (+   281us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:28.302953 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:28.303010 (+    57us) raft_consensus.cc:1645] Marking committed up to 16
0808 01:53:28.303050 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:28.303134 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:28.303303 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:53:28.305671  9635 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:28.306360  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3346) took 1105 ms. Trace:
I20260808 01:53:28.317369  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:28.318926  9635 rpcz_store.cc:276] 0808 01:53:27.201871 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:28.305029 (+1103158us) service_pool.cc:224] Handling call
0808 01:53:28.305447 (+   418us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:28.306084 (+   637us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:28.306297 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:53:28.272769 18880 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:28.327548  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:28.326165  9445 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:28.290905 18854 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:28.329450  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:28.330704  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 160779us
W20260808 01:53:28.318328  9524 leader_election.cc:336] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.867s (SENT)
I20260808 01:53:28.331837  9559 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:28.272581 18801 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.053s
I20260808 01:53:28.336568 18617 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.337898  9550 pending_rounds.cc:85] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 15
I20260808 01:53:28.338371  9550 pending_rounds.cc:107] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 23.16
I20260808 01:53:28.339792  9550 raft_consensus.cc:2889] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:28.338090 18617 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:28.347410  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:28.330819 18864 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:28.330574 18569 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:28.348546  9554 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:53:28.350478  9445 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:28.333163 18854 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:53:28.351862  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 322977us
I20260808 01:53:28.359014  9558 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:28.363405  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53208 (request call id 4178) took 1308 ms. Trace:
I20260808 01:53:28.363816  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:28.342777 18538 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.352046 18800 raft_consensus.cc:3055] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 LEADER]: Stepping down as leader of term 29
I20260808 01:53:28.363911  9556 rpcz_store.cc:276] 0808 01:53:27.050475 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:27.050611 (+   136us) service_pool.cc:224] Handling call
0808 01:53:28.358603 (+1307992us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b54003147d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:28.358815 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":337071872}
I20260808 01:53:28.371737  9635 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:28.352622 18864 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:53:28.373360  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 55071.2us
I20260808 01:53:28.374143  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:28.379108  9473 raft_consensus.cc:2410] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
I20260808 01:53:28.383669  9554 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: [23.12-23.12]
W20260808 01:53:28.378078  9524 leader_election.cc:336] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 2.031s (SENT)
I20260808 01:53:28.386585  9524 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.387290  9599 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.361124 18854 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:28.388584 18690 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:28.389413 18690 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:28.389894 18690 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.354024 18885 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260808 01:53:28.384481 18864 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:28.390905 18818 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260808 01:53:28.392373 18538 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:28.388613  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:28.395321  9524 leader_election.cc:336] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.735s (SENT)
I20260808 01:53:28.395658  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 1144) took 1200 ms. Trace:
I20260808 01:53:28.395776  9524 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.397017 18845 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260808 01:53:28.398536  9524 leader_election.cc:336] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.862s (SENT)
I20260808 01:53:28.386706 18566 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Leader election won for term 16
I20260808 01:53:28.399509 18566 raft_consensus.cc:697] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:53:28.399924  9524 leader_election.cc:336] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.905s (SENT)
W20260808 01:53:28.401430  9524 leader_election.cc:336] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 2.063s (SENT)
I20260808 01:53:28.400852 18566 consensus_queue.cc:237] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.374356 18800 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:28.403694 18800 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.405067 18800 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:28.406140 18690 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:28.406852  9524 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.407684 18690 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:28.357884 18569 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.396262  9629 rpcz_store.cc:276] 0808 01:53:27.195470 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:27.941495 (+746025us) service_pool.cc:224] Handling call
0808 01:53:27.942042 (+   547us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:28.045786 (+103744us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:28.045831 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:28.058775 (+ 12944us) log.cc:844] Serialized 52 byte log entry
0808 01:53:28.140657 (+ 81882us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:28.165974 (+ 25317us) log.cc:844] Serialized 14 byte log entry
0808 01:53:28.171821 (+  5847us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:28.171968 (+   147us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:28.395358 (+223390us) raft_consensus.cc:1685] finished
0808 01:53:28.395386 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:28.395581 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37521,"prepare.queue_time_us":25105,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":6309,"raft.queue_time_us":113065,"raft.run_cpu_time_us":393,"raft.run_wall_time_us":26797,"thread_start_us":47950,"threads_started":2,"wal-append.queue_time_us":188653}
I20260808 01:53:28.409456  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:28.410079 18569 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:28.376211 18887 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.040s
I20260808 01:53:28.357936 18880 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260808 01:53:28.426440  9552 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:28.430238 18831 raft_consensus.cc:3055] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:28.443296  9559 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:28.410192  9629 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:28.432266  9634 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260808 01:53:28.444970  9634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3348) took 1102 ms. Trace:
I20260808 01:53:28.445329  9634 rpcz_store.cc:276] 0808 01:53:27.342932 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:28.431528 (+1088596us) service_pool.cc:224] Handling call
0808 01:53:28.431867 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:28.444659 (+ 12792us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:28.444890 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:53:28.432667  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 60228.9us
W20260808 01:53:28.409570  9524 leader_election.cc:336] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 2.181s (SENT)
I20260808 01:53:28.447693  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:53:28.459777  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50230.2us
I20260808 01:53:28.466358  9477 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260808 01:53:28.467352  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 1151) took 1060 ms. Trace:
I20260808 01:53:28.468766 18519 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 16 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:28.470156 18713 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 22.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:53:28.473188  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 1148) took 1276 ms. Trace:
I20260808 01:53:28.468003  9629 rpcz_store.cc:276] 0808 01:53:27.407305 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:28.445121 (+1037816us) service_pool.cc:224] Handling call
0808 01:53:28.466747 (+ 21626us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:28.466953 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:28.466978 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:28.467021 (+    43us) raft_consensus.cc:1645] Marking committed up to 17
0808 01:53:28.467051 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:28.467128 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:28.467260 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:53:28.478996  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.483942  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.486135  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.473765  9630 rpcz_store.cc:276] 0808 01:53:27.198554 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:28.032628 (+834074us) service_pool.cc:224] Handling call
0808 01:53:28.032955 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:28.242376 (+209421us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:28.242422 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:28.242478 (+    56us) raft_consensus.cc:1645] Marking committed up to 16
0808 01:53:28.242520 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:28.472865 (+230345us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b540032c654. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:28.472926 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:28.473108 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":481042304}
I20260808 01:53:28.474352  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.488926  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:28.477406  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:28.489797  9559 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:28.473205 18751 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:28.490635  9521 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:28.495801  9481 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:28.496816  9478 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 24: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:28.442121 18831 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:28.501541 17932 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260808 01:53:28.498152  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:28.497504  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:28.484072  9634 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260808 01:53:28.504698  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.507148  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.507295  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.508267  9476 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:28.508447  9630 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:28.509968  9475 raft_consensus.cc:3055] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:28.510565  9475 raft_consensus.cc:740] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:28.455857 18698 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:28.471572 18880 consensus_queue.cc:1059] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.017s
W20260808 01:53:28.515681  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:28.515800  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:28.521762  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:28.529873  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:28.531054  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:28.531898  9478 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:28.535112  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.536101  9481 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:28.518476  9475 consensus_queue.cc:260] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.525820  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:28.501880 18845 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:28.547109  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:28.488831 18732 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.018s
I20260808 01:53:28.511869 18529 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:28.557901  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:28.543700  9524 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.518307 18611 raft_consensus.cc:2764] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election decision vote started in defunct term 23: won
I20260808 01:53:28.562780  9481 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260808 01:53:28.479540 18878 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.578075  9633 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:28.526652 18751 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:28.582023  9556 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:28.567931 17932 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260808 01:53:28.559427 18834 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.623s	user 0.023s	sys 0.014s
W20260808 01:53:28.591081 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:28.595544  9478 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
W20260808 01:53:28.598842  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 51889.1us
I20260808 01:53:28.599782  9601 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.600047  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:28.590549 18834 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.654s	user 0.023s	sys 0.014s
I20260808 01:53:28.603356  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:28.527971 18698 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.564702 18845 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:53:28.604803 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 127873us
I20260808 01:53:28.604934  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:28.592439 18878 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:28.604216  9481 raft_consensus.cc:2376] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27. Re-sending same reply.
I20260808 01:53:28.609295  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.558876 18824 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.043s
I20260808 01:53:28.607376 17932 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:28.616299  9478 raft_consensus.cc:2376] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25. Re-sending same reply.
I20260808 01:53:28.616791  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.607518  9524 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.612650  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:28.614332  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.560249 18854 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.582188 18751 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.603313 18566 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:28.628134  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:28.629740  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:28.630734  9629 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
I20260808 01:53:28.621475  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.609052 18569 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:28.639385 18732 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:28.626319 17932 raft_consensus.cc:3055] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:28.642881 17932 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:28.633173  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.633921  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.646641  9481 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 26: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:28.648277  9601 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:28.651897  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:28.653908  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.665545  9476 raft_consensus.cc:2393] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 28: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:28.623826 18824 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:28.661204 17932 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.668260  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:28.669883  9630 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 24: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:28.670842  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.646320 18878 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:28.672135  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:28.671923  9521 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:53:28.667289 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47945.6us
I20260808 01:53:28.673102  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:28.628674 18566 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:28.678661  9629 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 24: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:28.599650 18831 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.680260  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.660322 18519 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:28.652359 18732 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:28.663481 18538 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260808 01:53:28.694070  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 86642.3us
I20260808 01:53:28.695375 17932 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:28.696468  9524 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.683460  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.684008  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.685207  9481 raft_consensus.cc:2393] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 24: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:28.682278 18566 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:28.669667 18824 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:28.690562 18831 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:28.677402 18878 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:53:28.704692  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:28.653260 18896 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.046s
I20260808 01:53:28.700858  9524 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.686834 18611 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:28.698959 18738 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:28.720024 18738 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.721174 18230 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.047s
I20260808 01:53:28.721877 18611 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:28.722462 18611 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:28.710937 18255 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260808 01:53:28.708518 18700 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:28.716419 18878 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:28.728096 18854 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:28.730013  9524 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.730898 18699 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:28.731895  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:28.732587  9629 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:28.733008 18738 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:28.738397 18699 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260808 01:53:28.717427 18892 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260808 01:53:28.740360  9524 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.742192 18698 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:28.742746 18699 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:28.691859 18519 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:28.743840 18864 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.720130 18824 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.757529  9629 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
I20260808 01:53:28.699951 18732 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:28.759443 18894 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260808 01:53:28.758896  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:28.760890 18890 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
I20260808 01:53:28.761180  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:28.761596 18732 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:28.750607 18519 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:28.763046 18818 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:28.762008  9629 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 28: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:28.763963 18818 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.776285  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:28.773061 18834 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 29 LEADER]: Stepping down as leader of term 29
I20260808 01:53:28.786885  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 1150) took 1467 ms. Trace:
I20260808 01:53:28.790596  9635 rpcz_store.cc:276] 0808 01:53:27.319212 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:28.326183 (+1006971us) service_pool.cc:224] Handling call
0808 01:53:28.342412 (+ 16229us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:28.374869 (+ 32457us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:28.783714 (+408845us) spinlock_profiling.cc:243] Waited 29.1 ms on lock 0x7b540008d754. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:28.786781 (+  3067us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":66919680}
I20260808 01:53:28.792255  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:28.717803 18845 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.781596 18824 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:28.780714 18732 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:28.793831  9524 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.795369 17932 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:28.801754  9524 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.790380 18834 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:28.781781 18878 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.804178 18796 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:28.800258 18699 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:28.809756  9524 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.805696 17932 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:28.811944  9524 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:28.813968 18900 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260808 01:53:28.814703  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3570) took 1621 ms. Trace:
I20260808 01:53:28.815141  9632 rpcz_store.cc:276] 0808 01:53:27.193492 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:27.437728 (+244236us) service_pool.cc:224] Handling call
0808 01:53:27.467585 (+ 29857us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:27.619683 (+152098us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:28.813834 (+1194151us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b54001b49d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:28.814402 (+   568us) spinlock_profiling.cc:243] Waited 29.4 ms on lock 0x7b54001b49d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:28.814599 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":368176896}
I20260808 01:53:28.816207 18864 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:28.821888 18530 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:28.824033 18843 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:28.824671 18843 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:28.829243  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3827) took 1168 ms. Trace:
I20260808 01:53:28.829815  9480 rpcz_store.cc:276] 0808 01:53:27.660646 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:27.660833 (+   187us) service_pool.cc:224] Handling call
0808 01:53:28.210985 (+550152us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:28.211947 (+   962us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:28.828814 (+616867us) spinlock_profiling.cc:243] Waited 543 ms on lock 0x7b54000938d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:28.829159 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1248157440}
I20260808 01:53:28.809425 18818 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:28.811810 18732 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:28.844075 18566 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.832857 18699 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:28.848152 18878 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:28.849855  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:28.856739 18529 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:28.857419 18529 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.857542 18519 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.855357  9524 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.864964  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:28.866107  9524 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:28.868250  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 21 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:28.868707  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.869400  9474 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 28.
I20260808 01:53:28.868326  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.871903  9479 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:28.872692  9479 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:28.873178 18519 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:28.873519 18529 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:28.832366 18611 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.872037 18566 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:28.906822 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:28.908164 18615 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:28.909104 18615 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:28.877602  9524 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.909968 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:28.912617  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3347) took 1709 ms. Trace:
I20260808 01:53:28.913017  9633 rpcz_store.cc:276] 0808 01:53:27.206484 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:28.308419 (+1101935us) service_pool.cc:224] Handling call
0808 01:53:28.308828 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:28.578641 (+269813us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:28.912371 (+333730us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b540008c354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:28.912555 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":618630912}
I20260808 01:53:28.880345  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3182) took 1695 ms. Trace:
I20260808 01:53:28.914455  9558 rpcz_store.cc:276] 0808 01:53:27.184426 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:27.431456 (+247030us) service_pool.cc:224] Handling call
0808 01:53:28.358789 (+927333us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:28.359582 (+   793us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:28.880099 (+520517us) spinlock_profiling.cc:243] Waited 895 ms on lock 0x7b54003147d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:28.880266 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2058322816}
I20260808 01:53:28.881520 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:28.916225 18760 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:28.919649 10365 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260808 01:53:28.890467  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:28.891131  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:28.891808  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:28.895778  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 21 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:28.959198  9559 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 16.
I20260808 01:53:28.959375  9556 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:28.897634  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:28.960845  9554 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 24.
I20260808 01:53:28.902319  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3562) took 2637 ms. Trace:
I20260808 01:53:28.911106  9524 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:28.921252 18699 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:28.925051  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.926753  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.968531  9472 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 24.
I20260808 01:53:28.879113  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1572) took 2100 ms. Trace:
I20260808 01:53:28.909983 18615 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:28.980041 18611 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:28.979832 18615 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.927616 18865 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:28.962196  9636 rpcz_store.cc:276] 0808 01:53:26.265257 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:26.328262 (+ 63005us) service_pool.cc:224] Handling call
0808 01:53:26.341304 (+ 13042us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:27.357766 (+1016462us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:28.897105 (+1539339us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540027e054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:28.902207 (+  5102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2335666048}
I20260808 01:53:28.928488  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:28.985041  9474 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:53:28.930271 18539 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:28.986194 18539 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:28.987946 18796 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:28.988731 18539 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:28.989156 18712 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:28.885950  9475 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:53:28.921254  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:28.957715  9550 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 27.
I20260808 01:53:28.969810  9477 rpcz_store.cc:276] 0808 01:53:26.779083 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:26.881661 (+102578us) service_pool.cc:224] Handling call
0808 01:53:26.882058 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:28.466936 (+1584878us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:28.878703 (+411767us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b54002bc0d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:28.879012 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3629761920}
I20260808 01:53:28.932910 18809 raft_consensus.cc:3055] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:28.936051 18834 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:28.939052  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:28.997989  9633 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:53:28.997962 18834 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:28.942589  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:28.948175  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:29.000353  9630 raft_consensus.cc:2368] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 27. Current term is 28.
I20260808 01:53:28.950453  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:28.951038 17932 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260808 01:53:28.977089  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66047.2us
I20260808 01:53:28.996630  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:28.997291  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:29.005403 17932 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260808 01:53:29.003373 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 72845.7us
I20260808 01:53:29.006973  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:29.008417 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:29.009959 10360 raft_consensus.cc:2410] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 26 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260808 01:53:29.009457  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:29.011023  9478 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:28.873790  9479 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:29.003434 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 50259.8us
I20260808 01:53:29.013314  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:29.017290 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:29.019943  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:29.020946  9481 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:29.027314  9479 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:29.034365 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:29.038018 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:29.042944  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 144324us
I20260808 01:53:29.045614  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:29.047106  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:29.048125  9552 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:53:29.049722  9555 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 22. Current term is 23. Ops: []
W20260808 01:53:29.049963  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46565.6us
I20260808 01:53:29.052064  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:29.052280  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:29.053335  9559 raft_consensus.cc:2368] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 27. Current term is 28.
I20260808 01:53:29.056612 17932 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
W20260808 01:53:29.059556  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 9949664c54584858ae8fc74c9dc07a4a: Illegal state: Leader has not yet committed an operation in its own term
I20260808 01:53:29.064340 10332 leader_election.cc:400] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 27. Current term is 28.
I20260808 01:53:29.064740 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:29.064713 10332 leader_election.cc:403] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:53:29.068501 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:29.075327 10334 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:29.004812 18809 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:29.080888  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:29.082206  9600 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:29.083012  9478 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:29.084368 10360 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:29.085764 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:29.088878 17932 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:29.026294 18699 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:29.094187 10368 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260808 01:53:29.093726  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:29.046191 18901 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:29.097200 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:29.100919 10365 raft_consensus.cc:2368] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 24. Current term is 25.
I20260808 01:53:29.097292  9600 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:29.099081 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59860.4us
I20260808 01:53:28.988806 18796 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:29.116149  9481 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
I20260808 01:53:29.120175  9479 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
W20260808 01:53:29.107216  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:29.108052 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:29.102094 10334 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:29.121765  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:29.111750  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:29.122572  9476 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:53:29.125574 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:29.128473  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3803) took 1352 ms. Trace:
I20260808 01:53:29.131388 10362 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 16: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:29.131035  9479 rpcz_store.cc:276] 0808 01:53:27.775637 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:27.784001 (+  8364us) service_pool.cc:224] Handling call
0808 01:53:27.802852 (+ 18851us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:28.871734 (+1068882us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b5400190da0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490b1a0 00007f5d449955a0 00007f5d44996611 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c
0808 01:53:29.120896 (+249162us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:29.128405 (+  7509us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":417308544}
I20260808 01:53:29.096956 18569 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:29.049302 18712 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:29.116997 18895 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:29.135013 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:29.101722 18800 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Advancing to term 28
I20260808 01:53:29.064069 18865 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:29.099090 18759 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:29.160452 10365 raft_consensus.cc:2410] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 30 because replica has last-logged OpId of term: 29 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
I20260808 01:53:29.135263 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:29.155463 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:29.135012 18675 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:53:29.100520 18824 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.062s
W20260808 01:53:29.180588  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:29.170753  9475 raft_consensus.cc:1275] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260808 01:53:29.140988  9601 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:29.100571 18901 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:29.099418 18751 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260808 01:53:29.158667 18898 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260808 01:53:29.185822  9556 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 26.
I20260808 01:53:29.193349  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:29.193555  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:29.195674 10361 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
W20260808 01:53:29.135984 18631 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 26 is higher.
I20260808 01:53:29.200881 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:29.197858 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 95043.3us
I20260808 01:53:29.159504 18895 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:53:29.203641  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 62777.6us
I20260808 01:53:29.135228 18566 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:29.154976 18745 raft_consensus.cc:3055] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:29.111594 18539 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.091s
I20260808 01:53:29.096415 18809 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:29.213228 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:29.216535 18518 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:29.217901 18691 raft_consensus.cc:3055] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:29.209331  9523 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:29.220844 10369 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:29.205777 18901 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:29.277778 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:29.221078 10368 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260808 01:53:29.233248 18712 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:29.235025  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:29.283922  9637 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:29.284266  9472 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 24.
I20260808 01:53:29.236800 18796 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:29.239326 18825 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election won for term 25
W20260808 01:53:29.239517  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 924b85a2ee7e46ecbcfd57257902abef: Illegal state: Replica 259bcb3f0d254e8b9effac3277e654e0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 26 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:29.277240 18809 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:29.240101 18699 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 26.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:29.192924 18675 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:29.250852  9445 leader_election.cc:400] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 24. Current term is 25.
I20260808 01:53:29.294777  9445 leader_election.cc:403] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
W20260808 01:53:29.295615  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 45011us
I20260808 01:53:29.295536 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:29.261324 18745 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:29.258080  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 64615.6us
I20260808 01:53:29.259171 18895 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:29.262456  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:29.305608  9554 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:29.259950  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:29.267010  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:29.268352  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:29.307786  9559 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:29.271814  9552 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260808 01:53:29.272455  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:29.308635  9638 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260808 01:53:29.274290 18872 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260808 01:53:29.276176 18691 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:29.276621 18617 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260808 01:53:29.278582  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 69330.2us
I20260808 01:53:29.288777  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:29.310650  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50803.7us
I20260808 01:53:29.314013  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:29.315505  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:29.316111  9556 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 24.
I20260808 01:53:29.250146  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:29.301549  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:29.302976  9599 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:29.273487 18845 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260808 01:53:29.317584  9557 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:53:29.319834  9558 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 29. Current term is 30. Ops: [27.15-29.16]
I20260808 01:53:29.320556  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:29.322073 18872 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:29.322528 18732 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:29.322875  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:29.323349 18732 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:29.304064 18796 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.020s
I20260808 01:53:29.324581  9560 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260808 01:53:29.325232  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:29.319940  9632 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:29.327260  9633 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 27. Ops: []
I20260808 01:53:29.327811 18529 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:29.328316 18529 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:29.328780  9629 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260808 01:53:29.330961 18822 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260808 01:53:29.285714 18762 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:29.324640 18732 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:29.332082 18762 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:29.334980 18761 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:29.335641 18761 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:29.335290 18691 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:29.336457 18691 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:29.336596 18761 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:29.338029 18631 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 24. Current term is 25.
I20260808 01:53:29.338760 18825 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:29.338897 18712 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:29.339437 18712 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:29.339890 18712 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:29.339818 18825 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:29.326596  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:29.318710  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:29.311652  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:29.352308  9559 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
I20260808 01:53:29.353440  9554 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
W20260808 01:53:29.353031  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 52207.4us
I20260808 01:53:29.355911  9445 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:29.358136  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:29.360054  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:29.299530 18675 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:53:29.361853 18631 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:29.330539 18529 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:29.376459  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:29.386499  9550 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:29.387565  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:29.394160 18843 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.569s	user 0.011s	sys 0.003s
W20260808 01:53:29.395912  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 84334.8us
I20260808 01:53:29.322544 18872 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:29.402062  9637 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 3340) took 2207 ms (client timeout 2230 ms). Trace:
I20260808 01:53:29.401963 18872 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 25.21, Last appended by leader: 21, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:29.402565  9637 rpcz_store.cc:269] 0808 01:53:27.194629 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:27.398636 (+204007us) service_pool.cc:224] Handling call
0808 01:53:29.401845 (+2003209us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b54000df554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:29.401971 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4272709504}
I20260808 01:53:29.403545 18872 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:29.407774 18843 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:29.396816  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 93823.5us
I20260808 01:53:29.394439  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:29.400352 10361 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260808 01:53:29.415967  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:29.428581  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:29.410643 18843 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:29.422232 18759 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.010s
I20260808 01:53:29.432904 18745 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:29.361151 18738 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.039s
I20260808 01:53:29.400274 18751 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:53:29.400832 18627 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260808 01:53:29.400729 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:29.421276  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:29.368516 18845 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260808 01:53:29.435154  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 75214.9us
W20260808 01:53:29.485097  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 64016.5us
I20260808 01:53:29.485220  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:29.435837 18631 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260808 01:53:29.436902  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 81135.7us
I20260808 01:53:29.437597 18529 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:29.369376 18833 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:53:29.438797  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:29.506392  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:29.510314  9637 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:29.510697  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:29.511358  9475 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 30.
I20260808 01:53:29.511945  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:29.512593  9637 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 24.
I20260808 01:53:29.512742  9478 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260808 01:53:29.440826  9560 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260808 01:53:29.388330  9521 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:29.444878 18818 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:29.446722 10332 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:29.448714 18762 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:29.443463  9554 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: [23.12-23.12]
I20260808 01:53:29.396008 18865 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:29.450480 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:29.451360 18637 consensus_queue.cc:1048] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260808 01:53:29.458488 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:29.458755 18895 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:29.461483  9369 catalog_manager.cc:5697] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 19 to 21, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 21 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:29.491958  9556 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260808 01:53:29.497325  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:29.484225  9600 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:29.515116 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 68527.1us
I20260808 01:53:29.545862 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:29.518103  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:29.532042  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:29.523332  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:29.547210 10332 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:29.547669  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:29.527452  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:29.548753  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:29.527966  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:29.547415  9550 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 30.
W20260808 01:53:29.553809  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 69656.6us
I20260808 01:53:29.554881  9600 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:29.546976  9473 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:29.551132  9552 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
W20260808 01:53:29.555099  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 57878.9us
I20260808 01:53:29.555830 18519 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:29.543931 18530 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:29.562423 18519 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:29.564709 18895 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:29.546027 18877 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:29.567427 18877 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:29.538699 18745 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:29.544716 18834 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:29.574502 18834 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:29.557871 18800 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 27. Current term is 28.
I20260808 01:53:29.577865 18796 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260808 01:53:29.578943 18796 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:29.579653 18796 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 16 FOLLOWER]: Advancing to term 17
W20260808 01:53:29.583444 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:29.589609 18877 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:29.526713 18569 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election won for term 27
W20260808 01:53:29.591941 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 44806.8us
W20260808 01:53:29.593039  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:29.483574 18751 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:29.598366  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:29.604367  9480 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:29.608848 10332 leader_election.cc:400] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 27. Current term is 28.
I20260808 01:53:29.505730 18529 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:29.537582 18698 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:29.633251  9559 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:29.625453 10365 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28.
I20260808 01:53:29.627825 10332 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:29.629498  9479 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260808 01:53:29.631107  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:29.606851 18530 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:29.615547  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 61846.1us
W20260808 01:53:29.644917  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 90169us
I20260808 01:53:29.644939  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:29.602303 18747 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:29.602816 18627 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:29.611815 18751 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:53:29.633004 18529 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:29.551997 18872 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.103s
I20260808 01:53:29.653265  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:29.609884 18913 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:29.615991 18617 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:29.664507  9552 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:29.649525  9600 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:29.683245  9550 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260808 01:53:29.642396 18637 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:29.610592 18876 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.057s
I20260808 01:53:29.643653 18569 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:29.696772  9599 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:29.661792 18698 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election won for term 26
W20260808 01:53:29.703500  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 54089.1us
I20260808 01:53:29.666884 18759 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260808 01:53:29.685684 18834 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:29.669883 18913 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:29.693553 18755 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election won for term 23
W20260808 01:53:29.742877  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46240.2us
I20260808 01:53:29.745349  9599 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:29.748837 18878 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260808 01:53:29.749626 18878 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:29.750193 18878 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:53:29.757395 18530 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:29.701248 18627 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:29.764277 18627 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:29.766530  9369 catalog_manager.cc:5697] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 16 to 22, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 22 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:29.699028  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:53:29.769362 18569 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:29.693794 18637 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:29.722385 18912 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260808 01:53:29.776427 18755 raft_consensus.cc:697] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:53:29.781219  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 76725.3us
I20260808 01:53:29.782197  9600 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:29.780419  9523 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:53:29.790169  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:29.800305 18712 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:29.743856 18698 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:53:29.815949 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 188218us
I20260808 01:53:29.825127 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:29.826404  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 81186.1us
I20260808 01:53:29.745543 18913 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:29.827240  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:29.777918 18755 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:29.807971 18759 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260808 01:53:29.808427 18611 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election won for term 28
W20260808 01:53:29.832129  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:29.833047  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 444823us
I20260808 01:53:29.855207  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:29.843067  9480 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 25.
W20260808 01:53:29.854494 18901 consensus_meta.cc:287] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.549s	user 0.008s	sys 0.008s
I20260808 01:53:29.864548 18901 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:29.866130 10366 raft_consensus.cc:2368] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 28. Current term is 29.
I20260808 01:53:29.854707 18865 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:29.867172 18865 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:29.867695 18865 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:29.862282 10334 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:29.853631 18903 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:29.853983 18566 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:29.868294  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:29.894008  9550 raft_consensus.cc:2410] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 26 because replica has last-logged OpId of term: 24 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 17.
W20260808 01:53:29.871752  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:29.895011  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:29.897192  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:29.897905  9554 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:29.898970  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:29.900493 10361 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:29.873545 18637 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:29.878583 18700 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:29.912616  9523 leader_election.cc:400] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 28. Current term is 29.
I20260808 01:53:29.913141  9523 leader_election.cc:403] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:53:29.807394 18761 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.084s
W20260808 01:53:29.915086 18675 consensus_meta.cc:287] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.530s	user 0.013s	sys 0.002s
I20260808 01:53:29.915620 18675 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:29.916678 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 54530.4us
W20260808 01:53:29.911499 18809 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.605s	user 0.005s	sys 0.009s
I20260808 01:53:29.871215 18605 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.047s
I20260808 01:53:29.918112 18895 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:29.918737 18895 raft_consensus.cc:697] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:29.919673 18895 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:29.919770 18698 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 25.21, Last appended by leader: 21, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:29.923269 18901 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:29.926950 10368 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:53:29.927189  9476 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:29.938230  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:29.939086  9559 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:29.952642  9554 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 25.
I20260808 01:53:29.887286 18566 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:29.960981  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:29.880338 18627 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:29.872684 18712 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:29.873211 18903 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:29.966090  9555 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:29.890887 18611 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:29.947083 18700 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 29
W20260808 01:53:29.976006  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:29.861681 18896 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.105s
I20260808 01:53:30.000144 10361 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260808 01:53:29.878111 18796 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:29.969097 18566 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:29.926568 18529 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:29.954779 18615 consensus_queue.cc:1059] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.049s
I20260808 01:53:29.999805 18903 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:30.019088  9480 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0ea307d9adc442539683547d0176a05b"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:29.977370 18760 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:30.025826  9480 raft_consensus.cc:606] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:30.027726  9480 raft_consensus.cc:614] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:53:30.029222  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:30.031843 10366 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260808 01:53:30.031945  9474 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 24. Current term is 25. Ops: []
W20260808 01:53:30.034394  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:30.021170 18675 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:30.025188 18691 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:30.084555  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:30.090134  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 0ea307d9adc442539683547d0176a05b (from 6e7856bed1be43acaa821290d506ee97 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
W20260808 01:53:30.097601  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 122167us
I20260808 01:53:30.038005 18761 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:30.104137  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:30.102283  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 135453us
I20260808 01:53:30.025671 18909 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.064s
I20260808 01:53:30.106055  9599 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:30.087960 18825 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:30.031642 18760 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:30.135322  9480 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:30.136093  9480 raft_consensus.cc:606] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:30.136550  9480 raft_consensus.cc:614] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:30.136874  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:30.097698 18691 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:30.103332 18637 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.025s
W20260808 01:53:30.159580 18928 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260808 01:53:30.163903 18529 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:30.123992 18825 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:30.134011 18690 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:30.165836 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:53:30.167027 18922 log.cc:927] Time spent T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260808 01:53:30.165000  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:30.177249 18700 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 28. Current term is 29.
I20260808 01:53:30.168582 18948 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260808 01:53:30.179525  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 2f740cd3515641a8b2c4add8f809dd82 (from 6e7856bed1be43acaa821290d506ee97 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:30.102110 18751 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:30.055699 18611 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:30.181244  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:30.116669 18761 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:30.220778  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 115020us
W20260808 01:53:30.218848  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 105083us
I20260808 01:53:30.221655 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:30.169540 18796 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:30.168612 18760 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:30.222599 10363 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 30.
I20260808 01:53:30.217684 18948 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:53:30.239751  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 582102us
I20260808 01:53:30.240638  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:30.231086  9599 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:30.247368 10369 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:57156
I20260808 01:53:30.251116 10369 raft_consensus.cc:606] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:30.213881 18690 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:30.229425 18878 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:30.255960 18834 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:30.236637 18948 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
W20260808 01:53:30.260135  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:30.283408  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:30.265805  9555 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 29.
I20260808 01:53:30.286139  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:30.287338  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:30.288051  9475 raft_consensus.cc:3060] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:30.246198 18747 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:30.270073 18937 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260808 01:53:30.291780  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 56338.2us
I20260808 01:53:30.277107 18690 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:30.278419 18617 consensus_meta.cc:287] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.588s	user 0.015s	sys 0.000s
W20260808 01:53:30.262488  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 89171.1us
I20260808 01:53:30.280112 18751 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:30.285073 18530 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:30.288964  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:30.293512 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:30.295728  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:30.301774  9480 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:53:30.309537 18617 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:30.295152 18937 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:30.310662 18937 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:30.311439  9475 raft_consensus.cc:2468] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 29.
I20260808 01:53:30.315261 18745 raft_consensus.cc:2764] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260808 01:53:30.316265 18530 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:30.323061 18939 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260808 01:53:30.326104  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:30.335510 18825 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:30.327178 18872 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:30.327500 18809 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260808 01:53:30.323123  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:30.339203  9557 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 30.
I20260808 01:53:30.257648 18755 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.041s
I20260808 01:53:30.289614 18834 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:30.290592 18553 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 8, Time since last communication: 0.041s
I20260808 01:53:30.342913  9476 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:30.343622  9476 raft_consensus.cc:606] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:30.343971  9476 raft_consensus.cc:614] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:30.345893 10334 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:30.346397  9480 raft_consensus.cc:2468] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:53:30.332360 18747 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:30.335096  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:30.338423 18843 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:30.340929  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:30.351284 18529 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:30.363811 18530 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:30.373297  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:30.301765 18929 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
I20260808 01:53:30.382067 10334 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:30.381524  9476 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260808 01:53:30.386641  9557 raft_consensus.cc:3055] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:30.387540  9551 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 30: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:30.391837 18519 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:30.393790 18800 raft_consensus.cc:2804] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election won for term 19
W20260808 01:53:30.375520 18865 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.507s	user 0.003s	sys 0.011s
W20260808 01:53:30.395615  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 7b5ea129398a42fb9981fca652027cd6 (from 6e7856bed1be43acaa821290d506ee97 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:30.396509 18691 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:30.396680  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1592) took 1053 ms. Trace:
I20260808 01:53:30.340016 18745 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.026s
I20260808 01:53:30.387116  9557 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:53:30.403198  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 53858.4us
I20260808 01:53:30.366302 18901 raft_consensus.cc:3055] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 LEADER]: Stepping down as leader of term 21
I20260808 01:53:30.366568 18801 raft_consensus.cc:2804] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader election won for term 29
I20260808 01:53:30.412709 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:30.408244  9477 rpcz_store.cc:276] 0808 01:53:29.343610 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:29.343778 (+   168us) service_pool.cc:224] Handling call
0808 01:53:29.368439 (+ 24661us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:29.399662 (+ 31223us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:29.399714 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:29.467134 (+ 67420us) log.cc:844] Serialized 26 byte log entry
0808 01:53:29.851504 (+384370us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:29.851556 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:29.852499 (+   943us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:30.396428 (+543929us) raft_consensus.cc:1685] finished
0808 01:53:30.396450 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:30.396611 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29704,"raft.queue_time_us":29988,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":106,"thread_start_us":383748,"threads_started":1,"wal-append.queue_time_us":670056}
I20260808 01:53:30.417243  9557 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:30.407162 18529 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:30.420876 18913 consensus_meta.cc:287] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.559s	user 0.009s	sys 0.005s
I20260808 01:53:30.439325  9557 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:30.377332 18834 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:30.416704 18843 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:30.416323 18800 raft_consensus.cc:697] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:30.444154 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:30.418283 18901 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:30.441687  9600 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:30.441411 18745 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:30.411695 18566 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:30.455943  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 158751us
I20260808 01:53:30.416507 18865 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:30.438241 18801 raft_consensus.cc:697] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:30.446203  9551 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 25 index: 21. Preceding OpId from leader: term: 26 index: 22. (index mismatch)
I20260808 01:53:30.447523 18913 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:30.482092 18699 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:30.482582 10365 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 24. Current term is 25. Ops: [24.18-24.18]
I20260808 01:53:30.483083 18699 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:30.473738 18762 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:53:30.486826 18762 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:30.487336 18762 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:30.487727 18834 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:30.490895  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:30.506162  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 165319us
I20260808 01:53:30.509788 18834 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:30.512941  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:30.515583  9552 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 27. Ops: []
I20260808 01:53:30.517138  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:30.514536  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:30.519709  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:30.521489 18834 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:30.523326  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:30.526003  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:30.528535 18553 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:30.531922  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:30.532908  9556 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260808 01:53:30.534202 18530 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:30.536149 18530 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:53:30.539649  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:30.541380  9556 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:30.544157  9444 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:30.545683 18631 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260808 01:53:30.547102 18631 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:30.547336  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:30.546236  9556 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:30.550429  9560 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260808 01:53:30.468039 18843 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:30.471858 18691 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260808 01:53:30.474798 18927 raft_consensus.cc:3055] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:30.561779  9556 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:30.472289 18745 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:30.443617 18529 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:30.538151 18530 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.013s
W20260808 01:53:30.569657  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 52003.8us
W20260808 01:53:30.558234 18933 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260808 01:53:30.570087 18963 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:30.565932 18691 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:30.571192 18927 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:30.499042 18698 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.105s
I20260808 01:53:30.565371 18566 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:30.622112  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:30.501080 18877 consensus_queue.cc:1059] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.109s
I20260808 01:53:30.612872 18530 consensus_queue.cc:1243] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 27.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260808 01:53:30.633966  9474 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260808 01:53:30.618382  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:30.575472 18690 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:30.609009 18926 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:30.615398 18878 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:30.621116 18963 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:30.627614 18691 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:53:30.579414 18834 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.055s
I20260808 01:53:30.650417  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:30.657457  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:30.643347 18877 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:30.668531 18877 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:30.655547 18903 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.619s	user 0.012s	sys 0.003s
I20260808 01:53:30.614547 18901 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:30.664007 18529 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:30.651372 18878 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:30.653153 18843 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:30.665275  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:30.632174 18631 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.023s
I20260808 01:53:30.633551 18801 consensus_queue.cc:237] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:30.644618 18926 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:30.644919 18690 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:30.661070  9476 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:30.646971 18963 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:30.697917  9557 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260808 01:53:30.620575 18800 consensus_queue.cc:237] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:30.726146  9476 raft_consensus.cc:606] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:30.707062 18903 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:30.742964  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:30.732968 18960 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260808 01:53:30.733897 18259 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.125s
I20260808 01:53:30.742532 18948 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:30.735762 18962 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260808 01:53:30.658653 18927 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:30.754501 18927 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:30.741089 18690 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:30.745404 18865 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:30.723806 18878 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:30.722373 18699 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:30.770079  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 156778us
I20260808 01:53:30.781781 18878 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:30.767283 18617 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:30.783906 18617 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:30.787181 18948 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:30.796913 18865 raft_consensus.cc:3055] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Stepping down as leader of term 16
I20260808 01:53:30.797685 18865 raft_consensus.cc:740] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:30.789247 18926 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:30.819182  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:30.819919  9560 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:53:30.821651  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:30.822738  9559 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:30.824931  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:30.799000 18865 consensus_queue.cc:260] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:30.836468  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:30.837193  9554 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 24
I20260808 01:53:30.837713  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:30.833093  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:30.834442  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:30.806442 18877 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:30.839722  9638 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:30.845927  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:30.846747  9637 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:30.850075  9476 raft_consensus.cc:1240] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 18. Current term is 19. Ops: []
I20260808 01:53:30.851246  9475 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:30.850551  9559 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:30.854251  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:30.854460  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:30.855357  9479 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:30.856369  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:30.811475 18975 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:53:30.858582  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:30.859738  9475 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260808 01:53:30.873684  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:30.880584 18903 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:30.881434  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:30.882138  9559 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:30.885051  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:30.886874  9633 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: [23.12-23.12]
I20260808 01:53:30.887104  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:30.887882  9632 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:30.860603 18865 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 16 FOLLOWER]: Advancing to term 17
I20260808 01:53:30.890122  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:30.892046  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:30.893998  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:30.899232  9638 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:30.901026  9631 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27. Ops: [26.14-26.14]
I20260808 01:53:30.902537  9550 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27. Ops: [26.14-26.14]
I20260808 01:53:30.906971  9559 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 25.
I20260808 01:53:30.840981 18747 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:30.881290 18952 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:30.830910 18878 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.066s
W20260808 01:53:30.906487 18937 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.595s	user 0.015s	sys 0.001s
I20260808 01:53:30.903018 18617 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:30.939141 18937 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:30.941263  9629 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260808 01:53:30.914754  9524 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:30.914707  9554 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:30.921090  9637 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:30.913702 18675 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:30.921868 18952 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:30.943967 18675 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:30.945497 18952 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:30.950546 18675 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:30.951156 18923 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.072s	user 0.007s	sys 0.000s
I20260808 01:53:30.953327 18834 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:30.953812  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 3848) took 1329 ms. Trace:
I20260808 01:53:30.945996 18611 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.027s
I20260808 01:53:30.955616 18611 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:30.956393 18611 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:53:30.959707  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 45082.5us
I20260808 01:53:30.960608 18963 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:30.963049 18699 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:30.963809 18699 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:30.964437 18699 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:30.967311 18845 consensus_queue.cc:1059] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:30.954260  9481 rpcz_store.cc:276] 0808 01:53:29.624669 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:29.644564 (+ 19895us) service_pool.cc:224] Handling call
0808 01:53:30.953569 (+1309005us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54002dc3d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:30.953736 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2447509248}
I20260808 01:53:30.973273  9481 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260808 01:53:30.973770  9369 catalog_manager.cc:5697] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 13 to 19, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 19 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:30.974566  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:30.975132 10369 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260808 01:53:30.979943  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:30.985417 18529 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:30.985692 18907 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:53:30.986915 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:30.986152 18529 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:30.992213  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:30.996079  9369 catalog_manager.cc:5697] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 19 to 22, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 22 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:30.999190  9554 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:31.009320 10368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 1149) took 1257 ms. Trace:
I20260808 01:53:31.010125 18761 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:31.009768 10368 rpcz_store.cc:276] 0808 01:53:29.752449 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:29.752686 (+   237us) service_pool.cc:224] Handling call
0808 01:53:29.753069 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:29.943063 (+189994us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:31.009004 (+1065941us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b5400499654. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:31.009221 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":399255424}
I20260808 01:53:30.995456 18529 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:31.008750 18963 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:31.022760  9479 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 25 index: 21. Preceding OpId from leader: term: 26 index: 22. (index mismatch)
W20260808 01:53:31.023651 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:30.953953 18834 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:30.860476 18762 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:30.989352 18631 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:31.058096  9560 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 31.
I20260808 01:53:31.001358 18809 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:30.957870 18611 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:31.064682  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 58323.5us
I20260808 01:53:31.034307 18937 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:31.071966  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:30.920591 18747 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:30.916611 18975 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:31.069389  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:31.059839 18713 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:31.059527 18971 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:31.100885  9474 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:31.102383 18845 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.002s
I20260808 01:53:31.099962 18713 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:31.109968 18713 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:31.113168  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:31.113296  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:31.114288  9554 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:31.125249 18901 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:31.148164  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:31.128288 18996 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:31.128783 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:31.155071 10369 raft_consensus.cc:2410] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 24 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260808 01:53:31.066727 18675 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.034s
I20260808 01:53:31.130708 10361 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:31.131665 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:31.101959 18971 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:31.134406 18762 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:31.137045  9629 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:31.067229 18631 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:31.137923  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:31.170197  9637 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:31.139256 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:31.065610 18963 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 8, Time since last communication: 0.035s
I20260808 01:53:31.097545 18611 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:31.094980 18809 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:31.083952 18903 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.032s
W20260808 01:53:31.152328 18922 log.cc:927] Time spent T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260808 01:53:31.099800 18965 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:31.122290 18745 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.208218 18937 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:31.209236 18745 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:31.193672  9523 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:31.209333 18937 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.160921 18566 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:31.153316 18824 consensus_queue.cc:1059] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.042s
I20260808 01:53:31.259591  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:31.194648  9554 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:31.248615  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:31.266112  9638 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 18. Current term is 19. Ops: []
I20260808 01:53:31.192134 18972 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:31.192588 18958 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:31.279304  9631 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 21. Current term is 22. Ops: []
I20260808 01:53:31.212803 18955 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:31.252555 18927 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:53:31.293087 18366 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:31.300478 19002 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:31.301285 18366 raft_consensus.cc:697] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:31.300130  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:31.291867  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:31.308305  9638 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 30.
I20260808 01:53:31.284040  9637 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
W20260808 01:53:31.213554 18963 consensus_queue.cc:1175] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "259bcb3f0d254e8b9effac3277e654e0" responder_term: 27 status { last_received { term: 24 index: 13 } last_committed_idx: 13 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27. Ops: [26.14-26.14]" } } last_received_current_leader { term: 0 index: 0 } }
I20260808 01:53:31.195524 18809 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:31.309154  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 170106us
I20260808 01:53:31.275898 18759 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:31.316386  9554 raft_consensus.cc:3055] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:53:31.319819  9554 raft_consensus.cc:740] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:31.323716  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:31.324229  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:31.320719  9554 consensus_queue.cc:260] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:31.272007 18566 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.336637  9638 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:31.337832  9554 raft_consensus.cc:3060] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:53:31.318889 18834 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:31.352687 18631 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:31.355508 18971 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:31.355330 18843 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:31.357069 18843 consensus_queue.cc:1059] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:31.301784 19002 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.361096 18529 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.363107  9477 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 27. Current term is 28. Ops: [27.14-27.14]
I20260808 01:53:31.363260 19002 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:31.363503 18631 raft_consensus.cc:3055] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:31.364179 18631 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:31.364892  9638 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:31.366117  9633 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260808 01:53:31.366289  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:31.366446  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:31.367118  9638 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:31.367053  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:31.368063  9630 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:31.367884 18631 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.375005  9472 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 14. (term mismatch)
I20260808 01:53:31.375686  9472 pending_rounds.cc:85] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 13
I20260808 01:53:31.376000  9472 pending_rounds.cc:107] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 26.14
I20260808 01:53:31.376319  9472 raft_consensus.cc:2889] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:53:31.385332 18991 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.246s	user 0.000s	sys 0.001s
I20260808 01:53:31.388391 18566 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:31.328150 18759 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:31.392341 18759 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:31.394770 10367 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260808 01:53:31.395215 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:31.395991 10369 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:31.394847 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:31.295792 18972 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:31.398305  9600 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:31.399277 18972 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:31.399933 18972 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260808 01:53:31.400319 18566 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:31.400776 18566 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:31.401206 18566 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:31.400969 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:31.402583 18937 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:31.310868 18955 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:31.404351  9634 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:31.405535 18955 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:31.407485 18699 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.409926 18699 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:31.305398 18958 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:31.411967 18938 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.271s	user 0.005s	sys 0.000s
I20260808 01:53:31.413180 18958 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:31.415400 18809 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:31.418145 18975 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:31.419308 18999 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
I20260808 01:53:31.420540 18913 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:31.423091  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:31.430088 10365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 1035) took 1650 ms. Trace:
W20260808 01:53:31.431754  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 65400.3us
I20260808 01:53:31.431623 10365 rpcz_store.cc:276] 0808 01:53:29.779911 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:29.780389 (+   478us) service_pool.cc:224] Handling call
0808 01:53:29.780865 (+   476us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:30.483230 (+702365us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:31.429608 (+946378us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x7b5400681d54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:31.429979 (+   371us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1612934016}
I20260808 01:53:31.438108 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:31.439138 10365 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 25: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:31.439200  9634 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 25.
I20260808 01:53:31.445601 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:31.450317 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:31.452039 18628 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:31.452029 18747 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:31.347268 18366 consensus_queue.cc:237] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:31.460835  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 320105us
W20260808 01:53:31.346480 18569 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.549s	user 0.017s	sys 0.014s
I20260808 01:53:31.467921  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:31.468704 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:31.473955  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:31.478260 18691 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.821s	user 0.014s	sys 0.003s
I20260808 01:53:31.465315 18553 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Stepping down as leader of term 27
I20260808 01:53:31.479007 18553 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:31.478930 18691 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:31.480365 10368 raft_consensus.cc:2368] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 30. Current term is 31.
W20260808 01:53:31.466104 18569 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.668s	user 0.017s	sys 0.014s
W20260808 01:53:31.484722  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 159860us
I20260808 01:53:31.480264 18553 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.492064 18553 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:31.494144 18529 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:31.499264  9555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 1367) took 1363 ms (client timeout 1729 ms). Trace:
I20260808 01:53:31.499643  9552 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
W20260808 01:53:31.499780  9555 rpcz_store.cc:269] 0808 01:53:30.135764 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:30.136432 (+   668us) service_pool.cc:224] Handling call
0808 01:53:31.499024 (+1362592us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b54003269d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:31.499192 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":36138368}
I20260808 01:53:31.521754  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:31.538558  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:31.540303  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:31.538717  9555 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:53:31.499049 18566 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:31.524350  9369 catalog_manager.cc:5697] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 17 to 25. New cstate: current_term: 25 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:31.519594 18691 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 31 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:31.504415 18865 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.615s	user 0.014s	sys 0.002s
I20260808 01:53:31.544750  9369 catalog_manager.cc:5697] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 16 to 22, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 22 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:31.508481 18611 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:31.545219  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:31.551199  9629 raft_consensus.cc:2410] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260808 01:53:31.552856  9630 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260808 01:53:31.556893  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:31.556748  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:31.560798  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:31.562419  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:31.564158  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:31.564823  9633 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:31.566473  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:31.567168  9556 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:31.567014  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:31.547776 18800 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.011s	sys 0.004s
I20260808 01:53:31.504086 18996 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:31.580992  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:31.577224  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:31.582477 19011 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.017s
I20260808 01:53:31.586812  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:31.584609  9369 catalog_manager.cc:5697] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 17 to 22. New cstate: current_term: 22 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:31.565491 18529 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election won for term 25
W20260808 01:53:31.576467 18800 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.761s	user 0.011s	sys 0.004s
I20260808 01:53:31.585048  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:31.590549  9630 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:31.583616 18834 raft_consensus.cc:3055] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:31.573163 18747 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:31.597263  9369 catalog_manager.cc:5697] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:31.580013 18865 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:31.565927 18611 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.602866  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:31.605742  9632 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Advancing to term 31
W20260808 01:53:31.607798 19013 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260808 01:53:31.605113 18747 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:31.569998 19009 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
I20260808 01:53:31.611498 19009 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:31.612002 19009 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:31.613217 18611 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:31.612972 19009 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.615129 18865 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.599714 18529 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:31.619589 18529 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.584424 19008 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:31.621371 19008 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.626204 18907 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:31.630977  9556 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:31.635907  9632 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 31.
I20260808 01:53:31.637319  9555 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 31.
I20260808 01:53:31.584739 18913 consensus_queue.cc:1059] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 18.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.027s
I20260808 01:53:31.600008 18834 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:31.650578  9369 catalog_manager.cc:5697] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 17 to 23. New cstate: current_term: 23 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:31.596889 19011 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.056s
I20260808 01:53:31.650710 19009 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
W20260808 01:53:31.664191 18982 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.905s	user 0.005s	sys 0.000s
I20260808 01:53:31.665951 18747 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:31.667497 19016 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260808 01:53:31.673544 18834 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:31.682503 10334 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:31.650182 18907 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:31.691030  9599 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:31.638367 18952 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260808 01:53:31.611999  9367 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50064 (request call id 240) took 75070 ms (client timeout 15000 ms). Trace:
I20260808 01:53:31.692292  9599 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:31.709349  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:31.710770  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:31.712167  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:31.712878 18818 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader election won for term 31
I20260808 01:53:31.713135  9557 raft_consensus.cc:2410] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 27 because replica has last-logged OpId of term: 26 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 14.
W20260808 01:53:31.714951  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:31.715911  9521 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:31.761050  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:31.713140 18939 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:31.716216 18818 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:31.755050 18843 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:31.776175  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:31.716820 19014 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20260808 01:53:31.780808  9521 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:31.761793  9558 raft_consensus.cc:2368] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 30. Current term is 31.
I20260808 01:53:31.702266 18834 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:31.767887  9550 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:31.783161 18824 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:31.774925 18566 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:31.692258 18996 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.713698 18611 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:31.787947 18611 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.791695  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:31.792202  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:31.792518  9475 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:53:31.793437  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:31.794240  9558 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 25: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:31.754801 18699 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:31.783860  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:31.771744 18843 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.795534 18699 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.796202  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:31.797036  9550 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 25: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:31.795600  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:31.768287 18818 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.797045  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:31.717684 18761 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:31.793442  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:31.807879 18761 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:31.807215 18907 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:31.714603  9367 rpcz_store.cc:269] 0808 01:52:16.541789 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:16.618913 (+ 77124us) service_pool.cc:224] Handling call
0808 01:53:31.611470 (+74992557us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:53:19.478853 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:53:19.664212 (+185359us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:53:19.678472 (+ 14260us) write_op.cc:435] Acquired schema lock
0808 01:53:19.682765 (+  4293us) tablet.cc:662] Decoding operations
0808 01:53:20.284837 (+602072us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:53:20.284957 (+   120us) write_op.cc:641] Partition lock acquired for write op
0808 01:53:20.284988 (+    31us) tablet.cc:685] Acquiring locks for 19 operations
0808 01:53:20.292809 (+  7821us) tablet.cc:701] Row locks acquired
0808 01:53:20.299829 (+  7020us) write_op.cc:260] PREPARE: finished
0808 01:53:20.335601 (+ 35772us) write_op.cc:270] Start()
0808 01:53:20.360274 (+ 24673us) write_op.cc:276] Timestamp: P: 1786154000318130 usec, L: 0
0808 01:53:20.376759 (+ 16485us) op_driver.cc:348] REPLICATION: starting
0808 01:53:20.377627 (+   868us) log.cc:844] Serialized 7522 byte log entry
0808 01:53:21.886878 (+1509251us) op_driver.cc:464] REPLICATION: finished
0808 01:53:22.568765 (+681887us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4186fa5c 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d483e98d0
0808 01:53:23.314129 (+745364us) write_op.cc:301] APPLY: starting
0808 01:53:23.335804 (+ 21675us) tablet.cc:1367] starting BulkCheckPresence
0808 01:53:24.198576 (+862772us) tablet.cc:1370] finished BulkCheckPresence
0808 01:53:24.205666 (+  7090us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:53:24.932055 (+726389us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:53:24.932481 (+   426us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=38,key_file_lookups=38,delta_file_lookups=0,mrs_lookups=0
0808 01:53:24.932552 (+    71us) write_op.cc:312] APPLY: finished
0808 01:53:24.933064 (+   512us) log.cc:844] Serialized 172 byte log entry
0808 01:53:24.944025 (+ 10961us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:53:24.944191 (+   166us) write_op.cc:454] Released schema lock
0808 01:53:24.944255 (+    64us) write_op.cc:341] FINISH: Updating metrics
0808 01:53:24.944676 (+   421us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:53:24.944697 (+    21us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":604837,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":1288290,"apply.run_cpu_time_us":47888,"apply.run_wall_time_us":1663628,"cfile_cache_hit":78,"cfile_cache_hit_bytes":83661,"cfile_init":1,"lbm_read_time_us":671,"lbm_reads_lt_1ms":4,"num_ops":19,"prepare.queue_time_us":441976,"prepare.run_cpu_time_us":27268,"prepare.run_wall_time_us":1220514,"raft.queue_time_us":671131,"raft.run_cpu_time_us":59092,"raft.run_wall_time_us":1352718,"replication_time_us":1458866,"spinlock_wait_cycles":426872064,"thread_start_us":2080288,"threads_started":5,"wal-append.queue_time_us":223560,"wal-append.run_cpu_time_us":31136,"wal-append.run_wall_time_us":3092231}]]}
I20260808 01:53:31.811655 18926 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:31.808063  9475 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:31.817987  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:31.761231 19001 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260808 01:53:31.791793  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:31.830793  9557 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:31.818271  9369 catalog_manager.cc:5697] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 11 to 21, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 21 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:31.718904 19005 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:31.820142 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:31.844123 19005 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.816973 18926 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:31.851611  9475 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:31.859763  9558 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260808 01:53:31.874610  9476 raft_consensus.cc:1275] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260808 01:53:31.883004 18699 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:31.883518 19005 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:31.883841 18895 raft_consensus.cc:993] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: : Instructing follower 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:53:31.884442 18913 raft_consensus.cc:1081] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEADER]: Signalling peer 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:53:31.885128  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:31.889510 18761 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.877687 10332 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:31.848058 18691 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:31.899073  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:31.900254  9558 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 23: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:31.896206  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:31.835307 18809 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:31.906735 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:31.905040  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:31.918262  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:31.924139  9630 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:31.902114 18747 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:31.924810  9369 catalog_manager.cc:5697] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 11 to 21, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 21 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:31.876502 18963 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.039s
I20260808 01:53:31.909466 18895 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:31.932410  9630 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: [23.12-23.12]
I20260808 01:53:31.908377 18691 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:31.931769  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:31.929946  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:31.932919  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:31.912492 19005 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:31.929045  9369 catalog_manager.cc:5697] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 12 to 23, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 23 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:31.919269 18843 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:31.947325  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:31.948956  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:31.925320 19009 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:31.907956 18913 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.025s
I20260808 01:53:31.927296 18747 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:31.965256  9599 leader_election.cc:400] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 30. Current term is 31.
I20260808 01:53:31.937142 18824 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:31.969462 18824 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:31.972185 18824 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:31.906798 18926 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:31.932142 19011 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.007s
I20260808 01:53:31.937677 19008 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:31.968155  9599 leader_election.cc:403] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: Cancelling election due to peer responding with higher term
I20260808 01:53:31.981614  9369 catalog_manager.cc:5697] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 11 to 22, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 22 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:31.962353  9633 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
W20260808 01:53:31.996833  9367 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50084 (request call id 237) took 24082 ms (client timeout 15000 ms). Trace:
W20260808 01:53:31.997301  9367 rpcz_store.cc:269] 0808 01:53:07.891395 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:31.947201 (+24055806us) service_pool.cc:211] Skipping call since client already timed out
0808 01:53:31.973370 (+ 26169us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260808 01:53:32.000669  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:32.003125 18566 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:32.002473  9369 catalog_manager.cc:5697] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 16 to 19, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 19 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:32.002488  9599 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:31.968003 18611 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:31.980809 18747 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:32.005360  9369 catalog_manager.cc:5697] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 11 to 25, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 25 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:32.002717 18953 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader election lost for term 30. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 30. Current term is 31.
I20260808 01:53:32.007213 18975 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:32.009727  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:32.010051  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3865) took 1005 ms. Trace:
I20260808 01:53:32.033515  9481 rpcz_store.cc:276] 0808 01:53:31.005074 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:31.005261 (+   187us) service_pool.cc:224] Handling call
0808 01:53:31.016996 (+ 11735us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:31.248811 (+231815us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:31.248843 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:31.275097 (+ 26254us) log.cc:844] Serialized 26 byte log entry
0808 01:53:31.542711 (+267614us) raft_consensus.cc:1645] Marking committed up to 17
0808 01:53:31.542756 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:32.009615 (+466859us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b540009a6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:32.009666 (+    51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:32.009716 (+    50us) raft_consensus.cc:1685] finished
0808 01:53:32.009730 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:32.009964 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":142891,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":153,"spinlock_wait_cycles":429421440,"thread_start_us":266744,"threads_started":1,"wal-append.queue_time_us":299605}
I20260808 01:53:31.993842 18926 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
W20260808 01:53:32.045032  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 49731.1us
I20260808 01:53:32.010939 19008 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:32.016556 18872 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:32.048746 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:32.016793 18843 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:32.017819  9367 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50064 (request call id 242) took 23687 ms (client timeout 15000 ms). Trace:
I20260808 01:53:32.023824 18800 raft_consensus.cc:3055] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Stepping down as leader of term 18
W20260808 01:53:32.028828 18617 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 19 current term: 19 is higher.
W20260808 01:53:32.032070  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:32.062830 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:32.064949  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:53:32.022572  9601 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.193:39359 timed out after 1.876s (SENT) [suppressed 26 similar messages]
I20260808 01:53:32.037104 10363 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:32.049575  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 01:53:32.066883  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:32.067723  9475 raft_consensus.cc:2410] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 26 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260808 01:53:32.068506  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:32.068928  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:53:32.069622  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:32.052101  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:32.054623  9367 rpcz_store.cc:269] 0808 01:53:08.330276 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:32.017393 (+23687117us) service_pool.cc:211] Skipping call since client already timed out
0808 01:53:32.017704 (+   311us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260808 01:53:32.069824  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:53:32.071331  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:32.084187  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:32.045702 18699 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:32.046950 18952 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:32.096369  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:32.085621  9478 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:32.048851 18895 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:32.050513 19008 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:32.007266  9369 catalog_manager.cc:5697] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 15 to 19. New cstate: current_term: 19 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:32.044632 19009 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.018s
I20260808 01:53:32.103474 18699 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.062444 18800 raft_consensus.cc:740] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:32.107908 18631 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:32.072317 19004 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:32.071638 18843 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.112890 19011 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:32.161160 19011 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.162456 18843 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.114931 18553 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:32.164007 18553 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.117317  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:32.051409 18872 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:32.114517  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3368) took 1229 ms. Trace:
I20260808 01:53:32.125866 18901 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:32.128304  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:32.129199 18809 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 8, Time since last communication: 0.001s
I20260808 01:53:32.054124 18691 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.070920 18978 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.064s
I20260808 01:53:32.136121 19008 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.138952  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:32.067999 18529 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.070s
I20260808 01:53:32.160341 18699 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:32.163375 19011 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.169377  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:32.167526  9636 rpcz_store.cc:276] 0808 01:53:30.885410 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:30.888830 (+  3420us) service_pool.cc:224] Handling call
0808 01:53:30.889277 (+   447us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:31.016230 (+126953us) raft_consensus.cc:1534] Early marking committed up to index 16
0808 01:53:31.020708 (+  4478us) log.cc:844] Serialized 14 byte log entry
0808 01:53:31.182752 (+162044us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:31.187741 (+  4989us) log.cc:844] Serialized 52 byte log entry
0808 01:53:31.187974 (+   233us) raft_consensus.cc:1645] Marking committed up to 18
0808 01:53:31.624159 (+436185us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b180008cc80. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d41889e05 00007f5d483aada7 00007f5d4499dd87 00007f5d449ad49f 00007f5d449ad418
0808 01:53:31.775464 (+151305us) log.cc:844] Serialized 14 byte log entry
0808 01:53:31.776317 (+   853us) log.cc:844] Serialized 14 byte log entry
0808 01:53:32.008117 (+231800us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b48002379d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:53:32.113745 (+105628us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b180008cc80. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d44548f50 00007f5d4453c483 00007f5d44531eda 00007f5d4453242a 00007f5d4499de0a 00007f5d449ad49f 00007f5d449ad418
0808 01:53:32.114094 (+   349us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:32.114199 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:32.114249 (+    50us) raft_consensus.cc:1685] finished
0808 01:53:32.114265 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:32.114446 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":300375,"prepare.queue_time_us":320844,"prepare.run_cpu_time_us":153,"prepare.run_wall_time_us":158,"raft.queue_time_us":210121,"raft.run_cpu_time_us":4171,"raft.run_wall_time_us":1039757,"spinlock_wait_cycles":92918272,"thread_start_us":230231,"threads_started":2,"wal-append.queue_time_us":158301}
W20260808 01:53:32.180706  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 52506.3us
I20260808 01:53:32.181659  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:32.180792  9369 catalog_manager.cc:5697] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 11 to 26, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 26 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:32.200911  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:32.186519  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:32.172757 18553 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.192345  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:32.203657  9637 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:32.148109 18895 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.191277  9478 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:32.196487  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:32.179912 18690 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:32.167996 19004 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.227799 19004 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.186125 18809 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:32.194555 19011 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election won for term 28
I20260808 01:53:32.223435  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:32.223534  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:32.222009  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:32.194391 19008 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:32.198742 18529 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:32.236531  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:32.229121  9481 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:53:32.195737 18745 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:32.192513 18843 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:32.178915 18901 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.220328 18699 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:32.243577  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3856) took 1393 ms. Trace:
I20260808 01:53:32.244074  9474 rpcz_store.cc:276] 0808 01:53:30.850036 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:30.850174 (+   138us) service_pool.cc:224] Handling call
0808 01:53:30.879522 (+ 29348us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:31.116868 (+237346us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:32.243320 (+1126452us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b54004047d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:32.243511 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":508604800}
I20260808 01:53:32.238132 19009 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:32.253633  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:32.237934 19011 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:32.256132  9637 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260808 01:53:32.258659  9369 catalog_manager.cc:5697] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 13 to 18, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 18 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:32.230930 18690 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.265223  9481 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 31.
I20260808 01:53:32.265858  9475 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 25: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:32.240279 18745 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.246745 18699 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.251332 19009 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:32.292752 19009 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:32.267261 19011 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.294246 18699 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.268692 18901 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.272710  9637 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260808 01:53:32.273309  9369 catalog_manager.cc:5697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 14 to 19, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 19 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:32.201655 18800 consensus_queue.cc:260] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.291774 18745 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.239106 18529 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.268275 18997 raft_consensus.cc:3055] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:32.241469 18843 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.324852  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:32.300305 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 389932us
I20260808 01:53:32.330775  9634 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:32.335210 10332 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:32.341509 18690 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:32.342406 18800 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:53:32.324499 18747 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.349694 18747 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:32.351392  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:32.333002 18611 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:32.288748 18834 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.067s
I20260808 01:53:32.358413 18611 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.285039 18926 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.361202 18611 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:32.361601 18843 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:32.355571 18997 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:32.329907 18699 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:32.362516 18901 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260808 01:53:32.367856 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:32.383365  9477 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.019s	sys 0.050s
W20260808 01:53:32.383973  9477 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.019s	sys 0.050s
I20260808 01:53:32.364038  9369 catalog_manager.cc:5697] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 13 to 22, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 22 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:32.392393  9369 catalog_manager.cc:5697] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 11 to 20, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 20 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:32.402451  9369 catalog_manager.cc:5697] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 9 to 21, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 21 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:53:32.404060 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:32.418790  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:32.393743 18801 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Leader election won for term 31
I20260808 01:53:32.416870  9369 catalog_manager.cc:5697] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 15 to 20. New cstate: current_term: 20 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:32.402459 18611 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:32.405933 18895 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.373481 18699 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:32.477809  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 439040d168824496acc076b1c280bde3: Illegal state: Leader has not yet committed an operation in its own term
I20260808 01:53:32.479146 18699 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:32.480238 19000 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 1.206s	user 0.006s	sys 0.001s
I20260808 01:53:32.480312 18611 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.426898 18690 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:32.482260 18617 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:32.482599 18690 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.482915 18617 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.480485  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:32.484651  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3856) took 1852 ms. Trace:
I20260808 01:53:32.440575 18872 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:32.444550 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:32.420790 18997 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.467536 19032 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:32.473182 18529 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.518924 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:32.505810 18755 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:32.519277 19009 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.519703 18755 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.520576  9477 raft_consensus.cc:2393] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 28: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:32.520624 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:32.428107 18809 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:32.522320 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:32.523382 18865 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:32.524422 18675 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:32.522521 19009 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.524981 18675 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.474210 18926 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.527009  9479 raft_consensus.cc:2410] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260808 01:53:32.527519 18926 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:32.528049 18926 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.530251 18926 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:32.531736 18611 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:32.476243 18530 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:32.487633  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:32.532538 18611 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:32.533063  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45588.3us
I20260808 01:53:32.533411 18611 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.533981  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:32.485191  9480 rpcz_store.cc:276] 0808 01:53:30.632392 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:30.632592 (+   200us) service_pool.cc:224] Handling call
0808 01:53:30.633009 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:30.641430 (+  8421us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:30.641454 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:30.643725 (+  2271us) log.cc:844] Serialized 26 byte log entry
0808 01:53:31.022936 (+379211us) spinlock_profiling.cc:243] Waited 2.16 ms on lock 0x7b1800463de0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:53:31.097271 (+ 74335us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:53:31.097336 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:31.097441 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:32.479632 (+1382191us) spinlock_profiling.cc:243] Waited 7.66 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:32.484403 (+  4771us) raft_consensus.cc:1685] finished
0808 01:53:32.484427 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:32.484578 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32854,"raft.queue_time_us":10034,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":6511,"spinlock_wait_cycles":22595584,"thread_start_us":74061,"threads_started":1,"wal-append.queue_time_us":547745}
I20260808 01:53:32.534588 19010 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:32.535406 19010 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:32.535604  9480 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:32.505398 18976 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:32.537878 18976 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.540027 18976 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:32.542166 18690 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.466655 18965 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:32.543044 18965 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:32.543903 18965 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.544981 18965 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
W20260808 01:53:32.546303 19026 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
I20260808 01:53:32.551481 18865 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:32.509461  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:32.552094 18865 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.510862  9629 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: [23.12-23.12]
I20260808 01:53:32.511611  9369 catalog_manager.cc:5697] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 12 to 20, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 20 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:32.516750 18966 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:32.517201 10363 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260808 01:53:32.516721 19032 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.517766 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:32.555572 10366 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:32.556572 10361 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:32.556898 19032 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:32.556311  9369 catalog_manager.cc:5697] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 16 to 20, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 20 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:32.514459 18834 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:32.557818 18834 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.558331 18865 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:32.558983 18997 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:32.560106 18834 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.522967 18809 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.562374 18809 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.532586 18530 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.563669  9369 catalog_manager.cc:5697] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 18 to 19, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 19 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:32.566023  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:32.474141 18971 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:32.470336  9473 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260808 01:53:32.491359  9481 raft_consensus.cc:1240] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26. Ops: [25.13-25.13]
I20260808 01:53:32.548102 18617 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.568279 18971 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.547950  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:32.510519 18699 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:32.515676  9524 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:32.571193  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 55615.5us
I20260808 01:53:32.572576  9524 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:32.549798 18675 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.522346 18755 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:32.554456 18966 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.518070  9636 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:32.475641 18796 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260808 01:53:32.564378 18530 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:32.417255 18901 consensus_queue.cc:1048] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.064s
I20260808 01:53:32.572211 19005 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:32.577987  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:32.578655 10334 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.005s	sys 0.026s
I20260808 01:53:32.569345 18926 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:53:32.578945 10331 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.025s	sys 0.006s
I20260808 01:53:32.580138  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:32.575807 18690 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:32.581060  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:32.506037  9632 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.010s	sys 0.018s
I20260808 01:53:32.615694  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:32.626053  9632 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.011s	sys 0.019s
I20260808 01:53:32.581745  9473 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:32.601119  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:32.628086  9629 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:32.627895  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:32.581902 10365 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 17: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:32.628058  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:32.603099  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:32.596422 10369 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.016s	sys 0.002s
I20260808 01:53:32.629676  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:32.629755  9636 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
W20260808 01:53:32.626040 10331 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.025s	sys 0.006s
W20260808 01:53:32.630363 10369 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.017s	sys 0.002s
W20260808 01:53:32.636436 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:32.584388  9445 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:32.637524  9600 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:32.638336  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 54081.9us
I20260808 01:53:32.639307  9445 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:32.625819  9481 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
W20260808 01:53:32.639748  9601 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.009s	sys 0.033s
W20260808 01:53:32.641620  9601 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.009s	sys 0.033s
W20260808 01:53:32.641996  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 619425us
W20260808 01:53:32.581812 10334 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.006s	sys 0.026s
I20260808 01:53:32.642822  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:32.572288 18699 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.655646  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:32.656394  9481 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:32.630801  9479 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:32.658169  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:32.649730  9480 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260808 01:53:32.660143 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:32.663337  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:32.653587  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:32.642376  9445 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:32.666998  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:32.668323  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:32.670591  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:32.703382  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 61090.3us
W20260808 01:53:32.676337  9599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.003s	sys 0.023s
W20260808 01:53:32.681537  9476 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.006s	sys 0.018s
I20260808 01:53:32.577087 18617 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.091s
W20260808 01:53:32.642808 18980 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.006s	sys 0.037s
I20260808 01:53:32.640362 18809 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:32.633800 18796 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:32.605454 18872 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
W20260808 01:53:32.704344  9476 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.007s	sys 0.019s
W20260808 01:53:32.704627  9599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.003s	sys 0.023s
W20260808 01:53:32.743319  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 659025us
I20260808 01:53:32.634536 18690 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:32.636163 19005 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:32.705068  9481 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 25.
I20260808 01:53:32.707531  9635 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 22: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:32.707614 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:32.640074 18926 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:32.722998 18980 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.006s	sys 0.038s
W20260808 01:53:32.717219  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:32.717695 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:32.760289  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:32.712041 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:32.728821  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:32.761824  9633 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 28.
I20260808 01:53:32.758455 10366 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 17.
I20260808 01:53:32.732553  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:32.605257 18530 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.085s
I20260808 01:53:32.749639  9634 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 18. Current term is 19. Ops: []
I20260808 01:53:32.777607  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:32.778589  9552 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 16.
I20260808 01:53:32.732415 18569 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:32.748714 10361 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:32.789196  9445 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:32.765218 10365 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:32.724843  9629 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:45658
I20260808 01:53:32.792307  9629 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:53:32.792941  9629 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:32.717806 18976 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:32.803010 18976 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:32.803464 18976 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:32.803548  9555 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
W20260808 01:53:32.805477 19033 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
W20260808 01:53:32.737318 19017 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260808 01:53:32.772661  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:32.809003  9558 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 28: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:32.774451  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:32.809577  9630 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 25.
I20260808 01:53:32.809731  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1615) took 1052 ms. Trace:
I20260808 01:53:32.779904  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:32.811410  9477 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
W20260808 01:53:32.714562  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 77132us
I20260808 01:53:32.760741 19005 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:32.798206  9445 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:32.785965 18818 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:32.811978 18569 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:32.799685  9557 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260808 01:53:32.756877 18796 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.853410 18796 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.749244 18690 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:32.800999 19011 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:32.854422 19011 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:32.854995 19011 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:32.861006  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:32.862144  9558 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 25.
W20260808 01:53:32.801204 19035 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260808 01:53:32.802477 19039 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:32.817603  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:32.867728 19039 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:32.868499  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 51009.5us
I20260808 01:53:32.869529 18755 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260808 01:53:32.870046 19039 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.810114  9472 rpcz_store.cc:276] 0808 01:53:31.757604 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:31.828776 (+ 71172us) service_pool.cc:224] Handling call
0808 01:53:31.829193 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:31.849304 (+ 20111us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:31.849366 (+    62us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:31.879981 (+ 30615us) log.cc:844] Serialized 26 byte log entry
0808 01:53:32.371958 (+491977us) raft_consensus.cc:1645] Marking committed up to 18
0808 01:53:32.372013 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:32.372106 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:32.809494 (+437388us) raft_consensus.cc:1685] finished
0808 01:53:32.809517 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:32.809668 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14942,"raft.queue_time_us":11588,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":5758,"thread_start_us":452042,"threads_started":1,"wal-append.queue_time_us":700215}
I20260808 01:53:32.819165 18997 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:32.872228 18997 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:32.872700 18997 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:32.872964  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:32.873283  9629 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.865952 18569 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.816550 18818 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.825654 18699 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:32.832670  9600 leader_election.cc:400] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 30. Current term is 31.
I20260808 01:53:32.833581 18966 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.834277 18611 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:32.840750  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:32.875242 18699 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:32.876241 18611 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:32.876377 18818 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:32.876820 18611 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:32.764956 18971 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:32.870115 18755 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:32.878491 18755 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:53:32.877969 18611 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.869498  9558 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: []
W20260808 01:53:32.875094 19025 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.235s	user 0.003s	sys 0.000s
I20260808 01:53:32.847790  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:32.881458  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 49105.2us
I20260808 01:53:32.888199  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:32.890730  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:32.894176 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:32.895251  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:32.896296  9559 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 31: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:32.899904 18966 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:32.901847  9629 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:32.904618  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:32.907321  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:32.911974  9557 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:32.912029  9555 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260808 01:53:32.907131 18818 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:32.900784 18966 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:32.924094 18966 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:32.931588 19011 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.926205 18926 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:32.936419 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:32.937120 10369 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:32.942385  9478 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:32.946645 10369 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:32.948305  9445 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:32.948649  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 273200us
I20260808 01:53:32.958720 19011 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:32.959931 19011 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:32.960747 19011 raft_consensus.cc:697] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:32.961680 18961 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:32.971733  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:32.974642 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: true dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:32.975039  9632 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260808 01:53:32.975517 10365 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:32.975984 10365 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:32.976984 10365 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.978070 10365 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:32.983132  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:32.984030  9638 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:32.872889 18976 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:32.983225 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:32.991930 10369 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:32.993644 18976 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:32.993866 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:32.998409  9638 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:33.001837  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:33.003149  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:33.006210  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:33.029438  9559 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:33.015580  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:33.010092  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 7b5ea129398a42fb9981fca652027cd6: Illegal state: Replica 6e7856bed1be43acaa821290d506ee97 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 27 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:53:33.011058  9631 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.024s	sys 0.040s
W20260808 01:53:33.030500  9631 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.024s	sys 0.040s
I20260808 01:53:33.002075 18997 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:33.016427  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:33.022882  9600 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:33.033847  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:33.034236 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:33.034767  9638 raft_consensus.cc:2393] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 24: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:33.021821 10365 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:33.024916  9636 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:33.029469  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:33.035684  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:33.037150  9632 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:33.040661  9600 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:33.048758  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:33.036993  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 463584us
I20260808 01:53:33.051688  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:33.053030  9633 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 26: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:33.055205  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 4280) took 1344 ms. Trace:
I20260808 01:53:33.111074  9638 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:33.110158  9560 rpcz_store.cc:276] 0808 01:53:31.710608 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:31.710802 (+   194us) service_pool.cc:224] Handling call
0808 01:53:31.711064 (+   262us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:31.711265 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 14
0808 01:53:32.172305 (+461040us) log.cc:844] Serialized 14 byte log entry
0808 01:53:33.052750 (+880445us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:33.052823 (+    73us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:53:33.052885 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:33.054999 (+  2114us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:33.055141 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":705056,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":58,"raft.queue_time_us":331007,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"thread_start_us":1338678,"threads_started":3,"wal-append.queue_time_us":698744}
I20260808 01:53:33.060731  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:33.062068  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:33.062163  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:33.113862  9636 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:33.063377  9635 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 27. Current term is 28. Ops: [27.14-27.14]
I20260808 01:53:33.092120  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:33.072806 18965 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:33.116201 18965 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:33.099593 18809 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:32.962030 19011 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:33.103647 18997 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:33.103757  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:33.050187 18878 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:33.120846 18878 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:33.121263 18878 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:33.121585 18965 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:33.105088 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:33.106194  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:33.119812 18809 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:33.124593  9632 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260808 01:53:33.127534 18809 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:33.137362  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:33.137509 10365 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
W20260808 01:53:33.137697  9475 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.799s	user 0.035s	sys 0.040s
W20260808 01:53:33.173548  9475 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.835s	user 0.035s	sys 0.040s
I20260808 01:53:33.173842  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:33.064280 18801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.029s	sys 0.041s
I20260808 01:53:33.174605  9472 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 28.
I20260808 01:53:33.175237  9445 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:33.147248  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:33.053331 19010 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:33.012223 18755 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:33.155078  9478 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260808 01:53:33.158277  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:33.159852  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:33.195624  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:33.133307 10334 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:33.063717 18865 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:33.176334  9634 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:33.204715  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:33.178416  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:33.184281  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:33.184535 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:33.206233  9444 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:33.163931 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:33.190514 18976 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:33.190582 19005 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:33.192337  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 030eacff61b14455978d2f0de262a089: Illegal state: Replica 52ccf4dcca1843f7bb29e9172e2d4ab4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 27 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:33.216070  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:33.192916  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:33.218247  9638 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
W20260808 01:53:33.193698  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.667s	user 0.005s	sys 0.022s
I20260808 01:53:33.194411  9554 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 15. (term mismatch)
I20260808 01:53:33.219410  9554 pending_rounds.cc:85] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 14
I20260808 01:53:33.219767  9554 pending_rounds.cc:107] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 26.15
I20260808 01:53:33.220103  9554 raft_consensus.cc:2889] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:33.221642  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:33.196558  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:33.223116  9559 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:33.224442  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:53:33.225661  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.005s	sys 0.022s
W20260808 01:53:33.196763 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 63902.9us
I20260808 01:53:33.177448  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:33.190094 18801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.725s	user 0.030s	sys 0.042s
I20260808 01:53:33.207975 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:33.234023 10332 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:33.234230  9629 raft_consensus.cc:2368] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 17. Current term is 18.
I20260808 01:53:33.209926  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:33.238658  9558 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:33.242102  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:33.211009  9477 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 13. (term mismatch)
I20260808 01:53:33.244014  9477 pending_rounds.cc:85] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 12
I20260808 01:53:33.244459  9477 pending_rounds.cc:107] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 23.13
I20260808 01:53:33.244872  9477 raft_consensus.cc:2889] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:33.246438  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:33.247566  9554 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:33.249630  9474 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260808 01:53:33.250301  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:33.209424 19010 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:33.253067 18800 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:33.254228 19040 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:33.232860 18801 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:33.205924 18997 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election won for term 28
I20260808 01:53:33.215664 18843 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260808 01:53:33.204715 19056 raft_consensus.cc:2764] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260808 01:53:33.254798 19040 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:33.216017 18530 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:33.208410 18865 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:33.230291 18675 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:33.263113 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:33.281000 10365 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 25: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:33.255967  9478 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:33.265594 19010 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:33.274466  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:33.309921  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:33.270054 19040 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:33.270182 18631 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:33.277032 18530 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:33.280006 18675 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:33.295680  9558 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:33.311762 19001 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:53:33.409215 19058 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260808 01:53:33.390666 18675 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:33.352681 18631 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:33.301934 18690 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:33.352049 18530 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:33.453926  9478 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:33.299604 19061 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:53:33.390342 18366 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.013s	sys 0.038s
W20260808 01:53:33.404492 19049 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
W20260808 01:53:33.456143 18611 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.014s	sys 0.012s
W20260808 01:53:33.480551  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 17e220e6a25747939bdf18f26b732407: Illegal state: Replica 259bcb3f0d254e8b9effac3277e654e0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 28 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:53:33.480434  9552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.021s	sys 0.033s
W20260808 01:53:33.484097  9552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.022s	sys 0.033s
W20260808 01:53:33.475133 18366 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.013s	sys 0.041s
W20260808 01:53:33.486095 18611 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.014s	sys 0.012s
I20260808 01:53:33.456981 19001 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:33.505317 18617 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.748s	user 0.000s	sys 0.038s
I20260808 01:53:33.402529 18801 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:33.486378 19059 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:33.465252 19060 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.038s
W20260808 01:53:33.505761 18980 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.006s	sys 0.032s
I20260808 01:53:33.531564  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3398) took 1053 ms. Trace:
W20260808 01:53:33.514883 18966 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.004s	sys 0.014s
I20260808 01:53:33.390189 18865 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:33.503814 18755 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:33.487774 18878 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:33.545557  9475 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3873) took 1476 ms. Trace:
W20260808 01:53:33.517212 18617 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.766s	user 0.000s	sys 0.038s
I20260808 01:53:33.541309  9631 rpcz_store.cc:276] 0808 01:53:32.478315 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:32.482405 (+  4090us) service_pool.cc:224] Handling call
0808 01:53:32.482798 (+   393us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:32.483218 (+   420us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:32.483249 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:32.483911 (+   662us) log.cc:844] Serialized 52 byte log entry
0808 01:53:33.031004 (+547093us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:53:33.031051 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:33.031439 (+   388us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:33.531142 (+499703us) raft_consensus.cc:1685] finished
0808 01:53:33.531180 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:33.531467 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1254,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":9348,"thread_start_us":546653,"threads_started":1,"wal-append.queue_time_us":784923}
I20260808 01:53:33.514951 19005 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:33.473699 19061 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:33.503897  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 451934us
I20260808 01:53:33.521919 19010 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:53:33.540774 18980 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.006s	sys 0.032s
W20260808 01:53:33.563060  9556 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.684s	user 0.015s	sys 0.021s
W20260808 01:53:33.563860  9556 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.015s	sys 0.021s
W20260808 01:53:33.546779 18966 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.005s	sys 0.014s
W20260808 01:53:33.561848  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 354297us
W20260808 01:53:33.567030  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 329343us
I20260808 01:53:33.545769 18834 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:33.575788  9473 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:33.567592  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:33.576870  9524 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:33.592338  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:33.568049  9523 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:33.593793  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:33.577924  9636 raft_consensus.cc:2368] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23.
I20260808 01:53:33.592065  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:33.558713 18939 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:33.543731 19059 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:33.552423  9475 rpcz_store.cc:276] 0808 01:53:32.069200 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:32.326177 (+256977us) service_pool.cc:224] Handling call
0808 01:53:32.326634 (+   457us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:32.326953 (+   319us) raft_consensus.cc:1534] Early marking committed up to index 15
0808 01:53:33.174026 (+847073us) log.cc:844] Serialized 14 byte log entry
0808 01:53:33.544121 (+370095us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:33.544186 (+    65us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:33.544237 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:33.545267 (+  1030us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:33.545453 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1063602,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"thread_start_us":1205286,"threads_started":2}
I20260808 01:53:33.566514 19005 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:33.606132  9524 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:33.605456 19011 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:33.651686 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:33.614436 18631 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:33.566669 19010 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:33.603067 18611 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:33.656476 19054 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:33.607702 18939 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:33.667704  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 59276.9us
I20260808 01:53:33.712988 18965 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.008s
I20260808 01:53:33.684764 18611 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:33.723214 18611 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:33.729420 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:33.731139  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:33.731856  9476 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 25.
I20260808 01:53:33.732129 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:33.732424 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:33.733609 10361 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 25: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:33.734508 10334 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:33.739684 19062 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:33.740252 19062 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:33.740644 19062 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:33.742816 18690 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:33.685282 19063 raft_consensus.cc:2764] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260808 01:53:33.678018 18809 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:33.713636 19054 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:33.663282 19074 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:33.667517  9524 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:33.766290 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:33.604112 18997 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:33.658866 18976 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:33.697280 18631 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:33.799825  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:33.799852  9554 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260808 01:53:33.720289 18939 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:33.619787 19005 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:33.811437  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.003s	sys 0.015s
W20260808 01:53:33.810482  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 143101us
W20260808 01:53:33.818289  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.003s	sys 0.015s
I20260808 01:53:33.819545  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:33.825277 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:33.825923  9630 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.723s	user 0.007s	sys 0.012s
W20260808 01:53:33.826315  9630 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.007s	sys 0.012s
I20260808 01:53:33.770195 18809 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:33.614717 19011 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:33.771459 19075 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:33.845755 18809 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:33.810508 18631 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:33.824928 19001 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:33.844794 18796 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:33.867545  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:53:33.841475 19074 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:33.848399 19075 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:33.861428 18690 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:33.903990  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:33.909358  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:33.909842 18878 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:33.910036  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:33.915750 19059 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:33.863867 18631 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:33.877851 19083 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:33.949749 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:33.893023 19074 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:33.919749 19001 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:33.917168 19075 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:33.924190 19086 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:33.940670 19010 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:33.977144  9601 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.008s	sys 0.008s
W20260808 01:53:33.982301  9599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.010s	sys 0.005s
I20260808 01:53:33.949132 19061 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:33.991969  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 145011us
I20260808 01:53:33.992623 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:33.986162  9599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.010s	sys 0.005s
W20260808 01:53:34.003365 19072 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20260808 01:53:34.003780 19054 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.004467 19005 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:34.007458 19079 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
W20260808 01:53:34.008620 19040 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.655s	user 0.003s	sys 0.010s
I20260808 01:53:34.009120 19040 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:33.986214  9601 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.009s	sys 0.008s
W20260808 01:53:34.010327  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 673260us
W20260808 01:53:34.011102 19070 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.326s	user 0.000s	sys 0.001s
I20260808 01:53:33.920238 18690 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.013196 18690 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.014900  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:34.015668  9558 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 16 FOLLOWER]: Advancing to term 18
I20260808 01:53:34.016424 10365 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:33.896275 18796 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:34.021130  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:34.022197 18796 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:33.965381 18617 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260808 01:53:34.009639 19065 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.172s	user 0.000s	sys 0.001s
I20260808 01:53:34.030112  9601 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:33.993470 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:34.022141  9481 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:53:33.964344 19074 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:34.032318  9558 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:53:33.942144 19059 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.039s
I20260808 01:53:33.935475 18976 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:34.033252  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 4307) took 1264 ms. Trace:
I20260808 01:53:34.048202  9551 rpcz_store.cc:276] 0808 01:53:32.768986 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:32.785665 (+ 16679us) service_pool.cc:224] Handling call
0808 01:53:32.800480 (+ 14815us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:33.278429 (+477949us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:33.278482 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:33.279922 (+  1440us) log.cc:844] Serialized 52 byte log entry
0808 01:53:33.770527 (+490605us) raft_consensus.cc:1645] Marking committed up to 18
0808 01:53:33.985299 (+214772us) log.cc:844] Serialized 14 byte log entry
0808 01:53:33.994739 (+  9440us) log.cc:844] Serialized 14 byte log entry
0808 01:53:33.995124 (+   385us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:33.995213 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:34.032909 (+ 37696us) raft_consensus.cc:1685] finished
0808 01:53:34.032933 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:34.033186 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9580,"raft.queue_time_us":483620,"raft.run_cpu_time_us":452,"raft.run_wall_time_us":33466,"thread_start_us":1049474,"threads_started":3,"wal-append.queue_time_us":632992}
I20260808 01:53:34.038497 10369 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:34.050024  9558 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 23 index: 19. (index mismatch)
I20260808 01:53:33.918694 18997 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.041126  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:34.014675 18675 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.547s	user 0.008s	sys 0.003s
I20260808 01:53:34.027834  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:34.043565 18761 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260808 01:53:34.056344  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:34.054327  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:34.057603  9601 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:34.058584  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:34.073815  9477 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:34.076155  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:34.039930 18834 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:34.077754  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:34.076108  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48358.1us
I20260808 01:53:34.047469 19010 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:33.957403 19083 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:33.985847 19086 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.103221  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 4321) took 1078 ms. Trace:
I20260808 01:53:34.104771 19086 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.103814  9557 rpcz_store.cc:276] 0808 01:53:33.025203 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:33.025375 (+   172us) service_pool.cc:224] Handling call
0808 01:53:33.025870 (+   495us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:33.507435 (+481565us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:33.507503 (+    68us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:33.508213 (+   710us) log.cc:844] Serialized 26 byte log entry
0808 01:53:33.956745 (+448532us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:33.956800 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:33.956909 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:34.102868 (+145959us) raft_consensus.cc:1685] finished
0808 01:53:34.102900 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:34.103131 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":364761,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":15575,"thread_start_us":874588,"threads_started":2,"wal-append.queue_time_us":505376}
I20260808 01:53:34.078807 19057 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:34.106822 19085 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:34.108790  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:34.091154 18755 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:34.109673 18755 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:34.110203 18751 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Leader election won for term 18
I20260808 01:53:34.110875 18751 raft_consensus.cc:697] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:34.114351 19083 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.073062 18966 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:34.096087  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:34.093096 18761 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:34.098780 18971 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:34.101292 19011 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.052417 19005 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:34.111974 18611 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.074712 19062 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:34.119943  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:34.121089 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:34.133317  9445 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:34.156553 10369 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 19.
I20260808 01:53:34.141947 19088 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:34.091655 19096 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260808 01:53:34.147260 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 238423us
W20260808 01:53:34.154981 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:34.158958  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:34.149164 18761 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.160856 19105 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:34.161448 19085 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260808 01:53:34.161877  9551 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:34.172726 10332 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:34.157085 19005 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.162926 18751 consensus_queue.cc:237] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:34.164199 18611 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.180084  9476 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:34.156111 19040 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.165030 19011 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260808 01:53:34.157590  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 65766.3us
W20260808 01:53:34.141554  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.011s	sys 0.014s
I20260808 01:53:34.165920 18809 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.057432 18675 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:34.192610  9521 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:34.087224 18834 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.170375  9474 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:34.088035 19010 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.150627 19059 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 LEADER]: Stepping down as leader of term 27
I20260808 01:53:34.178210 18895 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260808 01:53:34.159736  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:34.192930  9551 raft_consensus.cc:1275] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260808 01:53:34.174386 19105 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:34.147841 18966 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.166637  9445 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:34.293143  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:34.227120  9560 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 LEADER]: Stepping down as leader of term 27
W20260808 01:53:34.363987  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 197451us
I20260808 01:53:34.364307  9560 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:53:34.364612  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 307098us
I20260808 01:53:34.365935  9601 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:34.295233  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:34.366557  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:34.311617  9478 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:34.368643 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:34.369210 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:34.369410 10365 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 17 FOLLOWER]: Advancing to term 18
I20260808 01:53:34.369822 10369 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:34.370015  9478 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 27. Current term is 28. Ops: [27.14-27.14]
I20260808 01:53:34.199328  9480 raft_consensus.cc:2410] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 26 because replica has last-logged OpId of term: 23 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
W20260808 01:53:34.193372  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.011s	sys 0.014s
W20260808 01:53:34.371438  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 772131us
I20260808 01:53:34.375185  9481 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:53:34.375670  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 183132us
I20260808 01:53:34.374366  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:34.367426  9474 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:34.192780  9557 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260808 01:53:34.383685 10361 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 23 index: 19. (index mismatch)
I20260808 01:53:34.386312  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:34.386061  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 226467us
I20260808 01:53:34.389577  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:34.365455  9560 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.400890 10365 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 18.
I20260808 01:53:34.401633  9560 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
W20260808 01:53:34.342518 19092 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.342s	user 0.004s	sys 0.000s
I20260808 01:53:34.183185 18800 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260808 01:53:34.245167 19066 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.670s	user 0.003s	sys 0.000s
W20260808 01:53:34.201088 19068 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.636s	user 0.001s	sys 0.003s
I20260808 01:53:34.411389  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:34.367311 18617 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:34.197537 18761 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:34.428471  9524 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:34.373991 19105 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:34.193012 19085 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:34.192829 19011 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:34.182891 19063 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:34.196653 18809 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.317806 19005 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.173472 19088 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.378381 19059 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:34.462502 19010 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.462311 19059 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.463414 18834 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.406746 19086 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.044s
I20260808 01:53:34.464143 19059 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:34.465011 19086 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:34.465523 19086 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:34.409637 19057 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:34.466306 19057 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:34.385243 18843 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:53:34.422933 19098 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:34.402866 19084 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:34.416747 10369 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:34.435328  9560 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 28 index: 14. (term mismatch)
I20260808 01:53:34.430007 18872 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:34.430945  9524 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:34.437085 18966 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.439296 19096 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:34.405923 18878 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.044s
W20260808 01:53:34.439618 19073 log.cc:927] Time spent T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.743s	user 0.002s	sys 0.017s
I20260808 01:53:34.440452 18631 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.456113 19011 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:34.414567 19083 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:34.443770 19067 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.840s	user 0.003s	sys 0.003s
W20260808 01:53:34.448069 19069 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.859s	user 0.009s	sys 0.001s
I20260808 01:53:34.448560 18611 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:34.449100  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 4319) took 1435 ms. Trace:
I20260808 01:53:34.460961 10366 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260808 01:53:34.433656 19001 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:34.462998 19088 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.399111 18699 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260808 01:53:34.384368 18971 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:34.429910 19061 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:34.472599  9524 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:34.497993  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:34.473620  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:34.476356  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37248 (request call id 1622) took 1831 ms. Trace:
I20260808 01:53:34.487161  9560 pending_rounds.cc:85] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 13
I20260808 01:53:34.502370  9560 pending_rounds.cc:107] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 27.14
I20260808 01:53:34.503250  9560 raft_consensus.cc:2889] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:34.501789  9479 rpcz_store.cc:276] 0808 01:53:32.645438 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:32.710917 (+ 65479us) service_pool.cc:224] Handling call
0808 01:53:32.711260 (+   343us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:32.711499 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:32.711527 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:32.778644 (+ 67117us) log.cc:844] Serialized 26 byte log entry
0808 01:53:33.438891 (+660247us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:53:33.438943 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:33.439044 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:34.340518 (+901474us) spinlock_profiling.cc:243] Waited 23.7 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:34.475999 (+135481us) raft_consensus.cc:1685] finished
0808 01:53:34.476025 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:34.476286 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":54543616,"thread_start_us":492259,"threads_started":1,"wal-append.queue_time_us":649973}
I20260808 01:53:34.504604  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3258) took 1314 ms. Trace:
I20260808 01:53:34.505363  9560 rpcz_store.cc:276] 0808 01:53:33.190043 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:33.191036 (+   993us) service_pool.cc:224] Handling call
0808 01:53:33.191437 (+   401us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:34.226980 (+1035543us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b5400086c20. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490b1a0 00007f5d449955a0 00007f5d44996611 00007f5d4499316f 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c
0808 01:53:34.504203 (+277223us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:34.504522 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":845973248}
I20260808 01:53:34.494874  9478 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:34.431820 18755 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:34.514241  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:34.430466 18617 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:34.515669  9559 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:53:34.477288  9550 rpcz_store.cc:276] 0808 01:53:33.013345 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:33.013519 (+   174us) service_pool.cc:224] Handling call
0808 01:53:33.014023 (+   504us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:33.498307 (+484284us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:33.498352 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:33.506118 (+  7766us) log.cc:844] Serialized 52 byte log entry
0808 01:53:33.881152 (+375034us) raft_consensus.cc:1645] Marking committed up to 11
0808 01:53:33.881585 (+   433us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:33.881822 (+   237us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:34.341659 (+459837us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:34.448646 (+106987us) raft_consensus.cc:1685] finished
0808 01:53:34.448681 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:34.449030 (+   349us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":327931,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":42986,"spinlock_wait_cycles":42661760,"thread_start_us":816931,"threads_started":2,"wal-append.queue_time_us":475180}
W20260808 01:53:34.476182 19071 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.709s	user 0.002s	sys 0.001s
W20260808 01:53:34.489909 19081 log.cc:927] Time spent T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.569s	user 0.000s	sys 0.002s
I20260808 01:53:34.513803  9524 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:34.510119 18530 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.037s	user 0.014s	sys 0.002s
W20260808 01:53:34.515254 18926 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 28 current term: 28 is higher.
I20260808 01:53:34.477833 18675 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.468992 19057 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:34.472792 19064 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:34.523430  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:34.475931 18631 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.469218 18843 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:34.471415 19062 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:34.532337  9478 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260808 01:53:34.512826 19060 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:34.479630 19001 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:34.591341  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 1384) took 1825 ms. Trace:
W20260808 01:53:34.560510  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:34.561429 10334 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:34.580778 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:34.515606 19061 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:53:34.472958 19084 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.079s
I20260808 01:53:34.591816  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:34.594894  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:34.470543 18872 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.477145 18611 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.595692  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:34.479696 18966 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.075s
I20260808 01:53:34.476687 19083 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.606824  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:34.456319 19085 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.598017  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:34.591828  9556 rpcz_store.cc:276] 0808 01:53:32.784686 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:32.784865 (+   179us) service_pool.cc:224] Handling call
0808 01:53:32.785272 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:32.796035 (+ 10763us) raft_consensus.cc:1534] Early marking committed up to index 19
0808 01:53:32.878890 (+ 82855us) log.cc:844] Serialized 14 byte log entry
0808 01:53:33.576638 (+697748us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:33.576704 (+    66us) raft_consensus.cc:1645] Marking committed up to 19
0808 01:53:33.576753 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:34.405023 (+828270us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b180036c3a0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4185ac56 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229
0808 01:53:34.591063 (+186040us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b54001ac554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:34.591128 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:34.591275 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10717,"prepare.queue_time_us":82980,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":10853,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":10077,"spinlock_wait_cycles":51428992,"thread_start_us":686609,"threads_started":1,"wal-append.queue_time_us":968118}
I20260808 01:53:34.550277 19059 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:34.599429  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:34.600550  9629 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 19: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:34.600474  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:34.557122  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 156343us
I20260808 01:53:34.620028  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:34.594578 19001 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
W20260808 01:53:34.603118  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 228833us
I20260808 01:53:34.566752  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:34.605505  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:34.557029 18843 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:53:34.629582  9637 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:34.616407  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:34.630035  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:34.610055  9600 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:34.563417 18961 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:34.478716 19063 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:34.593071 19060 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:34.557337 19064 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:34.643630  9601 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:34.596657 19062 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:53:34.466605 19086 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:34.469709 19098 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.637465  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:34.622597 19108 log.cc:927] Time spent T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260808 01:53:34.596738 18878 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.053s
I20260808 01:53:34.563874 18530 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.646070 18631 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260808 01:53:34.706041  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 68646.8us
I20260808 01:53:34.707235  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:34.647571 19059 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:34.635397 18976 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:34.668893  9637 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:34.592159 19057 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.050s
I20260808 01:53:34.655961  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:53:34.654031 19075 consensus_meta.cc:287] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.673s	user 0.008s	sys 0.005s
I20260808 01:53:34.700922  9477 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 27. Current term is 28. Ops: [27.14-27.14]
W20260808 01:53:34.701792  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 125392us
I20260808 01:53:34.655179 19060 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:34.663272 19062 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:34.642664 18834 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.064s
I20260808 01:53:34.711686  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:34.624387 19110 consensus_queue.cc:1059] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.077s
I20260808 01:53:34.665987 18971 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:34.657732  9601 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:34.711072 18997 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:53:34.700675 18755 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:34.719702  9475 raft_consensus.cc:2368] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 26. Current term is 27.
W20260808 01:53:34.729481  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 107843us
I20260808 01:53:34.715325 19086 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:34.713511 19010 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:34.650075 18961 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:34.727450 19074 consensus_meta.cc:287] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.672s	user 0.012s	sys 0.002s
I20260808 01:53:34.716212 19115 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:34.742146  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:34.722061 19062 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:34.752806  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:34.668021 19064 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:34.776659  9524 leader_election.cc:400] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 26. Current term is 27.
I20260808 01:53:34.777287  9524 leader_election.cc:403] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:53:34.740217 18755 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:34.720957 18976 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.740238 18834 consensus_queue.cc:1243] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 24.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260808 01:53:34.740603 18971 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:34.747121 18631 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260808 01:53:34.796521  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:34.783372 18699 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 27
W20260808 01:53:34.799612 19123 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260808 01:53:34.800279 19114 raft_consensus.cc:3055] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Stepping down as leader of term 22
W20260808 01:53:34.800737 19118 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260808 01:53:34.722245 19075 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.801522 18878 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:34.803206 19110 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260808 01:53:34.804764 19116 log.cc:927] Time spent T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Append to log took a long time: real 0.069s	user 0.000s	sys 0.002s
I20260808 01:53:34.792296 19122 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.023s
I20260808 01:53:34.807935 19083 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.810101 18699 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 26. Current term is 27.
I20260808 01:53:34.748736 19074 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.813382 19074 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.816032  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:34.752275 18997 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:34.821209  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:34.823530  9475 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
W20260808 01:53:34.825716 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 121947us
W20260808 01:53:34.835726 19113 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.135s	user 0.001s	sys 0.000s
I20260808 01:53:34.845553  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:34.843240  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:34.820045 18834 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:34.800747 19114 raft_consensus.cc:740] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:34.879206 18997 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:34.911463 19001 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.797047 18755 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:34.815222 19010 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:34.822381 19083 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:34.876718 19005 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:34.886173 18976 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.886497 18530 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:34.887532 19120 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
W20260808 01:53:34.888447 19126 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260808 01:53:34.889322  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:34.959820  9558 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
W20260808 01:53:34.840091 19078 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.838s	user 0.006s	sys 0.000s
I20260808 01:53:34.825142 18872 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.970232 10361 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 28. Current term is 29. Ops: [28.18-28.18]
I20260808 01:53:34.894516  9473 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
I20260808 01:53:34.970669  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3409) took 1959 ms. Trace:
I20260808 01:53:34.899502  9475 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:34.829061 19122 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:34.907275 18611 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.908679 18961 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:34.910817  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50140 (request call id 3901) took 1730 ms. Trace:
I20260808 01:53:34.972505  9472 rpcz_store.cc:276] 0808 01:53:33.180856 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:33.183865 (+  3009us) service_pool.cc:224] Handling call
0808 01:53:34.910562 (+1726697us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b540021e154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:34.910742 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3116180736}
W20260808 01:53:34.977610 19105 consensus_meta.cc:287] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.521s	user 0.009s	sys 0.004s
I20260808 01:53:34.986407 19039 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:34.987304 19039 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:34.992435  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:34.993358  9472 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 25.
I20260808 01:53:34.901410 18971 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:34.971088  9630 rpcz_store.cc:276] 0808 01:53:33.011421 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:33.011583 (+   162us) service_pool.cc:224] Handling call
0808 01:53:33.048076 (+ 36493us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:33.048484 (+   408us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:33.048539 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:33.064001 (+ 15462us) log.cc:844] Serialized 26 byte log entry
0808 01:53:33.826765 (+762764us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:53:33.826818 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:33.826897 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:34.814335 (+987438us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:34.970317 (+155982us) raft_consensus.cc:1685] finished
0808 01:53:34.970341 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:34.970603 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17594,"raft.queue_time_us":180,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":12284,"spinlock_wait_cycles":38316416,"thread_start_us":723514,"threads_started":1,"wal-append.queue_time_us":898297}
I20260808 01:53:35.000494 18755 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:34.953166 19098 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:35.009080 19063 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:35.011897 19075 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:35.013324 19085 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:34.931864  9444 leader_election.cc:400] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23.
I20260808 01:53:34.956303 19057 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:34.958225 19061 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 LEADER]: Stepping down as leader of term 27
I20260808 01:53:34.962596 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:34.962872  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:35.035516  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 103883us
I20260808 01:53:34.815093 19059 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:34.955973 18965 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:35.048410 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:35.047814  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:35.048120  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 300376us
I20260808 01:53:34.978293 19105 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:35.064893  9554 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260808 01:53:35.065034  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:35.066342  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:35.067633  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:35.067902  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:35.062511 19063 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:35.068956  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:35.069250 19063 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:35.070441  9550 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:35.070953  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:35.071359  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:35.071736  9552 raft_consensus.cc:2393] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 25: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:35.066983  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:35.081528  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:35.051734 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:35.063714  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:35.081708  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:35.084316  9558 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 23: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:35.051752 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:35.047127 19061 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:34.946591 19010 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:35.078166  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:35.112735 10366 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260808 01:53:35.124615  9555 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:35.124507 10334 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:35.103160 10365 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 25 index: 20. (index mismatch)
I20260808 01:53:35.103137  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:35.094568  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:35.137338  9475 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:35.098460  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:35.120436  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 462790us
I20260808 01:53:35.127655  9523 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:35.131803  9599 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:35.105587  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:35.091547  9556 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260808 01:53:35.092088  9524 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:35.145249  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 53233.1us
I20260808 01:53:35.146394  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:35.093082  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:35.146134  9524 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:35.062247 19104 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:35.063377 18926 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:35.120456 18617 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.570s	user 0.012s	sys 0.001s
I20260808 01:53:35.153930  9475 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:35.155270 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:35.155390 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:53:35.117096 19011 consensus_meta.cc:287] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.642s	user 0.009s	sys 0.002s
I20260808 01:53:35.197508  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:35.198333  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:35.122859 19096 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260808 01:53:35.238021  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 84062.4us
I20260808 01:53:35.084429 19001 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:35.166414 10368 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 25: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:35.240687  9601 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:35.108489 19088 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Stepping down as leader of term 25
W20260808 01:53:35.229717  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 59538.6us
I20260808 01:53:35.049098 19057 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:35.119207 19105 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:35.118299 19005 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:35.166208 18834 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:35.250604  9524 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:35.067243 18965 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:35.167995  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:35.239136 18699 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:35.199666 19104 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:35.087301 19119 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.146s
I20260808 01:53:35.151410 18872 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election won for term 28
I20260808 01:53:35.199345 19124 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:53:35.154978 19084 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:35.280925  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:35.238544 19039 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:35.117537 19064 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:35.150998 18961 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:35.144424 19074 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.114s
I20260808 01:53:35.169116 18926 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:35.251179 19088 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:35.292991  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:35.254195 18801 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:35.275454 19075 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:35.281965  9477 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:35.251283 19011 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:35.305243  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:35.327315  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:35.327865  9559 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 27: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:35.328764  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:35.087333 19114 consensus_queue.cc:260] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:35.274931 19056 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260808 01:53:35.318221  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 77645.3us
I20260808 01:53:35.197288 18617 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:35.326282 19085 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Stepping down as leader of term 28
I20260808 01:53:35.342159 19085 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:35.343075 19085 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:35.344539 19085 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:35.304495 18872 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:35.346431  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:35.347512 18872 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:35.347949  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:35.349251  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:35.350206  9634 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 29
I20260808 01:53:35.350883 19063 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:35.347535  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:35.353597  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:35.354172  9559 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:35.354889 19075 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:35.355333 18965 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:35.356177 19075 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:35.357586 19011 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:35.360217 19104 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:35.360776 19057 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:35.340723  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:35.299479 18699 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:53:35.313587 19086 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.565s	user 0.013s	sys 0.000s
I20260808 01:53:35.279992 19084 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:35.300943 18961 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:53:35.319836 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 195437us
I20260808 01:53:35.319271 18801 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:35.368989  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:35.374737  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:35.358098 19061 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:35.378014  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:35.379149  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:35.427330  9637 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:35.380717  9635 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260808 01:53:35.382287 18878 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260808 01:53:35.382718 18751 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
W20260808 01:53:35.383267 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 225302us
I20260808 01:53:35.390431  9634 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 29.
I20260808 01:53:35.368805  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:35.338253 19060 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.605s	user 0.008s	sys 0.003s
W20260808 01:53:35.337702 19139 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.327s	user 0.001s	sys 0.001s
I20260808 01:53:35.339488 18637 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260808 01:53:35.421257 19135 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
W20260808 01:53:35.340193 19141 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260808 01:53:35.423080  9638 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:35.376348  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:35.425221 18801 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Advancing to term 26
W20260808 01:53:35.446972 18843 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.797s	user 0.009s	sys 0.005s
I20260808 01:53:35.447187 19147 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:35.447549 18843 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:35.340126 19064 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:35.347855 19010 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:35.273422 18834 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:35.431351 19088 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:35.435750 18617 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:35.334237 18926 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:35.439245 18761 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:35.418430 19084 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:35.421914 18961 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:35.430843 19121 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:35.443153 19061 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:35.449823 18843 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:35.487562 19143 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260808 01:53:35.493479 19114 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:35.478657 19121 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:35.501791 19121 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:35.314566  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:35.566001  9472 raft_consensus.cc:2368] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 25. Current term is 26.
I20260808 01:53:35.364501  9558 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:35.571290  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:35.574798  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:35.340622  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:35.438141  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 69426.5us
I20260808 01:53:35.440377 10332 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:35.590989  9630 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
W20260808 01:53:35.580390  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:35.595429  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 317044us
W20260808 01:53:35.603492  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:35.604521  9524 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:35.606338  9524 leader_election.cc:400] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 25. Current term is 26.
I20260808 01:53:35.597747  9560 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:35.623729 10365 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:35.624845 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:35.625941 10361 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:35.626515 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:35.628973 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:35.594631  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:35.607438  9551 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260808 01:53:35.609601  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:35.638082  9473 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:35.645949  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:35.647943  9473 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 27: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:35.649106  9524 leader_election.cc:403] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260808 01:53:35.652272  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:35.653580  9481 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:35.654593 18611 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:35.655813 18611 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:35.661018  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:35.662446  9477 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:35.663595 18611 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:35.665252 18611 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:35.666052 18611 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:35.667071  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 72972.5us
I20260808 01:53:35.668566 18617 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:35.669673 18617 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:35.670109 18699 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:35.669651 18637 consensus_queue.cc:1059] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.002s
I20260808 01:53:35.672263 18637 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:35.673357 18617 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:35.676261 18617 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:35.676996 18617 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:35.662182  9599 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:35.704620 18611 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:35.684425  9523 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:35.684134 18690 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:35.677798 18617 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:35.715858 18690 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:35.717612 18834 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:35.719112  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:35.729231 19083 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:35.732712 19039 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260808 01:53:35.733997 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 109925us
I20260808 01:53:35.738838 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:35.740154 10366 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:35.720185  9473 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:35.677392 18637 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.066s
I20260808 01:53:35.742976 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:35.744794 19056 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:35.745527  9559 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 29.
I20260808 01:53:35.745995  9475 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:35.442999 19060 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:35.750941 19096 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:35.751295  9554 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 25: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:35.751593 19096 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:35.752139 19096 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:35.755726 19110 raft_consensus.cc:3055] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:35.756366 19110 raft_consensus.cc:740] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:35.757617 19124 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:35.759014 18926 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260808 01:53:35.755905  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 149959us
I20260808 01:53:35.761358  9524 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:35.745550 19056 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:35.763347 19105 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:35.763953 19105 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:35.764451 19105 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:35.765750 19088 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:35.767999  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:35.744848  9445 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:35.774653  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 112848us
I20260808 01:53:35.779878  9630 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
W20260808 01:53:35.781659  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 97540.3us
I20260808 01:53:35.783810 19060 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:35.783496 18690 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:35.784898 18690 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260808 01:53:35.786489  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:35.786834 18926 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:35.787268  9556 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:35.787474 18926 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:35.788908 18926 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:35.792405 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:35.796720 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:35.797497 18926 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:35.799731 10361 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:35.770087 19134 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260808 01:53:35.806898 19161 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.079s	user 0.004s	sys 0.000s
I20260808 01:53:35.770046 19153 raft_consensus.cc:2764] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260808 01:53:35.808642  9479 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 26.
I20260808 01:53:35.758174 19110 consensus_queue.cc:260] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:35.811115  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 1639) took 1214 ms. Trace:
I20260808 01:53:35.811664 19110 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:35.779448  9524 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:35.819617  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 75045.8us
I20260808 01:53:35.837975 19060 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:35.839089 19060 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:35.843648 19060 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:35.853698 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:35.730388 19083 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:35.876307  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 113766us
W20260808 01:53:35.880043  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:35.878794  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 110987us
I20260808 01:53:35.889569  9445 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:35.811652  9479 rpcz_store.cc:276] 0808 01:53:34.596284 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:34.597678 (+  1394us) service_pool.cc:224] Handling call
0808 01:53:35.810760 (+1213082us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x7b54002bc5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:35.810999 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2187713280}
W20260808 01:53:35.911702 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 333721us
I20260808 01:53:35.919950 18611 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260808 01:53:35.941064  9474 consensus_meta.cc:287] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.723s	user 0.013s	sys 0.011s
I20260808 01:53:35.941736  9559 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260808 01:53:35.943078 18834 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:35.943984 18834 raft_consensus.cc:697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:35.945384 18834 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:35.946735 18966 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260808 01:53:35.947140  9444 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.011s	sys 0.005s
W20260808 01:53:35.942989  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 69116.5us
W20260808 01:53:35.947619  9444 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.707s	user 0.011s	sys 0.006s
I20260808 01:53:35.941673  9474 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:35.947373 18966 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:35.955291 18865 consensus_queue.cc:1048] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.005s
I20260808 01:53:35.959232  9601 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:35.960414 19083 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:35.961539 18751 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:35.935727 10334 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:35.969955 18976 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
W20260808 01:53:35.970742 19059 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.004s	sys 0.009s
W20260808 01:53:35.971172 19059 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.004s	sys 0.009s
I20260808 01:53:35.972078 18976 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260808 01:53:35.972343 19056 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:35.973800 10366 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:35.985397  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:53:35.946786  9598 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.013s	sys 0.034s
I20260808 01:53:35.952804 18966 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:35.987094  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:35.990847  9598 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.014s	sys 0.035s
I20260808 01:53:35.990723  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:35.984614  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:36.000675 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:35.986316  9475 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
W20260808 01:53:36.002007  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 70202.4us
I20260808 01:53:35.974351  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:36.016923  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:36.017714 19083 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:36.019300 19083 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260808 01:53:36.041239 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 129158us
I20260808 01:53:36.043342 18675 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:36.043936 18675 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:36.045040 18675 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:36.055444  9475 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:35.935163 18971 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260808 01:53:36.071507 18801 consensus_meta.cc:287] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.561s	user 0.007s	sys 0.011s
W20260808 01:53:36.063674  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46912.1us
W20260808 01:53:36.038337 19084 consensus_meta.cc:287] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.561s	user 0.007s	sys 0.007s
I20260808 01:53:35.938024 18611 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:53:36.094172 19062 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 1.322s	user 0.014s	sys 0.007s
I20260808 01:53:36.096156 18926 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:36.116539 18926 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.117977  9475 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260808 01:53:36.105001  9481 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
I20260808 01:53:36.119124 18751 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:36.119951 18751 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Stepping down as leader of term 27
I20260808 01:53:36.120173 19056 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:36.120357 18751 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:36.121683 18751 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.123785 19129 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260808 01:53:36.125778 19137 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:36.126518 19137 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.128245 19124 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 25. Current term is 26.
W20260808 01:53:36.112561  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 153489us
I20260808 01:53:36.109290 18611 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:36.113482 19062 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:36.129128 19137 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:36.142022  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 65324.5us
I20260808 01:53:36.046018 18617 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.099244 19084 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.099438 18801 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:36.174486  9524 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:36.182865 10368 raft_consensus.cc:2368] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20.
I20260808 01:53:36.184531  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 4345) took 1042 ms. Trace:
W20260808 01:53:36.190595  9477 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.527s	user 0.009s	sys 0.009s
I20260808 01:53:36.186592  9558 rpcz_store.cc:276] 0808 01:53:35.142407 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:35.576463 (+434056us) service_pool.cc:224] Handling call
0808 01:53:35.577159 (+   696us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:35.578078 (+   919us) raft_consensus.cc:1534] Early marking committed up to index 18
0808 01:53:35.578736 (+   658us) log.cc:844] Serialized 14 byte log entry
0808 01:53:35.824397 (+245661us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:35.890670 (+ 66273us) log.cc:844] Serialized 26 byte log entry
0808 01:53:35.890915 (+   245us) raft_consensus.cc:1645] Marking committed up to 18
0808 01:53:35.890947 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:35.891042 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:36.184223 (+293181us) raft_consensus.cc:1685] finished
0808 01:53:36.184257 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:36.184447 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7340,"prepare.queue_time_us":158681,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":61,"raft.queue_time_us":91930,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":38703,"thread_start_us":242147,"threads_started":1,"wal-append.queue_time_us":331001}
W20260808 01:53:36.210988  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:36.212402  9551 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 27. Current term is 28. Ops: []
W20260808 01:53:36.203136  9555 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.517s	user 0.008s	sys 0.013s
W20260808 01:53:36.220074  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 109802us
I20260808 01:53:36.196733  9477 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28.
I20260808 01:53:36.260006 10369 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26. Ops: [25.20-25.20]
I20260808 01:53:36.262650  9555 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:36.263854  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:36.267014  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 80093us
W20260808 01:53:36.273814  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 69525.9us
W20260808 01:53:36.209141 19160 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260808 01:53:36.207558  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:36.277033  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:36.174778 19096 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.211915 18926 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:36.287474  9445 leader_election.cc:400] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20.
I20260808 01:53:36.220546 19059 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:36.208964 19084 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:36.215154 18751 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:36.286051  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53216 (request call id 3282) took 1073 ms. Trace:
I20260808 01:53:36.296821  9445 leader_election.cc:403] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
W20260808 01:53:36.304136  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:36.304919 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 108854us
I20260808 01:53:36.193686 19105 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:36.308807  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 45004.8us
I20260808 01:53:36.305608  9555 rpcz_store.cc:276] 0808 01:53:35.212636 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:35.318992 (+106356us) service_pool.cc:224] Handling call
0808 01:53:36.278738 (+959746us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:53:36.311439  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 128437us
I20260808 01:53:36.314994 18834 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260808 01:53:36.267774 19110 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.033s
I20260808 01:53:36.221969 18966 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:36.334959  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 66302.7us
I20260808 01:53:36.337123  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:36.331029  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:36.342877  9524 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:36.351930 19152 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:36.333299 18690 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:36.364621 19105 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:36.380023 19084 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:36.291872 10365 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d"
dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:57156
I20260808 01:53:36.343611 19124 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:36.337013 19115 consensus_queue.cc:1059] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.038s
I20260808 01:53:36.346580 19059 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:36.387174  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:36.386813 18834 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:36.396054 18834 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:36.393191 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:36.397202 10368 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Advancing to term 28
I20260808 01:53:36.371411 19153 raft_consensus.cc:3055] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:36.399699 19083 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:36.400066  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 63100.4us
I20260808 01:53:36.400534 19083 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.368165 18966 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:36.409338 19178 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260808 01:53:36.411008 19137 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:36.412272 18865 raft_consensus.cc:3055] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:36.417986 18865 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:36.419415  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:36.419272 18865 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.420562 19088 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:36.421201 18865 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:36.380511 18801 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:36.423090  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:36.428791 19124 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.386888 19084 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:53:36.425159  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 54521.1us
W20260808 01:53:36.432423 19061 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.921s	user 0.010s	sys 0.008s
W20260808 01:53:36.433431 18961 consensus_meta.cc:287] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.846s	user 0.018s	sys 0.000s
I20260808 01:53:36.433758 19124 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:36.434309  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 127202us
I20260808 01:53:36.435050 18617 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:36.404219 18834 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:36.403710 19153 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:36.428100  9524 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:36.443631 19062 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:36.444392 19134 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:36.444465 19164 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.107s
I20260808 01:53:36.452514 10369 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:36.459978 10365 raft_consensus.cc:606] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
W20260808 01:53:36.464157  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 86157.6us
I20260808 01:53:36.447815 18966 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:36.478032 10368 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:36.478214  9524 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:36.447476 18961 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.468954 19083 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:36.471091 19134 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:36.493465  9445 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:36.483112 19164 consensus_queue.cc:1243] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 25.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260808 01:53:36.452495 19084 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:36.497089  9521 leader_election.cc:336] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.567s (SENT)
I20260808 01:53:36.487140 18617 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:36.497677 18637 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260808 01:53:36.498207 19096 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:36.479394 19088 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:36.515189 18637 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:36.518085 18637 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:36.497341 19059 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.502206 19083 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election won for term 28
I20260808 01:53:36.523192 18637 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.20, Last appended by leader: 20, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:36.524765 18637 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
W20260808 01:53:36.527186  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 58358.6us
I20260808 01:53:36.531090  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:36.509070 19164 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.023s
I20260808 01:53:36.501361 19137 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.034s
I20260808 01:53:36.538307 18961 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:36.539397  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:36.514535 19152 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20.
I20260808 01:53:36.541452 19152 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election won for term 29
I20260808 01:53:36.543251 19152 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 29 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:36.544672 18611 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.544574 19152 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.547211  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:36.543072  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:36.550782  9477 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:36.480396 18966 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:53:36.506143 19167 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.303s	user 0.003s	sys 0.000s
I20260808 01:53:36.506364  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:36.510988 18617 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:36.511823  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:36.512507 18755 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:36.514703 19114 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:36.502082  9521 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:36.515409 19088 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.468294 19124 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.022s
I20260808 01:53:36.517109 19134 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:36.500401 18926 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election won for term 28
W20260808 01:53:36.566579  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:36.565675  9475 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:36.567991  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:36.568898  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:36.569204 19114 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.652619  9630 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:53:36.567638  9480 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260808 01:53:36.642319  9477 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 26.
I20260808 01:53:36.644042  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3280) took 1433 ms. Trace:
W20260808 01:53:36.645519 19179 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
W20260808 01:53:36.647573  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:36.648203  9476 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:36.559193 18966 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
W20260808 01:53:36.670033  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 201253us
I20260808 01:53:36.672871 19105 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:36.678920  9473 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
I20260808 01:53:36.652925 18926 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:36.674211 19061 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:36.655926  9557 rpcz_store.cc:276] 0808 01:53:35.210250 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:35.658803 (+448553us) service_pool.cc:224] Handling call
0808 01:53:35.659497 (+   694us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:35.684302 (+ 24805us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:35.684355 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:35.685424 (+  1069us) log.cc:844] Serialized 52 byte log entry
0808 01:53:35.934080 (+248656us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:53:35.934150 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:35.934264 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:36.466382 (+532118us) spinlock_profiling.cc:243] Waited 19.7 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:36.643605 (+177223us) raft_consensus.cc:1685] finished
0808 01:53:36.643628 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:36.643845 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":24752,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":35164,"spinlock_wait_cycles":45304192,"thread_start_us":248041,"threads_started":1,"wal-append.queue_time_us":470035}
I20260808 01:53:36.686151  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:36.688827  9481 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:53:36.685544  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:36.695916  9479 raft_consensus.cc:2410] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 27 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260808 01:53:36.666497 18611 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:36.665376 18865 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:36.707695  9524 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:36.709919  9521 leader_election.cc:336] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.593s (SENT)
I20260808 01:53:36.716001  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:36.715832 18617 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.719017 19075 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:36.719827  9635 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260808 01:53:36.721249 18926 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.722836 18865 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.725241 18865 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:36.719986 19075 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260808 01:53:36.728392 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 180911us
I20260808 01:53:36.741074 19134 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:36.741812 19061 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:36.744998 19061 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:36.746769 19075 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:36.747375 19075 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:36.669281 18755 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:36.765344  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 226042us
I20260808 01:53:36.770298  9472 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:36.706332 19129 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:36.769863  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 65116.2us
I20260808 01:53:36.715858 19088 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:36.777560  9476 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:36.779647  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:36.781625  9524 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:36.798115  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:36.799053  9635 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:36.800200  9524 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:36.781224 19084 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:36.803303  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:36.805029  9636 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
W20260808 01:53:36.811955 19181 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.137s	user 0.000s	sys 0.001s
I20260808 01:53:36.813855  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:36.814921  9633 raft_consensus.cc:2410] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 26 because replica has last-logged OpId of term: 25 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 19.
I20260808 01:53:36.815348  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:36.816082  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:36.816228  9634 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:36.821127  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:36.823050 19114 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:36.823338  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:36.824999  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:36.825335 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:36.826139  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:36.830356  9558 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:36.845148  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:36.833392 18755 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:36.847575 19153 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 23.19, Last appended by leader: 19, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:36.849078 19153 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:36.804618 19129 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.855206 19084 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:36.835155  9633 raft_consensus.cc:2410] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 26 because replica has last-logged OpId of term: 25 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 19.
I20260808 01:53:36.835512 19190 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:36.835845  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:36.836447 18751 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.529s	user 0.009s	sys 0.003s
I20260808 01:53:36.838793  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:36.840763  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:36.842639  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:36.842832  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:36.843354 19185 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:36.830842 19074 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:36.919844 19136 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:36.920490 19136 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:36.921671 19136 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.909744 19152 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260808 01:53:36.923202  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 128399us
I20260808 01:53:36.925302  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:36.928653  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:36.902717  9475 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28.
I20260808 01:53:36.929958  9635 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 26.
W20260808 01:53:36.902716  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 45356.6us
W20260808 01:53:36.913956 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 103467us
I20260808 01:53:36.909755  9550 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
I20260808 01:53:36.918830 19185 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.905191 18926 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:36.920506 19074 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:36.936376  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:36.946998  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:36.952224  9636 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:36.948704  9524 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:36.964237 10332 leader_election.cc:336] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.773s (SENT)
I20260808 01:53:36.951426 19152 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election won for term 27
W20260808 01:53:36.982760  9521 leader_election.cc:336] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.713s (SENT)
I20260808 01:53:36.987614  9521 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:36.983213  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:36.985749  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:36.994761  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:36.961037 18926 raft_consensus.cc:697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:37.006632  9634 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 29.
I20260808 01:53:36.951200 18751 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.036s
I20260808 01:53:36.905335 18865 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 27.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.077s
I20260808 01:53:36.976243 19083 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:36.953073 18834 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:37.031776 18834 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:37.032672  9478 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 25: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:37.034467 18690 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260808 01:53:37.036908 19189 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260808 01:53:37.015139 19136 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
W20260808 01:53:37.019867 19187 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260808 01:53:37.024051 18926 consensus_queue.cc:237] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:36.999841 19105 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election won for term 28
I20260808 01:53:37.043506 19105 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:37.027462 19074 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:37.029003 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 64803.4us
I20260808 01:53:36.930827 19190 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:37.051375  9636 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:53:37.052709  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:37.056552  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:37.063473 19075 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:37.066159 19075 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:37.066030 19083 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:37.071388  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 88704.2us
I20260808 01:53:37.044735 19105 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:37.072717  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:37.097599  9479 raft_consensus.cc:2368] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20.
I20260808 01:53:37.016897 18611 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:37.099153  9556 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:37.112483  9557 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 28.
I20260808 01:53:37.097441  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:37.126003  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:37.137475  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:37.138006  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:37.142974  9560 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: [25.18-25.18]
W20260808 01:53:37.143078  9521 leader_election.cc:336] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.965s (SENT)
I20260808 01:53:37.150537  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:37.157935  9551 raft_consensus.cc:2410] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 27 because replica has last-logged OpId of term: 25 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 12.
I20260808 01:53:37.152123  9521 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:37.159164 19129 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:37.168340  9558 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:37.169445  9480 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:37.149482 19152 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:37.174515 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 93966.2us
I20260808 01:53:37.160902 19185 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:37.191558  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:37.152087 18966 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:37.193259  9480 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:37.194667  9479 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
W20260808 01:53:37.198153 19197 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260808 01:53:37.200886 19084 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:37.202618 19152 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:37.191836 19059 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:37.204660 19059 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:37.205979 19190 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:37.206875 19195 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:37.207654 19195 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:37.208250 18971 raft_consensus.cc:3055] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:37.208746 18971 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:37.209781 18971 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:37.212157  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 76352.8us
I20260808 01:53:37.212173 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:37.228025  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:37.262506 18971 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:37.263392  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:37.263484  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.264554  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.265111 19203 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:37.273859  9631 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:53:37.265790  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.266022 19059 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:37.265895  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:37.267153  9476 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 26. Current term is 27. Ops: []
W20260808 01:53:37.267550  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:37.176743 18611 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260808 01:53:37.268249 18966 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:37.268344  9481 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260808 01:53:37.269001 19195 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:37.269483  9473 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 27. Current term is 28. Ops: [27.14-27.14]
I20260808 01:53:37.293546 10365 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 29.
I20260808 01:53:37.265352  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.265120  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.282958  9635 raft_consensus.cc:2368] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 26. Current term is 27.
I20260808 01:53:37.295228  9630 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260808 01:53:37.295491  9477 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260808 01:53:37.290160  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.291821  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:37.296906  9629 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 24.
I20260808 01:53:37.296823  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.299080  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.301082  9632 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:37.302507  9481 raft_consensus.cc:3055] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:37.303010  9481 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:37.289682 19203 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:37.303879  9481 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:37.304980  9481 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:37.305475 19203 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:37.307340  9635 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 28. Current term is 29. Ops: [28.18-28.18]
I20260808 01:53:37.311190 19205 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:37.311626  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:37.313741  9555 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
W20260808 01:53:37.311554  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48288.2us
W20260808 01:53:37.317636  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:37.317584  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:37.318230  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:37.319020  9521 leader_election.cc:336] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.134s (SENT)
I20260808 01:53:37.319574  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:37.320079  9479 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:37.303170  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:37.321708  9636 raft_consensus.cc:2393] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 24: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:37.306427  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:37.323554  9480 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28.
I20260808 01:53:37.330173  9481 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 23 index: 18. (term mismatch)
I20260808 01:53:37.331353  9481 pending_rounds.cc:85] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 17
I20260808 01:53:37.297793  9637 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:37.298906 19194 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260808 01:53:37.333541  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:37.331634 18611 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:37.335896  9633 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:37.338239  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:37.331758  9481 pending_rounds.cc:107] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 22.18
I20260808 01:53:37.348284  9481 raft_consensus.cc:2889] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:53:37.347242 10332 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.473s (SENT) [suppressed 19 similar messages]
I20260808 01:53:37.352846  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.366070  9634 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:37.365475  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.365474  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:37.379669 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:37.381596  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62639us
I20260808 01:53:37.382628  9473 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 26. Current term is 28. Ops: []
I20260808 01:53:37.383783  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.384177  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:37.384734  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.387640  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.389281 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:37.332417 19205 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:37.392083  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:37.392729  9481 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:37.350123 18690 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:37.396009  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 57890.7us
I20260808 01:53:37.350463 19203 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:37.335947 19062 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:37.398046  9634 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:37.399377  9630 raft_consensus.cc:2410] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 28 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260808 01:53:37.401212  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:37.401995  9480 raft_consensus.cc:2393] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 26: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:37.381371 19195 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:37.402519  9635 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:37.405236  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.408535  9632 raft_consensus.cc:2410] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 28 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260808 01:53:37.413244  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:37.408210  9521 leader_election.cc:336] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.738s (SENT)
I20260808 01:53:37.423652  9635 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:37.424451  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:37.425374  9633 raft_consensus.cc:2368] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20.
I20260808 01:53:37.425521  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.436626  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:37.437407 10365 raft_consensus.cc:1240] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 26. Current term is 28. Ops: []
I20260808 01:53:37.442871  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:37.445359  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:37.446288  9636 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 26: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:37.422842 19062 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:37.453207  9631 raft_consensus.cc:2410] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 25 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260808 01:53:37.454607  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.465592  9629 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 26: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:37.467402  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.457494  9481 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
I20260808 01:53:37.474000  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:37.404033 18690 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:37.477319  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 50931us
I20260808 01:53:37.485670  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:37.486740  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 78507.7us
I20260808 01:53:37.490388 19136 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:37.491204 19136 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:37.493721 19136 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:37.494261  9478 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:37.494535  9475 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:37.494992  9478 raft_consensus.cc:606] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:37.495391  9478 raft_consensus.cc:614] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:53:37.497069  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:37.478080 19203 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:37.499091 10332 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:37.501408 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:37.444226 19195 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:37.503113 19195 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:37.504971 18980 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:37.506175 19195 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:37.480777  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:37.442004 19207 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.069s
I20260808 01:53:37.552121 10334 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:37.553225 10334 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:37.552207 19203 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:37.555305 19207 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:37.507989  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:37.506564 18980 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:37.510535  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:37.510828  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.515381  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.556022 19121 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader election won for term 20
I20260808 01:53:37.509176 18690 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:37.518508  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.557216  9630 raft_consensus.cc:2368] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 26. Current term is 27.
I20260808 01:53:37.557307 19121 raft_consensus.cc:697] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:37.557049  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.558147  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.520563  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:37.523382  9551 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260808 01:53:37.527669  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:37.517194  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.540135  9476 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:37.543515  9599 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:37.555002 19203 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:37.558856  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.555452  9559 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:37.556025 19207 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:37.477470  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:37.520315 19164 raft_consensus.cc:3055] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Stepping down as leader of term 27
I20260808 01:53:37.563717  9632 raft_consensus.cc:2368] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 26. Current term is 27.
I20260808 01:53:37.560180  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.560204 19202 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:37.561427 19153 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:37.563988 19164 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:37.564858  9634 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:37.565595  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:37.558820  9630 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28.
I20260808 01:53:37.545837 19062 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:37.557776  9635 raft_consensus.cc:2410] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 27 because replica has last-logged OpId of term: 25 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 12.
I20260808 01:53:37.559888  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:37.558768  9636 raft_consensus.cc:2410] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 29 because replica has last-logged OpId of term: 28 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 13.
W20260808 01:53:37.587592  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:37.560417  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:37.617566  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.438184 19205 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:37.620852  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.620828  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.577721 19195 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:37.622579  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:37.565595 19153 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:37.632130  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.558764 19121 consensus_queue.cc:237] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:37.636823  9637 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:37.637634  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.565521 18961 consensus_queue.cc:1059] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.073s
I20260808 01:53:37.617507 19194 raft_consensus.cc:2764] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
W20260808 01:53:37.639392  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 79590.2us
I20260808 01:53:37.633798  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.653275  9635 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260808 01:53:37.656256  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:37.565346 19136 consensus_queue.cc:1059] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.083s
I20260808 01:53:37.648274 19153 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:37.680274  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.640432 19119 consensus_queue.cc:1059] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.021s
W20260808 01:53:37.696861 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 195542us
I20260808 01:53:37.634562 19205 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:37.711593  9633 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 13. (term mismatch)
I20260808 01:53:37.715664  9633 pending_rounds.cc:85] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 12
I20260808 01:53:37.715973  9633 pending_rounds.cc:107] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 24.13
I20260808 01:53:37.716219  9633 raft_consensus.cc:2889] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:37.566911 19164 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:37.565320 19202 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:37.717401  9634 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:37.728700  9629 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28.
I20260808 01:53:37.683133 19056 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:37.735718  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:37.566378 19207 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:37.745327  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:37.728278 18834 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.021s
I20260808 01:53:37.736919 19203 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:37.765236  9481 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
W20260808 01:53:37.772275  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:37.785354  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.786720  9636 raft_consensus.cc:2410] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 27 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260808 01:53:37.787955  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:37.802739  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:37.804646  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59421.8us
I20260808 01:53:37.749765 19056 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:37.788529 19152 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:37.808853 19164 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:37.834544  9444 leader_election.cc:400] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 19. Current term is 20.
I20260808 01:53:37.866381  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:37.850630 10368 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:37.883250  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:37.814885 19136 consensus_queue.cc:1243] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 24.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260808 01:53:37.871865 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:37.835937 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:37.871865 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 126882us
I20260808 01:53:37.885941  9560 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:37.913597 10369 raft_consensus.cc:2393] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 26: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:37.915995  9638 raft_consensus.cc:2368] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 26. Current term is 27.
I20260808 01:53:37.917532  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:37.925549  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:37.926218  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:37.927699  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 93305us
I20260808 01:53:37.921247  9524 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:37.929368  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:37.852277 19084 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:37.837044 18865 raft_consensus.cc:3055] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:37.919065  9636 raft_consensus.cc:2376] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23. Re-sending same reply.
I20260808 01:53:37.836198 19180 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:37.963138 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:37.959487  9523 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:37.954068  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:37.854667 19105 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:37.851913 19202 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:37.836264 19056 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:37.968179  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:37.968967  9634 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:37.973105 18617 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:37.974182  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:37.855042 19124 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.088s
I20260808 01:53:37.977789  9638 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260808 01:53:37.973297  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:37.953034 19223 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260808 01:53:37.979843  9476 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:37.982337  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.986043  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:37.986564  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.837862 19194 consensus_queue.cc:1059] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.133s
I20260808 01:53:37.982537  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:37.978215  9521 leader_election.cc:336] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.226s (SENT)
I20260808 01:53:37.961764 19137 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:37.956583 18865 raft_consensus.cc:740] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:37.974315 19061 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election won for term 28
I20260808 01:53:37.970927 19152 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:38.070396  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:38.052744  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.053926 19059 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:53:37.962436 18611 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:38.051625  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:37.968967 19200 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:38.078960  9630 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:38.069235  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:37.964402 19084 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:38.083060  9473 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260808 01:53:38.071144  9557 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:38.082406  9476 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 25.
I20260808 01:53:38.099599  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.106038  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:38.101107  9637 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:37.983161 19105 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:38.109882  9478 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:38.080277 19059 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:37.979557 19180 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:38.120869  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.082841 19061 raft_consensus.cc:697] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:38.113675  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.121807  9630 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 29.
I20260808 01:53:38.123440  9631 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:38.093473  9634 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:38.126991  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:38.129252 19224 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260808 01:53:38.130141 19228 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260808 01:53:38.133260  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3291) took 1061 ms. Trace:
W20260808 01:53:38.130700 19220 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260808 01:53:38.136626  9472 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3930) took 1326 ms. Trace:
I20260808 01:53:38.139174  9480 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260808 01:53:38.133692  9556 rpcz_store.cc:276] 0808 01:53:37.071831 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:37.161757 (+ 89926us) service_pool.cc:224] Handling call
0808 01:53:37.162666 (+   909us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:37.171841 (+  9175us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:37.171874 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:37.176337 (+  4463us) log.cc:844] Serialized 26 byte log entry
0808 01:53:37.640789 (+464452us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b180017bfc0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:53:37.713279 (+ 72490us) raft_consensus.cc:1645] Marking committed up to 17
0808 01:53:37.740541 (+ 27262us) log.cc:844] Serialized 14 byte log entry
0808 01:53:37.740788 (+   247us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:37.740880 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:37.827801 (+ 86921us) spinlock_profiling.cc:243] Waited 2.82 ms on lock 0x7b180017bfc0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188ce62 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:53:38.132997 (+305196us) raft_consensus.cc:1685] finished
0808 01:53:38.133026 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:38.133188 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17926,"prepare.queue_time_us":29867,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":5781,"raft.queue_time_us":14369,"raft.run_cpu_time_us":371,"raft.run_wall_time_us":18931,"spinlock_wait_cycles":36412672,"thread_start_us":70055,"threads_started":1,"wal-append.queue_time_us":720213}
I20260808 01:53:38.137029  9472 rpcz_store.cc:276] 0808 01:53:36.809817 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:36.809973 (+   156us) service_pool.cc:224] Handling call
0808 01:53:36.912948 (+102975us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:37.152693 (+239745us) raft_consensus.cc:1534] Early marking committed up to index 12
0808 01:53:37.346435 (+193742us) log.cc:844] Serialized 14 byte log entry
0808 01:53:37.680413 (+333978us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:37.702882 (+ 22469us) log.cc:844] Serialized 26 byte log entry
0808 01:53:37.703123 (+   241us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:53:37.703179 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:37.703287 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:38.136343 (+433056us) raft_consensus.cc:1685] finished
0808 01:53:38.136372 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:38.136536 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3371,"prepare.queue_time_us":247415,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":110615,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":25276,"thread_start_us":643387,"threads_started":3,"wal-append.queue_time_us":446780}
I20260808 01:53:38.140488 19084 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:38.127995  9637 raft_consensus.cc:2393] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 24: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:38.147835  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.147526 18865 consensus_queue.cc:260] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:38.148847  9637 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:38.149366 18865 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 28
I20260808 01:53:38.149890 19061 consensus_queue.cc:237] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:38.113770 19200 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:38.152638  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.154853 19088 raft_consensus.cc:3055] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:38.155361 19088 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:38.156414 19088 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:38.161720  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:38.152910 19200 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:38.131057  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 152873us
W20260808 01:53:38.166476  9521 leader_election.cc:336] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.790s (SENT)
I20260808 01:53:38.166972  9636 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:38.183646  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:38.120024 19059 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:38.187294  9636 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260808 01:53:38.095672 18611 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:38.157825 19088 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
W20260808 01:53:38.157917 19227 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.062s	user 0.000s	sys 0.002s
I20260808 01:53:38.096634 19137 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:38.192683  9634 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:38.158598 19136 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:53:38.205202  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 20a4f92418934341a6ac4927db61dc87 (from 6e7856bed1be43acaa821290d506ee97 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:38.214083 18961 raft_consensus.cc:3055] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:38.171710 19084 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:53:38.220110  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 53696.1us
W20260808 01:53:38.226563  9521 leader_election.cc:336] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.822s (SENT)
I20260808 01:53:38.229717 19084 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:38.225365 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 253151us
I20260808 01:53:38.191202 19200 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:38.236545  9633 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
W20260808 01:53:38.243122  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 290843us
I20260808 01:53:38.246766  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.250412  9635 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:38.250890  9635 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:38.251545  9629 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260808 01:53:38.250938  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.257467 10334 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:38.267369  9556 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:38.222605 18961 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:53:38.278410  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:38.313484  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:38.309639  9476 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3759209338e14437aff5f92a2fff36da"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:38.314858  9476 raft_consensus.cc:606] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:38.315421  9476 raft_consensus.cc:614] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:38.309336  9635 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:38.286631 18617 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:38.346601 18761 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260808 01:53:38.347133  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 3759209338e14437aff5f92a2fff36da (from 6e7856bed1be43acaa821290d506ee97 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:38.347497 18761 consensus_queue.cc:1243] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 27.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260808 01:53:38.336364  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:38.348274 18611 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:38.285820 18834 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:38.345994 18617 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:38.349107 18834 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:38.351565 18834 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:38.358964 18617 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:38.364596 19180 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:38.370460  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 143919us
I20260808 01:53:38.378336 19105 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:38.379261 19105 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:38.379971 19105 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:38.383762  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:38.384280 19137 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:38.330466 19230 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.063s
I20260808 01:53:38.334980  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:38.396524  9481 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
W20260808 01:53:38.399922  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 440892us
I20260808 01:53:38.225399 19136 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:38.295567 19229 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:38.340474 19084 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:38.287721 19205 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:38.410068  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:38.321483 19209 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:38.419319  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.422328  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:38.424221  9632 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28.
I20260808 01:53:38.262766 19200 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:38.411921 19227 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260808 01:53:38.441730  9552 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 27. Current term is 28. Ops: []
W20260808 01:53:38.448103  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:38.453168  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:38.452230  9472 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260808 01:53:38.454056  9554 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 26.
I20260808 01:53:38.412030 19229 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:38.465219  9481 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:38.465921  9481 raft_consensus.cc:606] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:38.465986 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:38.466380  9481 raft_consensus.cc:614] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:38.414134 19105 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:38.475435  9634 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:38.474181 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:38.478404  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.480060  9636 raft_consensus.cc:2368] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 26. Current term is 27.
W20260808 01:53:38.469599 19153 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.728s	user 0.010s	sys 0.004s
I20260808 01:53:38.484688 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:38.464160 19229 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:38.436020 19205 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:38.494843  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 3b8e0e8fda6744ad877514b11658b9d9 (from 6e7856bed1be43acaa821290d506ee97 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
W20260808 01:53:38.496740 19204 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 1.180s	user 0.009s	sys 0.002s
W20260808 01:53:38.501072  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:38.444432 19242 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:38.509186  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:38.510885  9550 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 28 index: 14. (term mismatch)
I20260808 01:53:38.513350  9550 pending_rounds.cc:85] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 13
I20260808 01:53:38.513621  9550 pending_rounds.cc:107] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 27.14
I20260808 01:53:38.513908  9550 raft_consensus.cc:2889] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:38.484589 19121 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.030s
I20260808 01:53:38.515513  9480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3925) took 1208 ms. Trace:
I20260808 01:53:38.515940  9480 rpcz_store.cc:276] 0808 01:53:37.320892 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:37.551620 (+230728us) service_pool.cc:224] Handling call
0808 01:53:38.138885 (+587265us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:38.139646 (+   761us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:38.515296 (+375650us) spinlock_profiling.cc:243] Waited 587 ms on lock 0x7b54002bbbd0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:38.515449 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1349594496}
I20260808 01:53:38.517254  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:38.519153  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 407cb6b113ce432587849bc840ba6872: Illegal state: Replica 52ccf4dcca1843f7bb29e9172e2d4ab4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 25 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:38.450227 19209 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:38.524820 19209 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:38.530165 19225 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.556s	user 0.002s	sys 0.000s
I20260808 01:53:38.531136  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:38.533092  9560 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 25: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:38.534835  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:38.536396  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45201us
I20260808 01:53:38.538796  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:38.540642 19200 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:38.541858 19195 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:38.542212  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:38.542876 19242 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:38.545465 19242 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:38.539130 19202 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:38.489046 18961 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:38.547041  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 44803.7us
I20260808 01:53:38.549249  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:38.537926 19059 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:38.547940  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:38.550839  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:38.559974  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:38.560361  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.564045  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:38.565017  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 239150us
I20260808 01:53:38.562974  9480 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7"
dest_uuid: "6e7856bed1be43acaa821290d506ee97"
mode: GRACEFUL
new_leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
 from {username='slave'} at 127.0.0.1:45190
I20260808 01:53:38.568429  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:38.569749  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:38.555346  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:38.578490  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:38.582108  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:38.587693  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:38.583598  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:38.590060  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:38.590706  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:38.577775  9521 leader_election.cc:336] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.044s (SENT)
I20260808 01:53:38.593999  9521 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:38.589491  9560 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:38.588519  9631 raft_consensus.cc:2376] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25. Re-sending same reply.
I20260808 01:53:38.492796 19153 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:38.601306  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.602782  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:38.607389  9631 raft_consensus.cc:2376] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28. Re-sending same reply.
I20260808 01:53:38.562891 19202 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:38.602808 10366 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
W20260808 01:53:38.601037  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 267530us
I20260808 01:53:38.588394 18961 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:38.592998 19239 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.044s
I20260808 01:53:38.633786  9636 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:38.629940  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:38.636713  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:38.622068  9524 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:38.606287 18690 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20260808 01:53:38.603307 19137 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:38.603379 19253 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260808 01:53:38.648510 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 414173us
I20260808 01:53:38.643095  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.638324  9630 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 29: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:38.639835  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.681236  9480 raft_consensus.cc:606] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Received request to transfer leadership to TS 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:38.683672  9480 raft_consensus.cc:614] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260808 01:53:38.690764  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 68758.4us
I20260808 01:53:38.638804 19061 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.049s
I20260808 01:53:38.708963  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.720084  9638 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 25: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:38.721011  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.721622  9638 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
I20260808 01:53:38.685577 10332 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:38.640928 19124 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Stepping down as leader of term 26
W20260808 01:53:38.714718  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 136981us
W20260808 01:53:38.728968 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 43525.2us
I20260808 01:53:38.715477  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:38.644726 19238 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:38.616416 19152 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.089s
I20260808 01:53:38.731314  9551 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:38.661161 19202 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:38.727396  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.685586 19194 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:38.735484  9638 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260808 01:53:38.735957  9636 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:38.638156 19254 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260808 01:53:38.736852  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:38.750234  9633 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:38.723385 18690 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:38.755664  9560 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260808 01:53:38.753772  9631 raft_consensus.cc:2376] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25. Re-sending same reply.
W20260808 01:53:38.762409  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:38.729259 19136 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:38.682858 19088 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.058s
I20260808 01:53:38.728591 19124 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:38.766305  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:38.735992 18865 consensus_meta.cc:287] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.586s	user 0.018s	sys 0.000s
I20260808 01:53:38.767084  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.772430 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:38.773602 10366 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 24: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:38.772657  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:38.686813 19137 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:38.781412  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:38.733477 19240 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260808 01:53:38.783502  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 139180us
I20260808 01:53:38.783150  9550 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:53:38.784377  9599 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:38.787688  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:38.792105  9634 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260808 01:53:38.763006 19200 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:38.777858 18971 raft_consensus.cc:3055] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Stepping down as leader of term 27
I20260808 01:53:38.762545 19153 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:38.763411 18690 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:38.793381  9551 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
W20260808 01:53:38.800963  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:38.783210 19242 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:38.816982  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.741824 19238 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:38.828538  9633 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 28. Current term is 29. Ops: []
I20260808 01:53:38.798812 19061 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260808 01:53:38.751345 19202 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:38.803383 18926 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:53:38.801483 19200 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:38.798655 19136 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:38.847903  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3654) took 1312 ms. Trace:
I20260808 01:53:38.798666 18971 raft_consensus.cc:740] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:38.859807  9630 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260808 01:53:38.805706 18690 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:38.640283  9366 catalog_manager.cc:5697] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 19 to 21, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 21 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:53:38.884119 19247 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
I20260808 01:53:38.827023 19124 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:38.886770 19263 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260808 01:53:38.888937 19205 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:38.849910 19242 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:38.895668 18971 consensus_queue.cc:260] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:38.897962 18971 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:38.911350 18865 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:38.911767 19202 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:38.915252 19137 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:38.851643 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:38.855858  9632 rpcz_store.cc:276] 0808 01:53:37.561889 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:37.565313 (+  3424us) service_pool.cc:224] Handling call
0808 01:53:38.826455 (+1261142us) spinlock_profiling.cc:243] Waited 740 ms on lock 0x7b5400307fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:38.826576 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1701332864}
I20260808 01:53:38.911662 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:38.857649  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:38.925596 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:38.926466 10369 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:38.884500 19238 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:38.935423 19059 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:38.936441 10369 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 30.
I20260808 01:53:38.940040  9523 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:38.863049 19136 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:53:38.888310 19249 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.227s	user 0.000s	sys 0.001s
W20260808 01:53:38.942302  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:38.944972  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:38.951861  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:38.942991 19262 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260808 01:53:38.977198  9479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3943) took 1103 ms. Trace:
I20260808 01:53:38.948526 19136 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:38.983487 19200 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:38.985874 19200 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:38.949007 19059 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election won for term 30
I20260808 01:53:38.990365 19059 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:38.992110  9473 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3949) took 1065 ms. Trace:
I20260808 01:53:38.993311  9473 rpcz_store.cc:276] 0808 01:53:37.927200 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:38.110946 (+183746us) service_pool.cc:224] Handling call
0808 01:53:38.123149 (+ 12203us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:38.123402 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:38.123430 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:38.131019 (+  7589us) log.cc:844] Serialized 26 byte log entry
0808 01:53:38.420271 (+289252us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:38.420316 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:38.420386 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:38.991686 (+571300us) raft_consensus.cc:1685] finished
0808 01:53:38.991728 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:38.992002 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":288845,"threads_started":1,"wal-append.queue_time_us":442542}
I20260808 01:53:38.977823  9479 rpcz_store.cc:276] 0808 01:53:37.873918 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:37.885529 (+ 11611us) service_pool.cc:224] Handling call
0808 01:53:37.920844 (+ 35315us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:37.939773 (+ 18929us) raft_consensus.cc:1534] Early marking committed up to index 14
0808 01:53:38.503360 (+563587us) spinlock_profiling.cc:243] Waited 1.48 ms on lock 0x7b1800463de0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d483aada7 00007f5d4499dd87 00007f5d449ad49f 00007f5d449ad418
0808 01:53:38.503732 (+   372us) log.cc:844] Serialized 14 byte log entry
0808 01:53:38.705603 (+201871us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:38.723868 (+ 18265us) log.cc:844] Serialized 26 byte log entry
0808 01:53:38.724066 (+   198us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:53:38.724117 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:38.724213 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:38.976625 (+252412us) raft_consensus.cc:1685] finished
0808 01:53:38.976811 (+   186us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:38.977092 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30755,"prepare.queue_time_us":564501,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":49,"raft.queue_time_us":27078,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":33143,"spinlock_wait_cycles":3414912,"thread_start_us":192732,"threads_started":1,"wal-append.queue_time_us":273936}
I20260808 01:53:38.997882 19270 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:38.998929 19270 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.002770 19270 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:38.953586  9631 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28.
I20260808 01:53:38.974088 19229 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:39.007871  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:39.009821  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:39.023108  9554 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:39.028174  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:39.028998  9550 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 29.
I20260808 01:53:38.992089 19059 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:39.030227 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:39.036885  9554 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 27.
I20260808 01:53:39.037853 19253 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:39.038306 19253 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:39.038606  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3921) took 1733 ms. Trace:
I20260808 01:53:39.039062  9474 rpcz_store.cc:276] 0808 01:53:37.305305 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:37.538749 (+233444us) service_pool.cc:224] Handling call
0808 01:53:37.539306 (+   557us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:37.549354 (+ 10048us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:37.549405 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:37.550200 (+   795us) log.cc:844] Serialized 26 byte log entry
0808 01:53:37.712968 (+162768us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:53:37.732564 (+ 19596us) log.cc:844] Serialized 14 byte log entry
0808 01:53:37.732813 (+   249us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:37.851911 (+119098us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b18004cff60. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:53:37.913078 (+ 61167us) spinlock_profiling.cc:243] Waited 9.52 ms on lock 0x7b54002dbed4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:37.913151 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:38.529434 (+616283us) spinlock_profiling.cc:243] Waited 1.12 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:39.025684 (+496250us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b2400013a78. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41713594 00007f5d4171169b 00007f5d418cd6d9 00007f5d418cd365 00007f5d418cc8c0 00007f5d4455a638 00007f5d4455ad98 00007f5d44532588 00007f5d4452d95a 00007f5d4452cda6
0808 01:53:39.038379 (+ 12695us) raft_consensus.cc:1685] finished
0808 01:53:39.038405 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:39.038544 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":23134,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":6126,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":252,"spinlock_wait_cycles":67726464,"thread_start_us":162268,"threads_started":1,"wal-append.queue_time_us":423030}
I20260808 01:53:39.013890 19229 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:39.014992  9630 raft_consensus.cc:2376] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28. Re-sending same reply.
I20260808 01:53:39.019022  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:39.043268  9475 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 27
I20260808 01:53:39.020673  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:39.022327 19243 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.490s	user 0.003s	sys 0.000s
I20260808 01:53:39.047173  9634 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 29.
I20260808 01:53:39.011451  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:39.049489  9636 raft_consensus.cc:2393] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 25: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:39.056785  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:39.058373  9559 raft_consensus.cc:2393] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 26: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:39.065444 10332 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:39.069459  9475 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 27.
W20260808 01:53:39.075565 19250 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.388s	user 0.000s	sys 0.005s
I20260808 01:53:38.941458 18865 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:39.077291  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3462) took 1081 ms. Trace:
I20260808 01:53:39.079298 18865 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:39.077962  9637 rpcz_store.cc:276] 0808 01:53:37.996144 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:38.149670 (+153526us) service_pool.cc:224] Handling call
0808 01:53:38.150130 (+   460us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:38.150932 (+   802us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:38.150963 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:38.151553 (+   590us) log.cc:844] Serialized 26 byte log entry
0808 01:53:38.442871 (+291318us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:53:38.442933 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:38.443034 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:39.002494 (+559460us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:39.076774 (+ 74280us) raft_consensus.cc:1685] finished
0808 01:53:39.076813 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:39.077197 (+   384us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":131,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"spinlock_wait_cycles":43301632,"thread_start_us":290740,"threads_started":1,"wal-append.queue_time_us":455803}
W20260808 01:53:39.080289  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet affdd24b03ae4ee795a90703613305f7 (from 6e7856bed1be43acaa821290d506ee97 to 52ccf4dcca1843f7bb29e9172e2d4ab4) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:39.080925  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260808 01:53:39.062390 19253 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:39.095381 19242 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:39.096277  9474 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:39.098933  9632 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 24: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:39.100986  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3296) took 1961 ms. Trace:
I20260808 01:53:39.101821  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:39.104296  9480 raft_consensus.cc:1240] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:39.101392  9555 rpcz_store.cc:276] 0808 01:53:37.139596 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:37.321333 (+181737us) service_pool.cc:224] Handling call
0808 01:53:37.361109 (+ 39776us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:37.388257 (+ 27148us) raft_consensus.cc:1534] Early marking committed up to index 15
0808 01:53:37.408662 (+ 20405us) log.cc:844] Serialized 14 byte log entry
0808 01:53:37.809264 (+400602us) spinlock_profiling.cc:243] Waited 1.69 ms on lock 0x7b18002c1580. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d4453242a 00007f5d4499de0a
0808 01:53:38.476579 (+667315us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:38.476640 (+    61us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:38.476696 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:39.100751 (+624055us) spinlock_profiling.cc:243] Waited 26.9 ms on lock 0x7b54001aca54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:39.100800 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:39.100906 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17584,"prepare.queue_time_us":20221,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"raft.queue_time_us":161626,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":119,"spinlock_wait_cycles":65682688,"thread_start_us":649323,"threads_started":2,"wal-append.queue_time_us":929215}
I20260808 01:53:39.106751  9635 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:39.066448  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:39.110975  9551 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:39.111655  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:39.114274 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:39.115381 10369 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
W20260808 01:53:39.116771  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:39.030231  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:39.120834 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 93912.6us
I20260808 01:53:39.109788  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:39.134225  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:39.136261  9479 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260808 01:53:39.136511 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:39.137293 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 71964.8us
W20260808 01:53:39.139693  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 78891.8us
I20260808 01:53:39.141776  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:39.145620  9635 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
W20260808 01:53:39.146894  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:39.142459  9552 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:39.141322  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:53:39.152063  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:39.126549 19242 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:39.153052  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:39.149138  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:39.157411 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:39.162002  9521 leader_election.cc:336] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.159s (SENT)
I20260808 01:53:39.158025 19202 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:39.159018 10365 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260808 01:53:39.162725  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:39.168769  9479 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260808 01:53:39.170037  9481 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260808 01:53:39.165557 19202 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:39.195739 19105 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:39.149916 19240 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:39.237428  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:39.248729  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 75915us
I20260808 01:53:39.249826  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:39.238991 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 67170.7us
I20260808 01:53:39.250749  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:39.251554  9479 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
W20260808 01:53:39.246552  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 98332.2us
W20260808 01:53:39.255971  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 93994.6us
W20260808 01:53:39.256170  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 107125us
W20260808 01:53:39.257995  9521 leader_election.cc:336] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.212s (SENT)
I20260808 01:53:39.253938  9600 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:39.259495  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:39.265635 10332 leader_election.cc:336] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.057s (SENT)
W20260808 01:53:39.267998  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:39.249557 19269 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:39.247444 19202 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:39.273612  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:39.261700 19152 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:39.261950 19195 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:39.267810 19209 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:53:39.300083 19209 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:39.262235 19238 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:39.276945  9599 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:39.285274  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:39.285952  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:39.283998 18761 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:39.314019 18761 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:39.314481 18761 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:39.280440 19266 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.035s
I20260808 01:53:39.293936  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:39.294279 19152 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:39.316650 19152 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.295202 19059 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:39.296327 19124 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:39.297348 19136 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:39.323001 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 64122us
I20260808 01:53:39.276471 19269 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:39.324617 19136 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:39.298746 19195 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:39.277293 19240 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:39.261461 19180 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.023s
I20260808 01:53:39.326265 19240 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:39.300855  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:39.276381  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:39.304059  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:53:39.328349  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 52069.2us
I20260808 01:53:39.315217 19248 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:53:39.329244 18961 consensus_meta.cc:287] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.696s	user 0.012s	sys 0.000s
I20260808 01:53:39.329003 19248 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.330091  9629 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260808 01:53:39.330873 19088 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:39.331472 19205 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260808 01:53:39.331856 19271 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
W20260808 01:53:39.333060 19260 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.447s	user 0.002s	sys 0.003s
I20260808 01:53:39.333788 19203 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260808 01:53:39.334849 19270 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260808 01:53:39.335767 19270 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Stepping down as leader of term 28
I20260808 01:53:39.331391 19248 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:39.336143 19270 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:39.157006  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:53:39.340875  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:39.324985 19269 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
W20260808 01:53:39.324666 10332 leader_election.cc:336] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.072s (SENT)
I20260808 01:53:39.346746  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:39.351732  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:39.351894  9635 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:53:39.353287 19242 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:39.353592  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:39.353977  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:39.355350  9555 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260808 01:53:39.331339 19088 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:39.356256 19088 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:39.358217  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:39.309098 19209 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:39.326089 19195 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.358934  9633 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:39.368459 19253 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:39.370337 19261 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.462s	user 0.001s	sys 0.002s
I20260808 01:53:39.339448  9366 catalog_manager.cc:5697] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 19 to 20, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 20 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:39.371912  9476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3951) took 1052 ms. Trace:
I20260808 01:53:39.372417  9476 rpcz_store.cc:276] 0808 01:53:38.319676 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:38.340362 (+ 20686us) service_pool.cc:224] Handling call
0808 01:53:38.340866 (+   504us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:38.342523 (+  1657us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:38.342571 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:38.495292 (+152721us) spinlock_profiling.cc:243] Waited 51.6 ms on lock 0x7b2c00134ff8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:53:38.495942 (+   650us) log.cc:844] Serialized 52 byte log entry
0808 01:53:38.681047 (+185105us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:38.681093 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:38.685417 (+  4324us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:39.369503 (+684086us) spinlock_profiling.cc:243] Waited 27.5 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:39.371624 (+  2121us) raft_consensus.cc:1685] finished
0808 01:53:39.371653 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:39.371836 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":714,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":120,"spinlock_wait_cycles":181812736,"thread_start_us":184664,"threads_started":1,"wal-append.queue_time_us":309519}
I20260808 01:53:39.373629  9366 catalog_manager.cc:5697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:39.384100 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:39.383102  9551 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 29.
I20260808 01:53:39.387158  9559 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 28. Current term is 29. Ops: [28.19-28.19]
W20260808 01:53:39.393636  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 116778us
W20260808 01:53:39.397238  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 56439.9us
W20260808 01:53:39.398304 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 73678.8us
W20260808 01:53:39.414311  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 3b8e0e8fda6744ad877514b11658b9d9: Illegal state: Replica 6e7856bed1be43acaa821290d506ee97 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 28 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:39.340458 19238 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.336772 18761 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:39.424008  9599 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:39.393177 19105 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:39.439245  9366 catalog_manager.cc:5697] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 19 to 20, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 20 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:39.406713 19245 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:39.406339 19244 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260808 01:53:39.454406  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 67251.2us
W20260808 01:53:39.457863  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:39.441716  9366 catalog_manager.cc:5697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 16 to 21, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 21 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:39.469237  9601 leader_election.cc:400] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20.
I20260808 01:53:39.469604  9601 leader_election.cc:403] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260808 01:53:39.466742  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:39.471316  9550 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 25
I20260808 01:53:39.472576  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:39.339536 18690 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.473445  9557 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 27
I20260808 01:53:39.435111 19203 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:39.483690  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3681) took 1049 ms. Trace:
I20260808 01:53:39.490994  9629 rpcz_store.cc:276] 0808 01:53:38.420938 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:38.421411 (+   473us) service_pool.cc:224] Handling call
0808 01:53:38.449580 (+ 28169us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:39.330454 (+880874us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:39.470299 (+139845us) spinlock_profiling.cc:243] Waited 880 ms on lock 0x7b540003c5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:39.470554 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2024119552}
I20260808 01:53:39.491117 19203 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:39.493995 19203 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:39.498446  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 204618us
W20260808 01:53:39.501361  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:39.337203 19270 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.462031 18971 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election won for term 29
I20260808 01:53:39.468886 19194 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:39.509464  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:39.517992 10365 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260808 01:53:39.469612 19271 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:39.495743 19200 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.501s	user 0.013s	sys 0.000s
I20260808 01:53:39.516230  9551 raft_consensus.cc:1240] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:39.483767 19239 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:39.534876  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:39.535408 10332 leader_election.cc:336] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.113s (SENT)
I20260808 01:53:39.503300 19203 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.026s
I20260808 01:53:39.528770 19244 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260808 01:53:39.497483 19202 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.534178 19164 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 19. Current term is 20.
I20260808 01:53:39.550423 19136 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:39.577345  9369 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50068 (request call id 242) took 41366 ms (client timeout 15000 ms). Trace:
I20260808 01:53:39.525969 19194 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.576087 19267 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:39.583614  9473 raft_consensus.cc:2368] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 26. Current term is 27.
W20260808 01:53:39.585767  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 101264us
I20260808 01:53:39.548908 19239 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.557436 19205 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.025s
I20260808 01:53:39.531140 19271 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.576035 19136 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.591295 19152 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:39.615577 19238 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:39.615464 19152 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.617902 19152 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:39.618328 19271 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:39.591447 19202 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:39.575304  9366 catalog_manager.cc:5697] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:39.595448 19194 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:39.599817 19203 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:39.577903  9369 rpcz_store.cc:269] 0808 01:52:58.035263 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:04.620609 (+6585346us) service_pool.cc:224] Handling call
0808 01:53:39.382802 (+34762193us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:53:33.870522 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:53:34.178130 (+307608us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:53:34.178197 (+    67us) write_op.cc:435] Acquired schema lock
0808 01:53:34.178214 (+    17us) tablet.cc:662] Decoding operations
0808 01:53:34.195113 (+ 16899us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:53:34.195239 (+   126us) write_op.cc:641] Partition lock acquired for write op
0808 01:53:34.195267 (+    28us) tablet.cc:685] Acquiring locks for 26 operations
0808 01:53:34.297274 (+102007us) tablet.cc:701] Row locks acquired
0808 01:53:34.297302 (+    28us) write_op.cc:260] PREPARE: finished
0808 01:53:34.297574 (+   272us) write_op.cc:270] Start()
0808 01:53:34.297778 (+   204us) write_op.cc:276] Timestamp: P: 1786154014297475 usec, L: 0
0808 01:53:34.297819 (+    41us) op_driver.cc:348] REPLICATION: starting
0808 01:53:34.438209 (+140390us) log.cc:844] Serialized 10194 byte log entry
0808 01:53:35.341065 (+902856us) op_driver.cc:464] REPLICATION: finished
0808 01:53:35.513661 (+172596us) write_op.cc:301] APPLY: starting
0808 01:53:35.513795 (+   134us) tablet.cc:1367] starting BulkCheckPresence
0808 01:53:35.964518 (+450723us) tablet.cc:1370] finished BulkCheckPresence
0808 01:53:35.964731 (+   213us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:53:36.533581 (+568850us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:53:36.534087 (+   506us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=52,key_file_lookups=52,delta_file_lookups=0,mrs_lookups=0
0808 01:53:36.534154 (+    67us) write_op.cc:312] APPLY: finished
0808 01:53:36.534977 (+   823us) log.cc:844] Serialized 228 byte log entry
0808 01:53:36.739019 (+204042us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4186fa5c 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb
0808 01:53:36.916406 (+177387us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:53:36.916926 (+   520us) write_op.cc:454] Released schema lock
0808 01:53:36.917018 (+    92us) write_op.cc:341] FINISH: Updating metrics
0808 01:53:36.917659 (+   641us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:53:36.917693 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1920221,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":172132,"apply.run_cpu_time_us":84001,"apply.run_wall_time_us":1403933,"cfile_cache_hit":106,"cfile_cache_hit_bytes":112963,"num_ops":26,"prepare.queue_time_us":674702,"prepare.run_cpu_time_us":22315,"prepare.run_wall_time_us":776137,"raft.queue_time_us":500664,"raft.run_cpu_time_us":20971,"raft.run_wall_time_us":143725,"replication_time_us":1042940,"spinlock_wait_cycles":28987776,"thread_start_us":1279829,"threads_started":5,"wal-append.queue_time_us":710905,"wal-append.run_cpu_time_us":68495,"wal-append.run_wall_time_us":2483972}]]}
I20260808 01:53:39.605600 18761 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:39.559417 19200 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.020s
W20260808 01:53:39.608311 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 72941.4us
I20260808 01:53:39.607731 19239 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:39.634471  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:39.635474  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:39.651037  9478 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:39.636996  9557 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
I20260808 01:53:39.633554  9550 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
W20260808 01:53:39.672938  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:39.674129  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:39.674785  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:39.718868  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 44186.6us
I20260808 01:53:39.727420  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:39.728526 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:39.729278 10365 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:39.732400 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:39.632851 19271 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:39.587875 19267 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:39.649495 19222 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:39.740127  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 203982us
W20260808 01:53:39.741577  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:39.729350 19230 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:39.753372  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 248578us
I20260808 01:53:39.670552 19256 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.074s
I20260808 01:53:39.764129 10361 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260808 01:53:39.671265 19203 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.768460  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:39.726363 19194 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.042s
W20260808 01:53:39.770310  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 314745us
I20260808 01:53:39.746766 19202 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:39.778123  9366 catalog_manager.cc:5697] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 17 to 24. New cstate: current_term: 24 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 01:53:39.784014 19278 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.198s	user 0.001s	sys 0.000s
W20260808 01:53:39.785159 19284 log.cc:927] Time spent T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260808 01:53:39.784960 19088 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.788348 19088 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:39.780364  9366 catalog_manager.cc:5697] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 5 to 17, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 17 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:39.795733  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:39.796823 19238 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:39.798135 19238 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260808 01:53:39.753834 19222 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:39.803434 19222 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:39.808389 19267 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:39.812021  9478 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28.
I20260808 01:53:39.812171 19209 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.812984 19293 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260808 01:53:39.820936  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:39.759199 18690 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:39.825486 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:39.747128 19271 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.816556 19164 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:39.859023  9631 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 29. Current term is 30. Ops: [29.16-29.16]
I20260808 01:53:39.827097 19195 raft_consensus.cc:2764] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260808 01:53:39.864470 19205 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:39.865801  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:39.868161  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 91505us
I20260808 01:53:39.889460  9633 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 27
I20260808 01:53:39.869604 19291 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.014s
I20260808 01:53:39.872936 19084 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:53:39.876097 19269 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:39.876477 19209 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:39.877346  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:39.878122 19164 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.823804 19194 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.055s
I20260808 01:53:39.925318  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:39.879398 19136 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:39.926052  9630 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:39.925961  9366 catalog_manager.cc:5697] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:39.768693 19230 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:39.931448 19230 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:39.932196 10369 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:39.881588  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:39.933305  9637 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:39.882306 19271 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:39.891227 19303 consensus_queue.cc:1059] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.050s
I20260808 01:53:39.935338 19303 raft_consensus.cc:3055] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:39.935858 19303 raft_consensus.cc:740] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:39.928396  9366 catalog_manager.cc:5697] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:39.936875 19303 consensus_queue.cc:260] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.886770  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:39.888053  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:39.906988  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:39.941437  9636 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:39.941885 19164 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:39.943420 19203 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:39.907786 19205 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:39.945864 19205 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:39.908175 19248 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 8, Time since last communication: 0.035s
I20260808 01:53:39.939970  9550 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:39.946789 19105 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:39.940573  9557 raft_consensus.cc:2410] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 28 because replica has last-logged OpId of term: 26 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 13.
I20260808 01:53:39.939445  9366 catalog_manager.cc:5697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 10 to 15, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 15 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:39.902667 10365 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
W20260808 01:53:39.888578  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47738us
I20260808 01:53:39.964695  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:39.902191  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 145371us
I20260808 01:53:39.902539  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:39.902345  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:39.966969  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:39.967762  9481 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:53:39.900027  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:39.916849  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:39.968515  9629 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:39.920608  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:39.922318  9634 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260808 01:53:39.901942  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:39.900310  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:39.970420  9555 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:39.970646  9632 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:39.957930  9473 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:39.959702 19269 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:39.938390 19303 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:39.893646  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:39.893199  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:39.888932  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 116303us
I20260808 01:53:39.975399  9633 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:39.977367  9630 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:39.920248  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:39.988850  9599 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:39.978559  9550 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:39.992259 10334 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:39.997638  9472 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:40.024384 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:53:40.025686  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59410.6us
I20260808 01:53:40.025751  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:40.026896  9521 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:39.996748  9551 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 27. Current term is 28. Ops: []
W20260808 01:53:39.994915  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 101792us
I20260808 01:53:40.036276  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:40.037086  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 116900us
I20260808 01:53:40.043210  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:40.043591  9555 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:40.037451  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:40.001484  9600 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:39.976074 18971 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:39.982797 19245 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:40.055568  9521 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:39.994117  9445 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:40.057441  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 63456.4us
I20260808 01:53:40.028587 18865 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:40.062407  9521 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:40.064379  9601 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:40.065011  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 63659.4us
I20260808 01:53:40.039306 19203 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:40.039501 18690 raft_consensus.cc:2764] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260808 01:53:40.081040  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:39.997117 19205 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:40.047056 19271 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:40.045837 18617 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election won for term 28
I20260808 01:53:40.050930 19194 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:40.064755 19209 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260808 01:53:40.069631 19244 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:40.079066 19164 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:40.047644 18611 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
I20260808 01:53:40.063941 19309 raft_consensus.cc:2804] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:40.055804 19222 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.047s
I20260808 01:53:40.074395 19256 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:40.064222 19059 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.054s
W20260808 01:53:40.118209 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 59148.4us
I20260808 01:53:40.124914  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:40.067390 19297 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.051s
I20260808 01:53:40.105085 19194 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:40.113683 18761 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:40.076954 18865 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:40.144692  9601 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:40.148725 19296 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:40.149284 19296 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:40.149694 19296 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:40.154877 18761 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:40.156002 18761 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:40.124802 18971 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:40.145731 19211 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 LEADER]: Stepping down as leader of term 27
I20260808 01:53:40.169936 19211 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:40.144270 19194 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
W20260808 01:53:40.172260 19312 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
I20260808 01:53:40.145926 19231 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.028s
I20260808 01:53:40.176646 19269 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:40.104418 19205 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:40.124475 18617 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:40.063781 19245 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:40.138229 19084 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260808 01:53:40.132148 19164 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:53:40.218549  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 73959.7us
W20260808 01:53:40.218421 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 72124.6us
W20260808 01:53:40.223814 10332 leader_election.cc:336] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 2.473s (SENT)
I20260808 01:53:40.224298 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:40.225152  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:40.147620 19209 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:40.145098 19309 raft_consensus.cc:697] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:53:40.298835  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:40.349715 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 167372us
I20260808 01:53:40.228036 19254 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:40.349898  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 65148.2us
I20260808 01:53:40.251184 19256 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:40.349290 19310 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:40.355180 19248 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260808 01:53:40.411577 19314 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.273s	user 0.000s	sys 0.001s
W20260808 01:53:40.412722 19313 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.280s	user 0.001s	sys 0.000s
W20260808 01:53:40.414283 19316 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.184s	user 0.000s	sys 0.000s
I20260808 01:53:40.379637 19269 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:40.447033 19309 consensus_queue.cc:237] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:40.448719 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 224925us
I20260808 01:53:40.449272 19269 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:40.437700  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 43661.9us
I20260808 01:53:40.395684 18611 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:40.459693  9601 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:40.427891 19230 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260808 01:53:40.481047  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:40.485836  9601 leader_election.cc:400] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 26. Current term is 27.
I20260808 01:53:40.486371  9601 leader_election.cc:403] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:53:40.478343 19256 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260808 01:53:40.493470  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 440577us
I20260808 01:53:40.494493  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:40.495528 19269 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:40.496423  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:40.434127 19056 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:40.500432  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:40.503664  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:40.503698  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:40.506196  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:40.507081  9634 raft_consensus.cc:2376] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 26. Re-sending same reply.
I20260808 01:53:40.370193 19254 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:40.513304  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:40.514848 19257 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 26. Current term is 27.
I20260808 01:53:40.508882 19271 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.007s
I20260808 01:53:40.516288 19296 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:40.509024 19202 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:40.495963 19269 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:40.512341  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:40.501888 18761 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:40.517838 19238 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:40.434846 19164 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 23.19, Last appended by leader: 19, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:40.429558 18617 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:40.506610 18834 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:40.535326  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:40.518087 19202 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:40.480504 18611 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:40.528658  9633 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260808 01:53:40.533389  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:40.534080  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:40.536013  9472 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
I20260808 01:53:40.539045  9558 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260808 01:53:40.539213  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:40.626906  9629 raft_consensus.cc:2376] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 29. Re-sending same reply.
I20260808 01:53:40.539461  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:40.540453 19238 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:40.545998  9638 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260808 01:53:40.547283  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:40.628705  9637 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:40.629334  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:40.547956 18865 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:40.551934 19194 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:40.605069  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:40.637471  9555 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:40.624462  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:40.625684  9551 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
W20260808 01:53:40.639433  9521 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.018s	sys 0.024s
I20260808 01:53:40.553459 19239 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:40.524292 19271 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:40.646663  9556 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 15. (term mismatch)
I20260808 01:53:40.627197  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:40.647171  9556 pending_rounds.cc:85] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 14
I20260808 01:53:40.629001 19238 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:40.514284  9634 raft_consensus.cc:2376] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20. Re-sending same reply.
I20260808 01:53:40.651535  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:40.652853  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:40.654134  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:40.657846 19239 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:40.635644  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 145c5a9982a24c8fa4e86d0d99cbbdfb: Illegal state: Replica 6e7856bed1be43acaa821290d506ee97 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 27 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:40.639273  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:40.522600 19269 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:40.663898 19329 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s
W20260808 01:53:40.635054 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 184969us
I20260808 01:53:40.642905  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:40.666157  9478 raft_consensus.cc:3060] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
W20260808 01:53:40.643273  9521 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.516s	user 0.018s	sys 0.024s
I20260808 01:53:40.535028 18834 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:40.647542  9556 pending_rounds.cc:107] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 26.15
I20260808 01:53:40.680447  9556 raft_consensus.cc:2889] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:40.639140  9636 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:40.655638  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:40.659755 19303 consensus_meta.cc:287] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.667s	user 0.013s	sys 0.002s
W20260808 01:53:40.668834  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 606482us
I20260808 01:53:40.615024 19293 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:40.602267 19202 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:40.687247  9637 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:40.703071  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:40.703594  9521 leader_election.cc:400] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 26. Current term is 27.
I20260808 01:53:40.706183  9478 raft_consensus.cc:2468] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
I20260808 01:53:40.708600  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:40.710947  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:40.723132 19324 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
W20260808 01:53:40.724478 19322 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.297s	user 0.000s	sys 0.002s
I20260808 01:53:40.725867 19296 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:40.727156 18611 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:40.727458  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:40.689363 18834 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:40.647115 19271 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:40.667364 19239 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:40.721271  9476 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24. Ops: [23.18-23.18]
I20260808 01:53:40.690523 19205 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:40.721899 19323 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260808 01:53:40.700891 19293 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:40.717360 19222 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:40.769311  9633 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260808 01:53:40.718793 19319 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:40.749116 19271 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
W20260808 01:53:40.776695  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:40.774658  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 128449us
I20260808 01:53:40.780007  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:40.781327 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 136354us
I20260808 01:53:40.781822  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:40.782986  9633 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
W20260808 01:53:40.777607 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 110769us
I20260808 01:53:40.788223 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:53:40.787084  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 68693.6us
I20260808 01:53:40.769961 19266 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:40.773119 19222 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:40.797605  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:40.758328 19269 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:40.766618 19254 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:40.775152 19194 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:40.799146  9478 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260808 01:53:40.802166  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:40.814980 19164 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:40.820056 19164 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:53:40.781302 19319 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:40.826105 19319 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:40.832599 19293 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:40.834535 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:40.832670 19222 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:40.847860 19203 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:40.848667 19203 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:40.848929 19222 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:40.856473  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:40.856030 18834 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:40.858002  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:40.859694  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 835681us
W20260808 01:53:40.810654 10332 leader_election.cc:336] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.645s (SENT)
I20260808 01:53:40.839178  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:40.869930 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59315.9us
I20260808 01:53:40.870182  9633 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Advancing to term 30
W20260808 01:53:40.841585  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 43747.8us
I20260808 01:53:40.807262 19266 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:40.800474 19328 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:40.885782  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:40.894186  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:40.900084  9630 raft_consensus.cc:2393] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 27: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:40.895784  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
W20260808 01:53:40.904754  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 94759.3us
I20260808 01:53:40.896633 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:40.906929  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:40.910981  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:40.909885  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:40.914728  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:40.915144  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:40.919813  9559 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:40.935596  9630 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 25 index: 20. Preceding OpId from leader: term: 26 index: 20. (term mismatch)
I20260808 01:53:40.936144  9630 pending_rounds.cc:85] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 19
I20260808 01:53:40.936441  9630 pending_rounds.cc:107] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 25.20
I20260808 01:53:40.936741  9630 raft_consensus.cc:2889] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:40.940871  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:40.942731  9558 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260808 01:53:40.898874 18834 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:40.952080 19269 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:40.951632  9633 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 30.
I20260808 01:53:40.925274 19256 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:40.924863 19195 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election won for term 24
W20260808 01:53:40.978645  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 52746.2us
W20260808 01:53:40.963902  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 58359.4us
I20260808 01:53:40.940438 19242 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:40.992594  9633 raft_consensus.cc:1275] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260808 01:53:41.024135  9521 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:40.954396 19256 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:41.035149 19328 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:41.024030  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:41.033624  9635 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
W20260808 01:53:41.009922 19335 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260808 01:53:41.042783  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:41.061277 19269 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:40.998108 19242 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:41.004901 19254 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260808 01:53:41.064707 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 241371us
W20260808 01:53:41.068197  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 44168.9us
I20260808 01:53:41.077258  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:41.035168 19195 raft_consensus.cc:697] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 24 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:41.078500  9559 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:41.084329  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:41.089757  9521 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:41.092227  9521 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:41.036550 19338 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:41.098779  9521 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:41.062083 19269 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:41.067992 19293 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:41.081171 19242 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:41.085803 19205 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:41.094151 19253 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260808 01:53:41.133224  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 231260us
I20260808 01:53:41.102370 19331 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:41.082078 19341 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.036s
W20260808 01:53:41.137439  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 228268us
I20260808 01:53:41.115605 19269 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:41.120570 19254 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:41.106071 19200 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.030s
I20260808 01:53:41.115823 19293 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:41.119769 19205 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:41.112131 19271 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:41.109270 19338 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:41.179098 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 282537us
I20260808 01:53:41.164361 19266 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:41.168246 19341 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260808 01:53:41.169243 19293 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:41.178579 19205 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:41.198484 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:41.215986 19341 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:41.194211 19319 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:41.233435  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:41.234998  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:41.236248  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:41.241348  9559 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:41.238562  9481 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24. Ops: [23.18-23.18]
I20260808 01:53:41.246255  9481 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260808 01:53:41.248958  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:41.203473 19266 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.042s
I20260808 01:53:41.189405 19195 consensus_queue.cc:237] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:41.277208 19256 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:41.243842 19319 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:41.346551  9559 raft_consensus.cc:2410] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 28 because replica has last-logged OpId of term: 26 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 13.
I20260808 01:53:41.511443  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:41.503650  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 321434us
W20260808 01:53:41.529469  9637 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.000s	sys 0.023s
W20260808 01:53:41.550387  9637 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.000s	sys 0.024s
W20260808 01:53:41.309976 19342 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.270s	user 0.002s	sys 0.000s
W20260808 01:53:41.465705  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 315022us
I20260808 01:53:41.510897  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:41.527570  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 428887us
W20260808 01:53:41.307616 19324 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260808 01:53:41.388619 19357 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.170s
I20260808 01:53:41.560997  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:41.562227  9635 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:41.562904  9521 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:41.564985  9521 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:41.544704 19348 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.253s	user 0.001s	sys 0.000s
I20260808 01:53:41.574520  9478 raft_consensus.cc:2410] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 29 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
I20260808 01:53:41.389205 19271 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:41.520390 19328 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:41.520659 19352 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.067s
I20260808 01:53:41.327600 19337 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 29 LEADER]: Stepping down as leader of term 29
I20260808 01:53:41.591100  9521 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:41.585678 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 345428us
I20260808 01:53:41.575471 19244 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:53:41.385179 19338 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:41.583555  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3494) took 1031 ms. Trace:
I20260808 01:53:41.610544 10332 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:41.591384  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:41.621474  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 55967.8us
I20260808 01:53:41.622215  9521 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:41.622941  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:41.627244  9635 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:41.628783  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:41.620435  9631 rpcz_store.cc:276] 0808 01:53:40.552591 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:40.677420 (+124829us) service_pool.cc:224] Handling call
0808 01:53:40.677878 (+   458us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:40.678165 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 20
0808 01:53:40.914896 (+236731us) log.cc:844] Serialized 14 byte log entry
0808 01:53:41.583064 (+668168us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:41.583143 (+    79us) raft_consensus.cc:1645] Marking committed up to 20
0808 01:53:41.583207 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:41.583305 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:41.583477 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":427925,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":146429,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":22503,"thread_start_us":883687,"threads_started":3,"wal-append.queue_time_us":390675}
I20260808 01:53:41.630743  9521 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:41.634027  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:41.609613 19337 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:41.620857 19271 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:41.609228 19056 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:41.620384 19244 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:41.642915  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:41.580322 19351 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.053s
I20260808 01:53:41.645174  9479 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:41.630625 19253 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:41.625905 19200 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:41.586066 19339 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.047s
I20260808 01:53:41.651242  9635 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
W20260808 01:53:41.649556  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 50882.7us
I20260808 01:53:41.635193 19338 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:41.637832 19364 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:41.666357  9477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 3969) took 2032 ms. Trace:
I20260808 01:53:41.631738 19254 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.025s
I20260808 01:53:41.670958  9477 rpcz_store.cc:276] 0808 01:53:39.633691 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:39.633838 (+   147us) service_pool.cc:224] Handling call
0808 01:53:39.634160 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:39.824236 (+190076us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:39.824268 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:39.824312 (+    44us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:53:39.824345 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:41.666022 (+1841677us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b5400399c54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:41.666090 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:41.666277 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":436662656}
I20260808 01:53:41.650254 19056 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:41.679263 19366 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260808 01:53:41.688033  9635 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
W20260808 01:53:41.691785 19355 log.cc:927] Time spent T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.297s	user 0.000s	sys 0.001s
I20260808 01:53:41.696558 19319 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:41.697737 19319 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:41.700096 19337 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:41.701033 19373 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260808 01:53:41.703573 19200 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:41.705246 19211 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:41.631829 19331 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.040s
I20260808 01:53:41.712828 19245 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:41.715022 19222 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.865s	user 0.009s	sys 0.011s
W20260808 01:53:41.717111 19202 consensus_meta.cc:287] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.002s	user 0.016s	sys 0.001s
I20260808 01:53:41.653483 19271 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:41.721200 19271 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:41.722271  9474 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260808 01:53:41.675582 19364 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:41.651979 19244 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:41.633538 19269 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:41.627683 19328 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:41.660841 19253 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:41.756255 19253 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:41.717880 19202 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:41.765470 19202 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:41.766002  9472 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:41.772236 19211 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:41.698347 19319 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:41.702351 19337 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:41.754844 19366 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:41.750118 19328 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:41.777480  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 154663us
I20260808 01:53:41.783919  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:41.765453 19269 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:41.785487 19203 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.936s	user 0.011s	sys 0.007s
I20260808 01:53:41.780617 19319 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:41.832122  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:41.836331 19364 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:41.843802  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:41.838058  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 217453us
W20260808 01:53:41.846884  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 216233us
I20260808 01:53:41.815903 19328 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:41.848472 18611 consensus_queue.cc:1048] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260808 01:53:41.853207 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 224424us
W20260808 01:53:41.834297 19242 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.711s	user 0.015s	sys 0.000s
I20260808 01:53:41.855554 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:41.848855  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:41.827078 19293 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.611s	user 0.009s	sys 0.005s
I20260808 01:53:41.857018 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:41.858011  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:41.849104 19309 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.234s	user 0.021s	sys 0.021s
I20260808 01:53:41.849432 19364 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260808 01:53:41.844277 19203 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:41.856243 19328 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:41.866714 19203 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:41.799886 19366 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:41.868196 19328 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:41.859251 19242 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:41.874164 19242 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:41.880242 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:41.883009  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:41.870239 19366 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:41.887342 19269 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:41.888204 19269 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:41.897264  9480 raft_consensus.cc:1240] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 20. Current term is 21. Ops: []
W20260808 01:53:41.898927 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:41.902617 19318 consensus_queue.cc:1059] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 20.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:53:41.903954 19318 raft_consensus.cc:3055] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:53:41.904474 19318 raft_consensus.cc:740] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:53:41.860733 19309 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.324s	user 0.022s	sys 0.022s
I20260808 01:53:41.918902 19269 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:41.909169 19366 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:41.920152 19366 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:41.922201  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:41.924273  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:41.924197  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:41.933611  9557 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:41.923198 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:41.923197  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:41.887638  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:41.937062 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:41.937348  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:41.940025  9552 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 24: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:53:41.941951  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 281410us
I20260808 01:53:41.943154  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:41.945648  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:41.947153  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:41.949874 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:41.953333  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 1209) took 1392 ms. Trace:
I20260808 01:53:41.859213 19293 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:41.958576  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 1211) took 1351 ms. Trace:
I20260808 01:53:41.999151  9476 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:41.959661  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:41.929821 19372 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.335s	user 0.000s	sys 0.003s
I20260808 01:53:41.961769  9558 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:41.954276  9636 rpcz_store.cc:276] 0808 01:53:40.560959 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:40.682136 (+121177us) service_pool.cc:224] Handling call
0808 01:53:40.682639 (+   503us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:40.683165 (+   526us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:40.683207 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:40.691000 (+  7793us) log.cc:844] Serialized 26 byte log entry
0808 01:53:41.139640 (+448640us) raft_consensus.cc:1645] Marking committed up to 17
0808 01:53:41.167504 (+ 27864us) log.cc:844] Serialized 14 byte log entry
0808 01:53:41.923577 (+756073us) spinlock_profiling.cc:243] Waited 3.96 ms on lock 0x7b1800410200. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667 00007f5d449ae5e8
0808 01:53:41.952593 (+ 29016us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:41.952714 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:41.952899 (+   185us) raft_consensus.cc:1685] finished
0808 01:53:41.953035 (+   136us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:41.953240 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8479,"prepare.queue_time_us":27500,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":2941,"raft.queue_time_us":138793,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":11073,"spinlock_wait_cycles":9104256,"thread_start_us":412137,"threads_started":2,"wal-append.queue_time_us":492641}
W20260808 01:53:41.965201  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 234177us
I20260808 01:53:41.922574 19366 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:41.980052 19248 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:41.982698 10369 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 19. (term mismatch)
I20260808 01:53:42.002882 10369 pending_rounds.cc:85] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting all ops after (but not including) 18
I20260808 01:53:41.995008 19351 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260808 01:53:41.930212 19205 consensus_meta.cc:287] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.709s	user 0.012s	sys 0.000s
I20260808 01:53:41.922749 19352 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:42.008718  9557 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:42.010139  9480 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:41.999547  9637 rpcz_store.cc:276] 0808 01:53:40.607555 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:40.700306 (+ 92751us) service_pool.cc:224] Handling call
0808 01:53:40.700855 (+   549us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:40.701236 (+   381us) raft_consensus.cc:1534] Early marking committed up to index 16
0808 01:53:40.912910 (+211674us) log.cc:844] Serialized 14 byte log entry
0808 01:53:41.574635 (+661725us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:41.579350 (+  4715us) log.cc:844] Serialized 26 byte log entry
0808 01:53:41.579577 (+   227us) raft_consensus.cc:1645] Marking committed up to 17
0808 01:53:41.579920 (+   343us) log.cc:844] Serialized 14 byte log entry
0808 01:53:41.921138 (+341218us) spinlock_profiling.cc:243] Waited 1.58 ms on lock 0x7b180018ec80. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667 00007f5d449ae5e8
0808 01:53:41.957312 (+ 36174us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:41.957429 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:41.958265 (+   836us) raft_consensus.cc:1685] finished
0808 01:53:41.958294 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:41.958476 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61618,"prepare.queue_time_us":405338,"prepare.run_cpu_time_us":152,"prepare.run_wall_time_us":6374,"raft.queue_time_us":711585,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":198,"spinlock_wait_cycles":3643264,"thread_start_us":778829,"threads_started":3,"wal-append.queue_time_us":448216}
I20260808 01:53:42.002981 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:42.003364 10369 pending_rounds.cc:107] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting uncommitted NO_OP operation due to leader change: 27.19
I20260808 01:53:42.000105  9554 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 18.
I20260808 01:53:41.945479 19203 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:42.006605  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 63604.1us
I20260808 01:53:42.008508  9477 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:42.011281 10369 raft_consensus.cc:2889] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:41.964011 19195 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:42.008528  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:42.015117  9599 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:42.009272  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:42.016247  9524 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:42.016732  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:42.020035 19387 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:42.014412 19352 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:42.017302  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 51303.6us
I20260808 01:53:42.022979 19352 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:42.022571 19203 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.018496 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:42.023563 19358 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:42.024744 19084 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:42.008719 19248 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.025569 19084 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:42.026154 19195 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:42.026801 19195 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:42.029436 19203 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:42.008248 19328 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:42.026264 19084 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:42.047477 19328 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:42.052635 19351 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:42.053756 19367 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Stepping down as leader of term 27
I20260808 01:53:42.053524 19351 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.017288  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:42.019124  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3496) took 1323 ms. Trace:
I20260808 01:53:42.034538 19385 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:53:42.020644 19387 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:42.037532 19248 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:42.054379 19367 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:41.956650 19318 consensus_queue.cc:260] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:42.055969 19386 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:42.061415  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:42.011950 19242 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:53:42.063496  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46310.9us
I20260808 01:53:42.059198  9630 rpcz_store.cc:276] 0808 01:53:40.696183 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:40.977165 (+280982us) service_pool.cc:224] Handling call
0808 01:53:40.977746 (+   581us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:41.181675 (+203929us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:41.181730 (+    55us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:41.182618 (+   888us) log.cc:844] Serialized 52 byte log entry
0808 01:53:41.392381 (+209763us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:53:41.392435 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:41.392537 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:41.929012 (+536475us) spinlock_profiling.cc:243] Waited 6.39 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:42.018787 (+ 89775us) raft_consensus.cc:1685] finished
0808 01:53:42.018811 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:42.019056 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":108793,"raft.run_cpu_time_us":353,"raft.run_wall_time_us":14991,"spinlock_wait_cycles":14695296,"thread_start_us":337035,"threads_started":2,"wal-append.queue_time_us":325341}
W20260808 01:53:42.069787  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 54795.3us
I20260808 01:53:42.086812  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:42.012554 19366 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:42.085305  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:42.088852  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:42.089386  9477 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 28: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:42.090826  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:42.096899  9633 raft_consensus.cc:2376] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 30. Re-sending same reply.
I20260808 01:53:42.098268  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:42.101186  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:42.091192  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:42.092200  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:42.013546 19211 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.067s
I20260808 01:53:42.063115 19341 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:53:42.012209 19205 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.117718  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:42.118412 10365 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 19. Preceding OpId from leader: term: 25 index: 20. (index mismatch)
I20260808 01:53:42.110077  9559 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 18. (term mismatch)
I20260808 01:53:42.119486  9559 pending_rounds.cc:85] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 17
I20260808 01:53:42.119784  9559 pending_rounds.cc:107] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 24.18
I20260808 01:53:42.120071  9559 raft_consensus.cc:2889] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:42.120498  9633 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 25. Current term is 26. Ops: [25.13-25.13]
I20260808 01:53:42.106197  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:42.114641  9480 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:42.115849  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:42.080667 19238 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.423s	user 0.009s	sys 0.014s
I20260808 01:53:42.122210  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:42.122968  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:42.077015 19358 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:42.127851  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:42.086285 19203 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:42.131426  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:42.131667  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:42.132476  9637 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260808 01:53:42.109671 19309 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:42.140225  9560 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:42.090837 19328 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:42.126904 19238 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.469s	user 0.010s	sys 0.014s
I20260808 01:53:42.117250 19341 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:42.124436 19318 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:42.147496 19200 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260808 01:53:42.148152 19371 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:42.142155 19384 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:42.150296  9475 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260808 01:53:42.151686 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:42.152971  9632 raft_consensus.cc:2468] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:42.152637  9523 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:42.154327  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:42.130131 19358 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:42.159677  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:42.135928 19239 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:42.170710  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:42.110390 19366 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:42.126015 19059 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.027s
I20260808 01:53:42.182451  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:42.129127 19331 consensus_queue.cc:1059] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.027s
I20260808 01:53:42.184695  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:42.186342  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:42.187841  9558 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 19. Preceding OpId from leader: term: 25 index: 20. (index mismatch)
I20260808 01:53:42.107354 19242 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:42.196856  9559 raft_consensus.cc:1275] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260808 01:53:42.095481 19386 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:42.175604 19352 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260808 01:53:42.169108 19358 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:42.171972  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:42.157459 19341 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:53:42.202643  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48403.7us
I20260808 01:53:42.207906  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:42.210532  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:42.206876  9554 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:42.160993 19203 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:42.188462 19200 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:53:42.214836  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:42.216696 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:42.219717  9632 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:42.222651  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:42.154451 19244 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.224677  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:42.200939 19383 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:42.205245 19238 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:42.187935 19248 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.035s
I20260808 01:53:42.239104  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:42.188890 19239 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.201197 19253 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:42.213143 19271 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:42.205725 19293 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:42.176820 19371 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:42.245800 19152 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260808 01:53:42.183097 19257 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.046s
I20260808 01:53:42.257890  9555 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 29. Current term is 30. Ops: []
I20260808 01:53:42.241288  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:42.177974 19384 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.253249 19202 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:42.350508  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:42.265748 19326 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 1.665s	user 0.002s	sys 0.012s
I20260808 01:53:42.267450 19329 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:42.268879 18617 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:53:42.272101 19136 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:42.272897 19368 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:42.276336 19370 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:42.277521 19382 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:42.277940 19394 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260808 01:53:42.254112 19343 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:42.229389 19203 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:42.222141 19164 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:53:42.284195 19056 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.533s	user 0.011s	sys 0.008s
I20260808 01:53:42.303241 19371 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:42.303679 19397 log.cc:927] Time spent T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
W20260808 01:53:42.305130 19398 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260808 01:53:42.306816 19400 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:42.307040 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:42.308339 19396 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260808 01:53:42.316498 19239 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:42.317481 19205 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:42.319576 19253 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:42.142300 19367 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.320194 19244 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:42.257752 19319 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.288460 19088 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:42.330436 19332 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 1.666s	user 0.000s	sys 0.014s
I20260808 01:53:42.280252 19310 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:42.333184 19366 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:42.335116 19351 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:42.337100 19084 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.248004 19383 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:42.340366 19386 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:42.262159 19200 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:42.344105 19365 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.810s	user 0.000s	sys 0.003s
I20260808 01:53:42.346889  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:42.349277  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:42.346982  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 167446us
I20260808 01:53:42.298396 19270 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:42.288932 19385 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.117s
W20260808 01:53:42.416934  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 280493us
I20260808 01:53:42.363730 19387 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:42.401909 19329 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:42.409729 19202 raft_consensus.cc:697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:42.460495  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 180150us
W20260808 01:53:42.478420  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 206665us
I20260808 01:53:42.406895 19136 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.424536 19331 raft_consensus.cc:3055] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:42.425388 19343 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:42.410887 19368 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:42.518590  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:42.428498 19056 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:42.522326  9476 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 28.
I20260808 01:53:42.523758  9632 raft_consensus.cc:2393] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 24: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:42.415194 19382 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.530753  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:42.531028  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:42.531370  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:42.549167  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:42.428196 19164 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:42.549932  9478 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 18.
I20260808 01:53:42.533489 10368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 1189) took 2489 ms. Trace:
I20260808 01:53:42.506909 19361 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:42.514642 19229 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:42.412686 19370 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.513558 19088 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.581394 10365 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:42.583809  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:42.584333  9477 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 30.
I20260808 01:53:42.551728 10368 rpcz_store.cc:276] 0808 01:53:40.034052 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:40.034220 (+   168us) service_pool.cc:224] Handling call
0808 01:53:40.053859 (+ 19639us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:40.217201 (+163342us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:40.217243 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:40.220701 (+  3458us) log.cc:844] Serialized 52 byte log entry
0808 01:53:40.424185 (+203484us) raft_consensus.cc:1645] Marking committed up to 17
0808 01:53:40.424249 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:40.424366 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:42.265037 (+1840671us) spinlock_profiling.cc:243] Waited 3.8 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:42.515175 (+250138us) raft_consensus.cc:1685] finished
0808 01:53:42.515199 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:42.523602 (+  8403us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14673,"raft.queue_time_us":47448,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":3907,"spinlock_wait_cycles":8744320,"thread_start_us":160316,"threads_started":2,"wal-append.queue_time_us":222035}
I20260808 01:53:42.554982  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:42.587042  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:42.524374 19200 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.588523  9632 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:53:42.586257  9634 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:42.543627 19331 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:42.551193 19386 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:42.629837  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 114474us
I20260808 01:53:42.529943 19310 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:42.574110 19394 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.035s
I20260808 01:53:42.638691  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:42.532176 19203 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:42.632339 19195 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.605s	user 0.007s	sys 0.009s
I20260808 01:53:42.659145  9472 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:53:42.638387  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:42.665714  9632 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:42.674100  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:42.674579  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 132051us
I20260808 01:53:42.580737 19229 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:42.572422 19361 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:42.699707  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:42.630079 19408 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:42.709344  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:42.656497 19385 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:42.721306  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:42.722451  9475 raft_consensus.cc:2410] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 28 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260808 01:53:42.654152 19403 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:42.726405  9632 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:42.643855 19253 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.064s
I20260808 01:53:42.747771  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:42.642392 19386 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.712047 19384 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:42.751835  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:42.754781  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:42.680426 19195 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.636714 19329 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:42.779058 19412 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.264s	user 0.000s	sys 0.001s
I20260808 01:53:42.721884 19385 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:42.646317 19202 consensus_queue.cc:237] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.642251 19343 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:42.787817 19318 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.626s	user 0.011s	sys 0.002s
I20260808 01:53:42.758841 19257 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:42.758041 19319 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:42.763203 19384 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:53:42.793292 19385 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:42.781388 19084 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:42.813470 19406 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.177s	user 0.001s	sys 0.000s
I20260808 01:53:42.814724 19238 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:42.817193 19271 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:42.801870 19088 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:42.817595 19229 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:42.818039 19402 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.193s	user 0.000s	sys 0.001s
I20260808 01:53:42.747951 19403 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:42.801544 19239 raft_consensus.cc:3060] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:42.722754 19408 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:42.792196 19369 consensus_queue.cc:1048] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.251s
I20260808 01:53:42.818364 19423 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:42.746536 19331 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.808429 19257 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.834319 19271 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:42.819774 19254 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:42.813575 19437 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260808 01:53:42.832847 19341 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.610s	user 0.014s	sys 0.000s
I20260808 01:53:42.828617 19383 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:42.862548 19254 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:42.876020 19254 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.877899 19254 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:42.835669 19382 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:42.853976 19370 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:42.885402 19403 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:42.891927  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3339) took 1049 ms. Trace:
I20260808 01:53:42.892464  9555 rpcz_store.cc:276] 0808 01:53:41.842389 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:41.846325 (+  3936us) service_pool.cc:224] Handling call
0808 01:53:41.846826 (+   501us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:42.345351 (+498525us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:42.891643 (+546292us) spinlock_profiling.cc:243] Waited 411 ms on lock 0x7b5400325ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:42.891851 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":944608640}
I20260808 01:53:42.855479 19435 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:42.894131 19435 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:42.843797  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:42.857741 19411 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
I20260808 01:53:42.839843 19238 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.900797 19238 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:42.859268 19331 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
W20260808 01:53:42.870400 19416 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.252s	user 0.000s	sys 0.001s
I20260808 01:53:42.905398 19408 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:42.905640  9481 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 29 index: 19. (term mismatch)
I20260808 01:53:42.906119  9481 pending_rounds.cc:85] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 18
I20260808 01:53:42.906378  9481 pending_rounds.cc:107] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 28.19
I20260808 01:53:42.906623  9481 raft_consensus.cc:2889] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:42.897513  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:42.907359 18834 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:42.863441 19271 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:42.908188  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:42.928992  9629 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:42.906804 19386 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:42.867774 19409 log.cc:927] Time spent T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.227s	user 0.001s	sys 0.000s
I20260808 01:53:42.853308 19200 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:42.932363  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 380921us
I20260808 01:53:42.941924  9601 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:42.897557 19310 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:42.850065 19410 consensus_queue.cc:1059] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 27.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.266s
I20260808 01:53:42.931349 18834 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:42.956610 19257 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:42.939131 19406 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
W20260808 01:53:42.940353 19358 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.719s	user 0.005s	sys 0.009s
I20260808 01:53:42.882553 19341 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:42.953089 19402 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
W20260808 01:53:43.011475 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:42.939692 19164 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:43.023761 10368 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 18. Current term is 19. Ops: []
I20260808 01:53:43.020780 19341 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:43.028066  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:43.026173 19200 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:43.045133  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:43.046108  9636 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:43.042878 19410 raft_consensus.cc:3055] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Stepping down as leader of term 27
W20260808 01:53:43.047550  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 74487.2us
I20260808 01:53:43.049669  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:43.051074  9601 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:42.975307 19445 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260808 01:53:43.055269  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:43.056612  9601 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:43.037889 19056 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:43.025236 19358 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:43.051560 19195 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:43.067863  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3325) took 2538 ms. Trace:
I20260808 01:53:43.050343 19410 raft_consensus.cc:740] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:43.070807  9556 rpcz_store.cc:276] 0808 01:53:40.525799 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:40.526236 (+   437us) service_pool.cc:224] Handling call
0808 01:53:40.526573 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:40.681231 (+154658us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:43.064115 (+2382884us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54000b4ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:43.064323 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":268368640}
I20260808 01:53:43.042868 19200 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:43.057762 19257 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:43.049212 19310 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.020s
I20260808 01:53:43.061398 19442 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:43.079177  9477 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24. Ops: [23.18-23.18]
I20260808 01:53:43.071553  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:43.080034  9475 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:43.067523 19294 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:43.088418  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:43.089514  9521 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:43.089794  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:43.091007  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:43.072903 19435 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:43.093071  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:43.093164  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:43.094043  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:43.099813  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:43.100215 18611 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260808 01:53:43.100703  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:43.101472 18611 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:43.101722  9521 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:43.103374  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:43.103691  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:43.092762 19442 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:43.104743 19403 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:43.104283  9475 raft_consensus.cc:2393] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 21: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:43.108058  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50140 (request call id 3959) took 1564 ms. Trace:
I20260808 01:53:43.108493  9481 rpcz_store.cc:276] 0808 01:53:41.543950 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:41.544104 (+   154us) service_pool.cc:224] Handling call
0808 01:53:41.544449 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:42.907174 (+1362725us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:43.107793 (+200619us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54000938d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:43.107993 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3129508736}
I20260808 01:53:43.097393 19408 raft_consensus.cc:2804] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:43.095429 19386 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:43.109681  9521 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:43.112767 18611 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:43.113217 18611 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:43.113534 18611 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
W20260808 01:53:43.127689 19447 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260808 01:53:43.128800 19200 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:43.134927 19164 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:43.135564 19164 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:43.134707 19445 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:43.136381 19442 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:43.136626 19445 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:43.137817 19164 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:43.139098 19445 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:43.140995 19310 consensus_queue.cc:1059] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:43.107394 19410 consensus_queue.cc:260] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:43.146404 18611 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:43.142419 19445 raft_consensus.cc:3055] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Stepping down as leader of term 25
W20260808 01:53:43.149700  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:43.149816 19443 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.007s
I20260808 01:53:43.158212 19358 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:43.112007 19294 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:43.159160 19294 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:43.161108 19294 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:43.151736 19445 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:43.163232 19445 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:43.164676 19445 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:43.167996 19410 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:43.168249  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3354) took 1068 ms. Trace:
I20260808 01:53:43.169049 19444 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:43.169746 19444 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:43.148093 19408 raft_consensus.cc:697] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:43.171517 19195 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:43.171630 19408 consensus_queue.cc:237] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:43.172256 19195 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:43.174055 19341 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:43.174728  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:43.174660 19341 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:43.181852  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:43.175797 18611 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:43.176040  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:43.168748  9557 rpcz_store.cc:276] 0808 01:53:42.099825 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:42.099988 (+   163us) service_pool.cc:224] Handling call
0808 01:53:42.100316 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:42.215713 (+115397us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:42.215765 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:42.215833 (+    68us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:53:42.215880 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:43.167952 (+952072us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b54000b45d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:43.168011 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:43.168179 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":264695168}
I20260808 01:53:43.183499  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:43.183619 10368 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260808 01:53:43.184193 19239 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:43.190704 19386 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:43.191663 19449 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260808 01:53:43.191439 19386 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:43.192076  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:43.192905  9555 raft_consensus.cc:2393] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 25: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:43.193861 19386 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:43.196025 19440 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
W20260808 01:53:43.199357  9445 consensus_peers.cc:597] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b335bc0170854e26a77dbec13dbe9eac. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:43.202258  9557 raft_consensus.cc:1240] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:43.184266  9632 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:53:43.204952  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3719) took 1085 ms. Trace:
I20260808 01:53:43.197291 19136 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:43.185424  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:43.198009 19341 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:43.206707  9631 rpcz_store.cc:276] 0808 01:53:42.119605 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:42.119732 (+   127us) service_pool.cc:224] Handling call
0808 01:53:42.120073 (+   341us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:42.138323 (+ 18250us) raft_consensus.cc:1534] Early marking committed up to index 11
0808 01:53:42.146650 (+  8327us) log.cc:844] Serialized 14 byte log entry
0808 01:53:42.531557 (+384907us) log.cc:844] Serialized 14 byte log entry
0808 01:53:42.576465 (+ 44908us) log.cc:844] Serialized 14 byte log entry
0808 01:53:42.582231 (+  5766us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:42.582804 (+   573us) log.cc:844] Serialized 52 byte log entry
0808 01:53:42.584028 (+  1224us) raft_consensus.cc:1645] Marking committed up to 13
0808 01:53:42.598380 (+ 14352us) log.cc:844] Serialized 14 byte log entry
0808 01:53:42.885672 (+287292us) spinlock_profiling.cc:243] Waited 2.8 ms on lock 0x7b18001eda20. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:53:43.204251 (+318579us) spinlock_profiling.cc:243] Waited 1.01 ms on lock 0x7b18001eda20. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d483aada7 00007f5d4499dd87 00007f5d449ad49f 00007f5d449ad418
0808 01:53:43.204472 (+   221us) log.cc:844] Serialized 14 byte log entry
0808 01:53:43.204704 (+   232us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:43.204771 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:43.204798 (+    27us) raft_consensus.cc:1685] finished
0808 01:53:43.204804 (+     6us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:43.204905 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":418097,"prepare.queue_time_us":562564,"prepare.run_cpu_time_us":149,"prepare.run_wall_time_us":152,"raft.queue_time_us":384204,"raft.run_cpu_time_us":899,"raft.run_wall_time_us":27336,"spinlock_wait_cycles":15294848,"thread_start_us":316285,"threads_started":3,"wal-append.queue_time_us":183091}
I20260808 01:53:43.212347  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:43.213725 18834 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:43.215384  9631 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 30: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:43.216409  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:43.217097  9631 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:43.149066 19310 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.059s
I20260808 01:53:43.221920  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:43.181351 19056 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:43.235297  9631 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28.
I20260808 01:53:43.237751  9521 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:43.239955 19205 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election won for term 28
I20260808 01:53:43.240566 19205 raft_consensus.cc:697] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:43.241539 19205 consensus_queue.cc:237] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:43.242143 19368 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:43.241894  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:43.239957  9554 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "030eacff61b14455978d2f0de262a089"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:51048
I20260808 01:53:43.243232  9554 raft_consensus.cc:606] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:43.243291  9631 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:43.243607  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:43.244333  9637 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 31.
I20260808 01:53:43.245741  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:43.246004  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:43.248351 19254 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260808 01:53:43.248607  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:43.248888 19254 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:43.249375 19254 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Advancing to term 32
I20260808 01:53:43.252588 19420 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:43.253302 19420 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:43.253780 19420 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:43.255205 19239 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:43.257419 19059 raft_consensus.cc:3055] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:43.257944 19059 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:43.265179 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:43.274032  9631 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:43.230192 19271 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:43.285399 18611 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:43.286751 18611 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:43.288038 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:43.288370 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:43.291864 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:43.217451 19385 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:43.296285  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:43.297580  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:43.298245  9552 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 18.
I20260808 01:53:43.302075  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:43.303637  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:43.304481  9637 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:43.295428 19059 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:43.306798 18611 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:43.307066 19271 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:43.233614 19417 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.069s
I20260808 01:53:43.308450 19195 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:43.233882 19056 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:43.310889 19056 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:43.313321  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3721) took 1192 ms. Trace:
I20260808 01:53:43.313741  9635 rpcz_store.cc:276] 0808 01:53:42.120910 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:43.059683 (+938773us) service_pool.cc:224] Handling call
0808 01:53:43.068561 (+  8878us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:43.072189 (+  3628us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:43.072219 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:43.072691 (+   472us) log.cc:844] Serialized 26 byte log entry
0808 01:53:43.182292 (+109601us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:43.279733 (+ 97441us) log.cc:844] Serialized 14 byte log entry
0808 01:53:43.283857 (+  4124us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:43.284008 (+   151us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:43.313096 (+ 29088us) raft_consensus.cc:1685] finished
0808 01:53:43.313118 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:43.313257 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70725,"prepare.queue_time_us":113433,"raft.queue_time_us":7687,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":15505,"spinlock_wait_cycles":12182272,"thread_start_us":76612,"threads_started":1,"wal-append.queue_time_us":196393}
I20260808 01:53:43.316051  9554 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "17e220e6a25747939bdf18f26b732407"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:51048
I20260808 01:53:43.316174  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:43.316787  9554 raft_consensus.cc:606] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:43.317131  9554 raft_consensus.cc:614] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:43.307598 19444 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:43.322649  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:43.329324 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:43.331355 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:43.332584 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:43.334408  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:43.333271 10365 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:43.338851  9632 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28.
W20260808 01:53:43.340225  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 17e220e6a25747939bdf18f26b732407 (from 52ccf4dcca1843f7bb29e9172e2d4ab4 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:43.343981 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:43.346248  9637 raft_consensus.cc:2468] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:43.347618  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:43.347527  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:43.348742  9637 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28.
W20260808 01:53:43.351286  9521 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.500s (SENT) [suppressed 24 similar messages]
I20260808 01:53:43.351631  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:43.352990  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:43.353909  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:43.299549 19331 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.046s
I20260808 01:53:43.365727  9637 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260808 01:53:43.366955  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3727) took 1241 ms. Trace:
I20260808 01:53:43.367425  9637 rpcz_store.cc:276] 0808 01:53:42.125195 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:43.349278 (+1224083us) service_pool.cc:224] Handling call
0808 01:53:43.365345 (+ 16067us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:43.366435 (+  1090us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:43.366865 (+   430us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:53:43.373498  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:43.389304 19329 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:43.400274  9555 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:51048
I20260808 01:53:43.401558  9555 raft_consensus.cc:606] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:43.402887  9555 raft_consensus.cc:614] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:43.396588 18961 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:43.431825  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:43.439239  9556 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
W20260808 01:53:43.444594  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 2c617cb141a34007b6c1b8af0d911923 (from 52ccf4dcca1843f7bb29e9172e2d4ab4 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
W20260808 01:53:43.451665  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:43.413571 18961 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:43.486521 19385 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:43.493137 19367 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:43.497381  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:43.514909  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:43.516382  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:43.518550  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:43.519129  9552 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:51048
I20260808 01:53:43.519886  9552 raft_consensus.cc:606] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:43.521719  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:43.523110  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:43.524096  9632 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 27.
I20260808 01:53:43.527196  9472 raft_consensus.cc:2393] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 24: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:43.527813  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:43.529837  9473 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 20. Current term is 21. Ops: []
I20260808 01:53:43.529688  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:43.502599 19382 raft_consensus.cc:3055] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:43.534808  9556 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:43.526223 19444 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260808 01:53:43.538111 19420 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:43.539090 19417 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260808 01:53:43.545814  9478 raft_consensus.cc:1240] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:43.547536  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:43.520826 19195 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:53:43.551458  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:43.536058 19254 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:43.559059  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 207782us
I20260808 01:53:43.561360  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:43.542869 19382 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:43.552433 19195 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:43.570195 19382 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:43.574333  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:43.574426  9555 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:51048
I20260808 01:53:43.575591  9555 raft_consensus.cc:606] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:43.575995  9555 raft_consensus.cc:614] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:43.573580 19195 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 18 FOLLOWER]: Advancing to term 19
W20260808 01:53:43.585652  7831 auto_leader_rebalancer.cc:467] leader transfer for tablet 924b85a2ee7e46ecbcfd57257902abef (from 52ccf4dcca1843f7bb29e9172e2d4ab4 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:43.586354  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260808 01:53:43.587430 18961 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:43.588655  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:53:43.589977  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:53:43.595733  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 1227) took 1393 ms. Trace:
I20260808 01:53:43.596274  9632 rpcz_store.cc:276] 0808 01:53:42.201958 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:43.581557 (+1379599us) service_pool.cc:224] Handling call
0808 01:53:43.595125 (+ 13568us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:43.595365 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:43.595392 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:43.595441 (+    49us) raft_consensus.cc:1645] Marking committed up to 16
0808 01:53:43.595473 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:43.595533 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:43.595662 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:53:43.596860 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:43.597476 19420 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:43.599484 19382 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:43.620343 10334 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:43.611776 19254 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:43.622619 18980 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:43.623889 18980 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:43.624289 18980 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:43.630728  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3726) took 1505 ms. Trace:
I20260808 01:53:43.631253  9635 rpcz_store.cc:276] 0808 01:53:42.124808 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:43.317347 (+1192539us) service_pool.cc:224] Handling call
0808 01:53:43.317702 (+   355us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:43.318367 (+   665us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:43.318407 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:43.320771 (+  2364us) log.cc:844] Serialized 26 byte log entry
0808 01:53:43.543242 (+222471us) raft_consensus.cc:1645] Marking committed up to 16
0808 01:53:43.558354 (+ 15112us) log.cc:844] Serialized 14 byte log entry
0808 01:53:43.558674 (+   320us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:43.558806 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:43.630433 (+ 71627us) raft_consensus.cc:1685] finished
0808 01:53:43.630459 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:43.630636 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":14342,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":2262,"raft.queue_time_us":224,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":88,"spinlock_wait_cycles":3760896,"thread_start_us":179231,"threads_started":1,"wal-append.queue_time_us":229848}
I20260808 01:53:43.623208  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:43.635142 19242 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:43.636200  9632 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260808 01:53:43.637602  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3509) took 1430 ms. Trace:
I20260808 01:53:43.638028  9632 rpcz_store.cc:276] 0808 01:53:42.207388 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:43.635591 (+1428203us) service_pool.cc:224] Handling call
0808 01:53:43.635961 (+   370us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:43.637338 (+  1377us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:43.637545 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:53:43.633841  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:43.643862  9635 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Advancing to term 32
I20260808 01:53:43.646492 19361 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:43.648108 19303 raft_consensus.cc:3055] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:43.648525 19303 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:43.649463 19303 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:43.658046  9635 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 32.
I20260808 01:53:43.664304  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3514) took 1427 ms. Trace:
I20260808 01:53:43.656538 19303 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:43.667304 19256 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:43.671303 19256 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:43.673667 19256 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:43.674144  9635 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260808 01:53:43.669915  9632 rpcz_store.cc:276] 0808 01:53:42.237316 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:43.638816 (+1401500us) service_pool.cc:224] Handling call
0808 01:53:43.657617 (+ 18801us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:43.663705 (+  6088us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:43.663807 (+   102us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:43.663958 (+   151us) raft_consensus.cc:1645] Marking committed up to 18
0808 01:53:43.664004 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:43.664094 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:43.664240 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2890,"raft.queue_time_us":4981}
I20260808 01:53:43.661878 19358 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:43.675998  9635 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:43.676293 19358 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:43.678480  9635 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 31. Current term is 32. Ops: []
I20260808 01:53:43.681602  9635 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:43.686209 19242 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:43.686892 19242 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:43.689452 19242 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:43.689991 19242 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:43.690665 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:43.694481 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:43.699143  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:43.700205  9480 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 27. Current term is 28. Ops: [27.14-27.14]
I20260808 01:53:43.702617  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:43.703361  9473 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:43.704052  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:43.706027 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:43.697549 10332 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:43.695575 10365 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:43.708251  9601 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:43.709259 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:43.697729  9521 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:43.712646  9521 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:43.713574  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:43.715102 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:43.715801  9521 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:43.719497 19253 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:43.716476 19331 raft_consensus.cc:2804] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:43.720132 19254 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election won for term 28
W20260808 01:53:43.721673 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 77335.7us
W20260808 01:53:43.723337  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:43.723977  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:43.725195 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:43.726454 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:43.678535 19358 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:43.714893 19205 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:43.709564 19361 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:43.714474 19309 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:43.742012  9521 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:43.690552 19242 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:43.721313 19329 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:43.720098 19253 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:43.720499 19331 raft_consensus.cc:697] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:43.749454  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:43.798022 18611 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Leader election won for term 32
W20260808 01:53:43.798125  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 44855.9us
I20260808 01:53:43.809953 18611 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:43.739045 19254 raft_consensus.cc:697] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:43.716291 19341 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 31.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.030s
W20260808 01:53:43.818789  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:43.742909 19309 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:43.819605  9633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3720) took 1699 ms. Trace:
I20260808 01:53:43.753140 19253 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:43.756688 19305 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:43.747494 19205 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:43.747239 19297 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:43.748221 19329 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:43.822160  9633 rpcz_store.cc:276] 0808 01:53:42.120654 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:42.121413 (+   759us) service_pool.cc:224] Handling call
0808 01:53:42.121740 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:42.190295 (+ 68555us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:42.190349 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:42.199919 (+  9570us) log.cc:844] Serialized 26 byte log entry
0808 01:53:42.509186 (+309267us) raft_consensus.cc:1645] Marking committed up to 17
0808 01:53:42.628222 (+119036us) log.cc:844] Serialized 14 byte log entry
0808 01:53:42.633748 (+  5526us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:43.808861 (+1175113us) spinlock_profiling.cc:243] Waited 41.1 ms on lock 0x7b5400338954. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:43.808946 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:43.809025 (+    79us) raft_consensus.cc:1685] finished
0808 01:53:43.819293 (+ 10268us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:43.819529 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":641550,"prepare.queue_time_us":194211,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":55,"raft.queue_time_us":162180,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":10387,"spinlock_wait_cycles":94605184,"thread_start_us":358379,"threads_started":2,"wal-append.queue_time_us":366197}
I20260808 01:53:43.830883  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:43.837056  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:43.843261  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:43.744761 19361 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:43.857059  9634 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 3515) took 1612 ms (client timeout 1728 ms). Trace:
I20260808 01:53:43.829846 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:43.854962 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 142082us
I20260808 01:53:43.844843 19329 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:43.861739 18961 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:43.862658 18961 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:43.863452 19469 log.cc:927] Time spent T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260808 01:53:43.866294 10363 raft_consensus.cc:2410] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 21 because replica has last-logged OpId of term: 20 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 16.
W20260808 01:53:43.857679  9634 rpcz_store.cc:269] 0808 01:53:42.244517 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:42.346229 (+101712us) service_pool.cc:224] Handling call
0808 01:53:43.856803 (+1510574us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b540003b6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:43.856983 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":539992320}
W20260808 01:53:43.869464 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 137344us
I20260808 01:53:43.891252 10334 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:43.877197  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:43.872589  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:43.826833 19309 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:43.839186 19445 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:43.824815 19466 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
I20260808 01:53:43.926420  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:43.934105 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:43.939167  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:43.941053  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:43.954190  9636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3725) took 1829 ms. Trace:
I20260808 01:53:43.954478 19254 consensus_queue.cc:237] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:43.956952  9630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 1224) took 1868 ms. Trace:
W20260808 01:53:43.960438 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 69316us
I20260808 01:53:43.968430 10334 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:43.857982 19363 raft_consensus.cc:3055] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 LEADER]: Stepping down as leader of term 31
I20260808 01:53:43.962030  9636 rpcz_store.cc:276] 0808 01:53:42.124553 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:43.059723 (+935170us) service_pool.cc:224] Handling call
0808 01:53:43.060124 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:43.072045 (+ 11921us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:43.072084 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:43.072549 (+   465us) log.cc:844] Serialized 26 byte log entry
0808 01:53:43.154331 (+ 81782us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:43.232364 (+ 78033us) log.cc:844] Serialized 14 byte log entry
0808 01:53:43.952447 (+720083us) spinlock_profiling.cc:243] Waited 4.9 ms on lock 0x7b540043b820. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490eb13 00007f5d4499419f 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c
0808 01:53:43.952516 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:43.953785 (+  1269us) spinlock_profiling.cc:243] Waited 5.15 ms on lock 0x7b5400307fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:43.953846 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:43.953906 (+    60us) raft_consensus.cc:1685] finished
0808 01:53:43.953922 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:43.954111 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60025,"prepare.queue_time_us":129124,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":49,"raft.queue_time_us":6679,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":38435,"spinlock_wait_cycles":23100928,"thread_start_us":49967,"threads_started":1,"wal-append.queue_time_us":125615}
I20260808 01:53:43.974663  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:43.965660 19468 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
I20260808 01:53:43.999918  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:44.003789  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:44.005105  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:43.967680  9630 rpcz_store.cc:276] 0808 01:53:42.088791 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:42.089298 (+   507us) service_pool.cc:224] Handling call
0808 01:53:42.089606 (+   308us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:42.541346 (+451740us) raft_consensus.cc:1534] Early marking committed up to index 16
0808 01:53:42.646803 (+105457us) log.cc:844] Serialized 14 byte log entry
0808 01:53:42.792217 (+145414us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:42.797132 (+  4915us) log.cc:844] Serialized 26 byte log entry
0808 01:53:42.797287 (+   155us) raft_consensus.cc:1645] Marking committed up to 17
0808 01:53:43.206611 (+409324us) log.cc:844] Serialized 14 byte log entry
0808 01:53:43.207606 (+   995us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:43.633874 (+426268us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b48002379d8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d498ef211 00007f5d498e62a7 00007f5d498e613a 00007f5d499dd07f 00007f5d499dcd47 00007f5d499f136c 00007f5d499f1318 00007f5d499f1298 00007f5d499f1253 00007f5d499f04fc 00007f5d48918176
0808 01:53:43.956598 (+322724us) spinlock_profiling.cc:243] Waited 443 ms on lock 0x7b54001261d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:43.956662 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:43.956709 (+    47us) raft_consensus.cc:1685] finished
0808 01:53:43.956723 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:43.956885 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":960006,"prepare.queue_time_us":593762,"prepare.run_cpu_time_us":133,"prepare.run_wall_time_us":4384,"raft.queue_time_us":32071,"raft.run_cpu_time_us":12622,"raft.run_wall_time_us":1106957,"spinlock_wait_cycles":1027876992,"thread_start_us":79020,"threads_started":2,"wal-append.queue_time_us":211829}
I20260808 01:53:44.038627  9630 raft_consensus.cc:1240] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:43.932451 18611 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:44.019156  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:44.021113 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52835.2us
W20260808 01:53:44.044298 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:43.998840  9631 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:43.960171 19361 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:43.856463 19470 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.130s
I20260808 01:53:43.847556 19297 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:44.037642  9558 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
I20260808 01:53:43.974437 19363 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:44.067358 19371 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:44.067644 18961 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:44.068184 19371 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:44.070549  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:44.071178 19297 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:44.023145 19231 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260808 01:53:44.072288 19371 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:44.056077 18980 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:43.932991 19445 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.103782 19231 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:44.106396  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 305831us
I20260808 01:53:44.104765 19231 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:44.067701 19467 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.321s	user 0.001s	sys 0.000s
I20260808 01:53:43.917239 19331 consensus_queue.cc:237] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.088310 19466 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:44.139286  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:44.109942  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:44.140735  9474 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:44.119007  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:44.066073  9523 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:44.124331  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:44.144418  9558 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 26: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:44.151376  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:44.155218 19256 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:44.156055 19256 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.159641  9632 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260808 01:53:44.164499  9634 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260808 01:53:44.106657 19371 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:44.178544  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:44.179534  9555 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 20.
W20260808 01:53:44.172541  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 106591us
W20260808 01:53:44.172564  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:44.184775  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 235038us
I20260808 01:53:44.185765  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:44.186527  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:44.143121 19448 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:44.177203  9599 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:44.144117 19361 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:44.160068 19367 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.666s	user 0.014s	sys 0.000s
I20260808 01:53:44.200714 19361 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.203145  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:44.203846 19297 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:44.204581 19297 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.205245 18834 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260808 01:53:44.206485  9634 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:44.207238  9638 raft_consensus.cc:2410] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 28 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260808 01:53:44.208437  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:44.212528  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:44.215627  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:44.219118 19256 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:44.219568  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:44.220084 18834 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:44.220327  9554 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:44.220749 18834 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:44.058447 19205 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:44.223536  9555 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:44.219134  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:44.227157 19056 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:44.227191 18834 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:44.227957 19056 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:44.229650  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:44.233889  9554 raft_consensus.cc:2410] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260808 01:53:44.188939 19371 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:44.235283  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:44.236562  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 53793us
I20260808 01:53:44.146862 19341 consensus_queue.cc:1059] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.050s
I20260808 01:53:44.263710 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:44.184338 19408 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:44.165941 19305 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:44.247033  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 70103us
W20260808 01:53:44.236632  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:44.240018  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 53564.1us
I20260808 01:53:44.289853  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:44.283957 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:44.285415  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:44.190778 19242 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:44.295114  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:44.255276 19468 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260808 01:53:44.303350  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:44.254521 19245 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:44.308174  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:44.308606  9474 raft_consensus.cc:2468] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 29.
W20260808 01:53:44.292562 19326 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
W20260808 01:53:44.300716 19303 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.585s	user 0.010s	sys 0.004s
I20260808 01:53:44.316921  9600 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:44.313827  9634 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260808 01:53:44.261798 19475 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:44.284451 19408 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:44.324834 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 213138us
I20260808 01:53:44.353838  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:44.312726 19293 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:44.309202 19056 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:44.365954 10334 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:44.321633 19194 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260808 01:53:44.318142 19361 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:44.317521 19266 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:44.391659  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:44.392812  9474 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:53:44.364163 19473 raft_consensus.cc:3055] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:44.363402 19293 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:44.381788 19303 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:44.298120 19305 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.190308 19363 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:44.394696 19326 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260808 01:53:44.195573 19448 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.400565 19256 consensus_queue.cc:1059] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.016s
I20260808 01:53:44.327358 18834 consensus_queue.cc:1059] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.067s
I20260808 01:53:44.413543  9631 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260808 01:53:44.361191 19408 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:44.354934 19475 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:44.409446 19463 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.799s	user 0.003s	sys 0.006s
I20260808 01:53:44.327936 19203 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.076s
W20260808 01:53:44.425102 19253 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.577s	user 0.005s	sys 0.009s
I20260808 01:53:44.419924  9550 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 27: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:44.416432 19293 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:44.427317 19478 ts_tablet_manager.cc:933] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359)
I20260808 01:53:44.404116 19473 raft_consensus.cc:740] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:44.425701 19059 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:44.412598 19445 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:44.494992 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:44.496177 10368 raft_consensus.cc:2410] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 25 because replica has last-logged OpId of term: 23 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 17.
I20260808 01:53:44.437029 19485 raft_consensus.cc:2804] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader election won for term 29
I20260808 01:53:44.421305 19305 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:44.499294  9552 raft_consensus.cc:1240] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:44.476629 19253 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.506896  9555 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 28. Current term is 29. Ops: []
W20260808 01:53:44.507784 19382 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.885s	user 0.003s	sys 0.011s
I20260808 01:53:44.474462 19266 raft_consensus.cc:3055] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:44.467635 19231 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:44.508744 19266 raft_consensus.cc:740] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:44.509953 19266 consensus_queue.cc:260] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.498381 19485 raft_consensus.cc:697] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:44.512688 19485 consensus_queue.cc:237] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:44.516165 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:44.514546 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:44.516178  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:44.552222  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:44.557140  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:53:44.560416 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46104.7us
I20260808 01:53:44.562944 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:44.564224  9556 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:44.564616  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:44.565177  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:44.565335  9633 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260808 01:53:44.567997  9473 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 28. Current term is 29. Ops: [28.19-28.19]
W20260808 01:53:44.571453  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:44.565887  9634 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
I20260808 01:53:44.536161 19266 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:44.585289  9474 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:44.589877  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:44.586011  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:44.585343  9477 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:44.605957  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:44.606860  9633 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:53:44.612723  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:44.624270  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:44.625415  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:44.585632 19371 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:44.636384  9481 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 28. Current term is 29. Ops: []
W20260808 01:53:44.630326  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 272295us
I20260808 01:53:44.639750  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:44.640844 19253 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:44.646914 19195 consensus_meta.cc:287] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.052s	user 0.007s	sys 0.008s
W20260808 01:53:44.648015 19329 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.786s	user 0.013s	sys 0.000s
I20260808 01:53:44.648782  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:44.649763  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:44.650090  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:44.651098  9477 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 31.
I20260808 01:53:44.652347 19363 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Advancing to term 32
I20260808 01:53:44.652792  9474 raft_consensus.cc:3055] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Stepping down as leader of term 27
I20260808 01:53:44.653317  9474 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:44.655129  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:44.656174 19487 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260808 01:53:44.656201 19477 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:44.658097  9599 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:44.662041 19466 raft_consensus.cc:3055] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Stepping down as leader of term 20
W20260808 01:53:44.663738 19482 log.cc:927] Time spent T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260808 01:53:44.648520 19329 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:44.650631 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:44.647523 19195 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:44.670473  9550 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260808 01:53:44.612540 19489 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:44.627238 19084 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:44.654455  9474 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:53:44.696988  9474 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:44.499338 19473 consensus_queue.cc:260] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:44.687331  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 328878us
W20260808 01:53:44.678277  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 350417us
I20260808 01:53:44.695331  9634 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260808 01:53:44.668656 19466 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:44.704439  9559 raft_consensus.cc:1240] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:44.667718 19491 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260808 01:53:44.704453 10367 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:44.707234  9478 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 18. Current term is 19. Ops: []
I20260808 01:53:44.699019  9600 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:44.713511 19495 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:44.713706 19200 raft_consensus.cc:993] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: : Instructing follower 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:53:44.719856  9600 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:44.713819 19466 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:44.715626 19245 raft_consensus.cc:3055] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:53:44.691336 19489 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:44.705523 19498 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:44.740655  9524 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:44.742635  9600 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:44.748006  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:44.727690 19245 raft_consensus.cc:740] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:44.723881 19331 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:44.700141 19084 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:44.739214 19309 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.778s	user 0.013s	sys 0.002s
I20260808 01:53:44.755098 19364 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:44.758821 19475 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:44.766957 19319 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:44.767515 19253 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:44.758699 18834 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:44.742275 19200 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:44.742204 19499 raft_consensus.cc:1081] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Signalling peer 259bcb3f0d254e8b9effac3277e654e0 to start an election
I20260808 01:53:44.727865 19477 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:44.768991 19382 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:44.743161 19489 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:44.749959 19203 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:44.777226 19386 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:44.779140 19309 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.803273 19203 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.805897 19203 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:44.772776 19466 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:44.777258 19364 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.750366 19497 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:44.757952 19367 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:44.774811 19331 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:44.727249 19495 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.769563  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:44.782332 19492 log.cc:927] Time spent T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.084s	user 0.001s	sys 0.001s
I20260808 01:53:44.770375 19496 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:44.750301 19448 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:44.781762 19319 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.785392 18834 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.787156 19200 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.793407 19329 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:44.787446 19477 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:44.783677 19253 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.842166 19477 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:44.841744 19478 tablet_copy_client.cc:323] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.193:39359
I20260808 01:53:44.838493 18834 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:44.816534 19084 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:44.838306 19309 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:44.840219 19200 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:44.847409  9638 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3724) took 2723 ms (client timeout 1500 ms). Trace:
I20260808 01:53:44.837013 19319 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:44.847945  9638 rpcz_store.cc:269] 0808 01:53:42.124285 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:42.131413 (+  7128us) service_pool.cc:224] Handling call
0808 01:53:44.847110 (+2715697us) spinlock_profiling.cc:243] Waited 2.07 s on lock 0x7b540008c354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:44.847272 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4767243008}
W20260808 01:53:44.850231 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:44.794100 19245 consensus_queue.cc:260] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:44.851866 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:44.807938  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:44.773020 19438 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:44.825873  9600 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:44.718626 19500 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:44.832777  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 63330.3us
I20260808 01:53:44.853164  9630 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:44.863510 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:44.878556 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:44.887584  9474 raft_consensus.cc:1275] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 17. (term mismatch)
I20260808 01:53:44.888242  9474 pending_rounds.cc:85] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 16
I20260808 01:53:44.891332  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:44.833989 19448 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260808 01:53:44.892565  9638 raft_consensus.cc:2410] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 25 because replica has last-logged OpId of term: 23 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 17.
I20260808 01:53:44.861948 19367 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:44.888844 19245 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:53:44.862355 19319 consensus_queue.cc:1059] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.054s
I20260808 01:53:44.892231 19497 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:44.888576  9474 pending_rounds.cc:107] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 27.17
I20260808 01:53:44.937768  9474 raft_consensus.cc:2917] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Aborting config change with OpId 27.17: Aborted: Op aborted by new leader
I20260808 01:53:44.938289  9474 raft_consensus.cc:2889] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:53:44.896765  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:44.831403 19496 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.875397 19293 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.862771 19438 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:44.936821  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:44.935938  9638 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:44.940378  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:37234 (request call id 25) took 3097 ms. Trace:
I20260808 01:53:44.947629  9474 rpcz_store.cc:276] 0808 01:53:41.841070 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:41.841281 (+   211us) service_pool.cc:224] Handling call
0808 01:53:42.664445 (+823164us) log.cc:844] Serialized 506 byte log entry
0808 01:53:43.241339 (+576894us) spinlock_profiling.cc:243] Waited 815 ms on lock 0x7b54000212d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44999b15 00007f5d4998bb39 00007f5d42ed8334 00007f5d42ed82c2 00007f5d42ed81f1 00007f5d42ed816c 00007f5d42ed7422 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:44.938789 (+1697450us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2831,"raft.run_cpu_time_us":30,"raft.run_wall_time_us":30,"spinlock_wait_cycles":1874896768}
I20260808 01:53:44.946846  9633 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:44.948220  9631 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260808 01:53:44.951097  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:44.952543  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:44.953302  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:44.938548  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 197990us
I20260808 01:53:44.953995  9629 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:44.957993  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:44.862355 19500 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:44.959112  9632 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:44.943077  9636 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1"
dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:42824
I20260808 01:53:44.960553  9473 raft_consensus.cc:1240] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:44.963222  9636 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 01:53:44.963611  9636 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:44.963932  9629 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:44.960419  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:44.955260  9524 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:44.966038  9524 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:44.964676  9635 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:44.916726  9630 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 28.
I20260808 01:53:44.968204  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:44.969147  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:44.954865 19253 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:44.950234 19476 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:44.954504 19242 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:44.972930  9629 raft_consensus.cc:2410] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 29 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 13.
W20260808 01:53:45.040652  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 74668.8us
I20260808 01:53:45.045861  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:45.041873  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:45.059463  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3728) took 2934 ms. Trace:
I20260808 01:53:44.971506 19305 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:53:45.097491 19305 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:45.098119 19305 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:53:45.102438 19364 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:45.103462 19489 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:45.052660 19242 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.107698 19495 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:45.109457 19242 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:45.045879 19230 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:45.110324  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 68593.6us
I20260808 01:53:45.046298 19476 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.111055 19230 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:45.114120 19230 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:45.084080 19293 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:45.114449 19500 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:45.116747 19496 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:45.117246  9632 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260808 01:53:45.084226 18980 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:45.087304 19518 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
I20260808 01:53:45.094871 19514 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election won for term 30
I20260808 01:53:45.096803 19438 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:44.942515 19497 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:45.085286 19510 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
I20260808 01:53:45.118317  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:45.127318  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:45.135756  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:45.178802  9629 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 26: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:45.178921  9635 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 30.
I20260808 01:53:45.179702  9634 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28. Ops: [26.19-27.20]
W20260808 01:53:45.101325 19059 consensus_meta.cc:287] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.594s	user 0.007s	sys 0.005s
W20260808 01:53:45.160146  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 49049.7us
W20260808 01:53:45.169829  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:45.183909 19363 consensus_meta.cc:287] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.531s	user 0.013s	sys 0.001s
I20260808 01:53:45.154961  9636 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.106436 19489 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:45.177279  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:45.193656  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 308205us
I20260808 01:53:45.137495 19242 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260808 01:53:45.184393 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:45.184700 19497 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:45.185336  9557 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260808 01:53:45.198014  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:45.186839 19266 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.583s	user 0.011s	sys 0.004s
I20260808 01:53:45.060220  9637 rpcz_store.cc:276] 0808 01:53:42.125449 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:43.368511 (+1243062us) service_pool.cc:224] Handling call
0808 01:53:43.858173 (+489662us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:44.144152 (+285979us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:44.144184 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:44.160266 (+ 16082us) log.cc:844] Serialized 26 byte log entry
0808 01:53:44.464870 (+304604us) raft_consensus.cc:1645] Marking committed up to 16
0808 01:53:44.507265 (+ 42395us) log.cc:844] Serialized 14 byte log entry
0808 01:53:44.508329 (+  1064us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:44.508531 (+   202us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:44.939601 (+431070us) raft_consensus.cc:1685] finished
0808 01:53:44.939646 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:45.059226 (+119580us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b540003b6d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:45.059386 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":40281,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":67823,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":34041,"spinlock_wait_cycles":1047697792,"thread_start_us":528322,"threads_started":2,"wal-append.queue_time_us":418929}
I20260808 01:53:45.198719  9631 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 30.
I20260808 01:53:45.195031  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:45.184302 19408 consensus_meta.cc:287] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.699s	user 0.012s	sys 0.000s
I20260808 01:53:45.196542 10368 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:45.198256  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:45.200069  9636 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:45.206180  9524 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:45.206597  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:45.207091 10365 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 32.
I20260808 01:53:45.213649  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:45.198365 19254 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:45.215821  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:45.216807  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:45.217832 19254 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.217087 19438 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:45.200012 19514 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:45.219847  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:45.220206  9524 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:45.207463 19266 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:45.213294 19495 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:45.219324 19438 raft_consensus.cc:697] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:45.224598 19500 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:45.225325  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:45.213101 19442 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:45.223578 19203 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260808 01:53:45.232465  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:45.224912 19293 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:45.215698 19504 consensus_queue.cc:1059] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.015s
I20260808 01:53:45.224191 19516 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:45.207526 19408 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:45.215023 19242 consensus_queue.cc:1059] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.016s
I20260808 01:53:45.236012  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:45.220580 19515 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.013s
W20260808 01:53:45.234306  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:45.246459 10332 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:45.249779  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:45.236927 19495 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260808 01:53:45.252785  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:45.252938  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:45.252918 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:45.250036  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:45.226495 19364 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.009s
I20260808 01:53:45.256593  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:45.256331  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:45.257851  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:45.259052  9480 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 26: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:45.234979 19059 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:53:45.266727 19516 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:45.268566 19514 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.256316 19442 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.014s
I20260808 01:53:45.271035 19515 raft_consensus.cc:3055] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:45.270921 19438 consensus_queue.cc:237] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.263283  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:45.262458  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:45.263577 19500 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260808 01:53:45.272720 19500 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:45.273193 19500 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Advancing to term 32
I20260808 01:53:45.274938 19230 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260808 01:53:45.259994 19239 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.004s
I20260808 01:53:45.275813  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:45.277508  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:45.261471 18980 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:45.279206 18980 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:45.279610 18980 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:45.280256 19367 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:45.257061 19203 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:53:45.285521 19500 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.288091 19442 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:45.289726  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:45.291154 19475 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:45.291536  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:45.292022 19475 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:45.231261 19496 consensus_queue.cc:1059] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.021s
I20260808 01:53:45.293497 19496 raft_consensus.cc:3055] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:45.293915 19496 raft_consensus.cc:740] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:45.261670 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:45.266129  9560 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24. Ops: []
W20260808 01:53:45.295397 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:45.262334  9476 raft_consensus.cc:2410] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 31 because replica has last-logged OpId of term: 29 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
I20260808 01:53:45.291267 19203 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:45.292559 19239 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:45.271627 19515 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:45.265529 19408 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:45.265807  9521 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:45.274470 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:45.298028  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:45.303110  9477 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:45.298311  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:45.295738  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:45.294842  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:45.300403  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:45.294914 19496 consensus_queue.cc:260] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.305850 19363 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260808 01:53:45.306409 19363 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:45.306778 19496 raft_consensus.cc:3060] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
W20260808 01:53:45.306597  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet b335bc0170854e26a77dbec13dbe9eac: Aborted: Op aborted by new leader
I20260808 01:53:45.299638 19203 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:45.307864 19297 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:45.308090 19203 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:45.308637 19497 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:45.306893 19363 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:53:45.302469 10332 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:45.316265  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:45.314033  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:45.319118  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:45.319634  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:45.322551  9477 raft_consensus.cc:1275] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260808 01:53:45.324465  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:45.325224  9476 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:45.321444 19466 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader election won for term 28
I20260808 01:53:45.326994  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:45.327702  9632 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:45.328668  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:45.301402 19386 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:45.305768 19205 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:45.332690  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:45.334406  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:45.313169 19442 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260808 01:53:45.331467 19516 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.007s
I20260808 01:53:45.336938 19515 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.340139 19475 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:45.262652 19477 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:45.344516  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 645503us
I20260808 01:53:45.321287 19408 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.024s
I20260808 01:53:45.324522 19200 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:45.346875 19386 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:45.348703 19195 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:45.372458 19403 consensus_queue.cc:1059] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260808 01:53:45.372843  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:45.373100 19513 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:45.349975 10367 raft_consensus.cc:1240] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 28. Current term is 29. Ops: []
I20260808 01:53:45.373687 19513 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.357996 19442 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:45.375209 19442 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:53:45.376230 19513 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:45.362874 19448 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:53:45.352329 19253 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:45.353157 19476 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:45.332774  9521 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:45.358503 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:45.362773 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:45.402295  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 69669.8us
I20260808 01:53:45.367588  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:45.369932  9599 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:45.371551 19331 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:45.350909 10361 raft_consensus.cc:3055] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:45.327052 19466 raft_consensus.cc:697] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:45.407786 10361 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:45.410912  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:45.413137  9521 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:53:45.413861  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 44066.5us
I20260808 01:53:45.415409  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:45.417689  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 50199.6us
I20260808 01:53:45.372903 19515 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:45.374610 19499 raft_consensus.cc:3055] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 LEADER]: Stepping down as leader of term 27
I20260808 01:53:45.422991  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:45.381003 19200 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:45.419253  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:45.430168 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:45.429225 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:45.433873 10369 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:45.427316  9557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3385) took 1132 ms. Trace:
I20260808 01:53:45.435943  9557 rpcz_store.cc:276] 0808 01:53:44.300051 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:44.361437 (+ 61386us) service_pool.cc:224] Handling call
0808 01:53:44.361698 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:45.198109 (+836411us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:45.426872 (+228763us) spinlock_profiling.cc:243] Waited 823 ms on lock 0x7b5400409a54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:45.427195 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1893337856}
I20260808 01:53:45.438118  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:45.442252 19309 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:53:45.446246  9599 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:45.448067 19309 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:45.450549  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:45.454409 19309 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260808 01:53:45.463513 10367 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 24: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:45.410985 19205 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:45.474727  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:45.477087  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:45.466614 10361 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:45.481743  9636 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:53:45.412034 19310 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:45.365324 19364 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.061s
I20260808 01:53:45.506281  9632 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260808 01:53:45.481778  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:45.423316 19239 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:45.417274 19504 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:45.523366  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:45.504074  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:45.477366  9599 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:45.509193  9477 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 29: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:45.501927  9629 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:45.422268 18971 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:45.510509  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:45.432070 19499 raft_consensus.cc:740] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:53:45.513103  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:45.399852 19305 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:45.470600 19513 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:45.463325 19200 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:45.417353 19242 consensus_queue.cc:1048] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.093s
I20260808 01:53:45.480050 19331 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:45.511541  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:45.501467 19205 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:53:45.537050  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 59787.9us
I20260808 01:53:45.534696  9476 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 28.
I20260808 01:53:45.420583 19476 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.105s
I20260808 01:53:45.481421 19329 consensus_queue.cc:1059] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 18.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.064s
I20260808 01:53:45.483978 19253 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.051s
I20260808 01:53:45.536958 19371 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:45.555140  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:45.519541 19239 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:53:45.537932 19341 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.136s	user 0.003s	sys 0.033s
I20260808 01:53:45.561916  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3380) took 1268 ms. Trace:
I20260808 01:53:45.580564  9555 rpcz_store.cc:276] 0808 01:53:44.293241 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:44.297148 (+  3907us) service_pool.cc:224] Handling call
0808 01:53:44.316522 (+ 19374us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:44.507340 (+190818us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:45.561544 (+1054204us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b540029f4d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:45.561824 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":437303168}
I20260808 01:53:45.574473  9599 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:45.538264 19481 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260808 01:53:45.551318 19513 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:45.521102 19310 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:45.578707 19239 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
W20260808 01:53:45.578979 19341 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.183s	user 0.004s	sys 0.033s
I20260808 01:53:45.562909 19205 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:45.563103 19331 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:45.575238 18971 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:45.603201  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: true dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:45.605517  9477 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:45.598001  9491 tablet_copy_service.cc:140] P 6e7856bed1be43acaa821290d506ee97: Received BeginTabletCopySession request for tablet b335bc0170854e26a77dbec13dbe9eac from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:37248)
I20260808 01:53:45.608978  9479 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 28: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:45.612569 19230 raft_consensus.cc:3055] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Stepping down as leader of term 18
I20260808 01:53:45.605259 19331 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:45.622107 19500 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:45.627753  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 4458) took 1427 ms. Trace:
I20260808 01:53:45.632635 19242 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:45.635257  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:45.637414  9556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 1459) took 1656 ms. Trace:
I20260808 01:53:45.641867 10366 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260808 01:53:45.652803  9556 rpcz_store.cc:276] 0808 01:53:43.981063 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:43.984316 (+  3253us) service_pool.cc:224] Handling call
0808 01:53:44.053842 (+ 69526us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:44.580463 (+526621us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:45.637104 (+1056641us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b540008a7d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:45.637339 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1173193600}
I20260808 01:53:45.648226  9560 rpcz_store.cc:276] 0808 01:53:44.200782 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:44.218220 (+ 17438us) service_pool.cc:224] Handling call
0808 01:53:44.218717 (+   497us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:45.295271 (+1076554us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:45.627341 (+332070us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54000fee54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:45.627656 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2408297344}
I20260808 01:53:45.639981  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:45.644235  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 283026us
I20260808 01:53:45.648020 10365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 1203) took 1378 ms. Trace:
I20260808 01:53:45.592352 19513 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:45.542125 19466 consensus_queue.cc:237] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:45.655455 10365 rpcz_store.cc:276] 0808 01:53:44.263227 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:44.263398 (+   171us) service_pool.cc:224] Handling call
0808 01:53:45.636792 (+1373394us) spinlock_profiling.cc:243] Waited 927 ms on lock 0x7b5400512cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:45.641218 (+  4426us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2131295104}
I20260808 01:53:45.652357  9480 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 29.
I20260808 01:53:45.638432 19386 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:45.681146  9477 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:45.645115 19230 raft_consensus.cc:740] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:53:45.675100  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 251810us
W20260808 01:53:45.700074  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 280878us
I20260808 01:53:45.701493  9445 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:45.683188  9601 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:45.685954  9480 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 20. Current term is 21. Ops: []
I20260808 01:53:45.692102  9477 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260808 01:53:45.695392 10361 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:45.712559  9524 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:45.673864 18971 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:45.717124  9524 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:45.659335  9491 tablet_copy_service.cc:161] P 6e7856bed1be43acaa821290d506ee97: Beginning new tablet copy session on tablet b335bc0170854e26a77dbec13dbe9eac from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:37248: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-b335bc0170854e26a77dbec13dbe9eac
I20260808 01:53:45.665364 19242 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:45.712059 19496 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:45.735358  9521 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:53:45.740063 10333 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:45.735558  9524 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:45.655944 19499 consensus_queue.cc:260] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.712772 19504 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:45.745622 19496 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.745517 19230 consensus_queue.cc:260] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:45.747656 19230 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:53:45.748685 19496 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:45.719061 18971 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:45.750437 18971 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:45.719285 19438 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:45.754536 19386 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:53:45.739773 19477 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:45.764562  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:45.767795 19084 raft_consensus.cc:3055] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Stepping down as leader of term 28
I20260808 01:53:45.768509 19084 raft_consensus.cc:740] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:45.773485  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:45.776059 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:45.778126 19475 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:45.766490  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:45.785353  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 49914.7us
I20260808 01:53:45.786438  9524 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:45.769762 19084 consensus_queue.cc:260] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:45.789753  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:45.742457 19253 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:45.736591 19530 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:45.822613  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:45.827049 10361 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260808 01:53:45.814322  9524 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:45.778723 19475 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:45.840217 19205 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.779145 19529 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260808 01:53:45.783360 19504 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:45.791699 19473 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:45.788905 19500 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260808 01:53:45.798458 19386 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:45.866572 19084 raft_consensus.cc:3060] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
W20260808 01:53:45.867262  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 53025.1us
I20260808 01:53:45.872087  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:45.875627  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:45.871974  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:45.880221  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:45.881309  9480 raft_consensus.cc:2393] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 19: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:45.892423 19205 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:45.893375 19442 consensus_meta.cc:287] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.518s	user 0.000s	sys 0.013s
I20260808 01:53:45.882478 19253 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:53:45.894164 19253 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:45.893882 19442 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.894613 19253 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:45.957168 19495 raft_consensus.cc:3055] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Stepping down as leader of term 28
I20260808 01:53:45.957903 19495 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:53:45.958570 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:45.896615 19328 raft_consensus.cc:3055] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Stepping down as leader of term 27
I20260808 01:53:45.959587 19328 raft_consensus.cc:740] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:45.960961 19328 consensus_queue.cc:260] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:45.962257 19328 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:45.897040  9478 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 28. Current term is 29. Ops: []
I20260808 01:53:45.907910 19504 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.910429 19254 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:45.857949 19529 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:45.974121 19529 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 18 FOLLOWER]: Advancing to term 19
I20260808 01:53:45.859366 19481 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:45.979112 19481 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.981570 19305 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:45.981930 19481 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:45.983692  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:45.920490 19239 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.987228  9473 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 27: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:45.991254 19239 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:45.921630 19475 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.832258 19514 consensus_queue.cc:1048] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.051s
W20260808 01:53:45.869846 19472 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 24 current term: 24 is higher.
I20260808 01:53:45.924470 19245 raft_consensus.cc:3055] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:45.995059 19245 raft_consensus.cc:740] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:45.996325 19245 consensus_queue.cc:260] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:45.997745 19245 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:45.925243  9551 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 LEADER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 31. Current term is 32. Ops: []
I20260808 01:53:46.002058  9551 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60552 (request call id 1464) took 2001 ms. Trace:
I20260808 01:53:46.005504 19499 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:45.927799 19242 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:45.871452 19525 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
W20260808 01:53:45.928970 18980 consensus_meta.cc:287] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.649s	user 0.004s	sys 0.012s
I20260808 01:53:45.929593 19310 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:45.930822 19367 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:45.933730 18971 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.939368  9638 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 23. Current term is 24. Ops: []
W20260808 01:53:45.940191  9479 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 4003) took 1383 ms (client timeout 1837 ms). Trace:
I20260808 01:53:45.940757 19513 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:45.941434  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:53:45.875844 19363 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.551s	user 0.010s	sys 0.004s
W20260808 01:53:45.949330 19203 consensus_meta.cc:287] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.641s	user 0.006s	sys 0.007s
I20260808 01:53:46.006471 10367 raft_consensus.cc:1240] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 23. Current term is 24. Ops: []
I20260808 01:53:46.007313 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:46.009093 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:46.016805 19442 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:45.926913  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:46.105088  9636 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:46.065026 19477 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:46.101714 10363 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:46.019210  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:46.029176 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 119322us
I20260808 01:53:46.032909 19200 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:46.108752  9600 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:46.052354 19475 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:46.055599  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:46.117527  9556 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:46.057412  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:46.059054  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:46.061172  9523 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.006s	sys 0.025s
W20260808 01:53:46.119786  9523 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.007s	sys 0.025s
I20260808 01:53:46.120077  9557 raft_consensus.cc:2410] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
W20260808 01:53:46.120460  9630 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.015s	sys 0.037s
W20260808 01:53:46.120821  9630 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.016s	sys 0.037s
I20260808 01:53:46.087606 19495 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:46.092411 18971 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:46.093683  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:46.076771 18980 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:46.095695 19253 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:46.098556 19386 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:46.018553  9551 rpcz_store.cc:276] 0808 01:53:44.001029 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:44.001209 (+   180us) service_pool.cc:224] Handling call
0808 01:53:44.001552 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:45.999062 (+1997510us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:46.001704 (+  2642us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b540002cbd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:46.001944 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4423467264}
I20260808 01:53:46.145941  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:46.104283 19363 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:46.095770 19367 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:46.162016 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:46.137615 10365 raft_consensus.cc:1240] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:46.138867  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:46.148908  9638 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 27
W20260808 01:53:46.166435 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:46.169176  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:46.170436  9557 raft_consensus.cc:2410] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 29 because replica has last-logged OpId of term: 26 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 13.
I20260808 01:53:46.106585 19203 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:46.173739  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:46.111387 19477 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:46.143869 19495 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:46.171710  9555 raft_consensus.cc:1240] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 29. Current term is 30. Ops: []
I20260808 01:53:46.176186 10368 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 30. Current term is 31. Ops: []
I20260808 01:53:46.176571  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:46.181586  9552 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:46.131983 19472 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.041s
I20260808 01:53:46.185667  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:46.187762  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: true dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:46.188339 18980 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:46.188735  9560 raft_consensus.cc:3055] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:46.189244  9560 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:46.190294  9560 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:46.177531  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:46.179486  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:46.132763 19331 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:46.138090 19488 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:46.158407 19254 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:46.158836 19305 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:46.201054 19254 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:46.193367  9560 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:46.195134  9633 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:46.195356  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:46.156476 10369 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:46.196902  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:46.197876  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 89217.2us
I20260808 01:53:46.204207  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:46.215945  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:46.205068  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:46.217098  9557 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 27: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:46.128985  9636 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
I20260808 01:53:46.219358  9636 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260808 01:53:46.223328  9636 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260808 01:53:46.192292 19529 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:46.229413  9600 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:46.205897 19539 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:46.217837 19256 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:46.227823 18971 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:46.247437 19367 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:46.248570 19363 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 31 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:46.251720 19477 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:46.252302 19513 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:46.224701 19504 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:46.264006 19504 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:46.265316 19331 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:46.266311 19504 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:46.254205 19529 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:46.254613 19309 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:46.214183 19488 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:46.256935 18971 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:46.219269 19305 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:46.260866 19203 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:46.253760 19254 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:46.274062 19200 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:46.241597 19539 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:46.279949 19539 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:46.281021 19309 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:46.282006 19309 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:46.251767 19253 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:46.286369 19305 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:46.037811  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:46.339712  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 301759us
I20260808 01:53:46.223320  9556 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 26: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:53:46.001963  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:46.347249  9473 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:45.950392  9474 raft_consensus.cc:1275] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
W20260808 01:53:46.078081  9479 rpcz_store.cc:269] 0808 01:53:44.556574 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:44.556765 (+   191us) service_pool.cc:224] Handling call
0808 01:53:45.939814 (+1383049us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400399254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:45.940076 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2416442752}
I20260808 01:53:46.351534  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:46.353384  9558 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 28.
W20260808 01:53:46.382965  9524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.000s	sys 0.011s
W20260808 01:53:46.383797  9524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.000s	sys 0.011s
I20260808 01:53:46.391361 19328 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:46.392231 19254 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:46.393308 19254 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:46.383628 19442 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260808 01:53:46.392798 19328 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:46.398438  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 54034.2us
W20260808 01:53:46.411425 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:46.438410  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:46.439597  9479 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:46.459298  9479 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 22.
I20260808 01:53:46.469374  9552 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 29.
I20260808 01:53:46.475297  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:46.480227  9474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 4021) took 1221 ms. Trace:
I20260808 01:53:46.480893  9474 rpcz_store.cc:276] 0808 01:53:45.258875 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:45.313654 (+ 54779us) service_pool.cc:224] Handling call
0808 01:53:45.949895 (+636241us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:46.349295 (+399400us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:46.479734 (+130439us) spinlock_profiling.cc:243] Waited 636 ms on lock 0x7b5400399250. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:46.480006 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1462288768}
I20260808 01:53:46.492964 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:46.495362 19254 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:46.298262 19539 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:46.503314  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:46.504518  9474 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:46.507586 19363 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:46.508893 19328 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:46.525727  9550 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:46.530603 19552 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:46.541687  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:46.548834  9599 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:46.526463  9559 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:46.508571 19363 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:46.583654  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:46.584867 19552 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:46.590793 19553 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:46.591012  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:46.591935 19553 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:46.592850  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:46.593683  9557 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:46.597471  9491 tablet_copy_source_session.cc:215] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:53:46.608923  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:46.614612  9475 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:46.615161 19363 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:46.620217 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:46.621758 10363 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 19: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:46.623919  9491 rpcz_store.cc:275] Call kudu.tserver.TabletCopyService.BeginTabletCopySession from 127.0.0.1:37248 (request call id 1752) took 1158 ms. Trace:
I20260808 01:53:46.624496  9491 rpcz_store.cc:276] 0808 01:53:45.502963 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:45.518756 (+ 15793us) service_pool.cc:224] Handling call
0808 01:53:46.596204 (+1077448us) spinlock_profiling.cc:243] Waited 367 ms on lock 0x7b54000212d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4499d7d7 00007f5d49964e46 00007f5d4996e84e 00007f5d411e261b 00007f5d4996e7ca 00007f5d499698dc 00007f5d49964a8b 00007f5d499596e1 00007f5d48a0c5d4 00007f5d48a0c562
0808 01:53:46.623750 (+ 27546us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":843851520}
I20260808 01:53:46.636504 19552 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:46.637472 19552 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:46.639003  9560 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:46.640275 19552 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:46.646889  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:46.647943  9473 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:53:46.640357  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:46.660285  9475 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 29.
I20260808 01:53:46.652331 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:46.664788  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:46.662571 10363 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
W20260808 01:53:46.666978 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:46.635389 19553 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:46.672439  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:46.675668  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:46.676527 10368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 1205) took 1328 ms. Trace:
I20260808 01:53:46.677215 10368 rpcz_store.cc:276] 0808 01:53:45.348065 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:45.348888 (+   823us) service_pool.cc:224] Handling call
0808 01:53:45.349255 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:46.182853 (+833598us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:46.675834 (+492981us) spinlock_profiling.cc:243] Waited 827 ms on lock 0x7b5400690fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:46.676269 (+   435us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1901086592}
I20260808 01:53:46.678884  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:46.681062  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:46.682947  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:46.666347  9481 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 28.
I20260808 01:53:46.691008 19363 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260808 01:53:46.676934  9632 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:46.682209  9554 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:46.697252  9560 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 24: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:46.699839 19328 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:46.700549  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:46.700695 19328 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:46.703255 19328 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:46.705643  9633 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260808 01:53:46.708572  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:46.705441  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:46.711274  9557 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 29.
W20260808 01:53:46.710142  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 173058us
W20260808 01:53:46.715047  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 178670us
W20260808 01:53:46.718986 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:46.721868  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 173348us
I20260808 01:53:46.723706  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:46.725204  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:46.727684  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:46.731335  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:46.732177  9479 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:46.733248 19328 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:46.743767  9550 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 27: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:46.748111  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:46.749289  9472 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:53:46.751209  9473 raft_consensus.cc:2393] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 21: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:46.750520  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:46.751991 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:46.752704  9638 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:46.761191  9558 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260808 01:53:46.763638  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:46.764740  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:46.765225  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:46.766170  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:46.759232  9633 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:53:46.766522  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:46.767258  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:46.774924  9473 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 27: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:46.775724  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:46.776036  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:46.776445  9474 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:46.777925  9552 raft_consensus.cc:1275] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260808 01:53:46.779083  9472 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 23: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:46.779959  9552 raft_consensus.cc:1240] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 31. Current term is 32. Ops: []
I20260808 01:53:46.783404  9559 raft_consensus.cc:1240] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260808 01:53:46.784663  9473 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
W20260808 01:53:46.785018  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:46.763481 19476 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:46.795974 19564 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:46.793952 10363 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:46.798698 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:46.801113  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:46.801586  9477 raft_consensus.cc:2410] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 26 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260808 01:53:46.801959  9599 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:46.801424 19562 log.cc:927] Time spent T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.162s	user 0.002s	sys 0.000s
I20260808 01:53:46.812880  9478 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:46.813616  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:46.814988 19476 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:46.816074  9479 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
I20260808 01:53:46.816550  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:46.787545 19539 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:46.807461 10334 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:46.813704 19541 consensus_queue.cc:1048] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.012s
I20260808 01:53:46.793460 19488 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:46.824839 19564 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:46.842517  9472 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 30. Current term is 31. Ops: []
I20260808 01:53:46.842717  9476 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 28: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:46.840438  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:46.842579  9600 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:46.839195 19408 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:46.865796 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 175100us
W20260808 01:53:46.865795  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 106813us
I20260808 01:53:46.871493  9632 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 22.
I20260808 01:53:46.873099 18971 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:46.873512  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 85035.1us
I20260808 01:53:46.874785  9599 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:46.875823 18971 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:46.877437  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:46.878562 18971 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:53:46.878840 19554 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:46.879410 19554 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:46.879843 19554 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:46.883140  9638 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:46.884699 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:46.890982  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:46.892674  9521 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:46.892943  9481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37248 (request call id 1750) took 1485 ms. Trace:
I20260808 01:53:46.893622 19254 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:46.894243  9475 raft_consensus.cc:1275] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260808 01:53:46.862545 19552 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:46.898871  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:46.893386  9481 rpcz_store.cc:276] 0808 01:53:45.407789 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:45.640563 (+232774us) service_pool.cc:224] Handling call
0808 01:53:45.941220 (+300657us) spinlock_profiling.cc:243] Waited 1.99 ms on lock 0x7b5800005498. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:53:46.892710 (+951490us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54002bc5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:46.892857 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":585082880}
I20260808 01:53:46.898934  9476 raft_consensus.cc:1275] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 13. (term mismatch)
I20260808 01:53:46.900362 10363 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:46.900823  9476 pending_rounds.cc:85] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Aborting all ops after (but not including) 12
I20260808 01:53:46.901909 19488 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:46.901957  9476 pending_rounds.cc:107] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Aborting uncommitted NO_OP operation due to leader change: 24.13
I20260808 01:53:46.902643 19488 raft_consensus.cc:697] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:46.903017  9476 raft_consensus.cc:2889] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:53:46.904331 19540 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.718s	user 0.006s	sys 0.000s
I20260808 01:53:46.904088 19488 consensus_queue.cc:237] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:46.907269  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42840 (request call id 3561) took 1583 ms. Trace:
W20260808 01:53:46.908954  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:46.907706  9635 rpcz_store.cc:276] 0808 01:53:45.323505 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:45.323630 (+   125us) service_pool.cc:224] Handling call
0808 01:53:45.323901 (+   271us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:45.330494 (+  6593us) raft_consensus.cc:1534] Early marking committed up to index 15
0808 01:53:45.713042 (+382548us) log.cc:844] Serialized 14 byte log entry
0808 01:53:46.204166 (+491124us) log.cc:844] Serialized 14 byte log entry
0808 01:53:46.204582 (+   416us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:46.211135 (+  6553us) log.cc:844] Serialized 26 byte log entry
0808 01:53:46.211345 (+   210us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:46.211395 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:46.211517 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:46.903369 (+691852us) spinlock_profiling.cc:243] Waited 5.56 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:46.906776 (+  3407us) raft_consensus.cc:1685] finished
0808 01:53:46.906804 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:46.907046 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":91830,"prepare.queue_time_us":427121,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":100,"raft.queue_time_us":122075,"raft.run_cpu_time_us":528,"raft.run_wall_time_us":42490,"spinlock_wait_cycles":12798848,"thread_start_us":724287,"threads_started":2,"wal-append.queue_time_us":452132}
I20260808 01:53:46.909251  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:46.911965  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:46.913992  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:46.915239  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:46.902961 19564 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:46.918047  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:46.932663  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:46.934019 19363 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:46.942157  9633 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
W20260808 01:53:46.943104  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:46.949816  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 72656.6us
I20260808 01:53:46.954734 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:46.955761 19573 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:46.916692 19552 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:46.979453  9600 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:46.982430  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 65700us
I20260808 01:53:46.982575  9479 raft_consensus.cc:2393] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 28: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:46.948824  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:46.989473  9521 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:46.988379  9474 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:46.990825  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:46.991701  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:47.005627  9556 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260808 01:53:47.006848  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:47.008296  9637 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 27: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:47.009193 19478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b335bc0170854e26a77dbec13dbe9eac. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 01:53:47.006897 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 104669us
I20260808 01:53:47.011502  9638 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:53:46.986462 19552 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:47.006851  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:46.938894 19363 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:46.991343 19254 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:47.031167  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:47.032811 19564 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:47.033540 19564 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.041127  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:47.038857 10334 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:47.043977 19245 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:47.045966 19574 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:47.048880 19476 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:46.988358 19408 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.034619 19328 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.006s
I20260808 01:53:47.042392  9524 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:47.053885  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 99833.9us
I20260808 01:53:47.041389 10363 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 19.
I20260808 01:53:47.070088 19363 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:47.071717 18980 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:47.072207 18980 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:47.072645 18980 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:47.073333  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:47.075657  9521 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:47.075798 19408 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:47.077486 19476 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:47.078075 19476 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:47.078521 19476 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:47.079126 19541 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
W20260808 01:53:47.086767  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 60777.4us
I20260808 01:53:47.086999 19408 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:47.087639 19564 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:47.087812 19408 raft_consensus.cc:697] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:47.090822  9629 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23.
I20260808 01:53:47.092936  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:47.017822 19573 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:47.098714 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:47.108569 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:47.110236 10363 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 19: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:47.055678 19254 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:47.111016 10368 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:47.107286 19245 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:47.126011  9638 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 31.
I20260808 01:53:47.097383 19532 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 28.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.090s
W20260808 01:53:47.132496  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 71827.8us
W20260808 01:53:47.130991 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 71597.4us
W20260808 01:53:47.142849 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:47.143823 19572 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.099s	user 0.000s	sys 0.008s
I20260808 01:53:47.123083 19245 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:47.154922  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 108292us
I20260808 01:53:47.156139  9601 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:47.160876 19254 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:47.162955 19573 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:47.163545 19532 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Stepping down as leader of term 28
I20260808 01:53:47.164089 19532 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:47.165295 19472 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:47.165218 19532 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.165899 19472 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:47.189620  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:47.190797  9638 raft_consensus.cc:2393] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 25: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:47.102308 19408 consensus_queue.cc:237] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:47.200305 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 134154us
W20260808 01:53:47.202399  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 96892us
I20260808 01:53:47.187444 18971 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:47.199349 19328 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:47.213290  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45744.4us
I20260808 01:53:45.481068  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:53:47.219509  9601 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:47.231333 19328 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:47.204190 19488 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 18, Time since last communication: 0.018s
I20260808 01:53:47.195574 19554 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:47.241087  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:53:47.246927  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 92236.1us
I20260808 01:53:47.250422  9523 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:47.245057 18971 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260808 01:53:47.256963  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 43859.3us
I20260808 01:53:47.259032 19546 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260808 01:53:47.260703 18980 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:47.261447 19254 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260808 01:53:47.264739  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 186196us
I20260808 01:53:47.271368  9599 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:47.263980 18971 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:47.277894 18971 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:47.279165 18980 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:47.246944 19472 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:47.284062 19472 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:47.287968 19578 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:47.288532 19578 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:47.288950 19578 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:47.293160  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:47.293416 19476 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:47.296914 19328 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:47.246075 19580 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.163s
I20260808 01:53:47.297429 19585 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260808 01:53:47.299551 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 83062.7us
I20260808 01:53:47.299975 19580 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:47.300969 19580 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Stepping down as leader of term 30
I20260808 01:53:47.301553 19580 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:47.302582 19580 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.304757 19580 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:53:47.308312 19577 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:47.309209 19577 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:47.273110  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:47.326963  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 62088.2us
I20260808 01:53:47.335093  9475 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 26. Current term is 27. Ops: []
W20260808 01:53:47.340710  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 76197.2us
I20260808 01:53:47.345052  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:47.346118  9481 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 26.
I20260808 01:53:47.352979  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:47.355034 19581 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:47.356015 19581 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:47.361054 10334 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:47.367046 19573 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:47.367709 19573 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:47.368146 19573 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:47.374914 19203 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:47.375883 19203 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.383309  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:47.384331  9474 raft_consensus.cc:2393] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 26: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:53:47.384497  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:47.337077 19539 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election won for term 25
W20260808 01:53:47.396039  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 142886us
I20260808 01:53:47.400663  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:47.401599  9481 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:47.409528  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:47.415575  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:47.393333 19328 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:47.424754  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:47.426044  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:47.404515 19245 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:53:47.450073  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 101023us
I20260808 01:53:47.450006  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:47.462777  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 199930us
I20260808 01:53:47.382505 19475 consensus_queue.cc:1048] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.243s
W20260808 01:53:47.475881  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 221516us
I20260808 01:53:47.450963 19581 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:47.504654  9550 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53216 (request call id 3407) took 1093 ms. Trace:
W20260808 01:53:47.516474 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 154448us
I20260808 01:53:47.504817 10369 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260808 01:53:47.508543  9550 rpcz_store.cc:276] 0808 01:53:46.411636 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:46.747091 (+335455us) service_pool.cc:224] Handling call
0808 01:53:46.773709 (+ 26618us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:46.863757 (+ 90048us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:46.863810 (+    53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:46.863897 (+    87us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:53:46.863952 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:47.498985 (+635033us) spinlock_profiling.cc:243] Waited 82.9 ms on lock 0x7b54003264d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:47.504378 (+  5393us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:47.504573 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":190633216}
I20260808 01:53:47.523545  9523 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:47.456744 19552 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.516358 19608 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:47.539728 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:47.548779  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 226541us
I20260808 01:53:47.528002 19554 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:47.471019 19328 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.552855 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:53:47.554204  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 101984us
I20260808 01:53:47.538210 19539 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:53:47.556113  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 147008us
I20260808 01:53:47.538728 19546 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:47.559583 10368 raft_consensus.cc:2368] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22.
I20260808 01:53:47.556126  9637 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:47.506709 19245 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:47.553627 19309 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election won for term 29
I20260808 01:53:47.542850 19553 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:47.576319 19328 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:47.578787 19561 log.cc:927] Time spent T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
W20260808 01:53:47.580248  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 96344.5us
I20260808 01:53:47.562258 19476 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:47.535771 19408 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.031s
I20260808 01:53:47.543027 19584 consensus_queue.cc:1048] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.220s
I20260808 01:53:47.587952  9481 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 22.
I20260808 01:53:47.614847  9600 leader_election.cc:400] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22.
I20260808 01:53:47.633908  9600 leader_election.cc:403] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260808 01:53:47.564989 19608 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:47.582773 19592 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:53:47.641601  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 52364.7us
W20260808 01:53:47.611253 19591 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260808 01:53:47.588370 19577 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:47.660020  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:47.654491  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:47.642300 19309 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 29 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:47.668754 19552 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:47.669600 19616 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:47.669739 19552 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:47.665378 19203 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:47.644531 19553 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:47.615413 19605 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.050s
I20260808 01:53:47.679052 19589 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:47.701061  9554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 4486) took 1246 ms. Trace:
W20260808 01:53:47.703088 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:47.650413 19592 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.702620  9554 rpcz_store.cc:276] 0808 01:53:46.456889 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:46.772252 (+315363us) service_pool.cc:224] Handling call
0808 01:53:46.803172 (+ 30920us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:47.081519 (+278347us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:47.081558 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:47.081607 (+    49us) raft_consensus.cc:1645] Marking committed up to 16
0808 01:53:47.081642 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:47.700800 (+619158us) spinlock_profiling.cc:243] Waited 278 ms on lock 0x7b5400313dd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:47.700857 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:47.700996 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":639492864}
W20260808 01:53:47.708448  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 54086.8us
I20260808 01:53:47.702253 19203 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260808 01:53:47.711588  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:47.712457  9479 raft_consensus.cc:2368] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22.
I20260808 01:53:47.712212 18971 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.712656  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:47.713557  9476 raft_consensus.cc:2393] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 26: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:53:47.714152 19599 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260808 01:53:47.717875 19553 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:47.718993 19516 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Stepping down as leader of term 30
I20260808 01:53:47.718813 19589 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.719564 19516 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:47.721675 19589 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:47.722355 19554 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:47.722873 19554 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:47.723284 19554 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:47.729868 19245 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:47.732139 19612 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:47.734097 19516 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:47.673158 19539 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.737479 18971 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:47.738562 19309 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.760042  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:47.761483  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 122260us
I20260808 01:53:47.706295 19408 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:47.685820 19552 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:47.765141  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 164858us
W20260808 01:53:47.764973  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 55440.5us
I20260808 01:53:47.763142 10334 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:47.764899  9599 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:47.747534 19516 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:53:47.775382  9481 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260808 01:53:47.772614 19579 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:47.773861 19478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:56258 (request call id 1112) took 3922 ms. Trace:
W20260808 01:53:47.791214  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 267787us
I20260808 01:53:47.792623  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:47.795370 19478 rpcz_store.cc:276] 0808 01:53:43.850998 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:43.851175 (+   177us) service_pool.cc:224] Handling call
0808 01:53:44.492812 (+641637us) ts_tablet_manager.cc:934] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359)
0808 01:53:47.006990 (+2514178us) tablet_copy_client.cc:368] Tablet copy session begun
0808 01:53:47.556698 (+549708us) tablet_metadata.cc:665] Metadata flushed
0808 01:53:47.561677 (+  4979us) tablet_copy_client.cc:497] Wrote new tablet metadata
0808 01:53:47.771722 (+210045us) tablet_copy_client.cc:418] Wrote new consensus metadata
0808 01:53:47.772153 (+   431us) tablet_replica.cc:206] Creating consensus instance
0808 01:53:47.772571 (+   418us) ts_tablet_manager.cc:989] Replica registered.
0808 01:53:47.773726 (+  1155us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":54005,"tablet-copy.queue_time_us":532002,"thread_start_us":597125,"threads_started":2}
I20260808 01:53:47.797183  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:47.797446 19478 tablet_copy_client.cc:806] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:53:47.798182 19478 tablet_copy_client.cc:670] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:53:47.798856 19245 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:47.792608 19573 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:47.799541 19245 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.800395 19620 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:47.801110 19573 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:47.801040 19620 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.801718 19614 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:47.802177 10365 raft_consensus.cc:2393] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 27: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:47.805306 19616 consensus_queue.cc:1059] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260808 01:53:47.807171  9632 raft_consensus.cc:2410] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 25 because replica has last-logged OpId of term: 24 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
I20260808 01:53:47.806542 19616 raft_consensus.cc:3055] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 LEADER]: Stepping down as leader of term 29
I20260808 01:53:47.808619 19616 raft_consensus.cc:740] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:47.809759 19616 consensus_queue.cc:260] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.810390 19606 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.005s
I20260808 01:53:47.812343  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:47.812897 19620 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:47.814917 19552 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:47.817570  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:47.801816 19245 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:47.823694  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 271766us
I20260808 01:53:47.824702  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:47.824121  9523 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:47.838433  9479 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 27: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:47.773026 19564 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Leader election won for term 31
I20260808 01:53:47.824188  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:47.849503  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:47.851665  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:47.852417  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:47.764200 19408 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.857988  9550 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:47.858354  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:47.815290 19616 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:47.863375  9636 raft_consensus.cc:2410] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 33 because replica has last-logged OpId of term: 32 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 17.
I20260808 01:53:47.793046 19579 raft_consensus.cc:697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:47.869467  9554 raft_consensus.cc:2393] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 28: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:47.872550  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:47.871616  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:47.877480 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 103986us
I20260808 01:53:47.880611  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:47.881907  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:47.886145  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:47.891192  9601 leader_election.cc:400] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 election: Vote denied by peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359) with higher term. Message: Invalid argument: T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22.
I20260808 01:53:47.891230  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:47.841809 19577 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.035s
I20260808 01:53:47.896919  9599 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:47.891775 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:47.867722 19245 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:47.911744 19245 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:47.913713  9521 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:47.843376 19614 consensus_queue.cc:1059] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 31.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.051s
I20260808 01:53:47.915493  9521 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:47.904959 19256 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:47.878904 19564 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 31 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:47.917174 19256 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:47.918228 19408 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:47.917939 19564 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.919016 19408 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260808 01:53:47.919790 19408 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:53:47.919890 19256 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:47.920328 19408 raft_consensus.cc:697] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:47.920545 19256 raft_consensus.cc:3055] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 LEADER]: Stepping down as leader of term 31
I20260808 01:53:47.892726 19605 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:47.907702 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:47.921574 10363 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
I20260808 01:53:47.921466 19605 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:47.774151 19608 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.924316 19605 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:47.905315  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:47.909452 19609 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:47.907897 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:47.925402 19609 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:47.892457 19620 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:47.915369 19579 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:47.938169  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:47.928083  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 104069us
I20260808 01:53:47.929059  9599 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:47.930501  9557 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:47.920991 19256 raft_consensus.cc:740] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:47.943111  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:47.945417 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:47.927110 19585 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:47.947443 10366 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:47.948103 19585 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:47.948431 19552 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260808 01:53:47.896754 18971 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:47.948838 19585 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.949298 19621 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:47.947810 19620 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.962513 19621 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:47.964718 19620 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:47.948640 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:47.936012 19605 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:47.953570  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:47.964807 19621 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:47.953533  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:47.949460 19552 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:47.933022 19245 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:47.946496  9599 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:47.948226 19592 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:47.970973  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:47.991575  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:47.954948 19553 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260808 01:53:47.973340 19580 consensus_meta.cc:287] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.662s	user 0.013s	sys 0.004s
I20260808 01:53:48.000870 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:47.987737  9599 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:47.984671 10363 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:48.010107  9558 raft_consensus.cc:2393] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 29: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:53:48.011459  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 68316.2us
I20260808 01:53:48.012396  9523 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:48.013413  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 98064.6us
I20260808 01:53:48.014497 19477 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260808 01:53:48.014714  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 109503us
I20260808 01:53:47.981412 19552 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:48.027786 19552 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:48.028753 19585 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:48.029249 19552 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
W20260808 01:53:48.031119  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 43518.3us
I20260808 01:53:48.031922  9599 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:48.036646  9521 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:48.008901 19578 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.713s	user 0.010s	sys 0.007s
I20260808 01:53:48.051564  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:47.921550 19408 consensus_queue.cc:237] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:48.057101 19580 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:48.057760 19580 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:48.060279  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.060225  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.052500  9556 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:48.035774 19305 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader election won for term 19
I20260808 01:53:48.064308  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.042333 19620 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:48.073729  9552 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
W20260808 01:53:48.082659  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 120159us
I20260808 01:53:48.087474  9600 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:48.122658  9600 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:48.053381 19606 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:48.049389 19609 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:48.141148  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:48.141702  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 127591us
I20260808 01:53:48.135767  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:48.160198  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 147879us
I20260808 01:53:48.094930 19589 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:48.167795  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:48.159349  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:48.133648 19602 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:48.140193 19592 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:48.055949 19578 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:48.141252 19605 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:48.157087 19612 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 21. Current term is 22.
I20260808 01:53:48.133659 19608 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:48.145751 19580 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:48.247300  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 106853us
I20260808 01:53:48.142705 19620 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:48.252873 19626 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260808 01:53:48.208164 19589 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:53:48.249486  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 106956us
I20260808 01:53:48.261684  9521 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:48.265405  9557 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 31.
I20260808 01:53:48.217474 19592 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:48.232604 19611 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260808 01:53:48.275861 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 192912us
I20260808 01:53:48.234105 19305 raft_consensus.cc:697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:48.273846  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:48.156972 19256 consensus_queue.cc:260] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:48.229298 19602 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:48.271739 19589 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
W20260808 01:53:48.277201 19554 consensus_meta.cc:287] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.553s	user 0.009s	sys 0.006s
I20260808 01:53:48.267369  9521 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:48.273269 19628 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:48.276358 19592 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:48.336223  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:48.344521 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:48.311934 19585 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:48.277994 19608 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:48.360632  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 216496us
I20260808 01:53:48.313290 19554 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:48.362509  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:48.371690  9444 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.194:36095 timed out after 1.853s (SENT) [suppressed 20 similar messages]
I20260808 01:53:48.338491 19408 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:48.353865 19631 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:48.391081 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:48.356623 19611 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:48.359613 19256 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Advancing to term 32
W20260808 01:53:48.409147 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 87885.4us
I20260808 01:53:48.405756  9629 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 22.
I20260808 01:53:48.354184 19574 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:48.416134  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:48.421066  9556 raft_consensus.cc:2410] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 31 because replica has last-logged OpId of term: 30 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 16.
I20260808 01:53:48.427423  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.403911 19626 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:48.432520  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.430225  9552 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:48.420367 19585 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:48.438800 19585 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:48.446926 19546 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:48.447180  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.433275 19574 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:48.452322 19532 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:48.453518 19574 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:48.454346 19611 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:48.456445 10365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56258 (request call id 1138) took 1118 ms. Trace:
I20260808 01:53:48.456908 10365 rpcz_store.cc:276] 0808 01:53:47.337720 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:47.337976 (+   256us) service_pool.cc:224] Handling call
0808 01:53:48.456172 (+1118196us) spinlock_profiling.cc:243] Waited 255 ms on lock 0x7b5400553054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:48.456344 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":585884416}
I20260808 01:53:48.458041 19611 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:48.462421 19604 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:48.462873 19328 consensus_queue.cc:1059] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260808 01:53:48.463254  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 3577) took 1407 ms. Trace:
I20260808 01:53:48.463953  9637 rpcz_store.cc:276] 0808 01:53:47.056168 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:47.056424 (+   256us) service_pool.cc:224] Handling call
0808 01:53:48.462946 (+1406522us) spinlock_profiling.cc:243] Waited 460 ms on lock 0x7b540008d254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:48.463124 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1058497280}
I20260808 01:53:48.467080  9633 raft_consensus.cc:1275] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260808 01:53:48.351442 19620 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:48.466461  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:48.473654  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:48.478044  9633 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:48.479658  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:48.484828  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.492311  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.493753  9552 raft_consensus.cc:2393] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 25: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:48.477835  9637 raft_consensus.cc:2376] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 23. Re-sending same reply.
I20260808 01:53:48.442365 19585 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:48.464283 19604 raft_consensus.cc:3055] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Stepping down as leader of term 27
I20260808 01:53:48.410849 19408 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:48.530642  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 68564.8us
I20260808 01:53:48.536027  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.540182  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.481958 19578 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:48.542136  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.414745 19631 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:48.545190  9552 raft_consensus.cc:2393] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 29: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:48.538266  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:48.546468  9550 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 30: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:48.548645  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.547081 19628 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:48.553602  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.553715 19628 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:48.513440 19604 raft_consensus.cc:740] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:48.556216 19604 consensus_queue.cc:260] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:48.556965 19628 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:48.557930 19604 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:48.482594 19626 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:48.552009  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:48.560601  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:48.560976 19626 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:48.561321  9637 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
I20260808 01:53:48.554345  9558 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
I20260808 01:53:48.562453  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:48.543891 19574 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:48.567309 19574 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:48.567854  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:48.548252 19254 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:48.572167 19254 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:48.572974  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.573737  9551 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:48.570142  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:48.483309 19579 consensus_queue.cc:1048] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.062s
W20260808 01:53:48.580123 19616 consensus_meta.cc:287] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.707s	user 0.008s	sys 0.008s
I20260808 01:53:48.580235  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:48.581059  9477 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:48.447696 19546 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:48.585136 19546 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:48.586861 19635 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
W20260808 01:53:48.591753  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 229342us
I20260808 01:53:48.591645  9559 raft_consensus.cc:2410] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 31 because replica has last-logged OpId of term: 30 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 16.
I20260808 01:53:48.592581  9600 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:48.404942 19305 consensus_queue.cc:237] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:48.606642  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:48.570333 19574 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:48.612079  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:48.612144  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.612928  9551 raft_consensus.cc:2376] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28. Re-sending same reply.
I20260808 01:53:48.553344 19554 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:48.620802  9633 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 30.
I20260808 01:53:48.625605 19638 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:48.626613 19638 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:48.629158 19578 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:48.629688 19578 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:48.630061 19578 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:48.633487 19606 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:53:48.635134 19627 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.455s	user 0.003s	sys 0.003s
I20260808 01:53:48.635157  7831 auto_leader_rebalancer.cc:474] table: test-workload, leader rebalance finish, leader transfer count: 0
W20260808 01:53:48.640070  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 47563.9us
I20260808 01:53:48.647714 19608 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:48.657109  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.657716  9554 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 25.
I20260808 01:53:48.656599  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:48.664230  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.665300 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:48.670293 10366 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260808 01:53:48.672008 10365 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 31: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:48.673694  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.674369  9552 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 25.
I20260808 01:53:48.679035  9477 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 27.
I20260808 01:53:48.679939  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:48.681701 19630 log.cc:927] Time spent T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.363s	user 0.000s	sys 0.001s
I20260808 01:53:48.685137  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:48.685909  9631 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:53:48.686949  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:48.687568  9631 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28.
I20260808 01:53:48.688153 19408 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:48.688577  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:48.689205  9631 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 31.
I20260808 01:53:48.689527 19254 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:48.667897 19608 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:48.691098 19608 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:48.691412  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:48.691623 19608 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:48.692123  9634 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28.
I20260808 01:53:48.676595 19602 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:48.694646 19579 consensus_queue.cc:1048] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:53:48.695660  9480 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260808 01:53:48.696089 19631 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:48.695825  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:48.708976 19631 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:48.709494 19631 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:48.709930 19631 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:48.704973 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:48.724162  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:48.726743 19254 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:48.650000 19606 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:48.727541 19254 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:53:48.728653  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:48.729197 19254 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:48.730019 19606 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:48.731643  9631 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260808 01:53:48.732790  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:48.733500  9631 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:53:48.733661 19638 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:48.737638  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.738879 10334 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:48.740286 19488 raft_consensus.cc:2804] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader election won for term 27
W20260808 01:53:48.740615  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:48.740940 19488 raft_consensus.cc:697] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:53:48.741168  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:48.737488  9636 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:48.732753  9521 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:48.745543  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:48.747502  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:48.752864  9481 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 29.
I20260808 01:53:48.754139  9521 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:48.713656  9479 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 13, Current term: 20, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:48.759270  9521 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:48.744228 19579 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:53:48.748194 19574 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:48.768254  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:48.766589  9521 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:48.772133  9554 raft_consensus.cc:2393] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 24: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:48.775173 19647 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.029s
I20260808 01:53:48.777344 19592 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:48.779753 19634 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.461s	user 0.001s	sys 0.000s
I20260808 01:53:48.780733 19592 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:48.785638 19578 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:48.790141  9630 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:48.763425 19408 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:48.791571 19488 consensus_queue.cc:237] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:48.792263  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.797003  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:48.767554 19621 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:48.751780 19564 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:48.798297  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:48.798484 19564 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:48.800832 19564 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:48.760923 19541 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:48.803431 19541 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:48.803918 19541 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:48.733661 19589 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:48.807533  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.809705  9560 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 27: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:48.797685  9551 raft_consensus.cc:3060] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 27
I20260808 01:53:48.809530 19589 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:48.810484 19552 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.775s	user 0.016s	sys 0.001s
I20260808 01:53:48.765647 19579 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:48.851919 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:48.839103  9523 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:48.829242  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62806.1us
I20260808 01:53:48.860442  9632 raft_consensus.cc:2410] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260808 01:53:48.861541 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:48.863786  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.858206  9521 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:48.827703 19478 tablet_copy_client.cc:538] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:53:48.871794 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:48.857965 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:48.873451  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.873835  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.877780  9557 raft_consensus.cc:2468] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:48.872087  9521 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:48.860229 19579 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:48.890333 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:48.895455 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:48.894109  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.829635 19363 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:48.899281  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:48.904915  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:48.905711  9559 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 29.
I20260808 01:53:48.885056 19476 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:48.913434 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:48.918503  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:48.920197  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:48.931360  9477 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 29: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:48.935060  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:48.938742  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:48.944272  9480 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260808 01:53:48.911960 19621 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260808 01:53:48.954536 19671 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260808 01:53:48.929657 19476 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:48.930045 19580 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:53:48.967718 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:48.972853  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:48.967832 19573 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:48.979979 19573 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:48.980578 19573 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:48.982769 19618 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:48.973846  9480 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 28.
I20260808 01:53:48.999250  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.004616  9477 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:48.934350 19628 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.041s
I20260808 01:53:48.965839 19621 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:48.927201 19363 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:49.021059  9557 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27. Ops: []
W20260808 01:53:49.024940 19646 log.cc:927] Time spent T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.421s	user 0.002s	sys 0.000s
W20260808 01:53:49.029932 19661 log.cc:927] Time spent T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.284s	user 0.003s	sys 0.000s
W20260808 01:53:49.032550 19662 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.286s	user 0.001s	sys 0.001s
I20260808 01:53:48.971872 19476 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:49.041173  9555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 4497) took 1218 ms. Trace:
I20260808 01:53:49.051043 19621 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Advancing to term 32
I20260808 01:53:49.042608  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42824 (request call id 3808) took 1238 ms. Trace:
I20260808 01:53:49.060599  9552 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 31.
I20260808 01:53:49.047245 19578 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:49.063949  9551 raft_consensus.cc:2468] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 27.
I20260808 01:53:48.971546 19580 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:49.090863 19589 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:49.091430 19580 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:49.074112  9555 rpcz_store.cc:276] 0808 01:53:47.822885 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:47.841443 (+ 18558us) service_pool.cc:224] Handling call
0808 01:53:47.862124 (+ 20681us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:47.905515 (+ 43391us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:47.905558 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:47.905610 (+    52us) raft_consensus.cc:1645] Marking committed up to 12
0808 01:53:47.905649 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:49.040718 (+1135069us) spinlock_profiling.cc:243] Waited 27.8 ms on lock 0x7b540029e5d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:49.040775 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:49.040954 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":64009472}
I20260808 01:53:49.046898 19626 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:49.081552  9632 rpcz_store.cc:276] 0808 01:53:47.803745 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:47.808018 (+  4273us) service_pool.cc:224] Handling call
0808 01:53:48.134603 (+326585us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b5800231698. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:53:49.042381 (+907778us) spinlock_profiling.cc:243] Waited 718 ms on lock 0x7b54001b3ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:49.042537 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1675149312}
I20260808 01:53:49.081909 10369 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:53:49.106351 19541 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:49.111603  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:49.115200  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:49.116288 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 280486us
I20260808 01:53:49.128530 19541 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:49.131970 10367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56266 (request call id 1492) took 1196 ms. Trace:
I20260808 01:53:49.132503 10367 rpcz_store.cc:276] 0808 01:53:47.906565 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:47.908444 (+  1879us) service_pool.cc:224] Handling call
0808 01:53:47.908975 (+   531us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:47.909519 (+   544us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:47.909572 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:47.910212 (+   640us) log.cc:844] Serialized 26 byte log entry
0808 01:53:48.379487 (+469275us) raft_consensus.cc:1645] Marking committed up to 17
0808 01:53:48.379562 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:48.379668 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:49.023842 (+644174us) spinlock_profiling.cc:243] Waited 2.45 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:53:49.098822 (+ 74980us) raft_consensus.cc:1685] finished
0808 01:53:49.098845 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:49.102641 (+  3796us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1509,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":122,"spinlock_wait_cycles":5629824,"thread_start_us":468753,"threads_started":1,"wal-append.queue_time_us":580996}
I20260808 01:53:49.133443  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:49.137864  9551 raft_consensus.cc:2393] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 32: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:49.140040  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:49.141217  9554 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 28.
I20260808 01:53:49.142998  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:49.144985 19631 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:49.097543 19626 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:49.145547 19476 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:49.149696 19573 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:49.152532 19363 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:49.157437 10332 leader_election.cc:336] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Timed out: RequestConsensusVote RPC to 127.7.165.194:36095 timed out after 1.698s (SENT)
I20260808 01:53:49.159231  9477 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260808 01:53:49.141899 19580 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:49.178835  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 100611us
I20260808 01:53:49.182104  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:49.189891  9558 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:49.192567  9560 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260808 01:53:49.193066  9554 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:49.195178 19541 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:49.196064 19541 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:49.183653 19676 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:49.201720 19674 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:49.201992 19676 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:49.202433 19674 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:49.203346  9633 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260808 01:53:49.204926 19674 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:49.205786 19676 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:49.207837 19608 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:49.210994 19608 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:49.211208 19626 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:49.213434  9638 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 1258) took 1421 ms. Trace:
I20260808 01:53:49.215121 19539 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:49.215355  9637 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3807) took 1411 ms. Trace:
I20260808 01:53:49.215617 19539 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:49.216066 19539 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:49.217780 19541 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:49.219270  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:49.220402  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3806) took 1417 ms. Trace:
I20260808 01:53:49.223992 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:49.225054 19363 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:49.224059 19554 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:53:49.228220  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:49.225736 19363 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:49.220883  9635 rpcz_store.cc:276] 0808 01:53:47.803062 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:48.044780 (+241718us) service_pool.cc:224] Handling call
0808 01:53:48.075317 (+ 30537us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:48.208089 (+132772us) raft_consensus.cc:1534] Early marking committed up to index 15
0808 01:53:48.519629 (+311540us) log.cc:844] Serialized 14 byte log entry
0808 01:53:49.212878 (+693249us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:49.212947 (+    69us) raft_consensus.cc:1645] Marking committed up to 15
0808 01:53:49.213009 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:49.220083 (+  7074us) spinlock_profiling.cc:243] Waited 86.2 ms on lock 0x7b540027e554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:49.220148 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:49.220319 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":611789,"prepare.queue_time_us":100898,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":11190,"raft.queue_time_us":612518,"raft.run_cpu_time_us":55,"raft.run_wall_time_us":55,"spinlock_wait_cycles":198231808,"thread_start_us":324327,"threads_started":2,"wal-append.queue_time_us":97735}
I20260808 01:53:49.226455  9552 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 29.
I20260808 01:53:49.202478  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 1257) took 1410 ms. Trace:
I20260808 01:53:49.247159  9629 rpcz_store.cc:276] 0808 01:53:47.791593 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:47.798966 (+  7373us) service_pool.cc:224] Handling call
0808 01:53:47.953336 (+154370us) spinlock_profiling.cc:243] Waited 19.7 ms on lock 0x7b5800231698. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:53:49.202193 (+1248857us) spinlock_profiling.cc:243] Waited 284 ms on lock 0x7b54004204d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:49.202388 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":697736320}
I20260808 01:53:49.249426  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.250219  9472 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 25.
I20260808 01:53:49.249514  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.215744  9637 rpcz_store.cc:276] 0808 01:53:47.803455 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:48.578558 (+775103us) service_pool.cc:224] Handling call
0808 01:53:48.580796 (+  2238us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:48.581768 (+   972us) raft_consensus.cc:1534] Early marking committed up to index 18
0808 01:53:48.583571 (+  1803us) log.cc:844] Serialized 14 byte log entry
0808 01:53:49.214996 (+631425us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:49.215050 (+    54us) raft_consensus.cc:1645] Marking committed up to 18
0808 01:53:49.215095 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:49.215166 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:49.215294 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":601487,"prepare.queue_time_us":485,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":607763,"raft.run_cpu_time_us":51,"raft.run_wall_time_us":51,"thread_start_us":28658,"threads_started":1,"wal-append.queue_time_us":95593}
I20260808 01:53:49.214221  9638 rpcz_store.cc:276] 0808 01:53:47.791888 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:47.799230 (+  7342us) service_pool.cc:224] Handling call
0808 01:53:47.990070 (+190840us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b5800231698. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:53:47.990428 (+   358us) raft_consensus.cc:1471] Updating replica for 3 ops
0808 01:53:48.044445 (+ 54017us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:48.044500 (+    55us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0808 01:53:48.076088 (+ 31588us) log.cc:844] Serialized 78 byte log entry
0808 01:53:48.337295 (+261207us) raft_consensus.cc:1645] Marking committed up to 20
0808 01:53:48.353371 (+ 16076us) log.cc:844] Serialized 14 byte log entry
0808 01:53:48.604050 (+250679us) log.cc:844] Serialized 14 byte log entry
0808 01:53:48.810940 (+206890us) spinlock_profiling.cc:243] Waited 24.6 ms on lock 0x7b1800475a20. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188ce62 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:53:49.212532 (+401592us) spinlock_profiling.cc:243] Waited 6.48 ms on lock 0x7b1800475a20. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667 00007f5d449ae5e8
0808 01:53:49.212848 (+   316us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:49.213007 (+   159us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:49.213076 (+    69us) raft_consensus.cc:1685] finished
0808 01:53:49.213097 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:49.213348 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":303023,"prepare.queue_time_us":36255,"prepare.run_cpu_time_us":345,"prepare.run_wall_time_us":19917,"raft.queue_time_us":899888,"raft.run_cpu_time_us":536,"raft.run_wall_time_us":35838,"spinlock_wait_cycles":116023808,"thread_start_us":103682,"threads_started":1,"wal-append.queue_time_us":276024}
I20260808 01:53:49.253348 19621 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:49.254922  9550 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 31. Current term is 32. Ops: []
I20260808 01:53:49.231066 19609 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:49.227064 19554 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:49.256458 19476 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:49.250712 19363 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:49.259284 19614 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:49.259951 19638 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:49.259887 19614 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:49.260644 19638 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:49.262387 19614 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:49.262902 19592 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:49.263664 19592 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:49.264431 19614 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:49.265064 19614 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:49.267748 19614 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:49.268152 19573 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:49.269129 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 111736us
I20260808 01:53:49.271036 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:49.272471 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:49.272567  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 522033us
I20260808 01:53:49.273506 19649 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:49.273824  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:49.260980  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:49.277290 19641 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:49.277884 19641 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:49.278517 19641 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:49.287554  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:49.285153  9599 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:49.290609  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.292691  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.296535 10334 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:49.299165  9477 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 19.
I20260808 01:53:49.266433 19592 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:49.309937  9521 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:49.258549 19554 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:49.241423 19631 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:49.327409  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:49.333523  9633 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:49.336375  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:49.337894 19580 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:49.311708 19607 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:49.342815  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:49.343626  9552 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 28: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:49.300254 19578 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260808 01:53:49.344725  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:49.321815  9629 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260808 01:53:49.341621 10334 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:49.331141  9481 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50124 (request call id 4059) took 1457 ms (client timeout 1524 ms). Trace:
W20260808 01:53:49.354734  9481 rpcz_store.cc:269] 0808 01:53:47.873890 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:47.874051 (+   161us) service_pool.cc:224] Handling call
0808 01:53:49.330951 (+1456900us) spinlock_profiling.cc:243] Waited 870 ms on lock 0x7b54004042d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:49.331082 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2000720256}
I20260808 01:53:49.338523  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:49.302335 19616 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:49.362283  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:49.356353  9632 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:49.303223 19573 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260808 01:53:49.358848  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:49.365935  9635 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 29.
I20260808 01:53:49.357041  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:49.362818 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 90467.9us
I20260808 01:53:49.358232  9560 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260808 01:53:49.316672 19584 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:49.371464  9560 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
W20260808 01:53:49.367895  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 58229.6us
I20260808 01:53:49.355952  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:49.374329  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:49.374476  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:49.330411 19638 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:49.364128 19603 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:53:49.377991  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:49.347071 19620 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:49.383090  9558 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:49.360122 19578 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:49.389829  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:49.390842  9638 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 32.
I20260808 01:53:49.395386 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:49.371296 19631 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election won for term 30
I20260808 01:53:49.371191 19245 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:49.378877 19554 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:49.393791  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:49.375887 19584 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:49.398278  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:49.392673  9633 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28.
I20260808 01:53:49.403733 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:49.405813  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:49.382370 19476 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260808 01:53:49.381978 19603 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:49.414865  9632 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 29.
I20260808 01:53:49.396147 19578 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:53:49.418226  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:49.372392 19579 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:49.421262  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:49.435071  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:49.379040 19580 consensus_queue.cc:1059] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.028s
I20260808 01:53:49.428608  9599 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:49.407047 19649 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260808 01:53:49.416541 19584 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:49.407271 19554 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:49.447325 19477 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election won for term 29
W20260808 01:53:49.471549  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 53386.7us
I20260808 01:53:49.419597 19516 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:53:49.485832  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:49.486052  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:49.475535  9634 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260808 01:53:49.399328 19618 consensus_queue.cc:1059] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.050s
I20260808 01:53:49.426009 19539 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:49.478965  9637 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:49.479117  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:49.402114 19628 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.056s
I20260808 01:53:49.480170 10363 raft_consensus.cc:1240] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: []
I20260808 01:53:49.497174  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.463536 19631 raft_consensus.cc:697] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:53:49.483500 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 79891.9us
I20260808 01:53:49.423455 19603 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:49.488902  9601 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:49.491780 19677 log.cc:927] Time spent T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260808 01:53:49.447570 19256 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.042s
I20260808 01:53:49.495220 19638 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260808 01:53:49.495332 19539 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:49.411882 19245 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:49.480775  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.496282 19477 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:49.481393 19554 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:49.477651 19541 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260808 01:53:49.484423 19607 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:49.425138 19620 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:49.562423  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.475970 19647 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Leader election won for term 31
I20260808 01:53:49.488708 19516 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:53:49.527220  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 128993us
I20260808 01:53:49.467574 19623 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.037s
I20260808 01:53:49.540366  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.487365  9635 raft_consensus.cc:2393] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 30: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:49.495903 19669 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:49.558126  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 129573us
W20260808 01:53:49.558887  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 376844us
I20260808 01:53:49.559054  9637 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:49.521816  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.532498 19621 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:49.564142  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.567799  9481 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28.
I20260808 01:53:49.573901  9477 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260808 01:53:49.577382  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:49.579772  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:49.584355  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.585121  9477 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 28.
I20260808 01:53:49.584957  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.538632 19577 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.034s
I20260808 01:53:49.563194 19604 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:49.591205  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.566272 19607 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:49.568708 19516 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:49.595256  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.566655 19552 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:49.574543 19623 consensus_queue.cc:1243] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 26.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260808 01:53:49.602330  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:49.602654  9474 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.622586  9474 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:49.604127  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.624089  9479 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 29.
I20260808 01:53:49.604190  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.624951  9472 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 27: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:49.576133 19539 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
W20260808 01:53:49.609321 19686 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
I20260808 01:53:49.570971 19638 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.033s
I20260808 01:53:49.635257  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:49.593238 19644 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:49.581292 19647 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:53:49.626109  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 55535.6us
I20260808 01:53:49.629792  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:49.642251  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:49.616396 19607 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:49.616035 19442 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:49.662595 19604 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:49.576054 19618 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:49.663249 19604 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:49.663657 19618 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:49.644842 19678 log.cc:927] Time spent T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.239s	user 0.001s	sys 0.000s
W20260808 01:53:49.645809 19688 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260808 01:53:49.649441  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:49.651105  9478 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260808 01:53:49.651400  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:49.651682  9480 raft_consensus.cc:1240] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:49.689699  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:49.690548  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:49.646781 19684 consensus_queue.cc:1059] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.152s
I20260808 01:53:49.690734  9630 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:49.691356  9556 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 20.
I20260808 01:53:49.700275  9523 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:49.673206 19478 tablet_bootstrap.cc:492] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:53:49.665625 19604 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:49.704191  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:49.671020 19579 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:49.576906 19669 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:49.712839  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.713523  9480 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
W20260808 01:53:49.714234 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 134502us
I20260808 01:53:49.708644  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:49.705855  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:49.710762  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:49.665723 19618 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:49.715929  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 185955us
I20260808 01:53:49.718444  9555 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 27: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:49.717163  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:49.687564 19442 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:49.721712  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:49.726939  9560 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:49.631016 19631 consensus_queue.cc:237] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:49.671891 19644 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:49.743294  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:49.746294  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:49.744107  9480 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 31.
W20260808 01:53:49.748031  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 105866us
I20260808 01:53:49.748911  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:49.735123 19476 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:49.754115 19476 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:49.754711 19476 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:49.642978 19477 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:49.765549 19684 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:49.766386 19684 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:49.769035 19684 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:49.770062 19476 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:49.777101 19245 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:49.781524 19476 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:49.784953 19669 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:49.785362 19647 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:49.786428 19620 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:49.746191  9551 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0ea307d9adc442539683547d0176a05b"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:51048
I20260808 01:53:49.787583  9551 raft_consensus.cc:606] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:49.787963  9551 raft_consensus.cc:614] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:49.743356 19650 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.052s
I20260808 01:53:49.800758  9481 raft_consensus.cc:2410] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 28 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260808 01:53:49.746537 19574 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:49.810907  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:49.812209  9480 raft_consensus.cc:3060] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:49.830379  9560 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:49.839870  9638 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260808 01:53:49.805541 19604 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:49.864234 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:49.778136 19641 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:49.871155 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:53:49.871351  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 122516us
I20260808 01:53:49.872093 10365 raft_consensus.cc:3060] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:49.872474  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:49.870265  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.873909  9475 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 27.
I20260808 01:53:49.876335  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:49.877305 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:49.879995  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:49.880990  9638 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 32.
I20260808 01:53:49.885515  9475 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:49.817556 19574 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:49.820403 19578 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:49.922412  9629 raft_consensus.cc:1240] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:49.936713  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:49.913522 19604 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260808 01:53:49.956135  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 83554.9us
I20260808 01:53:49.965404  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.986353  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:49.989969  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:49.991760 19699 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260808 01:53:49.992547 19203 raft_consensus.cc:2764] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260808 01:53:49.951251 19618 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:49.993680 19618 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:49.838629  7831 auto_leader_rebalancer.cc:741] global leader rebalance: transfer for tablet 0ea307d9adc442539683547d0176a05b (from 52ccf4dcca1843f7bb29e9172e2d4ab4 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:49.950426 19604 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:49.995127  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:49.999516  9476 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:50.008337  9475 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 30.
I20260808 01:53:50.011196 10369 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
I20260808 01:53:50.013123 19203 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260808 01:53:50.022492  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:50.024036  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:50.024567 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:50.025350 10368 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:50.026404  9560 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "439040d168824496acc076b1c280bde3"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:51048
I20260808 01:53:50.026547  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:50.027102  9560 raft_consensus.cc:606] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:50.027505  9560 raft_consensus.cc:614] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:50.002444 19604 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:50.032967  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:50.022511  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:50.036593 19305 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260808 01:53:50.013279 19584 consensus_meta.cc:287] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.525s	user 0.014s	sys 0.000s
I20260808 01:53:50.039055 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:50.039851  9445 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:50.047972  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:50.048663  9475 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 19 index: 17. (index mismatch)
I20260808 01:53:50.054729  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:50.057276 10367 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260808 01:53:50.051342 19203 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:50.040642 19703 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:53:50.042001 19644 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:50.051730 19305 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:53:50.067034  9480 raft_consensus.cc:2468] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 29.
W20260808 01:53:50.064845  7831 auto_leader_rebalancer.cc:741] global leader rebalance: transfer for tablet 439040d168824496acc076b1c280bde3 (from 52ccf4dcca1843f7bb29e9172e2d4ab4 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:50.066905  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:50.053830 19628 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:53:50.069502  9476 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260808 01:53:50.063414 19203 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:50.069854 19442 raft_consensus.cc:2764] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election decision vote started in defunct term 28: won
I20260808 01:53:50.076416 19631 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260808 01:53:50.081154  9560 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:51048
I20260808 01:53:50.064534 19580 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:50.083283 19668 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:50.038975 19607 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:50.085563 19203 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:50.087189  9560 raft_consensus.cc:606] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:50.088078  9524 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:50.047429 19584 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:50.089339 10365 raft_consensus.cc:2468] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
W20260808 01:53:50.090220  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 67827.8us
I20260808 01:53:50.092674  9560 raft_consensus.cc:614] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:50.075948 19641 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:50.088821 19578 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:50.097007 19628 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:50.097731  9630 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27. Ops: []
W20260808 01:53:50.098294  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 381247us
I20260808 01:53:50.098811 19607 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:50.103832 19626 raft_consensus.cc:3055] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:50.104344 19626 raft_consensus.cc:740] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:50.105389 19626 consensus_queue.cc:260] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:50.106745 19606 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:50.107199 19606 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:50.107601 19606 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Advancing to term 29
W20260808 01:53:50.103611 19603 consensus_meta.cc:287] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.573s	user 0.009s	sys 0.005s
W20260808 01:53:50.111375  7831 auto_leader_rebalancer.cc:741] global leader rebalance: transfer for tablet 4c485e8f4266462e8413079adcf8faef (from 52ccf4dcca1843f7bb29e9172e2d4ab4 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
W20260808 01:53:50.113441 19554 consensus_meta.cc:287] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.544s	user 0.013s	sys 0.000s
I20260808 01:53:50.113945 19554 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:50.115360  9631 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 28. Current term is 29. Ops: []
I20260808 01:53:50.102016 19618 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:50.100497  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:50.120774 19698 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:50.121381 19698 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:50.119751 19618 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260808 01:53:50.122283 19618 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:50.122704 19618 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Advancing to term 33
I20260808 01:53:50.102206 19628 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:50.127009 19693 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:50.127431 19628 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:50.127705 19693 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:50.119696  9524 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:50.130000 19621 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:50.130470 19621 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:50.093784 19616 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.725s	user 0.015s	sys 0.000s
W20260808 01:53:50.137844  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 44086us
I20260808 01:53:50.106633 19626 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:50.131307  9524 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:50.140008  9551 raft_consensus.cc:1240] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:50.143467 19476 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election won for term 30
I20260808 01:53:50.143360  9555 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:51048
I20260808 01:53:50.144498  9555 raft_consensus.cc:606] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:50.145015  9555 raft_consensus.cc:614] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260808 01:53:50.116446 19554 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:50.139715  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:50.130926 19621 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:50.157456  9558 raft_consensus.cc:1275] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260808 01:53:50.130647 19693 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:50.160331  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:50.161470  7831 auto_leader_rebalancer.cc:741] global leader rebalance: transfer for tablet 8c49fd321f8b4f89868246239530e78b (from 52ccf4dcca1843f7bb29e9172e2d4ab4 to 259bcb3f0d254e8b9effac3277e654e0) failed: code: NOT_THE_LEADER status { code: ILLEGAL_STATE message: "not currently leader" }
I20260808 01:53:50.145678 19477 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260808 01:53:50.171375 19703 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:50.171998 19703 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:50.147161 19616 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:50.162251 19669 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:50.167968 19674 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:50.169149 19641 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.006s
I20260808 01:53:50.153702 19476 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:50.167876 19408 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260808 01:53:50.187768  9632 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:50.181715  9601 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:50.166216  9524 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:50.182425  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:50.117750 19603 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:50.192757 19669 raft_consensus.cc:2804] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:50.193673 19408 raft_consensus.cc:2804] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader election won for term 29
I20260808 01:53:50.199335 19476 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:50.217592  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:50.220441 19702 log.cc:927] Time spent T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.202s	user 0.003s	sys 0.000s
I20260808 01:53:50.191058  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:50.222904 19611 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:53:50.225530 19611 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:50.225157 19203 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:50.225932 19611 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:50.229712 19603 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:50.217541 19620 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:50.235438 19620 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:50.235894 19620 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:50.242895 10334 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:50.244802 19488 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:50.245405 19488 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:50.245848 19488 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:50.248067 19203 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:50.250200 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:53:50.251534  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:50.204314 19604 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:50.207396 19516 consensus_meta.cc:287] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.580s	user 0.012s	sys 0.001s
I20260808 01:53:50.271760 19516 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:50.207631 19254 raft_consensus.cc:3055] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:50.274916 19254 raft_consensus.cc:740] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:50.275938 19254 consensus_queue.cc:260] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:50.277386 19254 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:50.183895 19674 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:50.281039 19674 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:50.255568 19305 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:50.213654  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:50.283516 19305 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:50.284742  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:50.214438  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:50.257769 19408 raft_consensus.cc:697] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:50.288127  9476 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 29.
I20260808 01:53:50.210163 19703 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:50.292039  9479 raft_consensus.cc:1275] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260808 01:53:50.215287  9632 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 30.
I20260808 01:53:50.210911 19669 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:50.294251 19669 raft_consensus.cc:3060] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:50.255725 10363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 1225) took 1375 ms. Trace:
I20260808 01:53:50.298990 10363 rpcz_store.cc:276] 0808 01:53:48.879945 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:48.880376 (+   431us) service_pool.cc:224] Handling call
0808 01:53:48.880710 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:49.496497 (+615787us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:50.255411 (+758914us) spinlock_profiling.cc:243] Waited 599 ms on lock 0x7b54004c3454. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:50.255627 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1378002304}
I20260808 01:53:50.301486 19631 consensus_queue.cc:1059] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:50.302537 19631 raft_consensus.cc:3055] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Stepping down as leader of term 19
I20260808 01:53:50.302901  9599 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:50.304385 19554 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:50.255787 19589 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260808 01:53:50.305796  9599 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:50.303983 19488 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:50.306932 10367 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 28: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:50.307329 19553 raft_consensus.cc:2804] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election won for term 25
W20260808 01:53:50.314131 19710 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.172s	user 0.004s	sys 0.000s
I20260808 01:53:50.313673 19669 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:50.315236  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:50.316027  9556 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
I20260808 01:53:50.316321 19669 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:50.302904 19631 raft_consensus.cc:740] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:50.317886 19631 consensus_queue.cc:260] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:50.318707 19363 raft_consensus.cc:3055] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEADER]: Stepping down as leader of term 24
I20260808 01:53:50.319116 19363 raft_consensus.cc:740] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:50.319334 19631 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:50.288153 19408 consensus_queue.cc:237] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:50.320099 19363 consensus_queue.cc:260] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:50.321190 19363 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:50.316710 19553 raft_consensus.cc:697] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:50.323690 19553 consensus_queue.cc:237] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:50.326068 19707 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.319s	user 0.003s	sys 0.000s
I20260808 01:53:50.316085 19574 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:50.328912 10367 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: []
I20260808 01:53:50.183425 19477 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:50.329625 19477 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:50.330811  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:50.270732  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:50.188364 19693 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:50.332443 19693 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:50.332943 19693 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:50.333660 10332 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:50.282927 19584 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:50.335255  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:50.211434  9558 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cacb9622c47c4737beff104125e98b77"
dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
mode: GRACEFUL
new_leader_uuid: "259bcb3f0d254e8b9effac3277e654e0"
 from {username='slave'} at 127.0.0.1:51048
W20260808 01:53:50.338402  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:50.338843  9558 raft_consensus.cc:606] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Received request to transfer leadership to TS 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:50.344945  9479 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:50.350025  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:50.350235  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 133239us
I20260808 01:53:50.353150  9524 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:50.274381 19516 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:50.359323  9478 raft_consensus.cc:2410] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
W20260808 01:53:50.359450 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:50.361145  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:50.362437  9600 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:50.383539  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:50.286398 19305 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:50.380873  9472 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:50.367623 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:50.402207  9472 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:50.366454  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:50.403119  9560 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
W20260808 01:53:50.395442  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:50.402575  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:50.406857  9552 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 31: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:50.410041  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
W20260808 01:53:50.415850 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 82266.5us
I20260808 01:53:50.365221 19554 consensus_queue.cc:1059] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.056s
I20260808 01:53:50.425421 19488 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:50.426524  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:50.427245  9550 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:50.430913 19552 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:50.431224  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:50.432224 19584 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:50.435508  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:50.422415 19554 raft_consensus.cc:3055] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:50.436991 19554 raft_consensus.cc:740] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:50.437570  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:50.365283 19647 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:50.439608 19647 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:50.440788  9557 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 29. Current term is 30. Ops: []
I20260808 01:53:50.440995 19305 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:50.441664  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:50.442219 19647 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:50.442739  9599 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:50.444131  9599 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:50.445083 19578 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:50.445540 19578 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:50.446055 19578 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:50.440794 19516 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election won for term 28
I20260808 01:53:50.448908  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:50.444854  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:50.452579 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:50.449879  9556 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
I20260808 01:53:50.365814  9635 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 28. Current term is 30. Ops: []
I20260808 01:53:50.456862  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
W20260808 01:53:50.459755 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 74377.5us
W20260808 01:53:50.459834  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 236731us
I20260808 01:53:50.464982  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:50.384828 19574 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:53:50.472471  9481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:50.467334  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:50.467837 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:50.469960  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:50.384366 19698 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:50.377388 19609 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260808 01:53:50.402184 19616 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:50.480048 10369 raft_consensus.cc:1240] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26. Ops: []
W20260808 01:53:50.480697 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 62135.7us
I20260808 01:53:50.485077  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:50.501431  9475 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 28: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:50.487813  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:50.494331  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:50.449568 19516 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:50.438306 19611 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:50.529074 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:50.513152 10367 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260808 01:53:50.514147  7831 auto_leader_rebalancer.cc:751] global leader rebalance finished, scheduled 5 move(s)
I20260808 01:53:50.375910 19703 consensus_queue.cc:1059] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.101s
I20260808 01:53:50.414413 19580 consensus_queue.cc:1048] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.100s
I20260808 01:53:50.531482 10368 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:50.415277 19539 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.101s
I20260808 01:53:50.472611 19574 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:50.464121 19641 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:50.532735  9550 raft_consensus.cc:2468] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
W20260808 01:53:50.563849  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:50.575721  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:50.533188  7831 auto_leader_rebalancer.cc:844] All tables' leader rebalancing finished this round
I20260808 01:53:50.472218 19668 consensus_queue.cc:1059] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.081s
I20260808 01:53:50.530525 19685 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:50.542325 19698 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:50.438073 19554 consensus_queue.cc:260] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:50.595389  9479 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 30: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:50.508407 19203 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.076s
I20260808 01:53:50.473419 19614 consensus_queue.cc:1059] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.116s
I20260808 01:53:50.555037 19616 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:50.584342 19574 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:50.572628 19609 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:50.585456 19641 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:50.627990  9472 raft_consensus.cc:2468] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 26.
I20260808 01:53:50.629200  9477 raft_consensus.cc:1240] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26. Ops: [23.18-25.19]
W20260808 01:53:50.637461  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 137889us
I20260808 01:53:50.639848  9472 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:50.603945 19668 raft_consensus.cc:3055] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:50.616993 19589 raft_consensus.cc:3055] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Stepping down as leader of term 19
W20260808 01:53:50.647552  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 73069.9us
I20260808 01:53:50.638788 10334 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:50.657066  9472 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260808 01:53:50.594359 19696 consensus_queue.cc:1048] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.043s
I20260808 01:53:50.663002  9600 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:50.664737 19674 raft_consensus.cc:2804] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:50.665786 19674 raft_consensus.cc:697] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:50.665444  9367 catalog_manager.cc:5697] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:50.667222 19674 consensus_queue.cc:237] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:50.668272  9367 catalog_manager.cc:5697] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 22 to 23, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 23 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:50.626266 19609 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:50.641181 19641 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:50.679448  9367 catalog_manager.cc:5697] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 19 to 22, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 22 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:50.610132 19698 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:50.657375 19618 consensus_meta.cc:287] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.534s	user 0.007s	sys 0.006s
I20260808 01:53:50.646137 19668 raft_consensus.cc:740] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:50.611975 19716 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.076s
I20260808 01:53:50.607962 19685 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:50.697813  9367 catalog_manager.cc:5697] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 22 to 26, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 26 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:50.662493 19589 raft_consensus.cc:740] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:50.683570 19477 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:50.682785 19614 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:50.695796  9444 connection.cc:256] RPC call timeout handler was delayed by -1.2417206200188957s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1535s for a response
I20260808 01:53:50.683843 19609 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
W20260808 01:53:50.718499  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 248627us
W20260808 01:53:50.719600  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:50.724498  9367 catalog_manager.cc:5697] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 18 to 26. New cstate: current_term: 26 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:50.712064 19623 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260808 01:53:50.694720 19604 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:50.707914 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 163915us
I20260808 01:53:50.706096 19203 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:50.673731 19553 consensus_queue.cc:1059] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.061s
I20260808 01:53:50.763978 10332 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:50.774528  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:53:50.752638  9367 catalog_manager.cc:5697] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 22 to 23, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 23 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:50.753327 19638 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:50.677515 19516 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:50.712603 19618 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:50.744937 19716 consensus_queue.cc:1243] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 30.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
W20260808 01:53:50.791417  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:50.768124 19476 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:50.760895 19616 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:50.795446  9367 catalog_manager.cc:5697] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 22 to 24, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 24 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:53:50.800254  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:50.858166  9635 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260808 01:53:50.868466  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:50.831530  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:50.846139 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:53:50.798995  9366 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55058 (request call id 133) took 46331 ms (client timeout 15000 ms). Trace:
I20260808 01:53:50.832659  9472 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 26. Current term is 27. Ops: []
W20260808 01:53:50.850591 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:50.783610 19603 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260808 01:53:50.860694 19606 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.753s	user 0.012s	sys 0.003s
W20260808 01:53:50.814299  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:50.780149 19203 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:50.823993 19611 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:50.768651 19614 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:50.848793 19408 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:53:50.955927 19254 consensus_meta.cc:287] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.678s	user 0.009s	sys 0.005s
I20260808 01:53:50.956457 19408 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:50.957918 19614 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:50.799185 19685 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:50.922057 19592 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:50.959729 19592 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:50.960777 19621 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.797s	user 0.014s	sys 0.000s
I20260808 01:53:50.961494 19621 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:50.935422 19631 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.602s	user 0.007s	sys 0.007s
I20260808 01:53:50.964247  9560 raft_consensus.cc:1240] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:50.930365  9367 catalog_manager.cc:5697] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 22 to 25, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 25 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:53:50.997397  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:50.959842 19628 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.832s	user 0.014s	sys 0.000s
I20260808 01:53:51.001756 19254 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:51.017095 19254 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:51.017637  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:51.012171  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 436011us
I20260808 01:53:51.019274  9523 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:51.019968  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:51.022792  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:50.781481 19668 consensus_queue.cc:260] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:51.022830  9637 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260808 01:53:50.794340 19638 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:50.937879 19579 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 26.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.085s
I20260808 01:53:50.873207 19476 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.030634 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:51.030092  9367 catalog_manager.cc:5697] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 15 to 22, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 22 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:51.031270  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:50.940104 19606 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.030723  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:51.075189 19728 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260808 01:53:51.075380  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:51.076426  9554 raft_consensus.cc:2368] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 31. Current term is 32.
I20260808 01:53:51.020686 19604 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:51.074433  9551 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 26.
I20260808 01:53:51.080467  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:51.081369  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 213016us
I20260808 01:53:51.081225  9629 raft_consensus.cc:2410] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 28 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260808 01:53:51.086146  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:51.084483  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:51.089010  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:51.039247 19712 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.152s	user 0.001s	sys 0.000s
I20260808 01:53:51.094218  9552 raft_consensus.cc:1240] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27. Ops: []
W20260808 01:53:51.038908 19717 log.cc:927] Time spent T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260808 01:53:51.019428 19628 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.027643 19729 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:51.095829 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:51.099927  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:51.098255  9559 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260808 01:53:51.027688 19735 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:51.098865  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:51.021478 19641 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.101754 10367 raft_consensus.cc:2393] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 20: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:51.096412  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:51.182399  9638 raft_consensus.cc:1240] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:51.091926  9367 catalog_manager.cc:5697] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 16 to 26. New cstate: current_term: 26 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:51.103538  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:51.040931 19721 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:51.078729 19698 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:51.104413 10368 raft_consensus.cc:1240] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 19. Current term is 20. Ops: [19.17-19.17]
I20260808 01:53:51.063252 19592 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:51.081867 19553 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:53:51.145361  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:51.193024  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:51.107270 19621 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:51.198318  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:51.090776 19716 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:51.111325 19203 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:51.117404 19713 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.389s	user 0.006s	sys 0.009s
W20260808 01:53:51.110140 19578 consensus_meta.cc:287] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.663s	user 0.010s	sys 0.000s
I20260808 01:53:51.201196 19578 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.203569 19578 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:51.103673 19722 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:51.205111  9638 raft_consensus.cc:2393] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 28: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:51.174494 19735 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:50.871747  9366 rpcz_store.cc:269] 0808 01:53:04.355237 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:14.720599 (+10365362us) service_pool.cc:224] Handling call
0808 01:53:50.677177 (+35956578us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:53:40.450542 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:53:40.717531 (+266989us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:53:40.746519 (+ 28988us) write_op.cc:435] Acquired schema lock
0808 01:53:40.758907 (+ 12388us) tablet.cc:662] Decoding operations
0808 01:53:40.844308 (+ 85401us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:53:40.844495 (+   187us) write_op.cc:641] Partition lock acquired for write op
0808 01:53:40.844565 (+    70us) tablet.cc:685] Acquiring locks for 11 operations
0808 01:53:40.846457 (+  1892us) tablet.cc:701] Row locks acquired
0808 01:53:40.846485 (+    28us) write_op.cc:260] PREPARE: finished
0808 01:53:40.903183 (+ 56698us) write_op.cc:270] Start()
0808 01:53:40.913294 (+ 10111us) write_op.cc:276] Timestamp: P: 1786154020880181 usec, L: 0
0808 01:53:40.913325 (+    31us) op_driver.cc:348] REPLICATION: starting
0808 01:53:40.954118 (+ 40793us) log.cc:844] Serialized 4430 byte log entry
0808 01:53:41.722723 (+768605us) op_driver.cc:464] REPLICATION: finished
0808 01:53:41.788987 (+ 66264us) write_op.cc:301] APPLY: starting
0808 01:53:41.789227 (+   240us) tablet.cc:1367] starting BulkCheckPresence
0808 01:53:43.016296 (+1227069us) spinlock_profiling.cc:243] Waited 2.7 ms on lock 0x7b180041abe0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d446d171c 00007f5d446c8e1c 00007f5d4840afa3 00007f5d4840b5c2 00007f5d48455a13 00007f5d482fe7c3 00007f5d482fe08b 00007f5d482ff857 00007f5d483fe814
0808 01:53:43.223963 (+207667us) tablet.cc:1370] finished BulkCheckPresence
0808 01:53:43.231825 (+  7862us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:53:43.552379 (+320554us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:53:43.575259 (+ 22880us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=22,key_file_lookups=22,delta_file_lookups=0,mrs_lookups=0
0808 01:53:43.577158 (+  1899us) write_op.cc:312] APPLY: finished
0808 01:53:43.608474 (+ 31316us) log.cc:844] Serialized 105 byte log entry
0808 01:53:43.660496 (+ 52022us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:53:43.668749 (+  8253us) write_op.cc:454] Released schema lock
0808 01:53:43.668829 (+    80us) write_op.cc:341] FINISH: Updating metrics
0808 01:53:43.671303 (+  2474us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:53:43.671341 (+    38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1243481,"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":65777,"apply.run_cpu_time_us":55889,"apply.run_wall_time_us":1882357,"cfile_cache_hit":46,"cfile_cache_hit_bytes":50173,"num_ops":11,"prepare.queue_time_us":398011,"prepare.run_cpu_time_us":22005,"prepare.run_wall_time_us":832113,"raft.queue_time_us":327226,"raft.run_cpu_time_us":9030,"raft.run_wall_time_us":94453,"replication_time_us":809141,"spinlock_wait_cycles":6219648,"thread_start_us":600621,"threads_started":5,"wal-append.queue_time_us":476920,"wal-append.run_cpu_time_us":49437,"wal-append.run_wall_time_us":2462258}]]}
I20260808 01:53:51.206043 19552 raft_consensus.cc:3055] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 LEADER]: Stepping down as leader of term 29
I20260808 01:53:51.206542 19552 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:53:51.207459  9366 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50084 (request call id 238) took 17710 ms (client timeout 15000 ms). Trace:
I20260808 01:53:51.207598 19552 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.208901 19552 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:51.211450 19641 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:51.211952 19442 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 31 LEADER]: Stepping down as leader of term 31
I20260808 01:53:51.212391 19442 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:51.213564 19735 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:51.165397 19408 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:51.167114 19606 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:51.168398 19730 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:51.215137 19730 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.217720 19730 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:51.160897  9558 raft_consensus.cc:1240] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:51.164247 19628 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:51.219761 19574 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.579s	user 0.012s	sys 0.001s
I20260808 01:53:51.220330 19574 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:51.169147 19720 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.420s	user 0.008s	sys 0.004s
I20260808 01:53:51.221709  9555 raft_consensus.cc:1240] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 29. Current term is 30. Ops: [27.14-29.15]
I20260808 01:53:51.120236 19729 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:51.172220 19363 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.851s	user 0.014s	sys 0.000s
I20260808 01:53:51.172775 19476 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:51.170642 19693 consensus_meta.cc:287] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.837s	user 0.009s	sys 0.004s
I20260808 01:53:51.106037 10365 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
W20260808 01:53:51.107949 19719 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.236s	user 0.000s	sys 0.000s
I20260808 01:53:51.028980 19731 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260808 01:53:51.175684  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:51.040374 19724 consensus_queue.cc:1059] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.394s
I20260808 01:53:51.165167 19669 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Stepping down as leader of term 27
W20260808 01:53:51.223523 19584 consensus_meta.cc:287] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.778s	user 0.011s	sys 0.000s
I20260808 01:53:51.181118  9633 raft_consensus.cc:1240] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 24. Current term is 25. Ops: []
I20260808 01:53:51.234057  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:51.182096  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 77347.4us
W20260808 01:53:51.235592  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:51.179118 19739 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:51.234622  9599 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:51.238948  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:51.188781 19721 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.205425 19607 raft_consensus.cc:3055] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Stepping down as leader of term 25
W20260808 01:53:51.207923  9366 rpcz_store.cc:269] 0808 01:53:33.497650 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:51.207180 (+17709530us) service_pool.cc:211] Skipping call since client already timed out
0808 01:53:51.207373 (+   193us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260808 01:53:51.162961 19554 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:51.213279 19442 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.228356 19641 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:51.172909 19638 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:51.225088  9367 catalog_manager.cc:5697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 16 to 21, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 21 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:51.231194  9634 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
W20260808 01:53:51.223712 19620 consensus_meta.cc:287] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.963s	user 0.010s	sys 0.005s
I20260808 01:53:51.233755 19669 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:51.178712 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:51.235509  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:51.119992  9630 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:51.238116  9556 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: []
I20260808 01:53:51.251081  7831 auto_leader_rebalancer.cc:153] leader rebalance for table test-workload
I20260808 01:53:51.244000 10332 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:51.248831  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:51.093989 19685 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:51.255846  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:51.257156  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:51.258630  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:51.302333 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 58418.4us
I20260808 01:53:51.264526 19442 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Advancing to term 32
I20260808 01:53:51.267822 19721 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:51.269869 19729 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:51.272562 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:51.277300 19609 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.526s	user 0.005s	sys 0.007s
I20260808 01:53:51.277439 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:51.197357 19553 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:51.278280 19592 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:51.249008 19584 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:51.257140 19607 raft_consensus.cc:740] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:51.355674  9367 catalog_manager.cc:5697] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 14 to 21, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 21 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:51.233453 19628 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.046s
I20260808 01:53:51.359622  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:51.228931 19693 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.258134 19739 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.331854 19696 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:51.289726  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:51.301857 19735 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:51.233739 19578 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:53:51.302539  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 67127.3us
I20260808 01:53:51.381618  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:51.174120 19626 consensus_meta.cc:287] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.022s	user 0.011s	sys 0.002s
I20260808 01:53:51.304488  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:51.311801  9479 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260808 01:53:51.339980  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:51.230501 19606 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.057s
I20260808 01:53:51.232838 19730 consensus_queue.cc:1059] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.066s
I20260808 01:53:51.353025  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:51.355536  9480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:51.356413  9474 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 30 index: 15. (index mismatch)
I20260808 01:53:51.311928 19746 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:53:51.283206  9559 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260808 01:53:51.360626  9554 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:53:51.420528  9636 raft_consensus.cc:2410] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 30 because replica has last-logged OpId of term: 29 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 15.
I20260808 01:53:51.361443  9552 raft_consensus.cc:1240] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 29. Current term is 30. Ops: [29.16-29.16]
I20260808 01:53:51.290987 19724 raft_consensus.cc:2804] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:51.367146  9475 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:51.286408  9635 raft_consensus.cc:1240] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 28. Current term is 29. Ops: []
I20260808 01:53:51.294299  9478 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:51.422209  9478 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Advancing to term 33
I20260808 01:53:51.366288  9367 catalog_manager.cc:5697] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 21 to 24. New cstate: current_term: 24 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:51.377066  9551 raft_consensus.cc:2410] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 20 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260808 01:53:51.423305  9635 raft_consensus.cc:1275] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260808 01:53:51.250715 19716 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:51.429430 19716 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:51.299201 19641 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:51.383456  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:51.431773 19641 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:51.433805 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:51.379586 19696 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:51.435274 19696 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:51.436484  9634 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:53:51.421576 19574 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:51.400125  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:51.311100 19620 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.437755  9477 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:51.353418 19592 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:51.440116 19592 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:51.352244 19685 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.351850 19609 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:51.408064  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:51.442869  9629 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Advancing to term 32
I20260808 01:53:51.443086  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:51.413333  9473 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97" is_pre_election: true
I20260808 01:53:51.414853 19668 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:51.444443  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:51.445072  9552 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:51.416627 19604 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:51.333338 19738 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.204s
I20260808 01:53:51.418568 19578 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:51.447736 19604 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:51.448294 19604 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:51.450089 19578 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:51.424561  9633 raft_consensus.cc:2410] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260808 01:53:51.389598 19735 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.425967 19607 consensus_queue.cc:260] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 25.20, Last appended by leader: 20, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:51.453363 19735 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:51.456982  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:51.458050  9558 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:51.435282  9638 raft_consensus.cc:1275] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260808 01:53:51.467712  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:51.470144  9631 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 23. Current term is 24. Ops: [23.12-23.12]
I20260808 01:53:51.471458  9478 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 33.
I20260808 01:53:51.471676  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:51.472532  9638 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 31.
I20260808 01:53:51.484472  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:51.487232  9367 catalog_manager.cc:5697] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 23 to 25, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 25 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:51.491107  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:51.491667 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:51.496344  9638 raft_consensus.cc:2393] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 29: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:53:51.497300  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:51.472754 19724 raft_consensus.cc:697] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:51.481458 19735 raft_consensus.cc:2764] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 33 FOLLOWER]: Leader pre-election decision vote started in defunct term 31: won
I20260808 01:53:51.501824  9367 catalog_manager.cc:5697] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 6 to 22. New cstate: current_term: 22 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:51.507761  9552 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:51.500697  9629 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 32.
I20260808 01:53:51.505084  9524 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:51.477689 19716 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.027s
I20260808 01:53:51.513710  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:51.516911  9638 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
I20260808 01:53:51.527429  9367 catalog_manager.cc:5697] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 19 to 22, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 22 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 01:53:51.529989 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 142275us
I20260808 01:53:51.491935 19203 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 28.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260808 01:53:51.498862 19475 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.025s
I20260808 01:53:51.494807 19644 consensus_queue.cc:1059] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.024s
I20260808 01:53:51.510821 19722 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:51.498704 19611 consensus_queue.cc:1059] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.024s
I20260808 01:53:51.534972  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:51.502642 19628 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.023s
I20260808 01:53:51.510807 19726 consensus_queue.cc:1059] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.022s
I20260808 01:53:51.525909 19626 raft_consensus.cc:2804] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Leader election won for term 33
I20260808 01:53:51.538637  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:51.539499  9367 catalog_manager.cc:5697] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 19 to 22, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 22 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:51.542469  9635 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 21: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:51.543856  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:51.554141 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:51.555047  9476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6e7856bed1be43acaa821290d506ee97"
I20260808 01:53:51.555718  9476 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:51.556825 19732 consensus_queue.cc:1059] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:51.557135 19616 consensus_queue.cc:1059] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260808 01:53:51.557834 19616 consensus_queue.cc:1243] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 log is divergent from this leader: its last log entry 26.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260808 01:53:51.558347 19616 raft_consensus.cc:3055] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:51.558765 19616 raft_consensus.cc:740] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:51.559118 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:51.559324 19203 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Stepping down as leader of term 28
I20260808 01:53:51.559804 19203 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:51.559715 19616 consensus_queue.cc:260] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.560637 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:51.560750 19669 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.561215 19203 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.563512 19616 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:51.562529  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:51.565094  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:51.565382 19729 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260808 01:53:51.567442 19574 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:51.568254  9560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 4542) took 1163 ms. Trace:
I20260808 01:53:51.568133 19574 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:51.568740  9560 rpcz_store.cc:276] 0808 01:53:50.404577 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:50.404878 (+   301us) service_pool.cc:224] Handling call
0808 01:53:50.405319 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:50.964700 (+559381us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:51.568025 (+603325us) spinlock_profiling.cc:243] Waited 559 ms on lock 0x7b540015b154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:51.568192 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1284916480}
I20260808 01:53:51.569507 19305 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:51.570025 19305 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:51.575307 19726 raft_consensus.cc:3055] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:51.575898 19726 raft_consensus.cc:740] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:51.576190 19589 consensus_queue.cc:260] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 19.17, Last appended by leader: 17, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:51.576903 19726 consensus_queue.cc:260] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.578336 19726 raft_consensus.cc:3060] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:51.558475 19638 raft_consensus.cc:3055] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 29 LEADER]: Stepping down as leader of term 29
I20260808 01:53:51.587257 19638 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:51.588109 19698 raft_consensus.cc:3055] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:51.588668 19698 raft_consensus.cc:740] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:51.588351 19638 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:51.546729 19750 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260808 01:53:51.590106 19638 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:51.546108 19626 raft_consensus.cc:697] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:51.582235 19735 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:51.583129 19203 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:51.544811 19724 consensus_queue.cc:237] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.565270 19254 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:51.534511 19745 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.017s
I20260808 01:53:51.591806  7831 tablet_server.cc:179] TabletServer@127.7.165.193:0 shutting down...
W20260808 01:53:51.598258  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59685.8us
I20260808 01:53:51.589738 19698 consensus_queue.cc:260] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:51.635253 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:51.635891 19698 raft_consensus.cc:3060] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:51.637436 19604 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.639146  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:51.642755 19618 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 33 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:51.643586 19696 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:51.599395  9367 catalog_manager.cc:5697] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 11 to 20, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 20 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:51.544941  9638 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 28.
I20260808 01:53:51.599445  9599 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:53:51.646301  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 46926.1us
I20260808 01:53:51.646983  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:51.648320  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:51.541640 19738 raft_consensus.cc:3055] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Stepping down as leader of term 26
W20260808 01:53:51.591476  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:51.588547 19752 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
W20260808 01:53:51.616070  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:51.613902 19363 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:51.617063 19739 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:51.618131 19669 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:51.544034 19757 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.078s
I20260808 01:53:51.624486 19305 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:51.626264  9632 raft_consensus.cc:2410] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 27 because replica has last-logged OpId of term: 26 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 13.
I20260808 01:53:51.543697 19722 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.671132  9599 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:51.612520 19693 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:51.610755 19749 log.cc:927] Time spent T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260808 01:53:51.650705 19738 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:51.646324  9446 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:51.667299  9480 raft_consensus.cc:1240] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 23. Current term is 24. Ops: []
W20260808 01:53:51.612524 19478 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260808 01:53:51.669984  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:51.671056  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:51.675486  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:51.691172  9560 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 30: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:53:51.650151 19753 log.cc:927] Time spent T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.116s	user 0.003s	sys 0.000s
I20260808 01:53:51.681016  9557 raft_consensus.cc:1240] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:51.694373  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:51.682279  9556 raft_consensus.cc:1240] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 25. Current term is 26. Ops: []
I20260808 01:53:51.684424  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:51.608628 19589 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:51.693354  9638 raft_consensus.cc:2410] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 30 because replica has last-logged OpId of term: 29 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 15.
I20260808 01:53:51.695977 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:51.696651  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:51.699093  9634 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28. Ops: [27.15-27.15]
I20260808 01:53:51.650296 19552 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:51.663872 19363 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:51.664407 19254 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:51.720312  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:51.725330  9551 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:51.709018  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:51.709702  9444 leader_election.cc:400] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 31. Current term is 32.
I20260808 01:53:51.727756  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 32 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:51.702502  9367 catalog_manager.cc:5697] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 11 to 24, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 24 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:51.732379  9559 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 32: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:51.732522  9554 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 25: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:51.653961 19618 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:51.644229 19626 consensus_queue.cc:237] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.667218 19739 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:51.736733  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:51.663616 19574 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:51.677654 19716 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260808 01:53:51.736327  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:51.739708  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:51.745720  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:51.692399 19578 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:51.691956 19554 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:51.743723  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:51.750155  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:51.752312 10367 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:51.749163  9554 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:51.755213  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:51.720723 19735 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:51.760340  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:51.711721 19552 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:51.769366  9552 raft_consensus.cc:2393] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 28: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:51.768406  9558 raft_consensus.cc:2468] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 28.
I20260808 01:53:51.773442  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:51.769227  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:51.774371  9554 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:51.775100  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:51.767695  9367 catalog_manager.cc:5697] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 14 to 21, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 21 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:51.708514 19539 consensus_queue.cc:1059] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.052s
I20260808 01:53:51.747922 19611 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:51.763497 19442 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.032s
I20260808 01:53:51.764065 19554 raft_consensus.cc:697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:51.759650 19738 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.785084 19716 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260808 01:53:51.750501 19618 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:51.803892  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 126749us
I20260808 01:53:51.824669  9476 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 30.
W20260808 01:53:51.806928  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:51.788241  9559 raft_consensus.cc:1217] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Deduplicated request from leader. Original: 27.15->[29.16-31.17]   Dedup: 29.16->[31.17-31.17]
I20260808 01:53:51.740386 19254 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:51.751659 19578 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:51.906540  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:51.855393  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:51.923322  9551 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 28.
I20260808 01:53:51.918430  9556 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:51.896461  9632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42840 (request call id 3601) took 1065 ms. Trace:
I20260808 01:53:51.779541 19552 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:51.937070  9559 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 31. Current term is 32. Ops: [29.16-31.17]
I20260808 01:53:51.936064  9632 rpcz_store.cc:276] 0808 01:53:50.831049 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:50.831217 (+   168us) service_pool.cc:224] Handling call
0808 01:53:51.896115 (+1064898us) spinlock_profiling.cc:243] Waited 756 ms on lock 0x7b540008d254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:51.896368 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1739254528}
I20260808 01:53:51.947729  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:51.948786  9550 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:51.951826  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:51.896330 19478 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260808 01:53:51.780660 19644 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260808 01:53:51.979741  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 204801us
I20260808 01:53:51.795909  9367 catalog_manager.cc:5697] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 15 to 18, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 18 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 01:53:51.965296  9600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:51.825600 19606 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:51.882918 19693 raft_consensus.cc:3055] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:51.966809  9524 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:51.790187 19696 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:51.892982 19611 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:51.826040 19475 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:51.986258  9367 catalog_manager.cc:5697] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 21 to 24, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 24 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:51.864737 19684 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:51.984964 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 79844.5us
I20260808 01:53:51.998524 19442 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:51.976617 19722 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:51.981438 19488 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:51.998605 19693 raft_consensus.cc:740] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:52.040740  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:52.041262  9550 raft_consensus.cc:2468] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 27.
I20260808 01:53:52.018994  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:52.020303  9636 raft_consensus.cc:2410] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 26 because replica has last-logged OpId of term: 25 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 18.
I20260808 01:53:52.021695  9367 catalog_manager.cc:5697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 19 to 23. New cstate: current_term: 23 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:52.007323  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:52.042002  9551 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:51.771890 19735 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.038270  9599 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:52.052717  9554 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 25.
W20260808 01:53:52.059592 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:52.045763 10332 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:52.062397 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:52.008026 19604 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:52.064127 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:52.065001 10369 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:53:52.010135 19442 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.012858 19488 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:52.067992 19442 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:52.029191 19620 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:52.068710 19757 consensus_queue.cc:1059] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.012s
I20260808 01:53:52.068897 19488 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 26.22, Last appended by leader: 22, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:52.016788 19693 consensus_queue.cc:260] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.071163 19693 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:52.071363  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:52.073153  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:52.075810  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:52.076684  9634 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 21: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:52.038532 19574 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election won for term 30
I20260808 01:53:52.077867 19698 raft_consensus.cc:2804] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election won for term 28
W20260808 01:53:52.078255 19756 log.cc:927] Time spent T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.323s	user 0.000s	sys 0.001s
I20260808 01:53:52.078840 19698 raft_consensus.cc:697] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:52.079957  9632 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
W20260808 01:53:52.081067 19760 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260808 01:53:52.080476 19698 consensus_queue.cc:237] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.066512 19363 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:52.082439 19363 raft_consensus.cc:2804] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:52.017417 19738 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:52.084668 19746 raft_consensus.cc:3055] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Stepping down as leader of term 27
I20260808 01:53:52.085330 19746 raft_consensus.cc:740] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:52.017671 19553 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:52.058450  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:52.089780  9600 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:52.099480 19735 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:52.099334  9367 catalog_manager.cc:5697] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 22 to 24, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 24 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:52.101116  9367 catalog_manager.cc:5697] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 20 to 22, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 22 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:53:52.036933  9507 heartbeater.cc:646] Failed to heartbeat to 127.7.165.254:43565 (6 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.7.165.254:43565 timed out after 15.000s (SENT)
I20260808 01:53:52.036345 19724 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:52.078505 19574 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:53:52.104203 19758 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260808 01:53:52.082973 19363 raft_consensus.cc:697] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:52.105840 19539 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:52.003855 19606 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.109541 19620 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:52.118233 19674 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260808 01:53:52.146068  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:52.129581  9367 catalog_manager.cc:5697] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 17 to 23. New cstate: current_term: 23 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:52.088099 19684 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:52.016955 19475 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:52.159468  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:52.138495 19638 consensus_meta.cc:287] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.527s	user 0.012s	sys 0.001s
I20260808 01:53:52.134091 19539 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:52.095983 19746 consensus_queue.cc:260] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.146827 19620 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:52.112049 19305 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.039s
I20260808 01:53:52.123495 19724 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.153990 19674 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:52.149825 19604 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:52.168725 19757 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:52.175562 19539 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:52.192636  9367 catalog_manager.cc:5697] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 16 to 23. New cstate: current_term: 23 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:53:52.197671  9523 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:52.181375 19620 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:52.150032 19554 consensus_queue.cc:237] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.187470 19674 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Advancing to term 26
W20260808 01:53:52.212033  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:52.214145  9367 catalog_manager.cc:5697] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 20 to 22, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 22 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:53:52.188616 19604 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.232533  9367 catalog_manager.cc:5697] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 20 to 21, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 21 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:53:52.238648  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:52.241896  9367 catalog_manager.cc:5697] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 9 to 21, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 21 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:53:52.235100 19669 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.567s	user 0.015s	sys 0.002s
W20260808 01:53:52.246420  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 150721us
I20260808 01:53:52.224458 19254 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:52.209702 19757 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.196038 19574 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.14, Last appended by leader: 14, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.262048  9478 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260808 01:53:52.198315 19363 consensus_queue.cc:237] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:52.252696 19604 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:52.253587 19685 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:52.254900 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:52.263662 19607 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:52.241576 19735 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:52.283381  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 187358us
I20260808 01:53:52.288662  9599 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:52.280194 19442 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:52.295486 19442 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:52.323853  9630 raft_consensus.cc:1275] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 19. (term mismatch)
I20260808 01:53:52.324512  9630 pending_rounds.cc:85] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 18
I20260808 01:53:52.324801  9630 pending_rounds.cc:107] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 25.19
I20260808 01:53:52.325121  9630 raft_consensus.cc:2889] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:53:52.331905  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 225708us
W20260808 01:53:52.332569 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 45352.7us
I20260808 01:53:52.341492 10332 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:52.266633 19757 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:52.353883 19772 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260808 01:53:52.333824 19739 consensus_meta.cc:287] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.542s	user 0.009s	sys 0.006s
W20260808 01:53:52.343185 19478 log.cc:927] Time spent T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260808 01:53:52.382920  9524 consensus_peers.cc:597] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:53:52.416137  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 66576.8us
W20260808 01:53:52.417380  9524 consensus_peers.cc:597] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:53:52.418195  9524 consensus_peers.cc:597] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:53:52.383773  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 402119us
I20260808 01:53:52.419497 19620 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.424013 19757 consensus_queue.cc:1059] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260808 01:53:52.425693 19669 raft_consensus.cc:3055] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Stepping down as leader of term 25
I20260808 01:53:52.426196 19669 raft_consensus.cc:740] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:52.427448  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:52.427666 19669 consensus_queue.cc:260] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.289856 19735 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.353722 19631 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:52.429167 19669 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 26
W20260808 01:53:52.432974  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:52.344498 19727 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260808 01:53:52.355655 19751 log.cc:927] Time spent T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.867s	user 0.000s	sys 0.008s
I20260808 01:53:52.346830 19606 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:52.406527 19589 consensus_meta.cc:287] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.655s	user 0.015s	sys 0.000s
I20260808 01:53:52.471060  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:52.406858 19739 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:53:52.430284 19620 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:52.488807  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:52.457150 19631 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:53:52.489975  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 292373us
I20260808 01:53:52.500857 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:52.489289  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 195770us
W20260808 01:53:52.504953  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:52.505720  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:52.506529  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:52.501857 10367 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:53:52.387264 19475 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:52.511477  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:52.489385 19724 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:52.519516  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:52.531131  9551 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:52.522751  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:52.515033  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 403956us
W20260808 01:53:52.526049  9601 connection.cc:256] RPC call timeout handler was delayed by -1.2664963770075701s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1663s for a response
I20260808 01:53:52.528816  9523 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:52.509387 19631 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:52.491472 19757 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:52.508224 19770 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:53:52.489804 19771 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260808 01:53:52.547189  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:52.541225  9599 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:52.555305 19475 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:52.557231  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:52.562649  9554 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:52.564016  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:52.559157  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:52.534472 19764 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.327s	user 0.001s	sys 0.001s
I20260808 01:53:52.567565  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:52.567569  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:52.568843 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:52.573212  9633 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260808 01:53:52.573732 10367 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Advancing to term 28
I20260808 01:53:52.534179 19775 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:52.538162 19620 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election won for term 28
I20260808 01:53:52.584048  9551 raft_consensus.cc:1240] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28. Ops: [27.15-27.15]
W20260808 01:53:52.594561  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:52.585870  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:52.592072  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:52.596259  9632 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 24: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:52.543084 19609 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:52.552644 19757 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:52.597090  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:52.600011  9557 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260808 01:53:52.597553  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:52.562212 19475 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:52.612771  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:52.570453 19668 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:52.615517  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:52.617725  9632 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 30.
I20260808 01:53:52.583087 19772 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:52.598939 19779 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:52.570621 19203 consensus_queue.cc:1059] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.047s
I20260808 01:53:52.614778 19475 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:52.600497 19770 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:52.605973 19757 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:52.604398  9633 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 28. Ops: []
I20260808 01:53:52.631268  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:52.621910 19363 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:52.599987 19775 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:52.655066 19616 consensus_meta.cc:287] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.084s	user 0.013s	sys 0.003s
I20260808 01:53:52.659371 19770 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.664743 19774 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:52.630095 19668 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:52.628953 19620 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:53:52.643008  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 57472.5us
I20260808 01:53:52.644033 19779 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:52.676633 19363 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:52.677829 19442 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:52.698968  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:52.703806 19783 consensus_queue.cc:1059] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:53:52.705147 19783 raft_consensus.cc:3055] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Stepping down as leader of term 28
I20260808 01:53:52.705538 19783 raft_consensus.cc:740] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:53:52.707612 19768 log.cc:927] Time spent T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
W20260808 01:53:52.709740 19726 consensus_meta.cc:287] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.101s	user 0.012s	sys 0.001s
I20260808 01:53:52.684345 19203 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:52.713477  9521 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:52.683401 19774 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:52.642015 19772 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:52.691351 19735 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:52.762212  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 221066us
W20260808 01:53:52.771059  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:52.722955  9521 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:52.752401  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 212350us
W20260808 01:53:52.723963 19539 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.515s	user 0.014s	sys 0.000s
W20260808 01:53:52.774698 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 205791us
I20260808 01:53:52.716257 19363 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:52.776413  9601 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:52.706516 19783 consensus_queue.cc:260] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.719399 19779 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:52.732337 19774 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
W20260808 01:53:52.775731  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 52907.6us
I20260808 01:53:52.794662  9635 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 26.
I20260808 01:53:52.771785 19735 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:52.795768  9601 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:52.774107 19607 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:52.772950 19781 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:52.785269 19589 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260808 01:53:52.727200 19254 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:52.794193 19554 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260808 01:53:52.806228 19779 consensus_queue.cc:1048] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.004s
I20260808 01:53:52.811126 19770 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:52.811467 19784 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260808 01:53:52.811810 19770 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:52.812441 19770 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.813199  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:52.814602 19616 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:52.730656 19203 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:52.815860 19738 consensus_meta.cc:287] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.733s	user 0.016s	sys 0.001s
I20260808 01:53:52.816519 19738 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:52.816934 19203 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:52.817061 19738 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:52.817610 19203 consensus_queue.cc:1059] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260808 01:53:52.820770 19684 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:52.797089 19783 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:52.828943  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:52.830210  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:52.830734  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:52.831437  9634 raft_consensus.cc:2393] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 28: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:52.830139  9638 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 30 index: 15. (index mismatch)
I20260808 01:53:52.833760  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:52.834106  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:52.831516  9633 raft_consensus.cc:2368] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 27. Current term is 28.
I20260808 01:53:52.834800  9637 raft_consensus.cc:3060] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Advancing to term 33
W20260808 01:53:52.822499 19793 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
I20260808 01:53:52.839908  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:52.842996  9629 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 25.
I20260808 01:53:52.843367  9552 raft_consensus.cc:1275] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260808 01:53:52.844586  9521 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:52.849004  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:52.849895  9629 raft_consensus.cc:2376] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21. Re-sending same reply.
I20260808 01:53:52.849960  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:52.851480  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:52.852591  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:52.852918 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:52.853729  9632 raft_consensus.cc:2410] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 30 because replica has last-logged OpId of term: 29 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260808 01:53:52.788268 19539 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:52.852275  9629 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
I20260808 01:53:52.855669 10367 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 28.
I20260808 01:53:52.857527  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:52.860186 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:52.860711  9638 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
W20260808 01:53:52.862146  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:52.867820 10332 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:52.871970  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:52.832140 19726 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:52.805403 19735 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:52.872938  9634 raft_consensus.cc:2393] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 25: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:52.873528  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:52.876804  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:52.879043  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:52.832495 19770 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:52.891206 19770 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:52.891387 19780 log.cc:927] Time spent T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260808 01:53:52.891712 19770 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.892486 19794 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:52.808188 19607 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.894917 19809 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:52.895788 19631 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:52.894124 19770 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:52.900689 19770 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:52.901964 19770 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.880726 19442 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:52.903116 19442 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:52.904932 19770 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:52.884423  9521 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:52.852464 19779 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260808 01:53:52.907043 19779 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:52.907271  9521 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:52.907547 19779 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Advancing to term 32
I20260808 01:53:52.881388 19647 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260808 01:53:52.867842  9551 raft_consensus.cc:1275] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260808 01:53:52.889081 19305 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:52.864233 19795 consensus_queue.cc:1048] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.047s
I20260808 01:53:52.910498 19305 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:52.911137  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:52.913982 19305 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:52.896160 19607 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:52.840523 19778 consensus_queue.cc:1048] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.040s
I20260808 01:53:52.888260  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:52.916759 19735 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:52.897672 19668 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:52.888358 19772 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:52.918744 19798 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
W20260808 01:53:52.852674 19797 log.cc:927] Time spent T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260808 01:53:52.786478 19620 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:52.900910 19693 consensus_meta.cc:287] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.829s	user 0.006s	sys 0.007s
I20260808 01:53:52.901294  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:52.905447 19442 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:52.905575 19809 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260808 01:53:52.897327  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:52.897339 19552 consensus_meta.cc:287] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 0.930s	user 0.011s	sys 0.001s
I20260808 01:53:52.898185 19631 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:52.926054  9637 raft_consensus.cc:2468] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 33.
I20260808 01:53:52.927925  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:52.865051 19684 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260808 01:53:52.874631 19584 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:52.898294 19799 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260808 01:53:52.872012 19801 raft_consensus.cc:3055] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Stepping down as leader of term 27
I20260808 01:53:52.943691 19801 raft_consensus.cc:740] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:52.943367 19782 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:52.944847 19782 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:52.945369 19782 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:52.945180 19801 consensus_queue.cc:260] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.950440 19799 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260808 01:53:52.951159  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:52.951272 19799 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:52.951750 19799 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:52.955862  9521 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:52.957737 19776 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:52.960875 19799 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:52.967198 19578 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:52.968185  9521 leader_election.cc:400] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 27. Current term is 28.
I20260808 01:53:52.968896  9521 leader_election.cc:403] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:53:52.970042  9521 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:52.938225 19552 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.972446  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45658 (request call id 1271) took 1280 ms. Trace:
I20260808 01:53:52.972903  9635 rpcz_store.cc:276] 0808 01:53:51.692448 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:51.694098 (+  1650us) service_pool.cc:224] Handling call
0808 01:53:52.972240 (+1278142us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54003faa54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:52.972369 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2452724096}
I20260808 01:53:52.973294 19552 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:52.973582 19765 raft_consensus.cc:3055] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Stepping down as leader of term 20
I20260808 01:53:52.929775 19305 consensus_queue.cc:1059] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.010s
I20260808 01:53:52.974479 19305 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 28
I20260808 01:53:52.974999 19589 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260808 01:53:52.975486 19589 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:52.975881 19589 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:52.972867  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:52.976622  9636 raft_consensus.cc:2368] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24.
I20260808 01:53:52.939883 19775 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:52.808009 19781 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:52.922207 19726 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:52.980777  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 79576.3us
I20260808 01:53:52.970716 19799 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20260808 01:53:52.977404 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 92017.2us
I20260808 01:53:52.933310 19685 consensus_queue.cc:1048] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.004s
I20260808 01:53:52.974062 19765 raft_consensus.cc:740] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:52.991123 10332 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:52.941984  9478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50124 (request call id 4125) took 1368 ms. Trace:
W20260808 01:53:52.982442 10334 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:52.995745 10363 raft_consensus.cc:1240] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 LEADER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27. Ops: []
I20260808 01:53:52.996479  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:52.996868  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:52.982887 19746 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:52.983660 19611 raft_consensus.cc:3055] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Stepping down as leader of term 26
I20260808 01:53:52.984342 19723 raft_consensus.cc:3055] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Stepping down as leader of term 23
I20260808 01:53:52.984532 19781 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:52.986728 19726 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:52.990558 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:52.997294  9552 raft_consensus.cc:2368] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26.
I20260808 01:53:52.982952 19738 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:52.998090  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:53.014626  9557 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:52.995745  9478 rpcz_store.cc:276] 0808 01:53:51.573438 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:51.573622 (+   184us) service_pool.cc:224] Handling call
0808 01:53:51.574017 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:52.262637 (+688620us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:52.941688 (+679051us) spinlock_profiling.cc:243] Waited 687 ms on lock 0x7b54002bb6d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:52.941911 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1581182848}
I20260808 01:53:53.002856 19775 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:53.005960  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:53.007246  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:53.009452 19684 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:53.008286  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:53.000492 10334 consensus_peers.cc:597] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:53.011353  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:53.015647  9524 consensus_peers.cc:597] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:53.016809  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:53.018662  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:52.988725 19801 raft_consensus.cc:3060] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
W20260808 01:53:52.994323 19674 consensus_meta.cc:287] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.764s	user 0.012s	sys 0.003s
I20260808 01:53:53.021234  9637 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 21: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:53.005728 19254 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:52.994226 19809 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260808 01:53:52.984532 19539 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:53.031034 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:53.044134  9634 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
W20260808 01:53:53.044946 10334 consensus_peers.cc:597] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:53:53.056414 10334 consensus_peers.cc:597] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:53.005164 19611 raft_consensus.cc:740] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:53.004017 19778 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:53.053670  9557 raft_consensus.cc:2468] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 24.
I20260808 01:53:53.045082  9630 raft_consensus.cc:2410] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 30 because replica has last-logged OpId of term: 29 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 15.
I20260808 01:53:53.061678  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.062178  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.010851 19723 raft_consensus.cc:740] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:53.064047  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.068078  9560 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 32.
I20260808 01:53:53.064177 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:53.014297 19730 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.055s
I20260808 01:53:53.070245 10367 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
W20260808 01:53:53.084132  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:53.042484 19552 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:53.085345  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:53.045629 19726 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260808 01:53:53.089403 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:53.089968  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.090373  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:53.012923 19363 consensus_queue.cc:1048] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.060s
I20260808 01:53:53.093326  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:53.086265  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.096160  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:53.097425  9638 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:53:53.055032 19799 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:53.020437 19771 consensus_queue.cc:1059] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.060s
I20260808 01:53:53.099215 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:53.138129  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.141422  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.143530  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.082511 19684 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:53.099403  9554 raft_consensus.cc:2393] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 26: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:53.082000 19442 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:53.111277  9634 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:53.150108  9550 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 21: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:53.112027 19809 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:53.115031 19723 consensus_queue.cc:260] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.115018 19617 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.002s
I20260808 01:53:53.047891 19674 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:53.100320 19552 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:53.119581 19757 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.101012 19811 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:53.121527 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:53.103677 19770 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260808 01:53:53.196134  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 102906us
I20260808 01:53:53.125662  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.196843 10369 raft_consensus.cc:2393] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 20: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:53.197280  9558 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 20.
I20260808 01:53:53.197396  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:53.126027 19716 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:53.100615 19735 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:53.126644  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.092046 19802 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:53.129140 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:53.132016  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 21 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.201342  9556 raft_consensus.cc:2410] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 21 because replica has last-logged OpId of term: 19 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 16.
I20260808 01:53:53.123927 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:53.041597 19775 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.157117 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:53.159262  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:53.088900 19668 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:53.123982 19669 consensus_meta.cc:287] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.652s	user 0.008s	sys 0.006s
I20260808 01:53:53.092581 19726 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:53.117147 19475 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:53.168789 19723 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:53.170825 19442 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:53.122539 19774 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.147367 19799 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:53.084326 19254 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:53.185917 19798 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:53.124673 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:53.217531 10365 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 26: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:53.125847  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.217566  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:53.202741  9557 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:53.203755 10367 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 20.
W20260808 01:53:53.207949  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 48969.5us
I20260808 01:53:53.198611 19620 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:53.161602 19684 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:53.215099 19770 consensus_queue.cc:1059] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:53.222227  9521 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:53.203883 19735 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:53.157902 19611 consensus_queue.cc:260] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:53.206956 19809 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:53.200629 19781 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:53.233357 19589 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:53.238305  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 116069us
I20260808 01:53:53.191988 19782 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.245360  9599 leader_election.cc:400] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26.
I20260808 01:53:53.262917  9599 leader_election.cc:403] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260808 01:53:53.247900  9633 raft_consensus.cc:2368] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 27. Current term is 28.
I20260808 01:53:53.213119 19726 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:53:53.252578 19674 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:53.252815 19254 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:53.233412 19488 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:53.239316 19620 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.254781 19757 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:53.282298  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:53.283411  9636 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 26: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:53.246495 19771 consensus_queue.cc:1059] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.009s
I20260808 01:53:53.287032 19552 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:53.198463 19716 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.210573 19817 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260808 01:53:53.224116 19776 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:53.226414 19669 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260808 01:53:53.264947  9445 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:53.242511 19574 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:53.244385 19735 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:53.257793 10363 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
I20260808 01:53:53.201463 19811 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.272544  9599 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:53.258653 19779 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:53.255549 19798 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:53.297854  9559 raft_consensus.cc:2368] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 25. Current term is 26.
I20260808 01:53:53.278292 19618 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:53.301368 10334 consensus_peers.cc:597] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:53.206312 19802 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:53.323101  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 58262.9us
W20260808 01:53:53.320667 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 270905us
I20260808 01:53:53.222651 19668 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.350878  9445 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:53.265223 19818 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:53.357421  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:53.358103  9636 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:53.309456 19771 consensus_queue.cc:1243] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Peer 7201605bb2f54c949a4ebe6e9b3b4b2a log is divergent from this leader: its last log entry 27.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260808 01:53:53.269941 19606 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26.
I20260808 01:53:53.300519 19475 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:53.309934 19770 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:53.352286  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 79876us
I20260808 01:53:53.356930 19674 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:53.262661 19781 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.322542 19669 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:53.296527 19589 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:53.304774 19757 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:53.290477 19620 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:53.300241 19799 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:53.348308 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 291141us
I20260808 01:53:53.304623 19809 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:53.343614 19574 raft_consensus.cc:697] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:53.301666 19782 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:53.350049 19815 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.137s	user 0.002s	sys 0.000s
I20260808 01:53:53.329607 19604 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:53.325115 19305 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 27. Current term is 28.
I20260808 01:53:53.309281 19775 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:53.296993 19774 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:53.372020 10332 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:53.355724 19794 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:53.438756  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:53.439594  9554 raft_consensus.cc:3060] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 28
I20260808 01:53:53.363936 19644 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:53.363714 19732 raft_consensus.cc:2804] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:53:53.351015 19798 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:53.403306 10334 consensus_peers.cc:597] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:53:53.450824 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 51362.5us
I20260808 01:53:53.383227 19669 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 32 FOLLOWER]: Advancing to term 33
I20260808 01:53:53.413707  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.434052  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.396852 19819 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:53.396996 19442 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:53.367046 19818 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:53.444530  9551 raft_consensus.cc:1275] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260808 01:53:53.388330 19674 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:53.383673 19770 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:53.454080 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 82196.5us
W20260808 01:53:53.457176 10334 consensus_peers.cc:597] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:53.508998  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:53.394541 19757 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.463989  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:53.512079  9638 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:53.477685 19674 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:53.479934 19798 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:53.482098  9636 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:53.487246 19811 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:53.466962 19363 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260808 01:53:53.489780  9601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:53.468336 19732 raft_consensus.cc:697] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:53.443429 19644 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:53:53.495688 19821 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260808 01:53:53.499017 19478 tablet_bootstrap.cc:492] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=2 applied=16 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260808 01:53:53.499792  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:53.504750  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:53.505528 19684 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:53.506944 19574 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:53.511087 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 59724.1us
I20260808 01:53:53.458128  9558 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
W20260808 01:53:53.523249  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:53.525993  9636 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 30 index: 15. (term mismatch)
I20260808 01:53:53.539527  9636 pending_rounds.cc:85] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 14
I20260808 01:53:53.539841  9636 pending_rounds.cc:107] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 29.15
I20260808 01:53:53.539917  9631 raft_consensus.cc:2393] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 26: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:53.540171  9636 raft_consensus.cc:2889] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:53.527273 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:53.528004  9601 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 39 similar messages]
I20260808 01:53:53.528882  9521 leader_election.cc:400] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 27. Current term is 28.
I20260808 01:53:53.529140 10368 raft_consensus.cc:2410] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
W20260808 01:53:53.513559 19828 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260808 01:53:53.408761 19809 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.553117 19809 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:53.532694  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:53.471345 19819 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.558224 19716 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:53.541153 10334 leader_election.cc:336] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:53.548453 19757 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:53.549410  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:53.542785  9521 leader_election.cc:403] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:53:53.532720 19644 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:53.562139 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:53.563282 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:53.576051  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:53.565750  9524 consensus_peers.cc:597] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:53.570101  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.578877  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.573122  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:53.581597  9524 leader_election.cc:336] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:53:53.582497  9524 leader_election.cc:336] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:53.587710 19716 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260808 01:53:53.588064 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:53.588886 10366 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Advancing to term 26
I20260808 01:53:53.590517 19611 raft_consensus.cc:3060] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:53.591859  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.593499  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:53.594151  9560 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:53.596032 19770 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:53.596908 19478 tablet_bootstrap.cc:492] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:53:53.597910 19478 ts_tablet_manager.cc:1403] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 3.981s	user 0.080s	sys 0.030s
I20260808 01:53:53.599083 10367 raft_consensus.cc:2393] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 26: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:53.581663  9558 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 27: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:53.602478  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.604226  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.605050  9550 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 27.
I20260808 01:53:53.608407  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:53.581889 10334 consensus_peers.cc:597] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:53.611600  9550 raft_consensus.cc:1240] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:53.612895  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:53.615170  9558 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260808 01:53:53.618321  9556 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 31. Current term is 32. Ops: []
W20260808 01:53:53.564656 19746 consensus_meta.cc:287] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.554s	user 0.015s	sys 0.000s
W20260808 01:53:53.619297  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 93024.1us
I20260808 01:53:53.619860  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:53.620261  9521 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:53.623231 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:53.624365 10367 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:53.635222  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.636618 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:53.636863  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.623811 19305 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:53.639714 19475 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:53:53.639514 19305 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.634675  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.643442  9599 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:53.649091  9523 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:53.650750  9560 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 29.
I20260808 01:53:53.579015 19779 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:53.600924 19716 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:53.658468 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:53.651274 19785 consensus_queue.cc:1048] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.004s
I20260808 01:53:53.615725 19757 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:53.607952 19668 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:53.668401  9638 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
W20260808 01:53:53.670634  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:53.670030  9523 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:53.676832 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:53.674086 19818 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:53.679270 19254 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260808 01:53:53.679760 19254 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:53.661993 19256 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:53.680163 19254 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:53.680487 19256 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:53.680882 19256 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:53.688737  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:53.689323 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:53.648172 19802 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:53.667687 19777 log.cc:927] Time spent T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.145s	user 0.001s	sys 0.001s
I20260808 01:53:53.692785 10366 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 26.
I20260808 01:53:53.655972 19475 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:53.692797 10332 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:53.656863 19746 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:53.695463  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 167486us
I20260808 01:53:53.717434  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.719147 10369 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
I20260808 01:53:53.663301 19716 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Advancing to term 20
I20260808 01:53:53.702929 19745 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:53.703629 19824 consensus_queue.cc:1059] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260808 01:53:53.722745 10367 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
W20260808 01:53:53.723892 19828 log.cc:927] Time spent T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260808 01:53:53.691015 19817 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:53:53.692996 19305 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:53.752547  9523 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:53.743902  9444 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:53.727679 19442 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:53.730939 19746 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.656211 19732 consensus_queue.cc:237] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:53.781160 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 170629us
I20260808 01:53:53.719580 19781 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:53.687095 19757 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:53.718565 19802 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:53:53.771281 19819 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:53.745121 19818 consensus_queue.cc:1059] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.040s
I20260808 01:53:53.768705 19475 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:53.782169  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.782742  9523 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:53.750036 19745 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:53.796818 19745 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:53.797082 19618 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:53.797637 19618 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:53.798130 19618 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:53.798273 19757 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:53.807161 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:53.808096 10363 raft_consensus.cc:2468] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:53.809867  9445 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:53.753947 19809 consensus_queue.cc:1059] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.058s
I20260808 01:53:53.813227  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:53.809729 19618 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:53.814111  9632 raft_consensus.cc:2376] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 32. Re-sending same reply.
I20260808 01:53:53.818531 19746 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:53.822751  9552 raft_consensus.cc:1275] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260808 01:53:53.823132  9554 raft_consensus.cc:2468] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 28.
I20260808 01:53:53.825901  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.827070  9554 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:53.819017 19618 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:53.829608 10334 consensus_peers.cc:597] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:53:53.830483 10334 leader_election.cc:336] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:53.831156 10334 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:53.833427 19809 raft_consensus.cc:2804] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:53:53.833992 19809 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:53.834117 10334 leader_election.cc:336] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:53.836239 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
W20260808 01:53:53.838215  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 53239.3us
I20260808 01:53:53.780390 19738 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election won for term 29
I20260808 01:53:53.847714 19738 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 29 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:53.788080 19305 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:53:53.854776 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:53.856247 19696 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260808 01:53:53.856657 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:53.780860 19478 raft_consensus.cc:359] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:53.835294 19801 consensus_meta.cc:287] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.780s	user 0.005s	sys 0.009s
I20260808 01:53:53.823849 19589 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 27. Current term is 28.
I20260808 01:53:53.865824  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:53.869838  9601 leader_election.cc:336] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:53.872216 19696 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:53.786484 19442 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.877767  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:53.880582 19256 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.852268 19305 raft_consensus.cc:697] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:53.875610 19628 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.008s
I20260808 01:53:53.891846  9552 raft_consensus.cc:1240] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 31. Current term is 32. Ops: []
I20260808 01:53:53.895973 19305 consensus_queue.cc:237] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:53.896926 19703 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260808 01:53:53.897564 19256 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:53.895933 19628 consensus_queue.cc:1243] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 27.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260808 01:53:53.898936 19478 raft_consensus.cc:740] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: NON_PARTICIPANT
I20260808 01:53:53.895895 19442 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:53.900520  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:53.900094 19478 consensus_queue.cc:260] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.901219  9555 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:53.905867  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:53.906955  9550 raft_consensus.cc:2468] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 26.
W20260808 01:53:53.909608 19726 consensus_meta.cc:287] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.596s	user 0.009s	sys 0.003s
I20260808 01:53:53.844183 19819 raft_consensus.cc:3055] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Stepping down as leader of term 27
I20260808 01:53:53.916108 19819 raft_consensus.cc:740] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:53.849653 19830 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260808 01:53:53.861887 19488 consensus_queue.cc:1048] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.061s
W20260808 01:53:53.921629 19833 log.cc:927] Time spent T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Append to log took a long time: real 0.238s	user 0.000s	sys 0.003s
I20260808 01:53:53.925025 19644 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:53.928982 19735 consensus_meta.cc:287] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.575s	user 0.008s	sys 0.003s
I20260808 01:53:53.929312 19716 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:53.931566 19669 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.918040 19819 consensus_queue.cc:260] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.845613 19809 raft_consensus.cc:3060] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:53.933219 19819 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:53.929499 19735 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.937536 19738 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.941655 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:53.942657 10366 raft_consensus.cc:2368] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26.
I20260808 01:53:53.944046 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:53.944278  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:53.947486 10361 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 27. Current term is 28. Ops: []
I20260808 01:53:53.949894 10362 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260808 01:53:53.951004 19478 ts_tablet_manager.cc:1434] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.353s	user 0.004s	sys 0.009s
I20260808 01:53:53.952869 19757 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260808 01:53:53.953184  9524 consensus_peers.cc:597] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:53:53.953766  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 346718us
I20260808 01:53:53.955186 19254 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:53.953084 19801 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:53.956967 19801 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.910231 19726 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.952898  9444 leader_election.cc:400] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24.
I20260808 01:53:53.937469 19817 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:53.967839 19757 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:53.963135 19611 consensus_queue.cc:1175] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "259bcb3f0d254e8b9effac3277e654e0" responder_term: 27 status { last_received { term: 27 index: 15 } last_committed_idx: 15 last_received_current_leader { term: 27 index: 15 } }
W20260808 01:53:53.974565  9524 consensus_peers.cc:597] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:53:53.977190  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 63014.2us
W20260808 01:53:53.955283 10334 consensus_peers.cc:597] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:53.979691  9521 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:53.980064 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 144317us
I20260808 01:53:53.992930 19716 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:53.993723 19716 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:53.994014 19746 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:53.994792 19746 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:53.995162 19745 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:53.997167  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:53.921710 19684 raft_consensus.cc:3055] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 LEADER]: Stepping down as leader of term 31
I20260808 01:53:53.973861 19254 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:54.002656 10363 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 28: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:53.998404 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:54.011782 10367 raft_consensus.cc:2393] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 20: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
W20260808 01:53:53.976157  9524 consensus_peers.cc:597] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:53:54.014532  9524 leader_election.cc:336] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:54.035567  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:54.022022 19746 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:54.023813  9599 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:53:54.020407 10334 leader_election.cc:336] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:53.997965 19817 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Leader election won for term 32
I20260808 01:53:53.994288 19716 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:53.999382  7831 tablet_copy_service.cc:362] P 6e7856bed1be43acaa821290d506ee97: Destroying tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-b335bc0170854e26a77dbec13dbe9eac due to service shutdown
I20260808 01:53:54.013012 19620 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:53.976817 19757 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:54.000802 19684 raft_consensus.cc:740] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:53.932178 19765 consensus_queue.cc:260] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:54.013278 19606 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:54.044651  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:54.035842  9524 leader_election.cc:336] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:54.049913  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:54.049793  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:54.021407 19626 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.227s	user 0.020s	sys 0.026s
I20260808 01:53:54.051795  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:54.059425  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 189598us
W20260808 01:53:54.052897 10334 consensus_peers.cc:597] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:54.053543  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:54.056452  9630 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 32.
I20260808 01:53:54.071830  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:54.056965  9636 raft_consensus.cc:2368] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27.
W20260808 01:53:54.053786  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:54.072273  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:54.040954 19732 raft_consensus.cc:2804] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election won for term 28
I20260808 01:53:54.077137  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:54.079732  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:54.081416  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:54.057430  7831 tablet_copy_service.cc:434] P 6e7856bed1be43acaa821290d506ee97: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-b335bc0170854e26a77dbec13dbe9eac on tablet b335bc0170854e26a77dbec13dbe9eac with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:54.083695  9636 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:54.053467 19620 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:54.082202 19746 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:54.086405 19628 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:54.086802 19746 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:54.087078 19628 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:54.066370 19626 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.272s	user 0.021s	sys 0.027s
I20260808 01:53:54.088178 19620 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:54.054857 19817 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:54.089692 19628 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:54.089996 19817 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 30.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:54.088327  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:54.093834 19809 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:54.096602  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:54.085741 19732 raft_consensus.cc:697] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:54.087146  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:54.099398 19836 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260808 01:53:54.107564  9636 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 29.
I20260808 01:53:54.115799 19802 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:53:54.117100  9444 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:54.062047 19774 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:53:54.058449 19606 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:54.137724  9558 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
I20260808 01:53:54.146535  9637 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:54.146797  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:54.148710  9638 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 29.
I20260808 01:53:54.132797 19765 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:54.167546  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:54.181250 19802 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:54.203146  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:54.148630 19735 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:54.148973 19685 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:54.213932  9555 raft_consensus.cc:2468] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
W20260808 01:53:54.243989  9524 leader_election.cc:336] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:54.210317  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:54.159426 19626 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:54.247959  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:54.253386  9636 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
W20260808 01:53:54.254156  9524 consensus_peers.cc:597] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:53:54.255156  9524 consensus_peers.cc:597] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:54.206534  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
W20260808 01:53:54.260397  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 53999.1us
I20260808 01:53:54.262280  9599 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:54.263397  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:54.263830  9524 consensus_peers.cc:597] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:53:54.265883  9524 consensus_peers.cc:597] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:54.270184  9636 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260808 01:53:54.291363  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:54.301833  9524 leader_election.cc:336] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:54.302645  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:54.304692  9524 leader_election.cc:336] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:54.305375  9524 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:54.266875 19774 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:54.247705 19685 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:54.316548  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:54.271608 19553 raft_consensus.cc:2804] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:53:54.323060  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:54.356577  9600 leader_election.cc:400] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 election: Vote denied by peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455) with higher term. Message: Invalid argument: T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for earlier term 25. Current term is 26.
I20260808 01:53:54.359632  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:54.360785  9636 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:54.299517 19735 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:54.334884 19584 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:54.236750 19684 consensus_queue.cc:260] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:54.259729 19626 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:54.265245 19732 consensus_queue.cc:237] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:54.342893  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:54.344707 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:54.347829  9524 leader_election.cc:336] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:54.445524  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:54.445603 10367 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:54.349210 19716 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:54.351763 19305 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260808 01:53:54.314409 19730 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260808 01:53:54.404680  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:54.465684 19843 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
I20260808 01:53:54.371907  9630 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 28: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:54.438153 19442 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:54.461702 19574 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:53:54.465514  9445 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:54.351300 19774 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:54.478989 19771 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:54.479314 19578 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:54.479713 19771 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:54.480279 19771 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:54.442191 19584 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:54.482926 19584 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:54.480770 19842 log.cc:927] Time spent T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.216s	user 0.001s	sys 0.000s
I20260808 01:53:54.478870 19774 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:54.485560  9629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3882) took 1467 ms. Trace:
I20260808 01:53:54.478835 19442 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:54.438364 19606 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:54.486716 19802 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:54.479972 19578 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:54.487582  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:54.456384 19685 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:54.489153 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:54.488651  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 51080.3us
I20260808 01:53:54.486016  9629 rpcz_store.cc:276] 0808 01:53:53.017922 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:53.018083 (+   161us) service_pool.cc:224] Handling call
0808 01:53:53.018427 (+   344us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:53.021895 (+  3468us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:53.021928 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:53.022507 (+   579us) log.cc:844] Serialized 26 byte log entry
0808 01:53:53.933026 (+910519us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b1800266680. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:53:54.038276 (+105250us) raft_consensus.cc:1645] Marking committed up to 20
0808 01:53:54.051263 (+ 12987us) log.cc:844] Serialized 14 byte log entry
0808 01:53:54.051958 (+   695us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:54.052071 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:54.485148 (+433077us) raft_consensus.cc:1685] finished
0808 01:53:54.485270 (+   122us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:54.485485 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3013,"prepare.queue_time_us":12624,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":3240,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":21396,"spinlock_wait_cycles":25776896,"thread_start_us":104946,"threads_started":1,"wal-append.queue_time_us":1175099}
I20260808 01:53:54.438439 19735 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:54.464877 19684 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Advancing to term 32
I20260808 01:53:54.501830  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:54.430646 19781 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.057s
I20260808 01:53:54.506451  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:54.511791 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:54.512737 10367 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:54.521593 10365 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: []
I20260808 01:53:54.521580  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:54.524704 10368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56276 (request call id 1245) took 1398 ms (client timeout 1798 ms). Trace:
W20260808 01:53:54.525529 10368 rpcz_store.cc:269] 0808 01:53:53.126539 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:53.128758 (+  2219us) service_pool.cc:224] Handling call
0808 01:53:54.520252 (+1391494us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b540072ecd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:54.524610 (+  4358us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":754108672}
I20260808 01:53:54.528247 10362 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260808 01:53:54.532480  9444 leader_election.cc:400] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27.
I20260808 01:53:54.548491  9444 leader_election.cc:403] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:53:54.554102  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:54.559286  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:54.560237  9558 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:54.558337  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:54.563179  9555 raft_consensus.cc:2368] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21.
I20260808 01:53:54.564746  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:54.569587  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:54.572664  9523 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:54.533376 19819 consensus_meta.cc:287] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.600s	user 0.009s	sys 0.007s
I20260808 01:53:54.530161 19802 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:54.577211 19779 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:54.600559  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:54.602870 19779 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:54.603498 19779 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:54.603706  9558 raft_consensus.cc:2368] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24.
I20260808 01:53:54.532992 19305 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:54.609120  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 261319us
I20260808 01:53:54.534106 19716 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:54.550495 19781 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Stepping down as leader of term 27
I20260808 01:53:54.518491 19685 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.088s
W20260808 01:53:54.619040  9559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60552 (request call id 1522) took 1482 ms (client timeout 1572 ms). Trace:
W20260808 01:53:54.619634  9559 rpcz_store.cc:269] 0808 01:53:53.133179 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:53.137788 (+  4609us) service_pool.cc:224] Handling call
0808 01:53:54.612585 (+1474797us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b540008a7d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:54.615689 (+  3104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":364018048}
I20260808 01:53:54.577927 19730 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:54.617559 19781 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:54.617798 19819 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:54.624575 19802 raft_consensus.cc:3060] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Advancing to term 27
I20260808 01:53:54.648638 10367 raft_consensus.cc:2468] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 21.
I20260808 01:53:54.624356 19553 raft_consensus.cc:697] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:54.630614 19739 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:54.631371 19716 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:54.640093 19798 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:54.617293 19305 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:54.688717 19852 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:54.689270 19305 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:54.667157  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:54.695238  9559 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 30.
W20260808 01:53:54.695528 19850 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260808 01:53:54.668229  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:54.696179 19757 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:54.696579  9558 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 28: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:54.696861  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:54.669404 19626 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:54.697844 19853 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260808 01:53:54.698304 19853 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:54.698762 19853 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Advancing to term 31
W20260808 01:53:54.699438 19844 log.cc:927] Time spent T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.201s	user 0.001s	sys 0.000s
I20260808 01:53:54.700661 19726 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:54.669047 19553 consensus_queue.cc:237] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:54.666642 19716 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:54.670426 19781 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.21, Last appended by leader: 21, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:54.706593 19781 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:54.671891  9557 raft_consensus.cc:1275] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260808 01:53:54.673902 19669 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 33 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:54.711637 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:54.677187 19809 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:54.677596  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:54.659652 19819 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:54.683254 19849 log.cc:927] Time spent T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.152s	user 0.000s	sys 0.001s
W20260808 01:53:54.684020 19841 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260808 01:53:54.686369 19578 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:54.687044 19735 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:54.655941 19442 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:54.686780 19798 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:54.663115 19739 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:54.702356 19746 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:54.710595 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 134222us
I20260808 01:53:54.707994 19716 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:54.712194  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 179932us
I20260808 01:53:54.755713  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:54.682798  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:54.717633  9550 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:54.722432 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:54.756776 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:54.757440 10367 raft_consensus.cc:2410] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 for term 29 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260808 01:53:54.711683 19757 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:54.758512 19757 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:54.751160 19859 consensus_queue.cc:1048] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260808 01:53:54.761147  9599 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:54.765753  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:54.713028 19779 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:54.768280 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:54.771850 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:54.715788 19852 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:54.772629 10367 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:53:54.736505 19819 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:54.764262 19363 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:54.777400  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:54.764959 19716 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:54.780964  9635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45658 (request call id 1291) took 1775 ms. Trace:
I20260808 01:53:54.784364  9635 rpcz_store.cc:276] 0808 01:53:53.005916 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:53.006174 (+   258us) service_pool.cc:224] Handling call
0808 01:53:53.006585 (+   411us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:53.007106 (+   521us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:53.007138 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:53.022906 (+ 15768us) log.cc:844] Serialized 26 byte log entry
0808 01:53:53.929185 (+906279us) spinlock_profiling.cc:243] Waited 6.99 ms on lock 0x7b18002e7440. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:53:54.330186 (+401001us) raft_consensus.cc:1645] Marking committed up to 19
0808 01:53:54.330265 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:54.330506 (+   241us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:54.780615 (+450109us) raft_consensus.cc:1685] finished
0808 01:53:54.780641 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:54.780883 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":672,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":11409,"spinlock_wait_cycles":16074496,"thread_start_us":400676,"threads_started":1,"wal-append.queue_time_us":1478740}
I20260808 01:53:54.791505  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:54.792930 19735 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:54.771529 19860 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:54.797186  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:54.776921 19442 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:54.776980 19746 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:54.800781  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:54.775667 19862 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:54.803246  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:54.803788 19669 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:53:54.807601 19739 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:54.798734 19809 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260808 01:53:54.810040 19819 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:54.811551 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:54.813024 19857 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:54.814097 19857 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:54.814754 19442 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:54.816835 19774 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:54.818240 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:54.818555 19857 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:54.819180 19774 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:54.799366 19860 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:54.826272  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:54.791441 19716 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:54.827098 19765 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:53:54.829614 19716 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:54.805064 19735 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:53:54.822520 19771 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:54.840047 19716 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:54.840659 19801 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:54.820091 19798 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:54.844887 10367 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 32.
I20260808 01:53:54.817226 19442 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:54.845577  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:54.845623 19852 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:54.856369 19852 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:54.857926  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:54.862308 19852 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:54.862840 19852 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:54.863212 19852 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:54.851686 19745 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:54.843534 19771 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:54.854379 19853 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:54.869485  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:54.870230 19644 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:54.870791 19853 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:54.867966  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:54.830502 19860 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:54.870329  9557 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 28: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:54.879395 19779 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:54.835531 19735 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:54.880395 19735 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:54.881408 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:54.892195 10363 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260808 01:53:54.897228 19852 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:54.906126 10361 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 27.
W20260808 01:53:54.912815 10334 leader_election.cc:336] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:53:54.913244  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 45361.4us
I20260808 01:53:54.914744  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:54.913730 10334 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:54.956614  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:54.981895 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 73792.2us
I20260808 01:53:54.917615  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:55.001562  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 84021.9us
I20260808 01:53:54.915868 19771 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:55.003177  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:54.916201 19644 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260808 01:53:55.004714 10334 leader_election.cc:336] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:55.007697  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:55.008621  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:55.010848  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:55.009075  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:55.011480  9637 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:53:55.015542  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:55.016696  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:55.017395  9555 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:55.018512  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:55.018656  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:55.020279  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:55.023020  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:55.023423  9554 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260808 01:53:55.023867  9629 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 29: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:54.987191 19745 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260808 01:53:54.985680 19779 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:55.030922  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:55.032612  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:55.033296  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:55.034130 19771 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:55.034538 19771 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:55.027834  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:55.030191  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:55.035054 19771 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:55.035588  9556 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 29.
I20260808 01:53:54.985685 19618 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:55.033310  9631 raft_consensus.cc:2468] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:55.030022  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:55.036629 19618 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:55.037945  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:55.039258 19618 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:55.027084  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:55.040279  9635 raft_consensus.cc:2410] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 29 because replica has last-logged OpId of term: 27 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 20.
I20260808 01:53:55.040542  9629 raft_consensus.cc:2468] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:55.042121  9637 raft_consensus.cc:2410] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 31 because replica has last-logged OpId of term: 30 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 14.
I20260808 01:53:55.012174 19738 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:55.043179 19738 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:55.043692 19738 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:55.047698 19852 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:55.004477 19442 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:55.048949 19442 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:55.049451 19442 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:55.058818  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:55.059597  9633 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Advancing to term 33
I20260808 01:53:55.058938 19779 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:55.064814 19735 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:55.065954 19779 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:55.067353 19735 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:55.056728 19738 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:55.071038 19738 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:55.071139  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:55.060700  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:55.073725  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:55.056761 19802 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 26. Current term is 27.
I20260808 01:53:55.074961  9444 leader_election.cc:400] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21.
I20260808 01:53:55.073240 19647 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:55.075383  9444 leader_election.cc:403] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260808 01:53:55.075811 19647 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:55.076300 19735 raft_consensus.cc:3060] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 20 FOLLOWER]: Advancing to term 21
I20260808 01:53:55.082476  9444 leader_election.cc:400] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24.
I20260808 01:53:55.083298  9444 leader_election.cc:403] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260808 01:53:55.084532  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:55.085254 10365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56258 (request call id 1160) took 1727 ms. Trace:
I20260808 01:53:55.085354  9552 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 28.
I20260808 01:53:55.085909 10365 rpcz_store.cc:276] 0808 01:53:53.357604 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:53.358086 (+   482us) service_pool.cc:224] Handling call
0808 01:53:54.521275 (+1163189us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:54.522575 (+  1300us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:55.084992 (+562417us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b540072ecd0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:53:55.085178 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2674270592}
I20260808 01:53:55.090282  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:55.091014  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:55.091732  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:55.091127  9637 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:55.092306  9629 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:55.095889  9630 raft_consensus.cc:2368] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28.
I20260808 01:53:55.097616  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:55.098546  9558 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 28: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:55.098587  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:55.098758  9601 consensus_peers.cc:597] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:53:55.099712  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 1.03831e+06us
I20260808 01:53:55.099606  9552 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:55.097206  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:55.104516  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:55.100792  9601 consensus_peers.cc:597] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:55.088557 19647 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:55.106012 19802 raft_consensus.cc:2804] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260808 01:53:55.107427 19802 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:55.107734  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:55.108001 19802 raft_consensus.cc:3060] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:55.109134  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:55.106125  9601 leader_election.cc:336] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:53:55.111451  9601 leader_election.cc:336] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:55.109961  9550 raft_consensus.cc:2393] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 27: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:55.112041  9601 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:55.113389 19578 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260808 01:53:55.115083  9601 leader_election.cc:336] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:55.095912 19852 raft_consensus.cc:3060] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:53:55.115674  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:55.117067  9601 leader_election.cc:336] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:55.117493 19757 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:55.117511  9601 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:55.105286  9637 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 29.
W20260808 01:53:55.119777  9601 leader_election.cc:336] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:55.120409  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:55.113981  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:55.125425  9550 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:55.128578  9523 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:55.081624 19738 raft_consensus.cc:2804] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:53:55.118399 19620 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260808 01:53:55.133548 19578 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:55.135991  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:55.137387  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:55.137997  9558 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Advancing to term 28
W20260808 01:53:55.139784 19848 log.cc:927] Time spent T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.525s	user 0.001s	sys 0.011s
I20260808 01:53:55.140126  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:55.147940  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:55.148909  9560 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 32: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:55.149868  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:55.150450  9560 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:53:55.158772 19644 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260808 01:53:55.160176 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:55.160789 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:55.160913 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:55.161697 10365 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:55.161742 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:55.162775 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:55.163546 10360 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 28: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:55.156200 19618 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.008s
I20260808 01:53:55.162228 19620 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:55.156425 19703 raft_consensus.cc:2804] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election won for term 21
I20260808 01:53:55.168991 19620 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:55.169318 19703 raft_consensus.cc:697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:55.170727 19703 consensus_queue.cc:237] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.17, Last appended by leader: 17, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:55.172991  9557 raft_consensus.cc:1240] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260808 01:53:55.174655 19620 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:55.176153 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:55.152289  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:55.154332 19738 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:55.154991  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:55.155716  9444 leader_election.cc:400] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28.
I20260808 01:53:55.186206  9444 leader_election.cc:403] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
I20260808 01:53:55.157363  9552 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:55.176841 10363 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:55.179396 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:55.182586 10331 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:55.184942  9600 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:55.208024 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:55.211771  9560 raft_consensus.cc:2468] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 31.
W20260808 01:53:55.213980  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 58404.8us
I20260808 01:53:55.157099 19757 raft_consensus.cc:2804] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:55.215337  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:55.183665  9633 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 33.
W20260808 01:53:55.184760  9445 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:55.132568 19685 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:55.173252 19771 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:55.183419  9629 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 30.
I20260808 01:53:55.247437  9551 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260808 01:53:55.251606 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:55.240545 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:55.252887  9551 raft_consensus.cc:1275] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260808 01:53:55.240950  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:55.254861  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 45829.6us
I20260808 01:53:55.255029  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:55.253582 10360 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:53:55.256913  9445 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:55.266284  9550 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 30.
I20260808 01:53:55.218597 19757 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:55.207106 19254 raft_consensus.cc:2764] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election decision vote started in defunct term 27: won
I20260808 01:53:55.240001 19611 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Advancing to term 28
I20260808 01:53:55.281306  9554 raft_consensus.cc:1275] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260808 01:53:55.254313 19801 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:55.209348 19644 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:55.213964 19647 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:55.140548 19798 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.126s
I20260808 01:53:55.285547  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:55.211069 19738 raft_consensus.cc:3060] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:55.289337 19785 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:55.213581 19860 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:55.276572 19862 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:55.289268 10363 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:55.293355 19771 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:55.284688 19757 raft_consensus.cc:3060] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:55.254875 19856 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260808 01:53:55.257561 19685 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:55.295300  9558 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 28.
I20260808 01:53:55.296654 19735 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 20. Current term is 21.
I20260808 01:53:55.263782 19716 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election won for term 29
I20260808 01:53:55.281905  9445 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:55.347280  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:55.302018 19765 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 33 FOLLOWER]: Leader election won for term 33
I20260808 01:53:55.299383 19669 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:55.298653 19859 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:53:55.363986  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 93104.5us
I20260808 01:53:55.308876 19647 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting leader election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:55.380473  9443 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:55.380973  9445 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:55.348716 19256 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:55.343357 19644 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:55.311460 19801 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:55.379920 19771 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:55.355165 19856 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:55.419790 19772 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:53:55.420378  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 232916us
I20260808 01:53:55.420550 19772 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:55.378988 19774 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:55.424088  9559 raft_consensus.cc:1275] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260808 01:53:55.386775 19735 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260808 01:53:55.353797 19785 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:55.384586 19647 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:55.352972 19862 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:55.383615 19765 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 33 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:55.426824  9599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:55.425985 19771 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:55.444077 19644 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:55.444415 19779 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:55.387658 19859 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:55.426750 19856 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Advancing to term 32
I20260808 01:53:55.456413 19772 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:55.415925 19442 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:55.459940  9629 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:53:55.487660  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:55.488478  9558 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 21.
I20260808 01:53:55.460990 19819 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:55.437737 19774 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:55.468701 19771 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:55.402511 19256 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:55.466837 19852 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 23. Current term is 24.
W20260808 01:53:55.462448  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 177024us
I20260808 01:53:55.510003  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:55.505968  9599 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:55.520458  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:55.518462  9599 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:55.495002 19819 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:55.483554 19853 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:55.538610  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:55.546103  9635 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 28. Current term is 29. Ops: [28.16-28.16]
I20260808 01:53:55.481199 19779 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:55.556051  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:55.556761  9629 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 29.
I20260808 01:53:55.558590  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:55.552181  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:55.523087 19735 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:53:55.562233  9630 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 30. Current term is 31. Ops: []
W20260808 01:53:55.555377 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 75684.7us
I20260808 01:53:55.519927 19553 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:55.566309  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:55.570749  9633 pending_rounds.cc:85] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 13
I20260808 01:53:55.579520  9633 pending_rounds.cc:107] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 27.14
I20260808 01:53:55.579900  9633 raft_consensus.cc:2889] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:55.571139  9636 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 32. Current term is 33. Ops: []
I20260808 01:53:55.571523  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:55.582055  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:55.532238 19852 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:55.583920  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:55.590104  9629 raft_consensus.cc:2410] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a for term 29 because replica has last-logged OpId of term: 27 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 20.
I20260808 01:53:55.592283 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:55.593349 10368 raft_consensus.cc:2410] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260808 01:53:55.589972  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:55.572919  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:55.573001  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 192154us
I20260808 01:53:55.507736 19739 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.035s
I20260808 01:53:55.508057 19620 consensus_queue.cc:1048] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.040s
I20260808 01:53:55.576040  9634 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260808 01:53:55.530851 19819 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:55.597656  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:55.599308  9445 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:55.600723 19685 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:55.593088 19880 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election won for term 28
I20260808 01:53:55.562017 19669 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260808 01:53:55.602835 19874 log.cc:927] Time spent T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260808 01:53:55.531687 19644 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.051s
I20260808 01:53:55.548373 19785 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:55.566807 19606 raft_consensus.cc:2804] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader election won for term 31
I20260808 01:53:55.579586 19716 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 29 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:55.539919 19853 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:55.589262 19860 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:55.640093  9521 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:55.594234 19735 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:55.578588 19775 consensus_queue.cc:1048] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.045s
I20260808 01:53:55.532500 19765 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:55.664280 10360 raft_consensus.cc:1275] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 21 index: 18. (index mismatch)
W20260808 01:53:55.670094  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 136671us
I20260808 01:53:55.669356 10332 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:55.676230  9599 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:55.686332 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:55.687201 10368 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Advancing to term 27
W20260808 01:53:55.689013  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 49003.1us
I20260808 01:53:55.634424 19852 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader election won for term 30
I20260808 01:53:55.707679  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:55.708395  9638 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 24.
I20260808 01:53:55.709393  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:55.706334  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:55.715672 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 46452.7us
W20260808 01:53:55.718060  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 71394.3us
W20260808 01:53:55.718981  9601 leader_election.cc:336] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:55.667830 19883 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260808 01:53:55.723445  9445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 124269us
I20260808 01:53:55.725716  9445 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:55.735260 19611 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565) with higher term. Message: Invalid argument: T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 27. Current term is 28.
I20260808 01:53:55.740500  9632 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
W20260808 01:53:55.741959 19875 log.cc:927] Time spent T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.170s	user 0.000s	sys 0.003s
I20260808 01:53:55.694168 19801 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:55.707235 19589 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:53:55.747588  9445 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:55.668479 19730 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.047s
I20260808 01:53:55.694129 19606 raft_consensus.cc:697] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:55.698252 19880 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:55.720432  9601 consensus_peers.cc:597] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:53:55.765079  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 44992.6us
W20260808 01:53:55.765789  9601 leader_election.cc:336] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:55.766270  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:55.708220 19735 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:53:55.722129 19769 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260808 01:53:55.757431  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 159916us
I20260808 01:53:55.711298 19852 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 30 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:55.746826 19869 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260808 01:53:55.701593 19860 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election won for term 28
W20260808 01:53:55.776490  9601 leader_election.cc:336] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:55.778962  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:55.781968  9632 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260808 01:53:55.789513  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:55.809029  9629 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 28: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:55.757669 19589 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:55.813794  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:55.809571  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:55.750509 19611 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260808 01:53:55.811542  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:55.724553 19782 consensus_queue.cc:1048] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.063s
I20260808 01:53:55.822371  9635 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:55.772624 19735 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:55.791515 19880 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 26.22, Last appended by leader: 22, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:55.756207 19774 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:55.772964 19801 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:55.785085 19442 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:55.810024 19723 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Stepping down as leader of term 28
W20260808 01:53:55.831336  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 138099us
W20260808 01:53:55.833866  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 57360.9us
I20260808 01:53:55.792935 19685 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:55.822037 19644 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:55.818254 19589 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:55.812374 19859 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:55.832177 19735 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:55.817728 19730 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:55.845459  9601 leader_election.cc:336] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:55.846238  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:55.847429  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:55.871294  9601 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:55.849269  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:55.834975  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 143046us
I20260808 01:53:55.863250  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:55.873170  9556 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 24.
I20260808 01:53:55.832957 19860 raft_consensus.cc:697] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:55.875443  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:55.876369  9554 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:55.764643 19716 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:55.870668  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 174799us
W20260808 01:53:55.895444  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 50051.4us
W20260808 01:53:55.896443  9601 leader_election.cc:336] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:55.897079  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:55.901047  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:55.836294 19888 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:55.848392 19798 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:55.843221 19723 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:55.923139  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:55.923871  9558 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 21.
I20260808 01:53:55.925004  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6982ec010444a8854140f7b64e985d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:55.926259  9552 raft_consensus.cc:3060] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:53:55.873569 19801 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:53:55.933109  9601 leader_election.cc:336] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:55.878657 19779 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:55.933704  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:55.936872  9552 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
I20260808 01:53:55.938012  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:55.938586  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:55.913790 19685 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260808 01:53:55.866365 19644 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:55.852394 19606 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:55.901995 19774 consensus_queue.cc:1048] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.052s
W20260808 01:53:55.945461 19879 log.cc:927] Time spent T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.274s	user 0.000s	sys 0.001s
I20260808 01:53:56.009985  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:56.067135  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:56.069274 19442 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260808 01:53:56.011708  9599 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:56.072868  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:55.967129  9554 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 29.
I20260808 01:53:56.074301  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:56.000895  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 67800.8us
W20260808 01:53:55.978267  9524 leader_election.cc:336] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:56.076867  9524 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:56.077070  9601 leader_election.cc:336] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:55.872303 19852 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:56.030228  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:56.032675 19647 consensus_meta.cc:287] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent flushing consensus metadata: real 0.587s	user 0.013s	sys 0.000s
W20260808 01:53:56.107826  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 129636us
I20260808 01:53:55.922348 19798 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:56.032171 19723 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:56.033116 19885 log.cc:927] Time spent T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.283s	user 0.002s	sys 0.000s
W20260808 01:53:56.034075 19889 log.cc:927] Time spent T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.176s	user 0.001s	sys 0.000s
W20260808 01:53:56.034862 19890 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
I20260808 01:53:55.902968 19730 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:56.010299 19859 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260808 01:53:56.038300 19757 consensus_meta.cc:287] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.677s	user 0.007s	sys 0.007s
I20260808 01:53:56.043042 19892 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:55.969550 19895 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:55.985960 19893 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:56.160486 19892 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:56.161125 19893 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:56.137239 19578 raft_consensus.cc:2804] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election won for term 27
W20260808 01:53:56.059754 19877 log.cc:927] Time spent T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.349s	user 0.000s	sys 0.002s
I20260808 01:53:56.162621 19578 raft_consensus.cc:697] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:56.163985 19578 consensus_queue.cc:237] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 25.20, Last appended by leader: 20, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:56.060734 19860 consensus_queue.cc:237] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:55.953584 19801 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:56.066289  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 165303us
I20260808 01:53:56.077582  9601 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:56.081568 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:56.168475 19256 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:56.137370 19669 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260808 01:53:56.169176 19256 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:56.169682 19669 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:56.170243 19669 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:56.089988 19883 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:56.177481  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:56.087762 10368 raft_consensus.cc:2468] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
W20260808 01:53:56.087966  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 76348.2us
I20260808 01:53:55.958140 19853 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:56.092559  9559 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:55.960035 19611 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:56.109928 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:56.115170  9444 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
W20260808 01:53:56.131389 10334 leader_election.cc:336] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:56.108880 19442 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:56.152192  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:56.191036  9638 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Advancing to term 29
W20260808 01:53:56.188112  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 111061us
I20260808 01:53:56.180935 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:56.191864  9601 leader_election.cc:336] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:56.192427  9601 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:56.194362  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:56.189953 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 75127.1us
W20260808 01:53:56.230767  9601 leader_election.cc:336] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:53:56.232317  9601 leader_election.cc:336] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:53:56.233412  9601 leader_election.cc:336] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:56.233830  9601 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:56.154318 19254 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260808 01:53:56.243371  9601 leader_election.cc:336] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:56.243881  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:56.251552  9601 leader_election.cc:336] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:56.252203  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:56.252996 19888 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260808 01:53:56.204473 19802 consensus_meta.cc:287] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.096s	user 0.016s	sys 0.002s
I20260808 01:53:56.131238 19647 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:56.263911  9631 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.013s	sys 0.040s
W20260808 01:53:56.265210  9631 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.014s	sys 0.040s
I20260808 01:53:56.200502 19723 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
W20260808 01:53:56.264416  9601 consensus_peers.cc:597] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:56.208491  9600 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:56.202803 19574 consensus_queue.cc:1059] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 30.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.065s
W20260808 01:53:56.269659  9601 consensus_peers.cc:597] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:56.239895 19853 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260808 01:53:56.272011  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 63661.1us
W20260808 01:53:56.272712  9601 leader_election.cc:336] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:56.189267 19757 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:56.262879 19819 consensus_meta.cc:287] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.631s	user 0.014s	sys 0.006s
I20260808 01:53:56.206473 19256 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:56.256737 19620 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:53:56.230647 19611 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:56.263038 19883 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election won for term 29
I20260808 01:53:56.293653 10361 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 28. Current term is 29. Ops: []
I20260808 01:53:56.268638 19888 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260808 01:53:56.307534  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 198721us
I20260808 01:53:56.269044 19802 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:56.290357  9638 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 29.
I20260808 01:53:56.293292 19254 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.020s
I20260808 01:53:56.312493  9601 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:56.349504  9524 leader_election.cc:336] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:56.373183  9524 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:56.334579 19862 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:56.401304 19611 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:56.402200 19611 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:56.338833 19819 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:56.405591 19819 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:56.406980 19611 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:56.367749 19779 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:56.358637 19883 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 29 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:56.343221 19256 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:56.425091  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 152408us
I20260808 01:53:56.389209 19644 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:56.428581  9601 consensus_peers.cc:597] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:53:56.410270 19738 consensus_meta.cc:287] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.057s	user 0.016s	sys 0.002s
I20260808 01:53:56.409687 19611 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:56.441342 19802 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:56.443224  9601 leader_election.cc:336] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:56.427153 19781 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260808 01:53:56.451094 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 134403us
W20260808 01:53:56.488847  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 299385us
I20260808 01:53:56.444468 19896 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:56.444095 19881 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:56.498387 19730 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:56.498598 19881 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:56.501281 19881 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:56.444339 19899 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260808 01:53:56.444298 19902 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260808 01:53:56.507296 19905 log.cc:927] Time spent T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20260808 01:53:56.456161 19738 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:56.508220 19669 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:56.509140 19903 log.cc:927] Time spent T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20260808 01:53:56.447672 19644 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:56.511123 19644 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:56.457567 19897 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 14, Time since last communication: 0.222s
I20260808 01:53:56.488260 19917 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:56.515008 19883 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:56.516026 19917 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:56.520008 19757 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:56.520268  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:56.520931 19730 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260808 01:53:56.520982  9555 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 32.
I20260808 01:53:56.523254 19647 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:56.523849 19897 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 LEADER]: Stepping down as leader of term 32
I20260808 01:53:56.524428 19897 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:53:56.525493 19897 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:56.527434 19897 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Advancing to term 33
I20260808 01:53:56.535296 19738 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:56.538404 19917 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:56.539054  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:56.514333 19644 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:56.541328 19669 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:56.455437  9601 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:56.543606  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 100450us
I20260808 01:53:56.545892 19798 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:56.548457  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:56.457458 19876 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.367s	user 0.002s	sys 0.000s
W20260808 01:53:56.551579  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 202097us
W20260808 01:53:56.556257  9601 consensus_peers.cc:597] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:56.548223 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:56.580336  9601 leader_election.cc:336] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:53:56.580336  9524 leader_election.cc:336] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:53:56.581981  9601 leader_election.cc:336] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:56.582609  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:56.587616  9524 leader_election.cc:336] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:56.491708 19913 raft_consensus.cc:3055] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Stepping down as leader of term 30
I20260808 01:53:56.593911 19774 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:56.609184 19669 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:56.636989  9631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42824 (request call id 3910) took 1080 ms. Trace:
W20260808 01:53:56.644591  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 529496us
I20260808 01:53:56.577732 19896 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:56.650496 19812 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260808 01:53:56.728246 19919 log.cc:927] Time spent T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260808 01:53:56.692662 19892 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:56.722635 19801 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:56.641070 19738 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:56.639575 19895 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260808 01:53:56.798166  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 216197us
I20260808 01:53:56.806427  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:53:56.849704  9601 consensus_peers.cc:597] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:56.807600  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:56.808706  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:56.809798  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:56.852641  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:56.853647  9637 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Advancing to term 25
I20260808 01:53:56.809854  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:56.855126  9554 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:56.811403  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:56.812494  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:56.814625  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:56.816956  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:56.799969  9631 rpcz_store.cc:276] 0808 01:53:55.556642 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:55.557588 (+   946us) service_pool.cc:224] Handling call
0808 01:53:55.582521 (+ 24933us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:53:55.582884 (+   363us) raft_consensus.cc:1534] Early marking committed up to index 17
0808 01:53:56.265789 (+682905us) log.cc:844] Serialized 14 byte log entry
0808 01:53:56.636479 (+370690us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:53:56.636565 (+    86us) raft_consensus.cc:1645] Marking committed up to 17
0808 01:53:56.636630 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:56.636744 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:56.636904 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17747,"prepare.queue_time_us":91708,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":29455,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":4891,"thread_start_us":998268,"threads_started":2}
I20260808 01:53:56.823661 10365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 1266) took 1616 ms. Trace:
I20260808 01:53:56.847565 19765 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:56.858312  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:56.858745 19765 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:56.835784 19908 log.cc:927] Time spent T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.449s	user 0.002s	sys 0.004s
I20260808 01:53:56.822844 19812 raft_consensus.cc:493] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:56.861286 19812 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 FOLLOWER]: Advancing to term 33
I20260808 01:53:56.861531  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
W20260808 01:53:56.863597 19589 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 0.964s	user 0.012s	sys 0.023s
I20260808 01:53:56.864100 19589 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:56.867455  9555 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 31 index: 17. (term mismatch)
I20260808 01:53:56.868513  9555 pending_rounds.cc:85] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 16
I20260808 01:53:56.868824  9555 pending_rounds.cc:107] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 30.17
I20260808 01:53:56.869403  9555 raft_consensus.cc:2889] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:56.795650 19913 raft_consensus.cc:740] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:53:56.794590  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 207015us
I20260808 01:53:56.795611 19880 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:56.821456 19757 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:56.836617  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:56.837646 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:56.837568  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:56.838712  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:56.818621 19856 consensus_meta.cc:287] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: Time spent flushing consensus metadata: real 1.324s	user 0.013s	sys 0.005s
I20260808 01:53:56.843904 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:56.848554 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:56.799952 19781 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:56.857959 10365 rpcz_store.cc:276] 0808 01:53:55.207419 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:55.207975 (+   556us) service_pool.cc:224] Handling call
0808 01:53:55.208344 (+   369us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:55.208773 (+   429us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:55.208801 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:55.209435 (+   634us) log.cc:844] Serialized 52 byte log entry
0808 01:53:55.515566 (+306131us) raft_consensus.cc:1645] Marking committed up to 14
0808 01:53:55.515607 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:55.515674 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:56.823360 (+1307686us) raft_consensus.cc:1685] finished
0808 01:53:56.823395 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:56.823588 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4768,"raft.queue_time_us":4559,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":33382,"thread_start_us":243947,"threads_started":1,"wal-append.queue_time_us":377901}
I20260808 01:53:56.814374 19442 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:56.844278 19862 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:56.901558 10362 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:56.901826  9635 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Advancing to term 29
W20260808 01:53:56.906152 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 92309.1us
I20260808 01:53:56.904093  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:56.848395 19892 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:56.821671 19685 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.063s
I20260808 01:53:56.821331 19669 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:56.872185 19618 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.841s	user 0.024s	sys 0.017s
I20260808 01:53:56.871706 19738 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:56.942322 19618 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.911s	user 0.025s	sys 0.018s
I20260808 01:53:56.904680 19757 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:56.957844 19771 consensus_meta.cc:287] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.441s	user 0.026s	sys 0.001s
I20260808 01:53:56.960819 19771 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:56.934260 19862 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:56.903750 19856 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:56.962800 19893 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:56.900604 19880 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:56.966912 19912 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.163s	user 0.000s	sys 0.001s
I20260808 01:53:56.976126 19880 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:56.945822 19892 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:56.979004 19862 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:56.982160 19892 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:56.826862  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:57.048049  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:57.048542  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:57.052575  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:57.055891  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 176499us
I20260808 01:53:56.869783  9555 pending_rounds.cc:107] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 32.18
I20260808 01:53:57.054616  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:57.060109  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:57.060029  9555 raft_consensus.cc:2889] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:57.062023  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:56.922191  9633 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 27.
I20260808 01:53:57.062992  9556 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Advancing to term 25
W20260808 01:53:57.077165 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 131896us
I20260808 01:53:57.091061 10363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56276 (request call id 1270) took 1050 ms. Trace:
I20260808 01:53:57.091732 10363 rpcz_store.cc:276] 0808 01:53:56.040575 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:56.040902 (+   327us) service_pool.cc:224] Handling call
0808 01:53:56.041359 (+   457us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:53:56.041796 (+   437us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:53:56.041828 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:53:56.173559 (+131731us) spinlock_profiling.cc:243] Waited 25.6 ms on lock 0x7b2c0002f0f8. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4328ec71 00007f5d4329e5d2 00007f5d449cede1 00007f5d4499381a 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22
0808 01:53:56.229482 (+ 55923us) log.cc:844] Serialized 52 byte log entry
0808 01:53:56.647591 (+418109us) raft_consensus.cc:1645] Marking committed up to 16
0808 01:53:56.647650 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:56.647755 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:57.090281 (+442526us) raft_consensus.cc:1685] finished
0808 01:53:57.090320 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:57.090821 (+   501us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":415,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":102,"spinlock_wait_cycles":58872064,"thread_start_us":386079,"threads_started":1,"wal-append.queue_time_us":647998}
I20260808 01:53:57.096589 10365 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 26 index: 22. Preceding OpId from leader: term: 28 index: 23. (index mismatch)
I20260808 01:53:56.927425 19442 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:57.098420 19578 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:57.099861 19578 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:57.091094 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:57.104307 19578 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:57.107766 19442 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:57.113351 19589 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.113929 19738 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.108837 19442 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:57.116462 19589 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:57.117208 19669 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:57.117434 19738 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:57.118252 19669 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:57.126422 19757 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.116246 19442 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:57.118507 19738 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:57.131709 19771 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.127182 19730 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:57.117461 19589 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:57.148152 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:57.159894 19730 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:57.165256 19442 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:57.159200 10365 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:57.167929 19864 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:57.168833 19864 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:57.166893 19852 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:57.161496 19735 consensus_meta.cc:287] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 1.261s	user 0.006s	sys 0.031s
I20260808 01:53:57.169411 19669 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:57.180194 19589 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.184513 19916 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:57.189797 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:57.187140 19730 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.238601 19916 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:57.240033 19730 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:57.240983 19730 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:57.187340 19771 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.042s
I20260808 01:53:57.198005 19893 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:57.247053 19893 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:57.250239 19915 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:57.250543 19893 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:57.251188 19915 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:57.252925 19730 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:57.254863 19856 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:57.255255 19915 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.257604 10369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:57.260185  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:57.225509 19735 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:57.187665 19930 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:57.262123  9551 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 28.
I20260808 01:53:57.263320  9635 raft_consensus.cc:2410] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 29 because replica has last-logged OpId of term: 27 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 20.
I20260808 01:53:57.263213 19930 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:57.264533  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:57.265215 10332 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:57.265652  9555 raft_consensus.cc:2393] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 31: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:57.266434 19930 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.267241 19930 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:57.270375 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:57.276597  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:57.270457 10334 consensus_peers.cc:597] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:53:57.279166 19932 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:57.280326 19932 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:57.280978  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:57.284869  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 50207.8us
W20260808 01:53:57.286358  9601 consensus_peers.cc:597] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:57.186617 19738 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:53:57.287097  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 436756us
W20260808 01:53:57.288434  9601 leader_election.cc:336] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:57.289893 19864 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.291404  9637 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:57.236586 19852 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:57.297540 19852 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.222142 19812 raft_consensus.cc:515] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:57.259232 10369 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28.
I20260808 01:53:57.289062  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:57.268255 19930 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:57.294144 10362 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 29.
I20260808 01:53:57.305820 10361 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:57.374841  9558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53208 (request call id 4645) took 1444 ms. Trace:
I20260808 01:53:57.330188 19932 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:57.330653  9551 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260808 01:53:57.361752  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:57.376502  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:57.434849 19256 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:57.369995  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:57.423516 19933 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:57.286382  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:57.439182 10364 raft_consensus.cc:1240] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Rejecting Update request from peer 259bcb3f0d254e8b9effac3277e654e0 for earlier term 28. Current term is 29. Ops: []
I20260808 01:53:57.450640  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:57.361800 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:57.454985  9559 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 25 index: 20. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
I20260808 01:53:57.457190 10360 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 33: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:57.458971 10369 raft_consensus.cc:1275] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 25 index: 20. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
I20260808 01:53:57.461236  9600 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:57.463124 19802 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election won for term 28
I20260808 01:53:57.463975 19802 raft_consensus.cc:697] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:57.465076 19899 consensus_queue.cc:1059] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260808 01:53:57.465660 19802 consensus_queue.cc:237] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:57.464303 19888 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:57.469069 19930 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.304490 19916 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:57.467100 19618 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:57.470031 19618 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:57.470427 19618 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:57.472191 19765 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.463203 19785 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:57.475606 19785 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:57.476555 19888 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:53:57.480242 19578 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260808 01:53:57.481761 19620 raft_consensus.cc:3055] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 LEADER]: Stepping down as leader of term 28
I20260808 01:53:57.482607 19620 raft_consensus.cc:740] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:57.330169 19852 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:57.489809 19860 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260808 01:53:57.491665  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 203460us
W20260808 01:53:57.485456  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 199184us
W20260808 01:53:57.485404 19723 consensus_meta.cc:287] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Time spent flushing consensus metadata: real 1.209s	user 0.024s	sys 0.004s
I20260808 01:53:57.497628 19812 leader_election.cc:290] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 33 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.482255 19781 consensus_queue.cc:1048] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260808 01:53:57.453039  9555 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 32: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:53:57.355406 19735 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:57.467127  9524 leader_election.cc:336] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:57.399852 10366 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:57.306087 19669 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.521080  9556 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 25.
I20260808 01:53:57.528487  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:57.540875  9557 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 28.
I20260808 01:53:57.542392 10366 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:57.548202  9601 leader_election.cc:336] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:57.536307 19785 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:57.556382  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:57.557359 19913 consensus_queue.cc:260] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 30.15, Last appended by leader: 15, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:57.562085  9601 leader_election.cc:336] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:53:57.562439  9521 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:57.563076 19913 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Advancing to term 31
W20260808 01:53:57.583413  9601 leader_election.cc:336] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:57.583402 19765 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:57.584128  9601 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:57.564272  9550 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 16. (term mismatch)
I20260808 01:53:57.566004 19254 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:57.584910  9550 pending_rounds.cc:85] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting all ops after (but not including) 15
I20260808 01:53:57.585280  9550 pending_rounds.cc:107] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Aborting uncommitted NO_OP operation due to leader change: 28.16
I20260808 01:53:57.585919  9550 raft_consensus.cc:2889] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:53:57.548096 19852 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:57.571328 19781 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:57.428654  9558 rpcz_store.cc:276] 0808 01:53:55.930607 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:55.931003 (+   396us) service_pool.cc:224] Handling call
0808 01:53:55.931399 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:53:56.343200 (+411801us) raft_consensus.cc:1534] Early marking committed up to index 16
0808 01:53:56.620460 (+277260us) log.cc:844] Serialized 14 byte log entry
0808 01:53:56.828374 (+207914us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:53:56.829233 (+   859us) log.cc:844] Serialized 26 byte log entry
0808 01:53:56.829998 (+   765us) raft_consensus.cc:1645] Marking committed up to 16
0808 01:53:56.830040 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:53:57.266917 (+436877us) spinlock_profiling.cc:243] Waited 6.34 ms on lock 0x7b180040e280. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4185ac56 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229
0808 01:53:57.374087 (+107170us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x7b5400382a54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:53:57.374163 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:53:57.374274 (+   111us) raft_consensus.cc:1685] finished
0808 01:53:57.374297 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:53:57.374578 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20732,"prepare.queue_time_us":83713,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":56,"raft.queue_time_us":92399,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":2178,"spinlock_wait_cycles":646957824,"thread_start_us":579426,"threads_started":3,"wal-append.queue_time_us":337169}
W20260808 01:53:57.570789 19883 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.007s	sys 0.016s
I20260808 01:53:57.584285 19765 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:57.589565 19765 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:57.589306 19781 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:57.590489 19620 consensus_queue.cc:260] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:57.592489 19781 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:57.594849 19574 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:57.595820 19883 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.753s	user 0.008s	sys 0.017s
I20260808 01:53:57.595744 19574 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:57.598927 19852 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:57.579646  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:57.603770  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45208.1us
I20260808 01:53:57.604691  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:57.605587 19883 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260808 01:53:57.606696 19883 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:53:57.609485 19896 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.550776 10368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:57.606415  9444 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:57.592526 19620 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:57.650803 19781 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260808 01:53:57.658901  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:57.659969  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:57.666603 19881 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:57.667451 19881 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:57.670154 19881 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.666927 19765 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:57.676785 19765 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.661041  9560 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 30: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:53:57.666991  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 200027us
I20260808 01:53:57.684355 10366 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
W20260808 01:53:57.688637  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 105338us
W20260808 01:53:57.690004  9601 leader_election.cc:336] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:57.690666  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:57.697116 19945 log.cc:927] Time spent T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Append to log took a long time: real 0.262s	user 0.000s	sys 0.004s
W20260808 01:53:57.699033  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 92717.8us
W20260808 01:53:57.722798  9524 leader_election.cc:336] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:57.723472  9524 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:57.723781  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:57.724645 19896 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260808 01:53:57.727411  9524 leader_election.cc:336] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:57.732568  9600 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:57.732887 10332 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:57.733868  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:57.734030 19647 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:57.734551 19647 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:57.734694 19578 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:53:57.735045 19647 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:57.735599 19578 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
W20260808 01:53:57.699999 10334 consensus_peers.cc:597] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260808 01:53:57.748277 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 344967us
I20260808 01:53:57.748505 19913 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:57.749614 19933 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:57.749282 19913 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:57.750373 19933 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Advancing to term 29
W20260808 01:53:57.751773 10334 leader_election.cc:336] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:57.704232 19574 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.657711 19735 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260808 01:53:57.716516 19947 log.cc:927] Time spent T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260808 01:53:57.763216 19757 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:57.763417 19883 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:57.764664 19883 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:57.765873  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:57.768100 19883 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:57.769261  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:57.770004  9556 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Advancing to term 33
I20260808 01:53:57.779722 19735 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260808 01:53:57.780925 19735 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.001s
W20260808 01:53:57.753177 10334 leader_election.cc:336] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:57.784296 19618 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:57.793471  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:57.805315 19735 raft_consensus.cc:3060] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Advancing to term 31
W20260808 01:53:57.795622  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 68249.2us
I20260808 01:53:57.797297  9445 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:57.800225 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:57.818940  9556 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 33.
W20260808 01:53:57.813917  9524 leader_election.cc:336] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:57.825194 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:53:57.839115  9524 leader_election.cc:336] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:53:57.840703  9524 leader_election.cc:336] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:57.841714 19618 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:57.829386 19883 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election won for term 29
W20260808 01:53:57.861892  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 68616.4us
I20260808 01:53:57.867522 10332 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:57.872236  9524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:57.873775 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 26 index: 22 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:57.873456 19775 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:57.877271 19896 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:57.878319 19896 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:57.877219 19775 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:57.888027 19932 raft_consensus.cc:2804] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 33 FOLLOWER]: Leader election won for term 33
I20260808 01:53:57.890870  9444 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:57.874032 19883 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 29 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:57.873379 19895 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:57.906311  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:57.903092 19669 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:53:57.915762  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 141216us
W20260808 01:53:57.915938 10334 consensus_peers.cc:597] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:53:57.912318 19765 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader election won for term 25
I20260808 01:53:57.929870 19932 raft_consensus.cc:697] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 33 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:53:57.927212 10334 consensus_peers.cc:597] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:53:57.908545 19669 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:57.947763 19928 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:53:57.952538 10334 consensus_peers.cc:597] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:53:57.967314  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:57.968398  9637 raft_consensus.cc:2410] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 34 because replica has last-logged OpId of term: 33 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 17.
I20260808 01:53:57.969821  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:57.970639  9635 raft_consensus.cc:2468] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:57.971103 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:57.946224 19765 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:57.949568 19896 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.977758  9600 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:57.981170  9633 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 30 index: 15. (index mismatch)
I20260808 01:53:57.981835  9551 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 30 index: 15. (index mismatch)
I20260808 01:53:57.925384 19895 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:57.983119  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:57.984789  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:57.985082 19895 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:57.985577  9630 raft_consensus.cc:2468] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:57.986166  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:57.984807  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:57.986281  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:57.966308  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:57.988615  9638 raft_consensus.cc:2468] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 30.
I20260808 01:53:57.985423  9557 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 28. Current term is 29. Ops: []
I20260808 01:53:57.938848 19757 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.023s
I20260808 01:53:57.993378  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:58.004045  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.004828  9635 raft_consensus.cc:2468] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:57.956184 19775 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:58.002795 19256 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:58.007045  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:58.007978  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.013831 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:58.017411  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:58.012421  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:57.999441 19896 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.008s
I20260808 01:53:58.021098  9635 raft_consensus.cc:2393] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 29: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:53:58.019932  9637 raft_consensus.cc:2468] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:57.992969  9369 catalog_manager.cc:5697] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:58.022459 19765 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:58.023834  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.022522  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.025266  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:58.025418  9629 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:58.030535  9633 raft_consensus.cc:2410] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 31 because replica has last-logged OpId of term: 30 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 13.
I20260808 01:53:58.029563 19669 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:57.978063 19928 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:58.035934  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:58.038107  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.038834  9637 raft_consensus.cc:2468] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:58.041908 19647 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:58.043085  9638 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260808 01:53:58.043488  9552 raft_consensus.cc:2368] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 29. Current term is 30.
W20260808 01:53:58.043514  9524 leader_election.cc:336] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:58.049494 19735 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:53:58.058954 19928 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:58.030136 19932 consensus_queue.cc:237] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:58.060631  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:58.063450 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 70572.4us
I20260808 01:53:58.063989  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:58.064280 10334 consensus_peers.cc:597] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:53:58.064987 19647 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:58.066187  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:58.070149  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:58.071062  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:58.071731  9556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:58.085412  9559 raft_consensus.cc:2410] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 29 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
I20260808 01:53:58.071859  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:58.076287  9369 catalog_manager.cc:5697] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 22 to 23, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 23 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:58.010660 19883 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:58.096820  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:58.097239  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.097604  9637 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:53:58.079541  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:58.099128  9551 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Advancing to term 22
W20260808 01:53:58.083730  9524 consensus_peers.cc:597] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:58.091010  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:58.101840  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:58.036458 19669 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:58.113763 19669 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:58.114873 10334 consensus_peers.cc:597] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:53:58.117923 19669 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:58.095808  9556 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:58.095638  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.118108 19913 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:58.118898  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:58.119277  9630 raft_consensus.cc:2410] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 32 because replica has last-logged OpId of term: 30 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 14.
I20260808 01:53:58.096060  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.096378  9554 raft_consensus.cc:2468] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 28.
I20260808 01:53:58.091943  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:58.103513 19757 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:58.127040 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:58.128120 10362 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 29: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:58.127611 19757 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:58.104082 19738 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:58.107146  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.107591 19852 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:58.112130  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:58.095438  9555 raft_consensus.cc:2410] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for term 32 because replica has last-logged OpId of term: 30 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 14.
I20260808 01:53:58.123282  9559 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
W20260808 01:53:58.124431  9524 leader_election.cc:336] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:53:58.149507  9524 leader_election.cc:336] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:58.110021 19933 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:58.152369  9637 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:58.153344 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:58.155207 10362 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 22: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:53:58.154126  9524 leader_election.cc:336] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:58.158002  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.157202 19757 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:58.163543 19669 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:58.157262  9369 catalog_manager.cc:5697] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 21 to 22, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 22 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:58.167030 10360 raft_consensus.cc:1240] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 28. Current term is 29. Ops: []
I20260808 01:53:58.168941 19933 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:58.153046 19852 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:58.173839 19852 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:58.175068  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 63231.2us
I20260808 01:53:58.176107  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:58.177995  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:58.153206 19738 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:58.180373  9524 leader_election.cc:336] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:58.181582 19738 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:58.186120  9524 leader_election.cc:336] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:58.186651  9551 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 22.
I20260808 01:53:58.190196  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 28 index: 23 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:58.192333  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:58.193289  9633 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Advancing to term 29
I20260808 01:53:58.195878  9555 raft_consensus.cc:2393] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 29: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
W20260808 01:53:58.189922  9524 leader_election.cc:336] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:53:58.207337  9524 leader_election.cc:336] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:58.223436 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:58.202775 19852 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260808 01:53:58.227317  9369 catalog_manager.cc:5697] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 23 to 28, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 28 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:58.231609 19896 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:58.232401 19896 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:58.214504 19765 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:58.237447 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:58.210331 19775 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:58.242162 19775 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:58.243997 19779 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:58.244638 19779 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:58.248669 19779 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:58.259856 19896 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:58.260905  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 82992.8us
I20260808 01:53:58.262135 19852 consensus_queue.cc:1059] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.001s
I20260808 01:53:58.263579 19852 raft_consensus.cc:3055] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Stepping down as leader of term 28
I20260808 01:53:58.264081 19852 raft_consensus.cc:740] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:58.268842 10334 leader_election.cc:336] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:58.269315 19852 consensus_queue.cc:260] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 28.23, Last appended by leader: 23, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:58.230387  9369 catalog_manager.cc:5697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 20 to 25, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 25 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:58.291643  9556 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 29.
I20260808 01:53:58.292935  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:58.293668  9554 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 30.
I20260808 01:53:58.294992  9559 raft_consensus.cc:2468] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 29.
I20260808 01:53:58.273682 19852 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:58.255769 19775 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:58.304486  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:53:58.298784 10334 leader_election.cc:336] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:58.306905  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:58.313278  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:58.315933  9633 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 29.
I20260808 01:53:58.316195 19738 raft_consensus.cc:2804] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:58.316658 19738 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:58.317055 19738 raft_consensus.cc:3060] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:58.318981 19757 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260808 01:53:58.319953 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:58.306044 10334 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:58.322113 19933 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260808 01:53:58.302256 10332 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:58.323117  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:58.323840  9638 raft_consensus.cc:3060] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Advancing to term 33
I20260808 01:53:58.324142  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.324966  9635 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 32.
I20260808 01:53:58.327179  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.329223  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:58.322317  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.333355  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:58.323128 19933 raft_consensus.cc:2804] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader election won for term 29
I20260808 01:53:58.335302 19933 raft_consensus.cc:697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:58.336452  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:58.337817  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.339813 19765 raft_consensus.cc:2804] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:58.338835 19738 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:58.340333 19765 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:58.341022 19765 raft_consensus.cc:3060] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:58.344689 19933 consensus_queue.cc:237] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:58.305001 19775 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:58.347198 19775 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:58.346326  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:58.351308 19738 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:58.359653  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.362377  9636 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 29.
I20260808 01:53:58.363636 19669 raft_consensus.cc:2804] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:58.364171 19669 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
W20260808 01:53:58.364531  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 66117.9us
I20260808 01:53:58.361805  9369 catalog_manager.cc:5697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 21 to 24, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 24 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:58.368829  9638 raft_consensus.cc:2468] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 33.
I20260808 01:53:58.364595 19669 raft_consensus.cc:3060] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:58.370103  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.375216  9631 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 33: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:58.373361  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:58.372597 19757 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:58.365053 19899 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260808 01:53:58.377350 19757 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:58.379468 19899 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:58.382875 19899 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Advancing to term 31
W20260808 01:53:58.384410  9524 leader_election.cc:336] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:58.390713 19757 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:58.392048 19765 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:58.394457 19765 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:58.395226 19765 raft_consensus.cc:2804] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260808 01:53:58.395640 19765 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:58.396106 19765 raft_consensus.cc:3060] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:53:58.396019 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:58.384964  9524 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:58.398000  9524 leader_election.cc:336] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:53:58.399389  9524 leader_election.cc:336] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:58.395861  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:58.401921  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.402503  9524 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:58.404848  9369 catalog_manager.cc:5697] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:58.404121 19896 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260808 01:53:58.408619  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:58.411572 19913 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:58.416935  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:58.407265 19757 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:58.419183  9369 catalog_manager.cc:5697] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 17 to 21, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 21 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:58.424268 19883 raft_consensus.cc:2804] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:53:58.428174 19883 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:58.428635 19883 raft_consensus.cc:3060] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:58.431207  9444 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:58.431998 19735 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:58.428419  9369 catalog_manager.cc:5697] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 25 to 29. New cstate: current_term: 29 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:58.434640 19881 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:53:58.434422  9369 catalog_manager.cc:5697] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 20 to 24. New cstate: current_term: 24 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:58.436964  9369 catalog_manager.cc:5697] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 13 to 23, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 23 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:58.439460  9444 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:53:58.444327  9636 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260808 01:53:58.425848 19757 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:53:58.447293 19757 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:58.449689  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:58.450147  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:58.450649  9551 raft_consensus.cc:2393] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 29: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:58.438760  9369 catalog_manager.cc:5697] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 19 to 26, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 26 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:53:58.446480  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:58.459636  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.460084  9637 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260808 01:53:58.470028  9556 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260808 01:53:58.470430  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:58.472150  9559 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260808 01:53:58.477466  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.480898  9629 raft_consensus.cc:1240] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 30. Current term is 31. Ops: []
I20260808 01:53:58.486088 19644 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election won for term 29
I20260808 01:53:58.487171  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:58.487865  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:58.490526  9367 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50068 (request call id 243) took 35794 ms (client timeout 15000 ms). Trace:
I20260808 01:53:58.478753  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:58.492758  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:58.482884  9524 leader_election.cc:336] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:53:58.480307 10334 consensus_peers.cc:597] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:53:58.425194 19738 raft_consensus.cc:2804] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:58.501470  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:58.501881  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.471102 19896 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:58.505213  9369 catalog_manager.cc:5697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 16 to 20. New cstate: current_term: 20 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:58.529409  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.530582  9636 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 25: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
W20260808 01:53:58.527539 10332 leader_election.cc:336] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.668s (SENT)
I20260808 01:53:58.508294 19738 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:58.509213 19620 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:58.545033  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:58.540992 19881 raft_consensus.cc:697] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:58.535347  9524 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 186 similar messages]
W20260808 01:53:58.574045  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 127662us
I20260808 01:53:58.553911 19738 raft_consensus.cc:3060] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:58.527117 19896 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:58.559460 19620 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:58.584151 19896 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:58.551225  9369 catalog_manager.cc:5697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 15 to 18, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 18 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:53:58.577540  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:58.588265  9524 leader_election.cc:336] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:58.566072 19578 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:58.603698  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:58.605290 19774 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:58.605958 19774 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:58.607131 19881 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:58.608372 19774 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:58.616688  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.617394  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.618256  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.618634  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.619014  9637 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 29.
I20260808 01:53:58.620071  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.616920  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.620095  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.626379 19883 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:58.630505 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 111028us
I20260808 01:53:58.633591 19977 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260808 01:53:58.634554 19775 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:58.619583 19757 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:58.636720 19757 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:58.589461 19644 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 29 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:58.641723 19644 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:58.645958 19883 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:58.651062 19578 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:58.653307 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:58.653795 19578 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:58.655177 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:58.685078  9629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:58.685858  9629 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:58.687088  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.687122  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 34 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.689992  9637 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 33.
I20260808 01:53:58.687341 19981 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:58.697674  9638 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 22: Already voted for candidate 6e7856bed1be43acaa821290d506ee97 in this term.
I20260808 01:53:58.691044 19618 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:53:58.491225  9367 rpcz_store.cc:269] 0808 01:53:22.696569 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:32.151361 (+9454792us) service_pool.cc:224] Handling call
0808 01:53:58.490203 (+26338842us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:53:53.027244 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:53:53.027910 (+   666us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:53:53.027971 (+    61us) write_op.cc:435] Acquired schema lock
0808 01:53:53.028004 (+    33us) tablet.cc:662] Decoding operations
0808 01:53:53.268861 (+240857us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:53:53.283070 (+ 14209us) write_op.cc:641] Partition lock acquired for write op
0808 01:53:53.283129 (+    59us) tablet.cc:685] Acquiring locks for 30 operations
0808 01:53:53.372785 (+ 89656us) tablet.cc:701] Row locks acquired
0808 01:53:53.372819 (+    34us) write_op.cc:260] PREPARE: finished
0808 01:53:53.398613 (+ 25794us) write_op.cc:270] Start()
0808 01:53:53.420023 (+ 21410us) write_op.cc:276] Timestamp: P: 1786154033388069 usec, L: 0
0808 01:53:53.436326 (+ 16303us) op_driver.cc:348] REPLICATION: starting
0808 01:53:53.498247 (+ 61921us) log.cc:844] Serialized 11746 byte log entry
0808 01:53:54.287970 (+789723us) op_driver.cc:464] REPLICATION: finished
0808 01:53:54.766668 (+478698us) write_op.cc:301] APPLY: starting
0808 01:53:54.772246 (+  5578us) tablet.cc:1367] starting BulkCheckPresence
0808 01:53:55.094862 (+322616us) tablet.cc:1370] finished BulkCheckPresence
0808 01:53:55.095032 (+   170us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:53:56.975286 (+1880254us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:53:56.975779 (+   493us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=60,key_file_lookups=60,delta_file_lookups=0,mrs_lookups=0
0808 01:53:56.975867 (+    88us) write_op.cc:312] APPLY: finished
0808 01:53:56.976525 (+   658us) log.cc:844] Serialized 260 byte log entry
0808 01:53:57.378598 (+402073us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:53:57.458267 (+ 79669us) write_op.cc:454] Released schema lock
0808 01:53:57.458493 (+   226us) write_op.cc:341] FINISH: Updating metrics
0808 01:53:57.459409 (+   916us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:53:57.459446 (+    37us) write_op.cc:454] Released schema lock
0808 01:53:57.695785 (+236339us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
Metrics: {"mutex_wait_us":5208,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":419204,"apply.run_cpu_time_us":124856,"apply.run_wall_time_us":2703721,"cfile_cache_hit":122,"cfile_cache_hit_bytes":129707,"num_ops":30,"prepare.queue_time_us":287565,"prepare.run_cpu_time_us":56257,"prepare.run_wall_time_us":613318,"raft.queue_time_us":311479,"raft.run_cpu_time_us":30900,"raft.run_wall_time_us":403927,"replication_time_us":822617,"spinlock_wait_cycles":31408000,"thread_start_us":766459,"threads_started":5,"wal-append.queue_time_us":456685,"wal-append.run_cpu_time_us":17680,"wal-append.run_wall_time_us":1397005}]]}
I20260808 01:53:58.724718  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:53:58.727109  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 149730us
I20260808 01:53:58.761735  9555 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260808 01:53:58.762900  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:58.766099  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.702380 19765 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:58.705708 19981 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:58.736696 19735 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260808 01:53:58.776933  9524 leader_election.cc:336] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:53:58.776281 19976 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
I20260808 01:53:58.795051 19644 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:58.796953 19765 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:58.798130 19899 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:58.915110  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:58.802583  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.916030  9636 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 31.
I20260808 01:53:58.781985 19620 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:53:58.922966  9521 leader_election.cc:336] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Timed out: RequestConsensusVote RPC to 127.7.165.195:34565 timed out after 1.704s (SENT)
I20260808 01:53:58.923308 19899 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:58.800199 19738 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:58.925863 19738 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:58.815128 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:58.927037 10362 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 29.
I20260808 01:53:58.821523  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:58.928785  9600 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:53:58.829545  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 66772.6us
I20260808 01:53:58.726536 19618 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:58.929841 19899 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:53:58.930321 19899 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:58.930737 19899 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:58.932243 19618 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:58.858942  9523 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:53:58.936578 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:58.937638 10360 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 29: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:58.788651 19984 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.036s
I20260808 01:53:58.917936  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:53:58.940066 19669 consensus_meta.cc:287] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: Time spent flushing consensus metadata: real 0.568s	user 0.010s	sys 0.009s
W20260808 01:53:58.940824  9367 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50068 (request call id 244) took 17368 ms (client timeout 15000 ms). Trace:
I20260808 01:53:58.940662 19669 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:53:58.941293  9367 rpcz_store.cc:269] 0808 01:53:41.580914 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:58.940542 (+17359628us) service_pool.cc:211] Skipping call since client already timed out
0808 01:53:58.940735 (+   193us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260808 01:53:58.919414  9629 raft_consensus.cc:2468] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 30.
I20260808 01:53:58.920397  9635 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 30. Current term is 31. Ops: [30.15-30.15]
I20260808 01:53:58.950518 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:58.923561  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:58.957732  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:58.958487  9560 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:53:58.960201 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:58.955133 10334 consensus_peers.cc:597] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:53:58.965029 10360 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260808 01:53:58.972901  9558 raft_consensus.cc:1275] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260808 01:53:58.978863 19618 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:58.979022 19765 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260808 01:53:58.951104 19738 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:53:58.981169 19738 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:58.984149 19738 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:58.977962 19985 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:58.988215 19981 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:58.985073 19669 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:58.989909  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 66987.3us
I20260808 01:53:58.990808 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:58.991814  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:58.967748 19968 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260808 01:53:58.968262 19644 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.037s
W20260808 01:53:59.020303  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 45468.5us
I20260808 01:53:59.010782  9560 raft_consensus.cc:2468] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 31.
W20260808 01:53:59.045148  9524 consensus_peers.cc:597] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:53:59.045918 19618 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:59.014019 19968 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:59.056901  9444 leader_election.cc:400] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 29. Current term is 30.
I20260808 01:53:59.059732  9444 leader_election.cc:403] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Cancelling election due to peer responding with higher term
W20260808 01:53:59.059604 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 99444us
I20260808 01:53:59.054287 19618 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:59.037618 19981 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:53:59.063180 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:59.065966  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:59.069484  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:59.062180 19968 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 FOLLOWER]: Advancing to term 33
I20260808 01:53:59.081570  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:59.102051  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:59.104257  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 92341.2us
I20260808 01:53:59.067252 19644 raft_consensus.cc:3060] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:59.073086 19669 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:59.110095  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:59.077961 19647 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260808 01:53:59.067838 19899 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:59.088755 19881 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260808 01:53:59.121105 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 60061us
I20260808 01:53:59.124958  9555 raft_consensus.cc:1275] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260808 01:53:59.108759 19738 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:59.125813  9599 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:53:59.133847 19981 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:53:59.143678 19618 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:59.146910 19578 raft_consensus.cc:2804] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader election won for term 31
I20260808 01:53:59.148944  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0
I20260808 01:53:59.149757 19685 consensus_queue.cc:1048] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
W20260808 01:53:59.151014 10334 consensus_peers.cc:597] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:59.159545 19981 raft_consensus.cc:2804] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:53:59.160271 19981 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:59.160753 19981 raft_consensus.cc:3060] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:53:59.170761  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:59.174072  9601 leader_election.cc:336] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:59.183092 19669 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:59.114984 19978 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:53:59.185184  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:59.201254  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:59.197849 19852 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260808 01:53:59.206856 19757 raft_consensus.cc:3060] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
W20260808 01:53:59.205009 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 73801.9us
I20260808 01:53:59.217810  9444 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:59.223302  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 94151.8us
I20260808 01:53:59.220117 19978 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.006s
I20260808 01:53:59.227890  9521 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:59.228999 19574 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:59.229293  9521 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:59.230149 19254 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:53:59.230636  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:59.235984  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:59.242861  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:59.244349  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:59.246970  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:59.247056  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:59.242870  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:59.253337  9558 raft_consensus.cc:1240] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 30. Current term is 31. Ops: [30.15-30.15]
I20260808 01:53:59.254503  9521 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:59.244529 20003 consensus_queue.cc:1048] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260808 01:53:59.258924  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:59.260538 19644 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader election lost for term 29. Reason: Vote denied by peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095) with higher term. Message: Invalid argument: T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 for earlier term 29. Current term is 30.
I20260808 01:53:59.262429  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:59.277549 19802 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:59.278544  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:59.278386 19802 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:59.279951  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:59.280928 20009 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260808 01:53:59.283089 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:59.283452 19913 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:53:59.283951 19913 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:59.284348 19913 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Advancing to term 30
W20260808 01:53:59.261739  9524 leader_election.cc:336] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:59.213148 19883 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.039s
I20260808 01:53:59.286990  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:59.287623  9632 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:53:59.288231 19883 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:53:59.291235  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:59.292078  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:59.293922  9601 leader_election.cc:336] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:59.295120  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:59.300009  9631 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260808 01:53:59.296550  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:59.296765  9629 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260808 01:53:59.300009  9552 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260808 01:53:59.252549 19978 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:59.304841  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:59.243806 19578 raft_consensus.cc:697] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:53:59.252704 19765 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260808 01:53:59.312263  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 57871.4us
I20260808 01:53:59.303097 19644 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:59.281414 19802 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:59.313803  9521 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:59.304971 19883 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260808 01:53:59.330768  7831 thread.cc:527] Waited for 1000ms trying to join with heartbeat (tid 9507)
W20260808 01:53:59.346190  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52316.7us
I20260808 01:53:59.289165 19981 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:59.343593 19578 consensus_queue.cc:237] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:59.312006 19896 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260808 01:53:59.360100  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 46391.3us
I20260808 01:53:59.368552  9521 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:59.315512 19978 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:59.306274 20009 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.044s
W20260808 01:53:59.350747  9601 leader_election.cc:336] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:59.379791  9521 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:59.386119  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:59.363420 19779 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:59.363108 19735 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:59.383054 19978 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:59.391518 19899 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:53:59.391871 20008 log.cc:927] Time spent T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: Append to log took a long time: real 0.082s	user 0.002s	sys 0.000s
I20260808 01:53:59.393461 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:53:59.393675 19757 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:59.425806 10362 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260808 01:53:59.433053  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 82375.3us
W20260808 01:53:59.433805 20002 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260808 01:53:59.435881 19781 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:59.436475 19781 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:59.440515 19779 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260808 01:53:59.441133 19779 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:53:59.441488 19779 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Advancing to term 34
W20260808 01:53:59.443849 19996 log.cc:927] Time spent T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
W20260808 01:53:59.433914  9601 consensus_peers.cc:597] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:53:59.444813  9601 leader_election.cc:336] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:59.445238  9601 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:53:59.446630  9601 leader_election.cc:336] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:53:59.448727  9601 leader_election.cc:336] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:53:59.449819  9601 consensus_peers.cc:597] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:59.450019 19978 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:59.451264 19779 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:59.450376  9601 consensus_peers.cc:597] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:53:59.454170  9601 consensus_peers.cc:597] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:53:59.451769 20004 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260808 01:53:59.456627  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 70638.6us
I20260808 01:53:59.468715 19981 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:59.469349 19978 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
W20260808 01:53:59.492393  9601 leader_election.cc:336] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:53:59.497542 10334 consensus_peers.cc:597] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260808 01:53:59.496055  9524 consensus_peers.cc:597] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:53:59.502784  9601 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:59.454203 19779 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 34 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:59.524415  9632 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 23.
I20260808 01:53:59.522871 19775 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:53:59.513952 20002 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.075s	user 0.002s	sys 0.000s
W20260808 01:53:59.540961  9524 leader_election.cc:336] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:59.547701 20016 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:53:59.513465 20019 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260808 01:53:59.567260  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 262470us
I20260808 01:53:59.570472  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:59.575749  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 83471.9us
W20260808 01:53:59.579138  9601 leader_election.cc:336] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:59.579664  9601 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:59.574492 19978 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260808 01:53:59.584869  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:59.589641  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:53:59.584132 20010 log.cc:927] Time spent T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.213s	user 0.005s	sys 0.000s
I20260808 01:53:59.579401 19981 consensus_queue.cc:1059] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 28.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.010s
I20260808 01:53:59.600123 19981 raft_consensus.cc:3055] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 30 LEADER]: Stepping down as leader of term 30
I20260808 01:53:59.600570 19981 raft_consensus.cc:740] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:53:59.601547 19981 consensus_queue.cc:260] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 30.15, Last appended by leader: 15, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:59.588068 20017 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260808 01:53:59.593010 20009 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:53:59.593045 19669 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260808 01:53:59.611649  9632 raft_consensus.cc:1275] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260808 01:53:59.613466  9633 raft_consensus.cc:1240] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Rejecting Update request from peer 7201605bb2f54c949a4ebe6e9b3b4b2a for earlier term 22. Current term is 23. Ops: []
I20260808 01:53:59.615149  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:59.587339 19578 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:59.644121  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:59.641139  9637 raft_consensus.cc:2376] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 33. Re-sending same reply.
W20260808 01:53:59.649405  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 59851.4us
I20260808 01:53:59.650614  9444 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:59.651024  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:59.652132  9633 raft_consensus.cc:2393] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 29: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:59.654229 19781 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:53:59.658057  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c49fd321f8b4f89868246239530e78b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:59.659000  9633 raft_consensus.cc:2393] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 33: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:53:59.660454  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:59.662626  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:59.665216  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:59.671629  9558 raft_consensus.cc:1240] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 34 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 33. Current term is 34. Ops: []
I20260808 01:53:59.675609  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:59.678062  9633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:59.674564  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:59.681120  9559 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260808 01:53:59.681154  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:59.683151  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:53:59.684046  9635 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 28.
I20260808 01:53:59.685242 20014 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:59.685845 20014 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:59.687889 19932 consensus_queue.cc:1059] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260808 01:53:59.688567 20014 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:53:59.689095 19932 raft_consensus.cc:3055] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Stepping down as leader of term 22
I20260808 01:53:59.689514 19932 raft_consensus.cc:740] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:53:59.685145  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:59.690841  9638 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260808 01:53:59.685093 20003 consensus_queue.cc:1048] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20260808 01:53:59.691998  9630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:53:59.694048 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 23 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:53:59.694751 10367 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 29.
I20260808 01:53:59.696303  9445 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:53:59.697162 20009 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:53:59.697569 20009 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:59.697938 20009 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:53:59.703053  9632 raft_consensus.cc:1240] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 29. Current term is 30. Ops: []
W20260808 01:53:59.703665  9601 leader_election.cc:336] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:59.656021 20023 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260808 01:53:59.656481 19981 raft_consensus.cc:3060] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:53:59.590561 19968 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:59.680883 19735 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader election won for term 23
I20260808 01:53:59.705987  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:59.734863 10363 raft_consensus.cc:1240] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 33 FOLLOWER]: Rejecting Update request from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 for earlier term 32. Current term is 33. Ops: []
I20260808 01:53:59.745663  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:59.737176 19735 raft_consensus.cc:697] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:53:59.751708 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 49865.7us
I20260808 01:53:59.716207 20014 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:53:59.768505  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:59.773216  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:59.748947 19932 consensus_queue.cc:260] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:59.787488 19735 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:59.768113 19644 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:59.818672  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:59.820649  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47533.5us
I20260808 01:53:59.821542  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:59.816793 19765 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260808 01:53:59.828465 19978 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260808 01:53:59.826169 19669 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:53:59.826887 20016 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260808 01:53:59.834627 20016 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:53:59.835153 20016 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Advancing to term 32
W20260808 01:53:59.832537 20027 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
W20260808 01:53:59.847399 10334 consensus_peers.cc:597] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260808 01:53:59.830660  9524 consensus_peers.cc:597] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260808 01:53:59.847237  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 124753us
I20260808 01:53:59.834314 19669 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:53:59.774380 19913 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:59.861704  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 158034us
W20260808 01:53:59.850473  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 71664us
W20260808 01:53:59.864634  9601 leader_election.cc:336] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:53:59.849479  9524 consensus_peers.cc:597] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:53:59.871613  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:53:59.879015  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 57545.2us
I20260808 01:53:59.883397  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:59.868261 20021 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:53:59.888718  9551 raft_consensus.cc:1240] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 29. Current term is 30. Ops: []
I20260808 01:53:59.888173  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:59.893072  9550 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260808 01:53:59.889360  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 23 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:53:59.895360  9558 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 29.
I20260808 01:53:59.899196  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:53:59.909224  9521 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:59.882617 19899 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:59.875324 20017 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.036s
I20260808 01:53:59.882462 19644 consensus_queue.cc:1059] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 33.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.031s
I20260808 01:53:59.875687 19978 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260808 01:53:59.918090  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:53:59.910270 19852 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:53:59.924153 19738 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:53:59.916220 19932 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:53:59.932814  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 68245.8us
I20260808 01:53:59.929612  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:53:59.921823 19685 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260808 01:53:59.935590  9524 consensus_peers.cc:597] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260808 01:53:59.937075  9601 consensus_peers.cc:597] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:53:59.943521  9601 leader_election.cc:336] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:53:59.920486 19981 consensus_queue.cc:1059] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.018s
I20260808 01:53:59.944635  9601 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:59.960860 19981 consensus_queue.cc:1243] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 27.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260808 01:53:59.937680 19802 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.025s
I20260808 01:53:59.973012 19968 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 33 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:59.973553 19738 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:53:59.957386 19620 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:53:59.950820 19852 consensus_queue.cc:1059] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.023s
I20260808 01:53:59.971104 20024 consensus_queue.cc:1059] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: INVALID_TERM, Last received: 32.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.089s
I20260808 01:53:59.996804 19913 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:53:59.999735 19669 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:00.000500 19669 raft_consensus.cc:3055] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 29 LEADER]: Stepping down as leader of term 29
I20260808 01:54:00.000939 19669 raft_consensus.cc:740] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
W20260808 01:54:00.001531 20037 log.cc:927] Time spent T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260808 01:54:00.014935 20030 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260808 01:53:59.950418  9601 consensus_peers.cc:597] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:00.015947  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 65599.6us
W20260808 01:54:00.017059  9601 leader_election.cc:336] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:00.018904  9601 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:00.021729 19899 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:53:59.994587 19738 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:53:59.984612  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 86829.5us
I20260808 01:54:00.001848 19669 consensus_queue.cc:260] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:53:59.993475 19981 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260808 01:54:00.024489  9601 leader_election.cc:336] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:00.044041  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:53:59.988466 19765 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:54:00.055526  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 126009us
W20260808 01:54:00.056264  9601 leader_election.cc:336] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:00.056788  9601 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:00.051395 19765 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:00.060101 19899 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:00.060873 20034 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:53:59.985150 20009 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:00.064719 20009 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:00.065941 19669 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:54:00.045550 19852 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:00.073014 19852 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:54:00.073395  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 164257us
I20260808 01:54:00.074519  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:00.075598 20024 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:54:00.075768 19738 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:00.083555 20032 log.cc:927] Time spent T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.207s	user 0.000s	sys 0.002s
I20260808 01:54:00.085361 19852 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:00.092211  9521 leader_election.cc:304] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:00.093772  9521 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:00.060945 19899 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:00.097635 19978 raft_consensus.cc:3055] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 29 LEADER]: Stepping down as leader of term 29
I20260808 01:54:00.098304 19978 raft_consensus.cc:740] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:54:00.098954 20016 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:00.100227 20039 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:00.100942 20039 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:00.101730 19913 raft_consensus.cc:2749] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260808 01:54:00.102595 19913 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:54:00.104161  9521 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:00.105283 20023 raft_consensus.cc:3055] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 33 LEADER]: Stepping down as leader of term 33
I20260808 01:54:00.105870 20023 raft_consensus.cc:740] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:54:00.107368 19928 raft_consensus.cc:3055] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 LEADER]: Stepping down as leader of term 32
I20260808 01:54:00.107843 19928 raft_consensus.cc:740] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:54:00.108744 19928 consensus_queue.cc:260] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:00.109540 19978 consensus_queue.cc:260] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:00.098026 19899 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:00.106986 20023 consensus_queue.cc:260] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:00.112541 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 23 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:54:00.115676  9521 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:00.110397 19928 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Advancing to term 33
I20260808 01:54:00.125916 10363 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Advancing to term 30
W20260808 01:54:00.131366  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 74273.3us
I20260808 01:54:00.132902  9521 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:00.124840 19913 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260808 01:54:00.142519 20042 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:00.135805 19765 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:00.143426 20019 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:54:00.143599 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:00.143945 20019 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:00.144411 20019 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Advancing to term 29
W20260808 01:54:00.137619 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 64519.2us
I20260808 01:54:00.134366 19578 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:00.138238 20047 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:54:00.136098 19852 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.004s
I20260808 01:54:00.140205 19899 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:00.142311 19999 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:00.152146  9555 raft_consensus.cc:1275] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 31 index: 15. (index mismatch)
I20260808 01:54:00.152041 19999 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:00.152464 20016 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:00.147621 19578 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:00.154412  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:00.143111 20042 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:00.151369 20039 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:00.166338  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 23 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:54:00.167095  9558 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:54:00.157694  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:00.159014  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:00.160897  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:00.161863  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:00.163676  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:54:00.126020 19978 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:54:00.135918 19774 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:00.170605  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:00.170938  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:00.172114 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:00.172549  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:00.172920 10362 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 29.
I20260808 01:54:00.174732  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:54:00.175220  9601 leader_election.cc:336] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:00.173950 20023 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 33 FOLLOWER]: Advancing to term 34
I20260808 01:54:00.177981 19774 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:00.181882 19775 consensus_queue.cc:1048] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:54:00.183880 19774 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:00.180477  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:00.185408 19774 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:54:00.176321  9600 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:54:00.190137  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:00.190616  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:00.192121 19774 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:54:00.192647 19774 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:54:00.192883  9601 leader_election.cc:336] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:54:00.196947  9601 leader_election.cc:336] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:00.197579  9601 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:00.197125 20045 raft_consensus.cc:2955] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Committing config change with OpId 20.15: config changed from index 11 to 15. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:00.195101  9551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:54:00.199018  9601 leader_election.cc:336] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:00.200525 19785 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:54:00.202876  9601 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:54:00.211450  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 76406.9us
I20260808 01:54:00.185468 19578 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:00.226126  9601 leader_election.cc:336] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:00.230661  9601 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:54:00.236512  9601 leader_election.cc:336] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:54:00.245476 10334 consensus_peers.cc:597] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:00.248600  9559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:00.240666  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:00.251781  9444 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:00.243474  9632 raft_consensus.cc:2955] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Committing config change with OpId 20.15: config changed from index 11 to 15. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:54:00.253716 10334 consensus_peers.cc:597] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:00.260190  9601 leader_election.cc:336] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:00.195161 19774 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:54:00.260753  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:00.262046  9444 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:54:00.264858  9601 leader_election.cc:336] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:00.265652  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:00.235253 20039 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260808 01:54:00.273949  9558 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 30.
I20260808 01:54:00.271059  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:00.275736 20050 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260808 01:54:00.276070  9524 consensus_peers.cc:597] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:00.246161 19899 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:54:00.246176 19999 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:00.305088 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:54:00.306993  9369 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55058 (request call id 134) took 28185 ms (client timeout 15000 ms). Trace:
I20260808 01:54:00.277745 19852 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:54:00.306888  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:00.272028 19775 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260808 01:54:00.310040 19647 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260808 01:54:00.313943 10362 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
W20260808 01:54:00.339361  9524 consensus_peers.cc:597] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:00.317092 20042 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:00.288491 19620 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:54:00.297961 19765 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:54:00.308624 20039 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:00.333925 10363 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 30.
I20260808 01:54:00.332525  9444 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:54:00.344463 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:54:00.339844  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 55524.4us
W20260808 01:54:00.353813  9524 leader_election.cc:336] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:54:00.349711  7831 thread.cc:527] Waited for 2000ms trying to join with heartbeat (tid 9507)
W20260808 01:54:00.307686  9369 rpcz_store.cc:269] 0808 01:53:32.111332 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:39.783093 (+7671761us) service_pool.cc:224] Handling call
0808 01:54:00.296623 (+20513530us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:53:58.982775 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:53:58.983563 (+   788us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:53:58.983628 (+    65us) write_op.cc:435] Acquired schema lock
0808 01:53:58.983649 (+    21us) tablet.cc:662] Decoding operations
0808 01:53:58.985093 (+  1444us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:53:58.985229 (+   136us) write_op.cc:641] Partition lock acquired for write op
0808 01:53:58.985261 (+    32us) tablet.cc:685] Acquiring locks for 14 operations
0808 01:53:59.002771 (+ 17510us) tablet.cc:701] Row locks acquired
0808 01:53:59.002817 (+    46us) write_op.cc:260] PREPARE: finished
0808 01:53:59.045306 (+ 42489us) write_op.cc:270] Start()
0808 01:53:59.064760 (+ 19454us) write_op.cc:276] Timestamp: P: 1786154039034235 usec, L: 0
0808 01:53:59.072666 (+  7906us) op_driver.cc:348] REPLICATION: starting
0808 01:53:59.214021 (+141355us) log.cc:844] Serialized 5512 byte log entry
0808 01:53:59.648829 (+434808us) op_driver.cc:464] REPLICATION: finished
0808 01:53:59.842988 (+194159us) write_op.cc:301] APPLY: starting
0808 01:53:59.843215 (+   227us) tablet.cc:1367] starting BulkCheckPresence
0808 01:54:00.107421 (+264206us) tablet.cc:1370] finished BulkCheckPresence
0808 01:54:00.107571 (+   150us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:54:00.188768 (+ 81197us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:54:00.189136 (+   368us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=0,mrs_lookups=0
0808 01:54:00.189221 (+    85us) write_op.cc:312] APPLY: finished
0808 01:54:00.193405 (+  4184us) log.cc:844] Serialized 129 byte log entry
0808 01:54:00.193635 (+   230us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:54:00.193822 (+   187us) write_op.cc:454] Released schema lock
0808 01:54:00.193885 (+    63us) write_op.cc:341] FINISH: Updating metrics
0808 01:54:00.195298 (+  1413us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:54:00.195334 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":211753,"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":193835,"apply.run_cpu_time_us":25931,"apply.run_wall_time_us":352265,"cfile_cache_hit":58,"cfile_cache_hit_bytes":62731,"num_ops":14,"prepare.queue_time_us":306977,"prepare.run_cpu_time_us":24188,"prepare.run_wall_time_us":347300,"raft.queue_time_us":213470,"raft.run_cpu_time_us":9121,"raft.run_wall_time_us":88929,"replication_time_us":575986,"thread_start_us":517862,"threads_started":4,"wal-append.queue_time_us":93128}]]}
I20260808 01:54:00.362836 20009 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader election won for term 30
I20260808 01:54:00.363552 20009 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 30 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:54:00.364550 20009 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 28.23, Last appended by leader: 23, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:00.343547 20049 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260808 01:54:00.363232  9524 consensus_peers.cc:597] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:00.374387 20051 log.cc:927] Time spent T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260808 01:54:00.380208  9600 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:00.284240 20019 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:00.349157 20050 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.078s
I20260808 01:54:00.424304  9555 raft_consensus.cc:1240] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Rejecting Update request from peer 6e7856bed1be43acaa821290d506ee97 for earlier term 28. Current term is 29. Ops: []
W20260808 01:54:00.467311  9524 leader_election.cc:336] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:00.410441 20019 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:00.410808 20034 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:54:00.354518 20039 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:00.504491 20039 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:00.507419  9366 catalog_manager.cc:5697] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 15 to 24, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 24 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:54:00.444808 20044 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260808 01:54:00.410566 20024 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260808 01:54:00.514835 10334 consensus_peers.cc:597] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:00.527333  9366 catalog_manager.cc:5697] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 23 to 25. New cstate: current_term: 25 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:54:00.538275  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 74852.6us
I20260808 01:54:00.557787  9560 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 29.
W20260808 01:54:00.563120 10334 consensus_peers.cc:597] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:00.506377 19774 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:00.573639  9366 catalog_manager.cc:5697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:00.569079  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:00.577548  9366 catalog_manager.cc:5697] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 22 to 28, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 28 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:00.579624  9366 catalog_manager.cc:5697] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 21 to 25. New cstate: current_term: 25 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
W20260808 01:54:00.584815  9524 consensus_peers.cc:597] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:00.594112 19781 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:00.595022 19774 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:00.594842 19781 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:00.585747 19981 consensus_queue.cc:1059] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.013s
I20260808 01:54:00.597265 19781 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:00.601361 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:00.602313 10362 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 33: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:54:00.603713  9366 catalog_manager.cc:5697] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 23 to 27, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 27 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:00.604859 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:54:00.605487 10367 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:54:00.605585  9366 catalog_manager.cc:5697] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 24 to 29. New cstate: current_term: 29 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:00.607110  9366 catalog_manager.cc:5697] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 21 to 23, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 23 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:00.608475  9366 catalog_manager.cc:5697] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 25 to 26, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 26 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:00.615192  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:54:00.617987  9554 raft_consensus.cc:3060] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:54:00.616498  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:00.618865  9560 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 33.
I20260808 01:54:00.616554 10360 raft_consensus.cc:2955] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Committing config change with OpId 20.15: config changed from index 11 to 15. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:54:00.625525  9524 consensus_peers.cc:597] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:00.629818 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 61520.7us
I20260808 01:54:00.622176  9366 catalog_manager.cc:5697] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 21 to 22, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 22 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:00.618827 20009 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:54:00.639683 20009 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:00.638947  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 57533us
I20260808 01:54:00.642607 20009 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:00.651626  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:00.639047  9366 catalog_manager.cc:5697] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:54:00.638243  9524 leader_election.cc:336] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:54:00.659572  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 44258.2us
I20260808 01:54:00.661988 10367 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 30.
I20260808 01:54:00.664086  9554 raft_consensus.cc:2468] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 30.
I20260808 01:54:00.664142  9600 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:54:00.668416  9636 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260808 01:54:00.669620 19899 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:00.671298 19802 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:00.670300 19899 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:00.673307 20023 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:00.674230 19899 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:00.666347 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:54:00.675140  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 57446.9us
I20260808 01:54:00.675895  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:54:00.676626  9636 raft_consensus.cc:2376] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 30. Re-sending same reply.
I20260808 01:54:00.677799  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:54:00.678915  9637 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Advancing to term 32
I20260808 01:54:00.678035  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:54:00.673997 20023 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:00.682619  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:54:00.682055  9366 catalog_manager.cc:5697] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:00.677006  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:00.691229  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:00.695128 20009 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:54:00.697315  9445 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:00.702555  9521 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:00.700081  9555 raft_consensus.cc:2955] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Committing config change with OpId 20.15: config changed from index 11 to 15. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:00.707198 20023 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:00.671942 19802 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:00.709252 19685 raft_consensus.cc:2804] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election won for term 30
I20260808 01:54:00.709297 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:00.714357 19578 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260808 01:54:00.714938 19578 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:00.715322 19578 raft_consensus.cc:3060] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Advancing to term 33
I20260808 01:54:00.722127 20050 raft_consensus.cc:3055] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Stepping down as leader of term 28
I20260808 01:54:00.722658 20050 raft_consensus.cc:740] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:54:00.723598 20050 consensus_queue.cc:260] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:00.724893 20050 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:54:00.729538 19685 raft_consensus.cc:697] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:54:00.709391 19802 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:00.730744 19685 consensus_queue.cc:237] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:00.737135 20064 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260808 01:54:00.662515 20045 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:37234 (request call id 27) took 12043 ms. Trace:
I20260808 01:54:00.738242 19578 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:00.738035 20045 rpcz_store.cc:276] 0808 01:53:48.593315 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:48.701834 (+108519us) service_pool.cc:224] Handling call
0808 01:53:48.788241 (+ 86407us) log.cc:844] Serialized 564 byte log entry
0808 01:53:49.208900 (+420659us) spinlock_profiling.cc:243] Waited 7.35 ms on lock 0x7b5400190120. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d4490c522 00007f5d448ed28a 00007f5d448f4133 00007f5d448f40b9 00007f5d448f4049 00007f5d448f4011 00007f5d448f31dd 00000000003aa554 00000000003aa399 00007f5d4188cdb5
0808 01:53:49.469959 (+261059us) spinlock_profiling.cc:243] Waited 3.73 ms on lock 0x7b1800304ea0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d41806d1a 00007f5d4af9a5d9 00007f5d4188f3a0 00007f5d41889e94 00007f5d41889e05 00007f5d448eceeb 00007f5d44971f43 00007f5d44999b32 00007f5d4998bb39
0808 01:53:49.766747 (+296788us) spinlock_profiling.cc:243] Waited 5.32 ms on lock 0x7b5400190120. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44913f06 00007f5d4490d87a 00007f5d4490caf4 00007f5d448ed28a 00007f5d448f4133 00007f5d448f40b9 00007f5d448f4049 00007f5d448f4011 00007f5d448f31dd 00000000003aa554
0808 01:54:00.632043 (+10865296us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":390254,"raft.queue_time_us":599545,"raft.run_cpu_time_us":9429,"raft.run_wall_time_us":1167774,"spinlock_wait_cycles":37748096,"thread_start_us":110065,"threads_started":1,"wal-append.queue_time_us":209929,"wal-append.run_cpu_time_us":2000,"wal-append.run_wall_time_us":1037579}
I20260808 01:54:00.740694 19913 raft_consensus.cc:2804] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election won for term 30
I20260808 01:54:00.744722 19913 raft_consensus.cc:697] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:54:00.741851 20023 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:54:00.741726 10362 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:54:00.746881 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 48990us
I20260808 01:54:00.742735  9599 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:54:00.754101  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 51806.5us
I20260808 01:54:00.756323  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:00.756601  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:00.757023  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:00.763243  9637 raft_consensus.cc:2468] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 32.
I20260808 01:54:00.764174 19781 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:54:00.790334 19981 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:54:00.813289 19981 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:00.815598  9366 catalog_manager.cc:5697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 23 to 24, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 24 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:00.818871  9366 catalog_manager.cc:5697] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 25 to 27. New cstate: current_term: 27 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:00.749342 19913 consensus_queue.cc:237] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:00.821556  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:54:00.836860  9444 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:00.837805 20014 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:54:00.840984  9552 raft_consensus.cc:2468] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 33.
I20260808 01:54:00.841646  9366 catalog_manager.cc:5697] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 20 to 25, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 25 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:00.849328  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:00.815770 19981 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:00.856210 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:00.844448  9599 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:54:00.869251  9366 catalog_manager.cc:5697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 16 to 25. New cstate: current_term: 25 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:00.861552 20014 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:00.866348 19774 raft_consensus.cc:2804] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Leader election won for term 33
I20260808 01:54:00.868491 19981 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:54:00.871945  9366 catalog_manager.cc:5697] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 21 to 24, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 24 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:00.890455 20014 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:00.881388  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:00.901401 20014 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:00.887346  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 117424us
I20260808 01:54:00.907512  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:54:00.875064  9601 consensus_peers.cc:597] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:00.905319  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:00.909479  9550 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 26 index: 22. Preceding OpId from leader: term: 30 index: 24. (index mismatch)
I20260808 01:54:00.911245  9444 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:54:00.911446  9601 leader_election.cc:336] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:00.912281  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:00.891691 19978 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:54:00.917235  9601 leader_election.cc:336] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:00.902388 19774 raft_consensus.cc:697] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:54:00.919679  9601 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:00.919868  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:00.926882  9366 catalog_manager.cc:5697] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 23 to 25. New cstate: current_term: 25 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:00.914870 19735 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:00.934486  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:54:00.935599  9601 leader_election.cc:336] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:00.936080  9601 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:00.919481 20039 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:54:00.943583 19774 consensus_queue.cc:237] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:00.937443  9366 catalog_manager.cc:5697] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 20 to 24. New cstate: current_term: 24 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:00.926182 19620 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:54:00.923461 20014 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:54:00.945420  9366 catalog_manager.cc:5697] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:00.930037 20050 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260808 01:54:00.937040 19735 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:00.923312 19978 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:00.945080 19785 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:00.954602 19618 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260808 01:54:00.940274 20017 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.026s
I20260808 01:54:00.967307 19578 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:00.968210 19978 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:00.974285 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:00.981873 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:54:00.949115 20009 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:00.962149  9366 catalog_manager.cc:5697] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 26 to 28. New cstate: current_term: 28 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:00.987435  9366 catalog_manager.cc:5697] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 reported cstate change: term changed from 18 to 22, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 22 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:00.993224 10360 raft_consensus.cc:2393] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 33: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:54:01.000581 10363 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 28 index: 23. Preceding OpId from leader: term: 30 index: 24. (index mismatch)
W20260808 01:54:01.009137  9601 leader_election.cc:336] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:01.009776  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:01.003274 20009 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:01.012719 20009 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:01.016386  9524 consensus_peers.cc:597] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:01.025480  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:01.026465  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:01.028138 19781 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:54:01.015623 19852 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.021s
I20260808 01:54:01.040459 19735 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:01.061736  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:01.064724 20009 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:01.065349 20009 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:01.067708 20009 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:01.072564  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:01.076088  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:01.077870  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:01.079607  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:54:01.080755 10334 consensus_peers.cc:597] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:01.082340 20009 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:54:01.083191 19852 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:01.083866 19852 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:01.087081  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:54:01.088799  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 138820us
I20260808 01:54:01.091566  9521 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:01.094746  9521 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:01.098217  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:01.085651 20023 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:01.103247  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:01.102746 19852 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:01.104161 20024 raft_consensus.cc:2804] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Leader election won for term 32
I20260808 01:54:01.106716  9444 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:01.099704 20021 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260808 01:54:01.101449 19978 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260808 01:54:01.122231  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:01.132965  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:01.139186  9636 raft_consensus.cc:2393] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6e7856bed1be43acaa821290d506ee97 in current term 32: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:54:01.136596 20050 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:54:01.108239 20023 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:01.125779 19852 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:54:01.124109 20024 raft_consensus.cc:697] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
W20260808 01:54:01.162371 20075 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260808 01:54:01.174901 20023 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:01.176059  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:01.177026  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:01.179759  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:01.180987 20023 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:54:01.187078 20024 consensus_queue.cc:237] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 30.15, Last appended by leader: 15, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:01.191982 20023 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:01.192803 20023 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:01.196623 10334 consensus_peers.cc:597] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:01.224938  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 3b8e0e8fda6744ad877514b11658b9d9: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:01.236850  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:01.244983 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:01.249410 10363 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260808 01:54:01.250445  9445 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:01.250560 20050 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:01.258882 19852 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:01.252291 20050 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:01.255232 19774 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.026s
I20260808 01:54:01.282015 20050 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:01.251286 20023 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:01.286614  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:01.287374  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:01.292407  9444 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:01.294092 20050 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260808 01:54:01.296945  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet e3814b146ccd472f9304a0be09ad3738: Illegal state: Replica 52ccf4dcca1843f7bb29e9172e2d4ab4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 24 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:01.302722 20003 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:01.303496 20003 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:01.305452  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:01.306115 20003 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:01.308604  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 2f740cd3515641a8b2c4add8f809dd82: Illegal state: Replica 52ccf4dcca1843f7bb29e9172e2d4ab4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 22 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:01.311324  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:01.312376  9636 raft_consensus.cc:2393] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 30: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
W20260808 01:54:01.315735  9524 consensus_peers.cc:597] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:01.319574 20050 raft_consensus.cc:3060] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Advancing to term 32
I20260808 01:54:01.320596 20009 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:54:01.340096 10334 leader_election.cc:336] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:01.323706 19578 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:54:01.357219  7831 thread.cc:527] Waited for 3000ms trying to join with heartbeat (tid 9507)
I20260808 01:54:01.358498 20009 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:01.359165 19578 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:01.360963 20009 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:01.362082 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:01.364009 10334 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:01.364159  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:01.364964  9554 raft_consensus.cc:2393] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 30: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:54:01.365717 20003 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Advancing to term 30
W20260808 01:54:01.366959  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 17e220e6a25747939bdf18f26b732407: Illegal state: Replica 259bcb3f0d254e8b9effac3277e654e0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 32 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:01.362391 19578 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:01.324798 20023 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:01.378086  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:54:01.379591 20080 log.cc:927] Time spent T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260808 01:54:01.380926  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:01.381881 19802 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:01.380419 20023 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:01.381451  9550 raft_consensus.cc:1275] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260808 01:54:01.386335  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:01.386883 19899 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:01.387555 19899 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:01.388263 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:01.387194  9637 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 30.
I20260808 01:54:01.389899 19899 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:01.390733  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:54:01.392385 20042 consensus_queue.cc:1048] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260808 01:54:01.393990 19802 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260808 01:54:01.394496 19802 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:01.395015 19802 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Advancing to term 31
W20260808 01:54:01.400466  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 78744.3us
I20260808 01:54:01.402834  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:01.386147 20023 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:01.405938  9550 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 34: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:54:01.407424  9637 raft_consensus.cc:1275] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260808 01:54:01.419157 20050 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260808 01:54:01.427610 19802 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:01.431809 20003 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:01.432303  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:54:01.432994  9637 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:54:01.433199 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:54:01.433719 19802 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 election: Requested vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:01.434986 10363 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 30.
I20260808 01:54:01.435312 19574 consensus_queue.cc:1048] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260808 01:54:01.458441  9637 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 31.
I20260808 01:54:01.459668  9444 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:54:01.464116  9601 leader_election.cc:336] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:01.464664  9601 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:54:01.466317  9601 leader_election.cc:336] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:01.466948  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:01.463222 20050 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Leader election won for term 31
I20260808 01:54:01.467700 20009 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:01.467583 20042 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:54:01.469893 19785 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Advancing to term 34
I20260808 01:54:01.488802  9550 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260808 01:54:01.479393 20009 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:01.477725 20050 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 31 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:54:01.498812 20009 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:01.500200 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:01.502372 20050 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:01.504731 19774 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260808 01:54:01.510130 10360 raft_consensus.cc:1275] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 33 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260808 01:54:01.516274  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:01.518317 19899 consensus_queue.cc:1048] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.003s
I20260808 01:54:01.520565  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:01.535019 20009 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:01.541923 20009 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:01.559231 20009 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:01.562319 20009 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:01.576659  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:01.578347  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:54:01.576162  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 74813.6us
W20260808 01:54:01.582316  9524 consensus_peers.cc:597] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:01.585747  9444 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:01.574656 19785 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260808 01:54:01.584932 20023 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:01.594202 20023 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:01.590612 20009 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260808 01:54:01.634166 10334 consensus_peers.cc:597] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:01.637320  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:01.639158 20042 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:01.634435 20023 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:01.644438  9524 consensus_peers.cc:597] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:01.643812  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:01.650471  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:01.655514 20017 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:54:01.663373 20009 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:01.673820 20009 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:01.679088  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:01.679768  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:01.681905  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:01.676255 20009 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:01.677235 20023 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:01.689532 19802 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260808 01:54:01.695218 10334 consensus_peers.cc:597] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:01.701532 20023 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:01.707885 20017 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:01.713343  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:01.714182  9555 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 34.
I20260808 01:54:01.711197 20017 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:01.716723  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:54:01.719051 20009 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260808 01:54:01.722144 20009 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:01.723038 20009 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 34 FOLLOWER]: Advancing to term 35
I20260808 01:54:01.727689 20017 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:01.728593 20023 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:01.731081  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:01.732520  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:01.736446  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:01.737154  9632 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 34.
I20260808 01:54:01.738972  9444 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:54:01.742467 20090 log.cc:927] Time spent T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260808 01:54:01.743003 20023 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:54:01.764045  9636 raft_consensus.cc:1275] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 30 index: 15. Preceding OpId from leader: term: 32 index: 16. (index mismatch)
I20260808 01:54:01.765928 20059 consensus_queue.cc:1048] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:54:01.777732 20009 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:01.786154 20009 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 35 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:01.787614  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:54:01.788122  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:54:01.788602  9555 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 34 FOLLOWER]: Advancing to term 35
I20260808 01:54:01.788987  9636 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 34 FOLLOWER]: Advancing to term 35
I20260808 01:54:01.808527  9367 catalog_manager.cc:5697] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 25 to 27. New cstate: current_term: 27 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:54:01.814173  9367 catalog_manager.cc:5697] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 23 to 29, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 29 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:54:01.816136  9367 catalog_manager.cc:5697] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 23 to 24, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 24 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:01.835217  9367 catalog_manager.cc:5697] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 24 to 26, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 26 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:01.836977  9367 catalog_manager.cc:5697] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 18 to 22. New cstate: current_term: 22 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:54:01.839672  9367 catalog_manager.cc:5697] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 29 to 32, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 32 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:01.842171  9367 catalog_manager.cc:5697] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 25 to 28, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 28 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:54:01.844213  9367 catalog_manager.cc:5697] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 18 to 19, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 19 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:54:01.837741  9636 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 35.
I20260808 01:54:01.855250  9444 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:54:01.856201 20023 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 35 FOLLOWER]: Leader election won for term 35
I20260808 01:54:01.856840 20023 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 35 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:54:01.857995 20023 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:01.846033  9367 catalog_manager.cc:5697] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 23 to 26, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 26 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:54:01.858105  9366 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50064 (request call id 243) took 24907 ms (client timeout 15000 ms). Trace:
I20260808 01:54:01.861685  9367 catalog_manager.cc:5697] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 26 to 28, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 28 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:01.871757  9367 catalog_manager.cc:5697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 01:54:01.863708  9366 rpcz_store.cc:269] 0808 01:53:37.042770 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:51.311483 (+14268713us) service_pool.cc:224] Handling call
0808 01:54:01.857997 (+10546514us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:54:01.142391 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:54:01.175914 (+ 33523us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:54:01.175949 (+    35us) write_op.cc:435] Acquired schema lock
0808 01:54:01.175966 (+    17us) tablet.cc:662] Decoding operations
0808 01:54:01.177678 (+  1712us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:54:01.177770 (+    92us) write_op.cc:641] Partition lock acquired for write op
0808 01:54:01.177797 (+    27us) tablet.cc:685] Acquiring locks for 22 operations
0808 01:54:01.220928 (+ 43131us) tablet.cc:701] Row locks acquired
0808 01:54:01.220950 (+    22us) write_op.cc:260] PREPARE: finished
0808 01:54:01.221100 (+   150us) write_op.cc:270] Start()
0808 01:54:01.221229 (+   129us) write_op.cc:276] Timestamp: P: 1786154041221049 usec, L: 0
0808 01:54:01.221251 (+    22us) op_driver.cc:348] REPLICATION: starting
0808 01:54:01.222003 (+   752us) log.cc:844] Serialized 8660 byte log entry
0808 01:54:01.460278 (+238275us) op_driver.cc:464] REPLICATION: finished
0808 01:54:01.535513 (+ 75235us) spinlock_profiling.cc:243] Waited 975 us on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4186fa5c 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d483e98d0
0808 01:54:01.544262 (+  8749us) write_op.cc:301] APPLY: starting
0808 01:54:01.544397 (+   135us) tablet.cc:1367] starting BulkCheckPresence
0808 01:54:01.694623 (+150226us) tablet.cc:1370] finished BulkCheckPresence
0808 01:54:01.694667 (+    44us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:54:01.737317 (+ 42650us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:54:01.737647 (+   330us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=44,key_file_lookups=44,delta_file_lookups=0,mrs_lookups=0
0808 01:54:01.737697 (+    50us) write_op.cc:312] APPLY: finished
0808 01:54:01.738161 (+   464us) log.cc:844] Serialized 196 byte log entry
0808 01:54:01.739091 (+   930us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:54:01.739299 (+   208us) write_op.cc:454] Released schema lock
0808 01:54:01.739361 (+    62us) write_op.cc:341] FINISH: Updating metrics
0808 01:54:01.740176 (+   815us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:54:01.740206 (+    30us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":83591,"apply.run_cpu_time_us":34640,"apply.run_wall_time_us":195908,"cfile_cache_hit":90,"cfile_cache_hit_bytes":96219,"num_ops":22,"prepare.queue_time_us":121429,"prepare.run_cpu_time_us":14360,"prepare.run_wall_time_us":99510,"raft.queue_time_us":162346,"raft.run_cpu_time_us":10088,"raft.run_wall_time_us":87982,"replication_time_us":236769,"spinlock_wait_cycles":2242432,"thread_start_us":301458,"threads_started":4,"wal-append.queue_time_us":52873}]]}
I20260808 01:54:01.887020  9555 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 35.
I20260808 01:54:01.918421 10363 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:54:01.931154 20050 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:01.932317 20050 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:01.979542  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:01.980301  9554 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 30.
I20260808 01:54:01.981979  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:54:01.983263 20009 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260808 01:54:01.983791 20009 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:01.984232 20009 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:54:01.999617  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:02.007007  9637 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 30.
I20260808 01:54:01.992069 20050 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:02.003964  9632 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 27 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260808 01:54:02.021373 20017 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
I20260808 01:54:02.022238 20017 consensus_queue.cc:1243] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Peer 259bcb3f0d254e8b9effac3277e654e0 log is divergent from this leader: its last log entry 27.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
W20260808 01:54:02.025312  9524 consensus_peers.cc:597] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:02.028012  9632 pending_rounds.cc:85] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Aborting all ops after (but not including) 20
I20260808 01:54:02.028712  9632 pending_rounds.cc:107] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: Aborting uncommitted NO_OP operation due to leader change: 27.21
I20260808 01:54:02.029117  9632 raft_consensus.cc:2889] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:54:02.036010 10363 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
W20260808 01:54:02.037432 10334 consensus_peers.cc:597] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:02.038272 20023 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260808 01:54:02.041368 20009 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:02.066099 20050 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:02.066812 20050 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:02.070250 20050 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:02.081562  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:02.082934  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:54:02.083542  9555 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:54:02.082751  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:02.087422  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:02.094592 20009 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:02.096184 20023 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260808 01:54:02.096742  9524 consensus_peers.cc:597] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:02.106532  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:54:02.107102 20050 raft_consensus.cc:493] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:02.107635  9637 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:54:02.097743  9555 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 31.
I20260808 01:54:02.107791 20050 raft_consensus.cc:515] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:02.116294 20050 leader_election.cc:290] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:02.122017  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:02.123801  9444 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:54:02.126178  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0ea307d9adc442539683547d0176a05b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:02.129959  9444 leader_election.cc:304] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:02.130735  9637 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 31.
I20260808 01:54:02.132799 20017 raft_consensus.cc:2749] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:54:02.127552 20023 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Leader election won for term 31
I20260808 01:54:02.133984 20023 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 31 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:54:02.140158 20023 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:02.144383 19852 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:02.145048 19852 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:02.149545 19852 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:02.155555  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:02.156193  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:02.159576  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:02.160588 19852 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260808 01:54:02.174983  9362 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 52406.7us
W20260808 01:54:02.175741  9601 consensus_peers.cc:597] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:02.180132  9524 consensus_peers.cc:597] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:02.198869  9362 catalog_manager.cc:4755] TS 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): ChangeConfig:REMOVE_PEER RPC failed for tablet 7c508218361c446898fede7c5682fa9e: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:02.207604  9367 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55058 (request call id 135) took 9997 ms. Trace:
I20260808 01:54:02.208168  9367 rpcz_store.cc:276] 0808 01:53:52.219192 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:53:58.942097 (+6722905us) service_pool.cc:224] Handling call
0808 01:54:02.207484 (+3265387us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:54:02.026881 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:54:02.027785 (+   904us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:54:02.027845 (+    60us) write_op.cc:435] Acquired schema lock
0808 01:54:02.027872 (+    27us) tablet.cc:662] Decoding operations
0808 01:54:02.056712 (+ 28840us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:54:02.056837 (+   125us) write_op.cc:641] Partition lock acquired for write op
0808 01:54:02.056886 (+    49us) tablet.cc:685] Acquiring locks for 11 operations
0808 01:54:02.069840 (+ 12954us) tablet.cc:701] Row locks acquired
0808 01:54:02.069860 (+    20us) write_op.cc:260] PREPARE: finished
0808 01:54:02.070040 (+   180us) write_op.cc:270] Start()
0808 01:54:02.070172 (+   132us) write_op.cc:276] Timestamp: P: 1786154042069980 usec, L: 0
0808 01:54:02.070189 (+    17us) op_driver.cc:348] REPLICATION: starting
0808 01:54:02.090326 (+ 20137us) log.cc:844] Serialized 4363 byte log entry
0808 01:54:02.132234 (+ 41908us) op_driver.cc:464] REPLICATION: finished
0808 01:54:02.138002 (+  5768us) write_op.cc:301] APPLY: starting
0808 01:54:02.138102 (+   100us) tablet.cc:1367] starting BulkCheckPresence
0808 01:54:02.151220 (+ 13118us) tablet.cc:1370] finished BulkCheckPresence
0808 01:54:02.151342 (+   122us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:54:02.186439 (+ 35097us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:54:02.186764 (+   325us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=22,key_file_lookups=22,delta_file_lookups=0,mrs_lookups=0
0808 01:54:02.186811 (+    47us) write_op.cc:312] APPLY: finished
0808 01:54:02.187148 (+   337us) log.cc:844] Serialized 105 byte log entry
0808 01:54:02.187330 (+   182us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:54:02.187492 (+   162us) write_op.cc:454] Released schema lock
0808 01:54:02.187547 (+    55us) write_op.cc:341] FINISH: Updating metrics
0808 01:54:02.189758 (+  2211us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:54:02.189808 (+    50us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":26649,"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":1389,"apply.run_cpu_time_us":16847,"apply.run_wall_time_us":49986,"cfile_cache_hit":44,"cfile_cache_hit_bytes":46046,"num_ops":11,"prepare.queue_time_us":128492,"prepare.run_cpu_time_us":7014,"prepare.run_wall_time_us":93002,"replication_time_us":49317,"spinlock_wait_cycles":217344,"thread_start_us":120120,"threads_started":1}]]}
I20260808 01:54:02.216943 20017 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:02.217686 20017 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:02.221592  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:02.222337  9554 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 29.
I20260808 01:54:02.224565  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:54:02.225531  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:02.226532 20050 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:54:02.227100 20050 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:02.228988 20050 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:54:02.236069  9369 catalog_manager.cc:5697] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 25 to 29. New cstate: current_term: 29 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:02.224552 20017 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:02.240476  9369 catalog_manager.cc:5697] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 23 to 26. New cstate: current_term: 26 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:02.242413 20017 raft_consensus.cc:493] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:02.243224 20017 raft_consensus.cc:515] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:02.245842 20017 leader_election.cc:290] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:02.247645  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:02.248286  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3814b146ccd472f9304a0be09ad3738" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:02.261170 19785 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:02.262034 19785 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:54:02.265205  9601 consensus_peers.cc:597] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:02.251209 20050 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:02.267977 20050 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:02.269484  9444 leader_election.cc:304] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:02.270898 19852 raft_consensus.cc:2749] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:54:02.272302  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:02.273375  9555 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 29: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:54:02.274394  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:54:02.275257  9554 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:54:02.290913 19785 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:02.276896  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:54:02.284538  9524 consensus_peers.cc:597] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:02.297504  9635 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Advancing to term 30
W20260808 01:54:02.313970  9601 consensus_peers.cc:597] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:02.317235  9369 catalog_manager.cc:5697] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 26 to 33. New cstate: current_term: 33 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:54:02.321316  9524 consensus_peers.cc:597] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:02.321632  9369 catalog_manager.cc:5697] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 24 to 27, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 27 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:02.339859  9369 catalog_manager.cc:5697] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 24 to 27, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 27 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:02.342876  9369 catalog_manager.cc:5697] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 22 to 24, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 24 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:02.336475  9635 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 30.
I20260808 01:54:02.329926 20050 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:02.348126 20050 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:02.347851  9369 catalog_manager.cc:5697] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 24 to 28. New cstate: current_term: 28 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:02.349530  9369 catalog_manager.cc:5697] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 21 to 27, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 27 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:02.351046  9444 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97; no voters: 
I20260808 01:54:02.351261  9369 catalog_manager.cc:5697] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 24 to 27, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 27 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:02.352306 20017 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader election won for term 30
I20260808 01:54:02.353825 20050 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:02.325635  9601 leader_election.cc:336] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:02.325276  9555 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 35 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
W20260808 01:54:02.359534  7831 thread.cc:527] Waited for 4000ms trying to join with heartbeat (tid 9507)
I20260808 01:54:02.360970  9601 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:02.362049 20042 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:54:02.356554  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:02.357328 20017 raft_consensus.cc:697] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 30 LEADER]: Becoming Leader. State: Replica: 6e7856bed1be43acaa821290d506ee97, State: Running, Role: LEADER
I20260808 01:54:02.373624  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:02.374075 20017 consensus_queue.cc:237] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:02.378171  9444 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:02.379180 20023 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:54:02.325474  9554 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7856bed1be43acaa821290d506ee97 in term 30.
W20260808 01:54:02.362188  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46218.4us
W20260808 01:54:02.381525  9601 consensus_peers.cc:597] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:02.362790 20050 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.000s
I20260808 01:54:02.353058  9369 catalog_manager.cc:5697] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 22 to 25, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 25 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:54:02.392263  9369 catalog_manager.cc:5697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 25 to 30, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 30 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:02.394292  9369 catalog_manager.cc:5697] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 27 to 28, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 28 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:02.397147  9369 catalog_manager.cc:5697] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 17 to 29, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 29 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:02.400678  9369 catalog_manager.cc:5697] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 22 to 28, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 28 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
W20260808 01:54:02.390399  9601 consensus_peers.cc:597] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:02.420559  9601 consensus_peers.cc:597] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:02.421876  9601 consensus_peers.cc:597] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:02.425313  9601 consensus_peers.cc:597] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:02.427523 20017 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:02.428328 20017 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:02.432717 20017 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:02.423674  9369 catalog_manager.cc:5697] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 14 to 31, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 31 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:02.437937  9369 catalog_manager.cc:5697] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 26 to 28. New cstate: current_term: 28 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:54:02.436355 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:02.439759  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:02.453205 20023 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:02.453904 20023 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:02.456740 20023 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:02.457659 20050 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:02.458168 20050 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:02.460878  9444 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:02.462931 20050 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:02.464038 10334 consensus_peers.cc:597] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:02.472527 10334 consensus_peers.cc:597] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260808 01:54:02.444556  9369 catalog_manager.cc:5697] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 24 to 27, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 27 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:02.478312  9369 catalog_manager.cc:5697] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 25 to 28, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 28 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:02.479362 19852 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:54:02.483454 20023 raft_consensus.cc:493] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:02.492252 20023 raft_consensus.cc:515] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:02.492654  9369 catalog_manager.cc:5697] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 22 to 25, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 25 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:02.494575  9369 catalog_manager.cc:5697] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 28 to 30, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 30 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:02.495251 20023 leader_election.cc:290] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:02.500607  9369 catalog_manager.cc:5697] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 27 to 28, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 28 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:02.503284 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:54:02.508253 10334 consensus_peers.cc:597] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:02.512966  9444 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 52169.3us
I20260808 01:54:02.515843  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:02.516387  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:02.515859  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:54:02.522663  9601 consensus_peers.cc:597] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:02.524034  9601 consensus_peers.cc:597] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:02.524837  9601 consensus_peers.cc:597] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:02.534843 10334 consensus_peers.cc:597] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260808 01:54:02.536350  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:02.538868  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "360b0cf59e7749058ca972bb487cbd10" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:02.539915  9369 catalog_manager.cc:5697] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 22 to 26, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 26 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:02.541620  9369 catalog_manager.cc:5697] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 23 to 28, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 28 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:54:02.559927  9601 consensus_peers.cc:597] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:02.567735  9369 catalog_manager.cc:5697] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 23 to 24, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 24 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:02.581198 19852 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:54:02.581935 19852 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:02.584563 19852 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:02.587215  9444 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:02.595750  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:02.597960 20023 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:02.598657 20050 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:02.598762 20023 raft_consensus.cc:493] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:02.599483 20023 raft_consensus.cc:515] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:02.601577 20023 leader_election.cc:290] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:02.602137  9601 consensus_peers.cc:597] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:02.590076 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:02.605062  9444 leader_election.cc:304] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:02.605958 20050 raft_consensus.cc:2749] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:54:02.618044  9636 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 35 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260808 01:54:02.619328  9637 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260808 01:54:02.620747  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:02.620872  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:02.621187 19852 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:02.621694 19852 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:02.622838  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d3a904863b545ffa227b964ee3a3f18" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:02.623951 19852 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:02.630030  9444 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:02.631006 20023 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260808 01:54:02.633037  9524 consensus_peers.cc:597] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:02.633327  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:54:02.633639  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 3b8e0e8fda6744ad877514b11658b9d9: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:02.634956  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:54:02.652148  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 7b5ea129398a42fb9981fca652027cd6: Illegal state: Replica 259bcb3f0d254e8b9effac3277e654e0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 28 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:02.653110 20050 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260808 01:54:02.659934  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 2c617cb141a34007b6c1b8af0d911923: Illegal state: Replica 52ccf4dcca1843f7bb29e9172e2d4ab4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 35 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:02.669044 20023 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260808 01:54:02.669168  9444 leader_election.cc:304] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:02.683495 20023 raft_consensus.cc:2749] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:54:02.687165  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:02.688732 20050 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260808 01:54:02.693457  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet fa45ab4daad84ee78514f98cb3738f23: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:54:02.713375  9524 consensus_peers.cc:597] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:02.713609  9554 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260808 01:54:02.725569 20050 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260808 01:54:02.758265 20050 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:02.760859 20050 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:02.767648 20050 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:02.771553  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:02.772740  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:02.775243  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:02.777559 20023 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260808 01:54:02.798019  9555 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260808 01:54:02.799816 20023 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260808 01:54:02.833940 20023 raft_consensus.cc:493] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:54:02.834609 20023 raft_consensus.cc:515] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:02.839347 20023 leader_election.cc:290] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:02.840236  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:02.848176 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57f5706d264426eb123cb26c6f3b89c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:54:02.851019  9524 consensus_peers.cc:597] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:02.856746  9445 leader_election.cc:304] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:02.861688 19852 raft_consensus.cc:2749] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260808 01:54:02.860143  9601 consensus_peers.cc:597] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:02.869655  9637 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Refusing update from remote peer 6e7856bed1be43acaa821290d506ee97: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260808 01:54:02.871979 20050 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260808 01:54:02.955735 10334 consensus_peers.cc:597] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260808 01:54:02.958107 20023 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:54:02.958899 20023 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:02.969866  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:02.976464 20023 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:02.978029 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:02.980152  9445 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:02.981585 20050 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260808 01:54:03.002348 20023 raft_consensus.cc:493] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:03.003116 20023 raft_consensus.cc:515] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:03.005760 20023 leader_election.cc:290] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:03.007793 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:03.008494  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cf7e7eadc2347aa84c98413849ebfc9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:03.010722 20023 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:54:03.011384 20023 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:03.016086 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:03.016640  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:03.019996  9444 leader_election.cc:304] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:03.021229 19852 raft_consensus.cc:2749] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:03.021422  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:03.025146 20050 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:03.026383 20023 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:03.072640 20023 raft_consensus.cc:493] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:03.073274 20023 raft_consensus.cc:515] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:03.076555  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:03.086229  9369 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50084 (request call id 239) took 5397 ms. Trace:
I20260808 01:54:03.086799  9369 rpcz_store.cc:276] 0808 01:53:57.688912 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:54:00.361317 (+2672405us) service_pool.cc:224] Handling call
0808 01:54:03.086145 (+2724828us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:54:02.668518 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:54:02.669086 (+   568us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:54:02.669119 (+    33us) write_op.cc:435] Acquired schema lock
0808 01:54:02.669136 (+    17us) tablet.cc:662] Decoding operations
0808 01:54:02.671051 (+  1915us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:54:02.671155 (+   104us) write_op.cc:641] Partition lock acquired for write op
0808 01:54:02.671187 (+    32us) tablet.cc:685] Acquiring locks for 24 operations
0808 01:54:02.673484 (+  2297us) tablet.cc:701] Row locks acquired
0808 01:54:02.673511 (+    27us) write_op.cc:260] PREPARE: finished
0808 01:54:02.673823 (+   312us) write_op.cc:270] Start()
0808 01:54:02.673953 (+   130us) write_op.cc:276] Timestamp: P: 1786154042673771 usec, L: 0
0808 01:54:02.673976 (+    23us) op_driver.cc:348] REPLICATION: starting
0808 01:54:02.674891 (+   915us) log.cc:844] Serialized 9447 byte log entry
0808 01:54:02.747935 (+ 73044us) op_driver.cc:464] REPLICATION: finished
0808 01:54:02.786478 (+ 38543us) write_op.cc:301] APPLY: starting
0808 01:54:02.786577 (+    99us) tablet.cc:1367] starting BulkCheckPresence
0808 01:54:02.872214 (+ 85637us) tablet.cc:1370] finished BulkCheckPresence
0808 01:54:02.872323 (+   109us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:54:02.940668 (+ 68345us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:54:02.940997 (+   329us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=48,key_file_lookups=48,delta_file_lookups=0,mrs_lookups=0
0808 01:54:02.941041 (+    44us) write_op.cc:312] APPLY: finished
0808 01:54:02.941526 (+   485us) log.cc:844] Serialized 212 byte log entry
0808 01:54:02.958926 (+ 17400us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:54:02.959192 (+   266us) write_op.cc:454] Released schema lock
0808 01:54:02.959274 (+    82us) write_op.cc:341] FINISH: Updating metrics
0808 01:54:03.004217 (+ 44943us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:54:03.004264 (+    47us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":38242,"apply.run_cpu_time_us":35237,"apply.run_wall_time_us":217703,"cfile_cache_hit":98,"cfile_cache_hit_bytes":104591,"num_ops":24,"prepare.queue_time_us":33073,"prepare.run_cpu_time_us":6657,"prepare.run_wall_time_us":16410,"replication_time_us":73764,"thread_start_us":61653,"threads_started":2}]]}
I20260808 01:54:03.080943  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3759209338e14437aff5f92a2fff36da" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:03.091928 20023 leader_election.cc:290] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:03.108260 19852 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:03.111917  9444 leader_election.cc:304] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:03.111763 19852 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:03.115960 19852 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:03.116765 19852 raft_consensus.cc:2749] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:54:03.119216  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:03.119868  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "030eacff61b14455978d2f0de262a089" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:03.121901  9444 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:03.123345 19852 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260808 01:54:03.129752 10334 consensus_peers.cc:597] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:03.196147  9524 consensus_peers.cc:597] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:03.206396  9370 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50084 (request call id 235) took 109747 ms (client timeout 15000 ms). Trace:
W20260808 01:54:03.207026  9370 rpcz_store.cc:269] 0808 01:52:13.459356 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:13.463575 (+  4219us) service_pool.cc:224] Handling call
0808 01:54:03.206140 (+109742565us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":51,"spinlock_wait_cycles":7424}
W20260808 01:54:03.211596  9601 consensus_peers.cc:597] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:03.217859  9369 catalog_manager.cc:5697] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 24 to 29, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 29 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:54:03.221282  9369 catalog_manager.cc:5697] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 21 to 27. New cstate: current_term: 27 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:03.223924  9369 catalog_manager.cc:5697] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:03.230476  9369 catalog_manager.cc:5697] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 11 to 15, term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:03.233294  9369 catalog_manager.cc:5697] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 26 to 30, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 30 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:03.236491  9369 catalog_manager.cc:5697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 24 to 31, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 31 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:03.238629  9369 catalog_manager.cc:5697] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 26 to 27, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 27 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:54:03.241921  9369 catalog_manager.cc:5697] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 26 to 33, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 33 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:54:03.244241  9369 catalog_manager.cc:5697] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 32 to 33, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 33 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:03.247030  9369 catalog_manager.cc:5697] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 24 to 26, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 26 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:03.248329  9369 catalog_manager.cc:5697] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 15 to 21. New cstate: current_term: 21 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:03.250030  9369 catalog_manager.cc:5697] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 28 to 30. New cstate: current_term: 30 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:03.251314  9369 catalog_manager.cc:5697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 22 to 23, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 23 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:03.253571  9369 catalog_manager.cc:5697] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 18 to 21, leader changed from 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 21 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:54:03.287688  9601 consensus_peers.cc:597] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:03.360066  7831 thread.cc:527] Waited for 5000ms trying to join with heartbeat (tid 9507)
W20260808 01:54:03.390389  9362 catalog_manager.cc:4755] TS 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): ChangeConfig:REMOVE_PEER RPC failed for tablet 7c508218361c446898fede7c5682fa9e: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:03.416013 20023 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:03.416718 20023 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:03.422938  9369 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:55058 (request call id 136) took 1098 ms. Trace:
I20260808 01:54:03.426397  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:03.427450 20023 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:03.429032 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:03.430940  9369 rpcz_store.cc:276] 0808 01:54:02.324939 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:54:03.138930 (+813991us) service_pool.cc:224] Handling call
0808 01:54:03.422835 (+283905us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:54:03.295282 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:54:03.295937 (+   655us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:54:03.295978 (+    41us) write_op.cc:435] Acquired schema lock
0808 01:54:03.295999 (+    21us) tablet.cc:662] Decoding operations
0808 01:54:03.297509 (+  1510us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:54:03.297607 (+    98us) write_op.cc:641] Partition lock acquired for write op
0808 01:54:03.297643 (+    36us) tablet.cc:685] Acquiring locks for 14 operations
0808 01:54:03.299724 (+  2081us) tablet.cc:701] Row locks acquired
0808 01:54:03.299749 (+    25us) write_op.cc:260] PREPARE: finished
0808 01:54:03.299944 (+   195us) write_op.cc:270] Start()
0808 01:54:03.300073 (+   129us) write_op.cc:276] Timestamp: P: 1786154043299892 usec, L: 0
0808 01:54:03.300094 (+    21us) op_driver.cc:348] REPLICATION: starting
0808 01:54:03.305786 (+  5692us) log.cc:844] Serialized 5575 byte log entry
0808 01:54:03.331167 (+ 25381us) op_driver.cc:464] REPLICATION: finished
0808 01:54:03.331873 (+   706us) write_op.cc:301] APPLY: starting
0808 01:54:03.331993 (+   120us) tablet.cc:1367] starting BulkCheckPresence
0808 01:54:03.345474 (+ 13481us) tablet.cc:1370] finished BulkCheckPresence
0808 01:54:03.345702 (+   228us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:54:03.393071 (+ 47369us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:54:03.393304 (+   233us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=0,mrs_lookups=0
0808 01:54:03.393378 (+    74us) write_op.cc:312] APPLY: finished
0808 01:54:03.393765 (+   387us) log.cc:844] Serialized 129 byte log entry
0808 01:54:03.399037 (+  5272us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:54:03.399496 (+   459us) write_op.cc:454] Released schema lock
0808 01:54:03.399564 (+    68us) write_op.cc:341] FINISH: Updating metrics
0808 01:54:03.419368 (+ 19804us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:54:03.419401 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":409,"apply.run_cpu_time_us":21164,"apply.run_wall_time_us":87433,"cfile_cache_hit":56,"cfile_cache_hit_bytes":58604,"num_ops":14,"prepare.queue_time_us":18449,"prepare.run_cpu_time_us":5191,"prepare.run_wall_time_us":17254,"replication_time_us":30879,"thread_start_us":17837,"threads_started":1}]]}
I20260808 01:54:03.429324  9370 catalog_manager.cc:5697] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 25 to 30. New cstate: current_term: 30 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:03.435317  9445 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:03.436982  9370 catalog_manager.cc:5697] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 25 to 31, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 31 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:03.438776 20023 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:03.439791 20017 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:03.441756 20017 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:03.444283 20017 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:03.445322  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:03.446826  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:03.457947  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:03.460556 20017 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:54:03.446851  9370 catalog_manager.cc:5697] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 28 to 31, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 31 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:03.464186  9370 catalog_manager.cc:5697] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 22 to 25, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 25 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:03.469547  9370 catalog_manager.cc:5697] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 24 to 32, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 32 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:03.472257  9370 catalog_manager.cc:5697] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 27 to 29. New cstate: current_term: 29 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:03.473794  9370 catalog_manager.cc:5697] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 24 to 29, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 29 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:03.475301  9370 catalog_manager.cc:5697] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 21 to 30, leader changed from 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 30 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
I20260808 01:54:03.478607  9370 catalog_manager.cc:5697] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 26 to 35, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 6e7856bed1be43acaa821290d506ee97 (127.7.165.193). New cstate: current_term: 35 leader_uuid: "6e7856bed1be43acaa821290d506ee97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } }
W20260808 01:54:03.519436  9524 consensus_peers.cc:597] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:03.540326 10334 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 355 similar messages]
I20260808 01:54:03.582197 20017 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:03.582821 20017 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:03.585139 20017 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:03.587649  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:03.594954  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:03.599658  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:03.601068 20023 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:54:03.665637 20017 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:03.666316 20017 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:03.668776 20017 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:03.671305  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:03.673594  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:03.675711  9444 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:03.677343 20023 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:54:03.708709 20017 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:03.709391 20017 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:03.715224 20017 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:03.717543  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:03.717515  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:03.720304  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:03.721843 20017 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260808 01:54:03.755271  9524 consensus_peers.cc:597] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:03.788488  9371 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50084 (request call id 234) took 146260 ms (client timeout 15000 ms). Trace:
W20260808 01:54:03.789033  9371 rpcz_store.cc:269] 0808 01:51:37.528014 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:51:37.528185 (+   171us) service_pool.cc:224] Handling call
0808 01:54:03.788207 (+146260022us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55096,"reported_tablets":51}
W20260808 01:54:03.824587 10334 consensus_peers.cc:597] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:03.899663  9372 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50068 (request call id 240) took 121449 ms (client timeout 15000 ms). Trace:
W20260808 01:54:03.900266  9372 rpcz_store.cc:269] 0808 01:52:02.514766 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:02.515104 (+   338us) service_pool.cc:224] Handling call
0808 01:52:02.787280 (+272176us) spinlock_profiling.cc:243] Waited 6.35 ms on lock 0x7b5800001720. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d41855a62 00007f5d4a821df9 00007f5d4a7bad56 00007f5d4a78c248 00007f5d4a888bfe 00007f5d48d424a4 00007f5d48d42432 00007f5d48d42361 00007f5d48d422dc
0808 01:54:03.898864 (+121111584us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27534,"reported_tablets":51,"spinlock_wait_cycles":14850816}
W20260808 01:54:03.963165  9524 consensus_peers.cc:597] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260808 01:54:04.060369 20050 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:04.061064 20050 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:04.066143 20050 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:04.066601 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:04.070508  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "626249dbd6eb4d66bfa8cce2e716cbe2" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 22 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:04.074285  9444 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:04.076355 20023 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:54:04.098409  9373 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50084 (request call id 236) took 80465 ms (client timeout 15000 ms). Trace:
W20260808 01:54:04.102087  9373 rpcz_store.cc:269] 0808 01:52:43.643108 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:43.643452 (+   344us) service_pool.cc:224] Handling call
0808 01:54:04.098166 (+80454714us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":109588,"reported_tablets":51}
I20260808 01:54:04.191212 20050 raft_consensus.cc:493] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:04.191814 20050 raft_consensus.cc:515] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:04.197489  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:04.202425  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f740cd3515641a8b2c4add8f809dd82" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:54:04.207036 10334 consensus_peers.cc:597] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:04.194317 20050 leader_election.cc:290] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:04.210008  9444 leader_election.cc:304] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:04.217764 20017 raft_consensus.cc:2749] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260808 01:54:04.227447  9524 consensus_peers.cc:597] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:04.360715  7831 thread.cc:527] Waited for 6000ms trying to join with heartbeat (tid 9507)
I20260808 01:54:04.418320  9362 catalog_manager.cc:5244] ChangeConfig:REMOVE_PEER RPC for tablet 7c508218361c446898fede7c5682fa9e with cas_config_opid_index 11: aborting the task: latest config opid_index 15; task opid_index 11
I20260808 01:54:04.873842 20123 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:04.875854 20123 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:04.906527 20123 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:04.942265 20130 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:04.969892 20132 raft_consensus.cc:493] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:04.970630 20132 raft_consensus.cc:515] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:04.974198 20132 leader_election.cc:290] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:04.956637 20130 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:04.983109 20130 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:05.009819  9426 auto_rebalancer.cc:644] Failed to schedule move for tablet 924b85a2ee7e46ecbcfd57257902abef: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:05.012248 20140 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:05.019742 20140 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:05.018043 20123 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:05.023381 20123 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:05.025084  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:05.025586 20123 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:05.018850 19852 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:05.026901 19852 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:05.027467 20133 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:05.028050 20133 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:05.029260 19852 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:05.029421  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:05.030447 20133 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:05.010243  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:05.022390 20140 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:05.034310  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:05.036456 20133 raft_consensus.cc:493] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:05.037181 20133 raft_consensus.cc:515] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:05.040635  9554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:05.046386 20133 leader_election.cc:290] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:05.063282  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 33 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:05.076777 20017 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:05.077540 20017 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:05.080607 10360 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:05.087409  9637 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:54:05.090229 20017 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:05.091259 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:05.092032 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:05.097399  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a4f92418934341a6ac4927db61dc87" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:05.104189  9445 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:05.112059 20136 raft_consensus.cc:2749] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:54:05.112999 20136 raft_consensus.cc:493] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:05.113582 20136 raft_consensus.cc:515] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:05.116333 20136 leader_election.cc:290] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:05.117214  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:05.126040  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:54:05.141551 10334 consensus_peers.cc:597] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:05.142258 10334 consensus_peers.cc:597] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:05.143105 10334 consensus_peers.cc:597] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260808 01:54:05.163718  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17e220e6a25747939bdf18f26b732407" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 33 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:05.165506  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90465c2f160641a7901ab0815e723ce1" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:05.206225 20141 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:05.206975 20141 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:05.209997 20141 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:05.217603  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:05.217877 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:05.220450  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9949664c54584858ae8fc74c9dc07a4a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:05.225020  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:05.227682  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03b17f21031f4ff4828c85b7332fcc2d" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:05.235836  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "439040d168824496acc076b1c280bde3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:05.238081  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b4331dfdd5b43448a400f2eb7b1be70" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:54:05.251746  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 49124.3us
W20260808 01:54:05.252007  9409 debug-util.cc:398] Leaking SignalData structure 0x7b0800348640 after lost signal to thread 7834
W20260808 01:54:05.253360  9409 debug-util.cc:398] Leaking SignalData structure 0x7b0800348780 after lost signal to thread 9412
W20260808 01:54:05.254101  9409 debug-util.cc:398] Leaking SignalData structure 0x7b08002b1800 after lost signal to thread 9506
W20260808 01:54:05.254406  9409 debug-util.cc:398] Leaking SignalData structure 0x7b08002a91e0 after lost signal to thread 9507
W20260808 01:54:05.256121  9409 debug-util.cc:398] Leaking SignalData structure 0x7b0800053f40 after lost signal to thread 9585
W20260808 01:54:05.256465  9409 debug-util.cc:398] Leaking SignalData structure 0x7b0800053f60 after lost signal to thread 9586
W20260808 01:54:05.258461  9409 debug-util.cc:398] Leaking SignalData structure 0x7b08000402c0 after lost signal to thread 9663
W20260808 01:54:05.260358  9409 debug-util.cc:398] Leaking SignalData structure 0x7b08000f9800 after lost signal to thread 10393
I20260808 01:54:05.262063 20144 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:05.262830 20144 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:05.265543 20144 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:05.266886 20143 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:05.267606 20143 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:54:05.271880  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 73234.8us
I20260808 01:54:05.277420  9444 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:05.279006 20121 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:54:05.289582  9444 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:05.291447  9444 leader_election.cc:304] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:05.292814  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b335bc0170854e26a77dbec13dbe9eac" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:05.295141 20017 raft_consensus.cc:2749] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:05.295818 20144 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:05.295125 20121 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260808 01:54:05.296326 20122 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:05.297247 20122 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:05.298161 20144 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:05.299834 20122 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:05.300810 20144 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:54:05.262168  9374 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50068 (request call id 238) took 190130 ms (client timeout 15000 ms). Trace:
I20260808 01:54:05.307806  9444 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:54:05.306728  9374 rpcz_store.cc:269] 0808 01:50:55.132078 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:50:55.132370 (+   292us) service_pool.cc:224] Handling call
0808 01:54:05.261920 (+190129550us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23481,"reported_tablets":51}
I20260808 01:54:05.313308  9444 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:54:05.313771  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 51267.2us
I20260808 01:54:05.315659  9444 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:05.316697 20121 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:05.321470 20136 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260808 01:54:05.322280  9524 consensus_peers.cc:597] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:05.324759  9524 consensus_peers.cc:597] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:05.325714  9524 leader_election.cc:336] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:54:05.328125  9377 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50068 (request call id 241) took 90362 ms (client timeout 15000 ms). Trace:
W20260808 01:54:05.328653  9377 rpcz_store.cc:269] 0808 01:52:34.968076 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:34.971782 (+  3706us) service_pool.cc:224] Handling call
0808 01:54:05.327857 (+90356075us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":309822,"reported_tablets":51}
I20260808 01:54:05.330713  9444 leader_election.cc:304] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:05.332113 20136 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:05.334347 20136 raft_consensus.cc:2749] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:05.335032  9444 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:05.336032 20136 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260808 01:54:05.336761  9444 leader_election.cc:304] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:05.338367 20136 raft_consensus.cc:2749] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:54:05.343411 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:05.348418 19578 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:05.349102 19578 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:05.349948 19785 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:05.350608 19785 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:54:05.351712  9600 consensus_peers.cc:597] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c485e8f4266462e8413079adcf8faef. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:05.352308 20143 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:05.352628 19578 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:05.354252 19785 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:05.361485  7831 thread.cc:527] Waited for 7000ms trying to join with heartbeat (tid 9507)
W20260808 01:54:05.364908  9524 consensus_peers.cc:597] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:05.403285  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 53896.5us
I20260808 01:54:05.409142  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "460e1059f5504707b891c4b0e0ac6a09" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:05.412010  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:54:05.424592  9524 consensus_peers.cc:597] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260808 01:54:05.456333 20121 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:54:05.456987 20121 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:05.460464 20122 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:05.461297 20122 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:05.464684 20122 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:05.467317  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:05.474625  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0db7b84c3b040c788e0a3f66657d912" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:05.479048 20122 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:05.479970 20122 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:05.481329 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:05.482334 20122 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:05.491519 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:05.493454  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:54:05.504175  9368 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50064 (request call id 241) took 78290 ms (client timeout 15000 ms). Trace:
W20260808 01:54:05.504992  9598 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:54:05.511039  9557 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260808 01:54:05.511827 20121 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:54:05.504696  9368 rpcz_store.cc:269] 0808 01:52:47.214018 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:52:47.214211 (+   193us) service_pool.cc:224] Handling call
0808 01:54:05.503923 (+78289712us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15913,"reported_tablets":51}
I20260808 01:54:05.514993  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bcbe89d624b472f8be5abc6b34c9026" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:05.516820  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:05.506253  9444 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:05.519865 20017 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:05.519569 20122 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:54:05.521202 20122 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:05.531422  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:05.535570  9552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b5ea129398a42fb9981fca652027cd6" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:05.543929  9599 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:05.546764 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:05.553896 20149 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:54:05.565518 19785 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260808 01:54:05.553418 20122 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:05.579299  9366 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50064 (request call id 244) took 7163 ms. Trace:
I20260808 01:54:05.588025  9521 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:54:05.591687 10334 consensus_peers.cc:597] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:05.594209 20142 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:54:05.597378  9521 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:05.598902 20142 raft_consensus.cc:2749] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:54:05.605530  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:05.611045 20122 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:54:05.606160  9366 rpcz_store.cc:276] 0808 01:53:58.415993 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:54:01.881697 (+3465704us) service_pool.cc:224] Handling call
0808 01:54:05.579221 (+3697524us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37408,"reported_tablets":51}
W20260808 01:54:05.610140  9601 consensus_peers.cc:597] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:05.618757  9601 consensus_peers.cc:597] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:05.619905  9601 consensus_peers.cc:597] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:05.635507 10334 consensus_peers.cc:597] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:05.642189  7831 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260808 01:54:05.646481  7831 tablet_replica.cc:333] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:05.651453  9367 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:50068 (request call id 245) took 7047 ms. Trace:
I20260808 01:54:05.651942  9367 rpcz_store.cc:276] 0808 01:53:58.603883 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:54:02.212335 (+3608452us) service_pool.cc:224] Handling call
0808 01:54:05.651383 (+3439048us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":51}
I20260808 01:54:05.678342 20023 raft_consensus.cc:493] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:05.679008 20023 raft_consensus.cc:515] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:05.681322 20023 leader_election.cc:290] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:05.683089 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:05.684500  7831 raft_consensus.cc:2243] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:05.684952  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3354328ed023401e83a2361925b02e81" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 27 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:05.685513  7831 raft_consensus.cc:2272] T 460e1059f5504707b891c4b0e0ac6a09 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:05.687793  9445 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:54:05.689671 10334 consensus_peers.cc:597] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:05.699980  7831 tablet_replica.cc:333] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:05.701220  7831 raft_consensus.cc:2243] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:05.701956  7831 raft_consensus.cc:2272] T 03b17f21031f4ff4828c85b7332fcc2d P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:05.701890 20023 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260808 01:54:05.708236  9601 leader_election.cc:336] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:05.708787  9601 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:54:05.710112  9601 leader_election.cc:336] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:05.710565  9601 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:54:05.714280  9524 consensus_peers.cc:597] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260808 01:54:05.715473 20144 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:54:05.716333 20144 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:54:05.722167 20023 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:05.723276 20023 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:54:05.724252  9601 leader_election.cc:336] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:05.729117  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:05.724834  9601 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:05.730703  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:05.732530  9444 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:05.733408 20121 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260808 01:54:05.735126 20149 raft_consensus.cc:2955] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Committing config change with OpId 24.17: config changed from index -1 to 17, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:54:05.736326 20023 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:05.738978  9552 raft_consensus.cc:2955] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Committing config change with OpId 24.17: config changed from index -1 to 17, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:54:05.740881  7831 tablet_replica.cc:333] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:05.741986  7831 raft_consensus.cc:2243] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:05.736310 20144 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:54:05.742898  7831 raft_consensus.cc:2272] T 7d3a904863b545ffa227b964ee3a3f18 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:05.744740  9601 leader_election.cc:336] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:05.745282  9601 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:05.746081 20144 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:54:05.746258  7831 tablet_replica.cc:333] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
W20260808 01:54:05.746542  9601 consensus_peers.cc:597] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:05.751951 20023 raft_consensus.cc:493] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:05.752696 20023 raft_consensus.cc:515] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:05.755578 20023 leader_election.cc:290] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:05.756379 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:05.764223  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c094b8e92d5e449aa2641c359d46d622" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:05.766197  9444 leader_election.cc:304] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:05.767632 20023 raft_consensus.cc:2749] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:05.767552  7831 raft_consensus.cc:2243] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 28 LEADER]: Raft consensus shutting down.
I20260808 01:54:05.769830  7831 raft_consensus.cc:2272] T ea7d6b03a61944ea8a8201984886f30a P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:05.773164  7831 tablet_replica.cc:333] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:05.774168  7831 raft_consensus.cc:2243] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:05.795288  7831 raft_consensus.cc:2272] T 0b4331dfdd5b43448a400f2eb7b1be70 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:05.796716  9366 catalog_manager.cc:5697] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index -1 to 17, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 24 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:05.802413  7831 tablet_replica.cc:333] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
W20260808 01:54:05.783272  9601 leader_election.cc:336] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:05.785552  9557 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 30.15, Last appended by leader: 14, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:54:05.806489  7831 raft_consensus.cc:2243] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:05.816835  7831 raft_consensus.cc:2272] T 5cf7e7eadc2347aa84c98413849ebfc9 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:05.825387  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:05.828460  7831 tablet_replica.cc:333] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
W20260808 01:54:05.834992  9523 consensus_peers.cc:597] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 145c5a9982a24c8fa4e86d0d99cbbdfb. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:05.836804  7831 raft_consensus.cc:2243] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:05.838549  7831 raft_consensus.cc:2272] T 360b0cf59e7749058ca972bb487cbd10 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:05.841190 20149 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260808 01:54:05.840454  9524 consensus_peers.cc:597] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:05.842541  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 59576.8us
I20260808 01:54:05.840883 20154 ts_tablet_manager.cc:933] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:05.849436  7831 tablet_replica.cc:333] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
W20260808 01:54:05.843357  9601 leader_election.cc:336] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:05.851840  9601 leader_election.cc:304] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:05.857249 20154 tablet_copy_client.cc:323] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.195:34565
W20260808 01:54:05.859467  9524 consensus_peers.cc:597] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260808 01:54:05.860042  7831 raft_consensus.cc:2243] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 29 LEADER]: Raft consensus shutting down.
W20260808 01:54:05.863780  9524 consensus_peers.cc:597] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260808 01:54:05.864603 20149 raft_consensus.cc:2749] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:54:05.867520  9637 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 30 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260808 01:54:05.870750  7831 raft_consensus.cc:2272] T 103b913d6c014692b6dea1780af2aa20 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:05.872728 10334 consensus_peers.cc:597] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:05.887975 20142 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260808 01:54:05.898751  7831 tablet_replica.cc:333] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:05.899937  7831 raft_consensus.cc:2243] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:05.907337  7831 raft_consensus.cc:2272] T 90465c2f160641a7901ab0815e723ce1 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:05.920550  7831 tablet_replica.cc:333] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:05.931597  7831 raft_consensus.cc:2243] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:05.937866  7831 raft_consensus.cc:2272] T e3814b146ccd472f9304a0be09ad3738 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:05.941975  7831 tablet_replica.cc:333] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:05.951542  9648 tablet_copy_service.cc:140] P 259bcb3f0d254e8b9effac3277e654e0: Received BeginTabletCopySession request for tablet 4c485e8f4266462e8413079adcf8faef from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:45658)
I20260808 01:54:05.952509  9648 tablet_copy_service.cc:161] P 259bcb3f0d254e8b9effac3277e654e0: Beginning new tablet copy session on tablet 4c485e8f4266462e8413079adcf8faef from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:45658: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-4c485e8f4266462e8413079adcf8faef
W20260808 01:54:05.955720  9524 consensus_peers.cc:597] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260808 01:54:05.988767  9648 tablet_copy_source_session.cc:215] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:54:06.001010 20121 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:06.001963 20121 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:06.002616  7831 raft_consensus.cc:2243] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.008054  7831 raft_consensus.cc:2272] T 3759209338e14437aff5f92a2fff36da P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.012912  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:06.016901 20154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c485e8f4266462e8413079adcf8faef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:06.016947  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4bce6ea2a8244811bf007638181a417a" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:06.023897  9444 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:06.024874 20023 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260808 01:54:06.022166 20121 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:06.061827 20154 tablet_copy_client.cc:806] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:54:06.073714 20154 tablet_copy_client.cc:670] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:54:06.075662  7831 tablet_replica.cc:333] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.077023  7831 raft_consensus.cc:2243] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.077983  7831 pending_rounds.cc:70] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Trying to abort 1 pending ops.
W20260808 01:54:06.083348 10334 consensus_peers.cc:597] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:06.087380  7831 pending_rounds.cc:77] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97: Aborting op as it isn't in flight: id { term: 27 index: 20 } timestamp: 7316086902010331136 op_type: NO_OP noop_request { }
I20260808 01:54:06.088232  7831 raft_consensus.cc:2889] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260808 01:54:06.094928  7831 raft_consensus.cc:2272] T e57f5706d264426eb123cb26c6f3b89c P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.098783  7831 tablet_replica.cc:333] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.100498  7831 raft_consensus.cc:2243] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Raft consensus shutting down.
W20260808 01:54:06.107589  9601 consensus_peers.cc:597] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:06.112214  7831 raft_consensus.cc:2272] T 20a4f92418934341a6ac4927db61dc87 P 6e7856bed1be43acaa821290d506ee97 [term 24 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:06.112450  9601 consensus_peers.cc:597] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:06.124539  9601 consensus_peers.cc:597] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:06.125741  9601 consensus_peers.cc:597] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:06.126499  9601 consensus_peers.cc:597] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:06.132977  7831 tablet_replica.cc:333] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.143842  7831 raft_consensus.cc:2243] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 30 LEADER]: Raft consensus shutting down.
I20260808 01:54:06.149544  7831 raft_consensus.cc:2272] T d2b2a88358244f93970835ea3120c380 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.163471  7831 tablet_replica.cc:333] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.164798  7831 raft_consensus.cc:2243] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.165915  7831 raft_consensus.cc:2272] T 0ea307d9adc442539683547d0176a05b P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.168066 19852 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:06.169099 19852 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:06.172101 19852 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:06.174719  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:06.176555  7831 tablet_replica.cc:333] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.176167 20143 raft_consensus.cc:2955] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Committing config change with OpId 30.16: config changed from index -1 to 16, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:54:06.177733  7831 raft_consensus.cc:2243] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.181376  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affdd24b03ae4ee795a90703613305f7" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:06.187719  9444 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:06.188992 19852 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:54:06.178614  7831 raft_consensus.cc:2272] T b335bc0170854e26a77dbec13dbe9eac P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.204692  7831 tablet_replica.cc:333] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.219702  7831 raft_consensus.cc:2243] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.220434  7831 raft_consensus.cc:2272] T 3354328ed023401e83a2361925b02e81 P 6e7856bed1be43acaa821290d506ee97 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.223847  7831 tablet_replica.cc:333] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.227445  7831 raft_consensus.cc:2243] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.228317  7831 raft_consensus.cc:2272] T 2f740cd3515641a8b2c4add8f809dd82 P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.231644  7831 tablet_replica.cc:333] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.232767  7831 raft_consensus.cc:2243] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.233464  7831 raft_consensus.cc:2272] T 17e220e6a25747939bdf18f26b732407 P 6e7856bed1be43acaa821290d506ee97 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.258623  9632 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:54:06.267454  9635 raft_consensus.cc:2955] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Committing config change with OpId 30.16: config changed from index -1 to 16, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:54:06.269119  7831 tablet_replica.cc:333] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.279224  7831 raft_consensus.cc:2243] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.280048  7831 raft_consensus.cc:2272] T d0db7b84c3b040c788e0a3f66657d912 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.280092  9367 catalog_manager.cc:5697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: config changed from index -1 to 16, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 30 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:06.292001  7831 tablet_replica.cc:333] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.293113  7831 raft_consensus.cc:2243] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.294059  7831 raft_consensus.cc:2272] T affdd24b03ae4ee795a90703613305f7 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.297610  7831 tablet_replica.cc:333] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.298810 20154 tablet_copy_client.cc:538] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:54:06.313619 20154 tablet_bootstrap.cc:492] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:54:06.314154  7831 raft_consensus.cc:2243] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus shutting down.
W20260808 01:54:06.329381  9600 consensus_peers.cc:597] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455): Couldn't send request to peer 7201605bb2f54c949a4ebe6e9b3b4b2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b8e0e8fda6744ad877514b11658b9d9. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:06.334887  9555 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260808 01:54:06.348378  7831 raft_consensus.cc:2272] T 030eacff61b14455978d2f0de262a089 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.348654 19852 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:06.349074 20149 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:54:06.349359 19852 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:06.351727 19852 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:06.352581  7831 tablet_replica.cc:333] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.354249  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:06.357683  7831 raft_consensus.cc:2243] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 35 LEADER]: Raft consensus shutting down.
I20260808 01:54:06.359081  7831 raft_consensus.cc:2272] T 2c617cb141a34007b6c1b8af0d911923 P 6e7856bed1be43acaa821290d506ee97 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.362368  7831 tablet_replica.cc:333] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.363497  7831 raft_consensus.cc:2243] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 29 LEADER]: Raft consensus shutting down.
I20260808 01:54:06.367388  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50fd024764ea49c7a1f16e992e7599f3" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:54:06.342609  9601 consensus_peers.cc:597] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:06.372004  9444 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
W20260808 01:54:06.372757  9601 consensus_peers.cc:597] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:06.371570  7831 raft_consensus.cc:2272] T 2973a134fcef4b8da54331bcd12f1173 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.375659 19852 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:54:06.386662  7831 tablet_replica.cc:333] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.388021  7831 raft_consensus.cc:2243] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.395267  7831 pending_rounds.cc:70] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Trying to abort 1 pending ops.
I20260808 01:54:06.395780  7831 pending_rounds.cc:77] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97: Aborting op as it isn't in flight: id { term: 25 index: 16 } timestamp: 7316086897763536896 op_type: NO_OP noop_request { }
I20260808 01:54:06.396245  7831 raft_consensus.cc:2889] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260808 01:54:06.396704  7831 raft_consensus.cc:2272] T c85ff498df484fc6ac0913aa57aa868b P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.400138  7831 tablet_replica.cc:333] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.401062  7831 raft_consensus.cc:2243] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.402063  7831 raft_consensus.cc:2272] T cab93716156d46d3b0bda8d98d1fc76c P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.408617  7831 tablet_replica.cc:333] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.411445  7831 raft_consensus.cc:2243] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 30 LEADER]: Raft consensus shutting down.
I20260808 01:54:06.412679  7831 raft_consensus.cc:2272] T 244e080726394210b5467bf688813cf0 P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.443503  7831 tablet_replica.cc:333] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.444444  7831 raft_consensus.cc:2243] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.449512 20149 raft_consensus.cc:2955] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 LEADER]: Committing config change with OpId 31.18: config changed from index -1 to 18, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:54:06.445330  7831 raft_consensus.cc:2272] T 32724e13cbcb49e991b11d18699e899a P 6e7856bed1be43acaa821290d506ee97 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.467347  9555 raft_consensus.cc:2955] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Committing config change with OpId 31.18: config changed from index -1 to 18, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:54:06.479280  7831 tablet_replica.cc:333] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.480285  7831 raft_consensus.cc:2243] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.480892  7831 raft_consensus.cc:2272] T 11c99c174e234329801ee9b1ceae68d9 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.483831  7831 tablet_replica.cc:333] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.484661  7831 raft_consensus.cc:2243] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 31 LEADER]: Raft consensus shutting down.
I20260808 01:54:06.485826  7831 raft_consensus.cc:2272] T 7b1e1cf1fbe745feb69372fed0557ce0 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.486720 20163 ts_tablet_manager.cc:933] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:06.492144  9601 consensus_peers.cc:597] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:06.493076  7831 tablet_replica.cc:333] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.494030  7831 raft_consensus.cc:2243] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.494747  7831 raft_consensus.cc:2272] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 6e7856bed1be43acaa821290d506ee97 [term 21 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:06.502998  9601 consensus_peers.cc:597] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:06.512820 20163 tablet_copy_client.cc:323] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.194:36095
I20260808 01:54:06.518558  7831 tablet_replica.cc:333] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.519728  7831 raft_consensus.cc:2243] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 33 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.520720  7831 raft_consensus.cc:2272] T 439040d168824496acc076b1c280bde3 P 6e7856bed1be43acaa821290d506ee97 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.562640  9571 tablet_copy_service.cc:140] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Received BeginTabletCopySession request for tablet 145c5a9982a24c8fa4e86d0d99cbbdfb from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:60552)
I20260808 01:54:06.563373  9571 tablet_copy_service.cc:161] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Beginning new tablet copy session on tablet 145c5a9982a24c8fa4e86d0d99cbbdfb from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:60552: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-145c5a9982a24c8fa4e86d0d99cbbdfb
W20260808 01:54:06.570591  9601 consensus_peers.cc:597] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:06.585057  7831 tablet_replica.cc:333] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.587806  9367 catalog_manager.cc:5697] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: config changed from index -1 to 18, NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) added. New cstate: current_term: 31 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } }
I20260808 01:54:06.586395  7831 raft_consensus.cc:2243] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus shutting down.
W20260808 01:54:06.607522  9601 consensus_peers.cc:597] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:06.608407  9601 consensus_peers.cc:597] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:06.628842  9571 tablet_copy_source_session.cc:215] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:54:06.647675  7831 raft_consensus.cc:2272] T 7b5ea129398a42fb9981fca652027cd6 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:06.611588  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 66301.1us
W20260808 01:54:06.661772 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 110448us
I20260808 01:54:06.671815  7831 tablet_replica.cc:333] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.672873  7831 raft_consensus.cc:2243] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.673413  7831 pending_rounds.cc:70] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Trying to abort 1 pending ops.
I20260808 01:54:06.673676  7831 pending_rounds.cc:77] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97: Aborting op as it isn't in flight: id { term: 30 index: 15 } timestamp: 7316086933131431936 op_type: NO_OP noop_request { }
I20260808 01:54:06.674069  7831 raft_consensus.cc:2889] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260808 01:54:06.674417  7831 raft_consensus.cc:2272] T fa45ab4daad84ee78514f98cb3738f23 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.690940  7831 tablet_replica.cc:333] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.691129 20163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 145c5a9982a24c8fa4e86d0d99cbbdfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:06.691967  7831 raft_consensus.cc:2243] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.692766  7831 raft_consensus.cc:2272] T 487114a18943467b8d3a6db94b17a84d P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.704105  7831 tablet_replica.cc:333] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.704968  7831 raft_consensus.cc:2243] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 22 LEADER]: Raft consensus shutting down.
I20260808 01:54:06.706640  7831 raft_consensus.cc:2272] T 7c508218361c446898fede7c5682fa9e P 6e7856bed1be43acaa821290d506ee97 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.709527  7831 tablet_replica.cc:333] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.725643  7831 raft_consensus.cc:2243] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.726572  7831 pending_rounds.cc:70] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Trying to abort 1 pending ops.
I20260808 01:54:06.730468  7831 pending_rounds.cc:77] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97: Aborting op as it isn't in flight: id { term: 27 index: 15 } timestamp: 7316086898380324864 op_type: NO_OP noop_request { }
I20260808 01:54:06.730897  7831 raft_consensus.cc:2889] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260808 01:54:06.731262  7831 raft_consensus.cc:2272] T ed168d59c43d4bfaa07d104032ad8977 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.734249  7831 tablet_replica.cc:333] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.735126  7831 raft_consensus.cc:2243] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 25 LEADER]: Raft consensus shutting down.
I20260808 01:54:06.736266  7831 raft_consensus.cc:2272] T cacb9622c47c4737beff104125e98b77 P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.742463 20163 tablet_copy_client.cc:806] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:54:06.744199 20163 tablet_copy_client.cc:670] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:54:06.747846  7831 tablet_replica.cc:333] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
W20260808 01:54:06.748858  9524 consensus_peers.cc:597] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:06.752274 20163 tablet_copy_client.cc:538] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:54:06.767104  7831 raft_consensus.cc:2243] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.775559  7831 raft_consensus.cc:2272] T 50fd024764ea49c7a1f16e992e7599f3 P 6e7856bed1be43acaa821290d506ee97 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.778506 20171 ts_tablet_manager.cc:933] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Initiating tablet copy from peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:06.786211  7831 tablet_replica.cc:333] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.796953 20171 tablet_copy_client.cc:323] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Beginning tablet copy session from remote peer at address 127.7.165.195:34565
I20260808 01:54:06.803617  7831 raft_consensus.cc:2243] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.804450  7831 raft_consensus.cc:2272] T 3b8e0e8fda6744ad877514b11658b9d9 P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.804553  9648 tablet_copy_service.cc:140] P 259bcb3f0d254e8b9effac3277e654e0: Received BeginTabletCopySession request for tablet 3b8e0e8fda6744ad877514b11658b9d9 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a ({username='slave'} at 127.0.0.1:45658)
I20260808 01:54:06.805311  9648 tablet_copy_service.cc:161] P 259bcb3f0d254e8b9effac3277e654e0: Beginning new tablet copy session on tablet 3b8e0e8fda6744ad877514b11658b9d9 from peer 7201605bb2f54c949a4ebe6e9b3b4b2a at {username='slave'} at 127.0.0.1:45658: session id = 7201605bb2f54c949a4ebe6e9b3b4b2a-3b8e0e8fda6744ad877514b11658b9d9
I20260808 01:54:06.807955  7831 tablet_replica.cc:333] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.808879  7831 raft_consensus.cc:2243] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.809700  7831 raft_consensus.cc:2272] T c094b8e92d5e449aa2641c359d46d622 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.816926  9648 tablet_copy_source_session.cc:215] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 01:54:06.827469 20171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8e0e8fda6744ad877514b11658b9d9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 01:54:06.827859  9524 consensus_peers.cc:597] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260808 01:54:06.836793  7831 tablet_replica.cc:333] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.853850 20154 tablet_bootstrap.cc:492] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=1 applied=17 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:54:06.837844  7831 raft_consensus.cc:2243] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 29 LEADER]: Raft consensus shutting down.
I20260808 01:54:06.855876  7831 raft_consensus.cc:2272] T c99678e59ec845b28293f811d76f24e8 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.871827  7831 tablet_replica.cc:333] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.873102  7831 raft_consensus.cc:2243] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 23 LEADER]: Raft consensus shutting down.
I20260808 01:54:06.874322  7831 raft_consensus.cc:2272] T 54a2156aecd649d391b0acd6dc223843 P 6e7856bed1be43acaa821290d506ee97 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.883177 20154 tablet_bootstrap.cc:492] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:54:06.884342 20154 ts_tablet_manager.cc:1403] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.571s	user 0.064s	sys 0.018s
I20260808 01:54:06.889529 20163 tablet_bootstrap.cc:492] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:54:06.893309 20131 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:06.894009 20131 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:06.898984 20171 tablet_copy_client.cc:806] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 0 data blocks...
I20260808 01:54:06.904791 20171 tablet_copy_client.cc:670] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Starting download of 1 WAL segments...
I20260808 01:54:06.905141 20154 raft_consensus.cc:359] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:54:06.906139 20154 raft_consensus.cc:740] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
W20260808 01:54:06.910480 10334 leader_election.cc:336] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:06.910750  7831 tablet_replica.cc:333] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:06.911662 20131 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:06.911788  7831 raft_consensus.cc:2243] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:06.912583  7831 pending_rounds.cc:70] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Trying to abort 1 pending ops.
I20260808 01:54:06.912850  7831 pending_rounds.cc:77] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97: Aborting op as it isn't in flight: id { term: 29 index: 17 } timestamp: 7316086908199686144 op_type: NO_OP noop_request { }
I20260808 01:54:06.918563 20171 tablet_copy_client.cc:538] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 01:54:06.913565  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:06.924696 20154 consensus_queue.cc:260] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:54:06.950943 20154 ts_tablet_manager.cc:1434] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.066s	user 0.009s	sys 0.002s
I20260808 01:54:06.932718 20151 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:06.953120 20151 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:06.954802  7831 raft_consensus.cc:2889] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260808 01:54:06.955327  7831 raft_consensus.cc:2272] T 9949664c54584858ae8fc74c9dc07a4a P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:06.955719 20151 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:06.940933 20144 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:06.959632  9648 tablet_copy_service.cc:342] P 259bcb3f0d254e8b9effac3277e654e0: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-4c485e8f4266462e8413079adcf8faef received from {username='slave'} at 127.0.0.1:45658
I20260808 01:54:06.960234  9648 tablet_copy_service.cc:434] P 259bcb3f0d254e8b9effac3277e654e0: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-4c485e8f4266462e8413079adcf8faef on tablet 4c485e8f4266462e8413079adcf8faef with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:06.961308  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:06.959017 20144 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:06.965517 20171 tablet_bootstrap.cc:492] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap starting.
I20260808 01:54:06.966261  9636 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 28.
I20260808 01:54:06.970016 20144 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:06.970196 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:06.970921 10367 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 28.
I20260808 01:54:06.972347  9600 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:54:06.975821 20144 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:54:06.977198 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:06.976282 20144 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:06.978168 20144 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Advancing to term 29
I20260808 01:54:06.978709 20167 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260808 01:54:06.987416 20167 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:06.987933 20167 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Advancing to term 29
W20260808 01:54:06.998816  9601 leader_election.cc:336] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:07.008580  7831 tablet_replica.cc:333] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:07.010098  7831 raft_consensus.cc:2243] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Raft consensus shutting down.
W20260808 01:54:07.012774  9524 leader_election.cc:336] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:07.013384  9524 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:07.013599 20144 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:07.015944 20143 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:07.017444 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:54:07.018234 20144 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:54:07.024240  9601 leader_election.cc:336] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:07.029053  7831 raft_consensus.cc:2272] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 6e7856bed1be43acaa821290d506ee97 [term 30 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:07.058219  9524 consensus_peers.cc:597] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:07.069270  7831 tablet_replica.cc:333] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:07.081404  7831 raft_consensus.cc:2243] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:07.082486  7831 raft_consensus.cc:2272] T 4c485e8f4266462e8413079adcf8faef P 6e7856bed1be43acaa821290d506ee97 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:07.087441  7831 tablet_replica.cc:333] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:07.088339  7831 raft_consensus.cc:2243] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 31 LEADER]: Raft consensus shutting down.
I20260808 01:54:07.089573  7831 raft_consensus.cc:2272] T 924b85a2ee7e46ecbcfd57257902abef P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:07.093884 20167 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:07.095381 10367 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 29: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:54:07.099419  9600 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:07.100813 20149 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260808 01:54:07.102377 10334 leader_election.cc:336] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:07.112565  7831 tablet_replica.cc:333] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:07.118863  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:54:07.119025  7831 raft_consensus.cc:2243] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:07.119870  9636 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 29: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:54:07.121016  7831 pending_rounds.cc:70] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Trying to abort 1 pending ops.
I20260808 01:54:07.121341  7831 pending_rounds.cc:77] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97: Aborting op as it isn't in flight: id { term: 30 index: 20 } timestamp: 7316086906965172224 op_type: NO_OP noop_request { }
I20260808 01:54:07.121521 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:07.121742  7831 raft_consensus.cc:2889] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260808 01:54:07.122327  7831 raft_consensus.cc:2272] T 4bce6ea2a8244811bf007638181a417a P 6e7856bed1be43acaa821290d506ee97 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:07.123486 20168 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260808 01:54:07.120301 20167 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 29 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:07.131520 19785 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:07.132257 19785 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:07.137284  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:07.138121  9557 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 29.
W20260808 01:54:07.139349  9601 leader_election.cc:336] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:07.140044  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:07.140836 19785 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:07.146536 19785 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:54:07.147133 19785 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:07.147590 19785 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:54:07.166954 19785 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:07.169721 19785 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:07.170269  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:54:07.171012  9557 raft_consensus.cc:3060] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Advancing to term 30
W20260808 01:54:07.173143  9601 leader_election.cc:336] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:07.179433  9557 raft_consensus.cc:2468] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 30.
I20260808 01:54:07.181082  9599 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:07.182493 19785 raft_consensus.cc:2804] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election won for term 30
I20260808 01:54:07.183147 19785 raft_consensus.cc:697] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:54:07.183750  7831 tablet_replica.cc:333] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:07.186156  7831 raft_consensus.cc:2243] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:07.184856 19785 consensus_queue.cc:237] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:07.188755  7831 raft_consensus.cc:2272] T fca874793205463dbb922993f57bf7f5 P 6e7856bed1be43acaa821290d506ee97 [term 29 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:07.220762 10334 consensus_peers.cc:597] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:07.230579 10334 consensus_peers.cc:597] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:07.328691  7831 tablet_replica.cc:333] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:07.332146  7831 raft_consensus.cc:2243] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:07.336864  7831 raft_consensus.cc:2272] T 82043e5428784a1eb53a98811ecbbbb4 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:07.373643 20163 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260808 01:54:07.371233  7831 tablet_replica.cc:333] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97: stopping tablet replica
I20260808 01:54:07.363260 20121 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:07.393664 20121 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:07.398940  7831 raft_consensus.cc:2243] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:07.448892 20121 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:07.448112  9366 catalog_manager.cc:5697] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 29 to 30, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 30 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:07.462926 20179 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:07.464020 20179 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:07.427809 20144 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:07.477975  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:07.452666  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "6e7856bed1be43acaa821290d506ee97" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:54:07.491871  9524 leader_election.cc:336] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:07.512394 20151 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:07.513365 20151 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:07.515203 20180 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:07.516194 20180 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:07.519181 20180 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:07.467351 20144 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:07.524554 20144 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:07.529784  9524 consensus_peers.cc:597] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:07.530977  9524 leader_election.cc:336] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:07.533818 20151 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:07.534955 20179 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:54:07.538869  9524 leader_election.cc:336] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:07.541365  9444 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e7856bed1be43acaa821290d506ee97; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:07.542594 20121 raft_consensus.cc:2706] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Received pre-election callback for term 28 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20260808 01:54:07.546164 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:07.547242 10367 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 30.
I20260808 01:54:07.577320 20163 tablet_bootstrap.cc:492] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=1 applied=16 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:54:07.584899 20163 tablet_bootstrap.cc:492] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:54:07.586346 20163 ts_tablet_manager.cc:1403] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.697s	user 0.068s	sys 0.012s
W20260808 01:54:07.549131 10334 consensus_peers.cc:597] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 26: this message will repeat every 5th retry.
I20260808 01:54:07.601537 20151 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:07.601603 19785 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:07.604652 19785 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:07.612802  7831 raft_consensus.cc:2272] T 407cb6b113ce432587849bc840ba6872 P 6e7856bed1be43acaa821290d506ee97 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:07.615728 20151 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:07.619024 20173 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:07.619850 20173 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:07.628408  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:07.630489 20181 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260808 01:54:07.641029 20181 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:07.641608 20181 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:54:07.654785 10363 raft_consensus.cc:1217] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEARNER]: Deduplicated request from leader. Original: 24.16->[24.17-24.17]   Dedup: 24.17->[]
I20260808 01:54:07.651513 19785 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:07.618352 20151 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:07.661055  9524 leader_election.cc:336] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:07.622432 20173 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:07.640605 20171 tablet_bootstrap.cc:492] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 01:54:07.668368 20171 tablet_bootstrap.cc:492] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Bootstrap complete.
I20260808 01:54:07.681962 20171 ts_tablet_manager.cc:1403] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent bootstrapping tablet: real 0.717s	user 0.076s	sys 0.012s
I20260808 01:54:07.713160 20171 raft_consensus.cc:359] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:54:07.714273 20171 raft_consensus.cc:740] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:54:07.715075 20171 consensus_queue.cc:260] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:54:07.729070 10363 raft_consensus.cc:1217] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 LEARNER]: Deduplicated request from leader. Original: 31.17->[31.18-31.18]   Dedup: 31.18->[]
W20260808 01:54:07.729497  9524 consensus_peers.cc:597] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:07.735448  9524 consensus_peers.cc:597] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:07.777127  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:07.778088  9636 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 29.
I20260808 01:54:07.778505  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:07.779480  9632 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 35.
I20260808 01:54:07.761513 20163 raft_consensus.cc:359] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:54:07.781015 20163 raft_consensus.cc:740] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 LEARNER]: Becoming Follower/Learner. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Initialized, Role: LEARNER
I20260808 01:54:07.781490 20171 ts_tablet_manager.cc:1434] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.099s	user 0.007s	sys 0.003s
I20260808 01:54:07.782001 20163 consensus_queue.cc:260] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } }
I20260808 01:54:07.784235  9648 tablet_copy_service.cc:342] P 259bcb3f0d254e8b9effac3277e654e0: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-3b8e0e8fda6744ad877514b11658b9d9 received from {username='slave'} at 127.0.0.1:45658
I20260808 01:54:07.785977  9648 tablet_copy_service.cc:434] P 259bcb3f0d254e8b9effac3277e654e0: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-3b8e0e8fda6744ad877514b11658b9d9 on tablet 3b8e0e8fda6744ad877514b11658b9d9 with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:07.793521 20189 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:07.800698  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:07.801599  9631 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 30.
I20260808 01:54:07.801759 20189 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:07.799938 20163 ts_tablet_manager.cc:1434] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a: Time spent starting tablet: real 0.213s	user 0.009s	sys 0.003s
I20260808 01:54:07.806241 20189 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:07.809369 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:54:07.809630  9524 consensus_peers.cc:597] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:07.826020 10334 consensus_peers.cc:597] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:07.790377 20181 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:07.831066 10363 raft_consensus.cc:1217] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 LEARNER]: Deduplicated request from leader. Original: 30.15->[30.16-30.16]   Dedup: 30.16->[]
I20260808 01:54:07.843133  9521 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:07.844213 20185 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260808 01:54:07.844252 10334 consensus_peers.cc:597] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:07.844830 20185 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:07.845796 20185 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:54:07.850345 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2b2a88358244f93970835ea3120c380" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 31 candidate_status { last_received { term: 30 index: 24 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:54:07.845752  9521 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:07.851120 10363 raft_consensus.cc:3060] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:54:07.851663 20188 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260808 01:54:07.852167 20188 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
W20260808 01:54:07.852548  9524 leader_election.cc:336] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:07.852561 20188 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 35 FOLLOWER]: Advancing to term 36
I20260808 01:54:07.866417  9571 tablet_copy_service.cc:342] P 52ccf4dcca1843f7bb29e9172e2d4ab4: Request end of tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-145c5a9982a24c8fa4e86d0d99cbbdfb received from {username='slave'} at 127.0.0.1:60552
I20260808 01:54:07.867162  9571 tablet_copy_service.cc:434] P 52ccf4dcca1843f7bb29e9172e2d4ab4: ending tablet copy session 7201605bb2f54c949a4ebe6e9b3b4b2a-145c5a9982a24c8fa4e86d0d99cbbdfb on tablet 145c5a9982a24c8fa4e86d0d99cbbdfb with peer 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:54:07.879878  9601 leader_election.cc:336] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:54:07.882386  9601 leader_election.cc:336] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:07.878304  9521 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:07.883566 20180 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260808 01:54:07.884021 20180 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:07.884423 20180 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:54:07.870393 20181 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:54:07.894232 10334 consensus_peers.cc:597] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 26: this message will repeat every 5th retry.
I20260808 01:54:07.907989  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:07.910498 19785 raft_consensus.cc:1064] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:54:07.919196 19785 consensus_queue.cc:237] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 15, Current term: 24, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:07.906301  9601 consensus_peers.cc:597] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:07.928788  9601 leader_election.cc:336] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:07.937553 20185 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:07.940048 20180 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:07.944278 10363 raft_consensus.cc:2468] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 31.
I20260808 01:54:07.947199  9555 raft_consensus.cc:2393] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 31: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:54:07.947042  9523 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:07.951234 20181 raft_consensus.cc:2804] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader election won for term 31
I20260808 01:54:07.953465 20181 raft_consensus.cc:697] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:54:07.954859 20181 consensus_queue.cc:237] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 30.24, Last appended by leader: 24, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:07.972512  9524 leader_election.cc:336] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:07.975219 20131 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:07.976004 20131 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:07.979316  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:07.989637  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:07.990787  9557 raft_consensus.cc:2393] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 30: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:54:07.993830  9554 raft_consensus.cc:1275] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260808 01:54:07.996214 20185 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:07.997575 20167 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:07.998324 20167 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:07.999107 10334 leader_election.cc:336] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:07.978602 20131 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:08.008941 10334 leader_election.cc:336] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:08.009584  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:54:08.009737  9524 leader_election.cc:336] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:08.009663 20167 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:08.019940 20189 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:08.020790 20189 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:08.024785 20189 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:08.026970 20173 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:08.027561 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:08.027777 20173 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:08.028312 10363 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:54:08.031944 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:08.030462 20173 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:08.037588 10332 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:08.038900 20131 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:54:08.041268  9632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "244e080726394210b5467bf688813cf0" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:54:08.042276  9632 raft_consensus.cc:3060] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:54:08.045511  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:08.052410 20180 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:08.054298  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2973a134fcef4b8da54331bcd12f1173" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:54:08.055058  9637 raft_consensus.cc:3060] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:54:08.059095 20131 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:08.057335  9599 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:08.059895 20131 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:08.062340 20183 consensus_queue.cc:1048] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260808 01:54:08.066149 10334 leader_election.cc:336] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:08.067902 20131 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:08.022192 20188 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:08.072232  9555 raft_consensus.cc:2393] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 36: Already voted for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in this term.
I20260808 01:54:08.072815 20188 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 36 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:08.075039 20179 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:08.075930 20179 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:08.076864  9637 raft_consensus.cc:2468] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 30.
I20260808 01:54:08.082984 20184 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:08.098009 20179 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:08.104784  9632 raft_consensus.cc:2468] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 31.
I20260808 01:54:08.101604 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:08.128355 10363 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 22.
I20260808 01:54:08.102097 10360 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEARNER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260808 01:54:08.111999  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:08.107944  9366 catalog_manager.cc:5697] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 30 to 31, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 31 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260808 01:54:08.099773  9524 leader_election.cc:336] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:08.113831  9555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:08.116617 20184 raft_consensus.cc:1064] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
I20260808 01:54:08.121134  9550 raft_consensus.cc:1275] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260808 01:54:08.130034  9555 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 22.
I20260808 01:54:08.135752 20183 raft_consensus.cc:3060] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 35 FOLLOWER]: Advancing to term 36
W20260808 01:54:08.131697  9524 leader_election.cc:336] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:08.133051 20189 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:54:08.137450 20173 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:54:08.137151 20184 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 16, Current term: 31, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:08.146447  9599 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:54:08.147465 20189 consensus_queue.cc:1048] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:54:08.149348 20189 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260808 01:54:08.149757 20189 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:08.150167 20189 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:54:08.155094 20144 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:08.155879 20144 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:08.156920 20180 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:08.157665 20180 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:08.159178 20144 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:08.161672 20180 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:08.163616  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:08.164449  9637 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 31.
I20260808 01:54:08.166211  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:08.168166  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:08.168850  9635 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 23.
W20260808 01:54:08.169589  9524 leader_election.cc:336] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:08.160171 20144 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:08.170965 20144 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:08.176684 20144 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:08.189144  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:08.195778  9550 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
W20260808 01:54:08.189417  9601 leader_election.cc:336] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:08.196627  9550 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 25.
I20260808 01:54:08.198899  9601 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:54:08.204243  9524 consensus_peers.cc:597] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:08.208192  9524 consensus_peers.cc:597] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:08.231081 20200 raft_consensus.cc:2749] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260808 01:54:08.236471 20183 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260808 01:54:08.253803 20181 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:08.254570 20181 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:08.282291 10363 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 LEARNER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260808 01:54:08.299489 20180 raft_consensus.cc:1064] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: attempting to promote NON_VOTER 7201605bb2f54c949a4ebe6e9b3b4b2a to VOTER
W20260808 01:54:08.306838  9524 leader_election.cc:336] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:08.308207 20180 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 14, Current term: 30, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:08.319641  9524 consensus_peers.cc:597] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:08.321579  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:54:08.326560  9550 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260808 01:54:08.261528 20181 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:08.332365 20144 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260808 01:54:08.329996  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 140624us
I20260808 01:54:08.331434 20183 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.010s
I20260808 01:54:08.348263 20200 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260808 01:54:08.350927  9560 raft_consensus.cc:2955] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Committing config change with OpId 24.18: config changed from index 17 to 18, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:54:08.357764  9524 consensus_peers.cc:597] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:08.339713 20144 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:08.360551 20144 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:54:08.363068 20173 raft_consensus.cc:2955] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Committing config change with OpId 24.18: config changed from index 17 to 18, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:54:08.384387 10334 consensus_peers.cc:597] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:08.385182 10334 consensus_peers.cc:597] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 26: this message will repeat every 5th retry.
I20260808 01:54:08.398540  9521 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:08.422956 20188 raft_consensus.cc:2804] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election won for term 30
I20260808 01:54:08.423697 20188 raft_consensus.cc:697] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:54:08.424927 20188 consensus_queue.cc:237] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:08.408555 10332 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:08.455442  9521 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:08.457561 20210 raft_consensus.cc:2804] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader election won for term 31
I20260808 01:54:08.458176 20210 raft_consensus.cc:697] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:54:08.459264 20210 consensus_queue.cc:237] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:08.459703 10360 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 LEARNER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260808 01:54:08.471078 10332 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:08.477504 20160 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260808 01:54:08.483631 20198 log.cc:927] Time spent T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260808 01:54:08.492549 20201 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:08.501435 20202 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260808 01:54:08.502141 20202 raft_consensus.cc:493] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:08.502661 20202 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Advancing to term 32
I20260808 01:54:08.504794 20186 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260808 01:54:08.505349 20186 raft_consensus.cc:493] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:08.505764 20186 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Advancing to term 24
I20260808 01:54:08.510716 20189 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:08.493376 20201 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:08.514935  9637 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 23: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:54:08.524564  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c617cb141a34007b6c1b8af0d911923" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:54:08.524849  9636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:08.525795  9636 raft_consensus.cc:2393] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 23: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:54:08.525617  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:08.528456  9638 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 29.
I20260808 01:54:08.533172  9635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:08.536492 10360 raft_consensus.cc:2955] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Committing config change with OpId 24.18: config changed from index 17 to 18, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:08.549657 20201 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:08.559695 20215 raft_consensus.cc:2955] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 LEADER]: Committing config change with OpId 31.19: config changed from index 18 to 19, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:54:08.569216  9524 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 343 similar messages]
I20260808 01:54:08.573638 20189 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:08.573860 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:54:08.574936 10363 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Advancing to term 23
W20260808 01:54:08.581085  9520 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:54:08.581476 20202 raft_consensus.cc:515] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:08.565948 20144 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:08.594398 20144 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:08.595539  9635 raft_consensus.cc:2393] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in current term 26: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:54:08.597694 20202 leader_election.cc:290] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:08.600725 10334 leader_election.cc:336] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:08.604074 10367 raft_consensus.cc:2955] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Committing config change with OpId 31.19: config changed from index 18 to 19, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:54:08.637890  9524 consensus_peers.cc:597] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:08.608320  9558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:54:08.639057  9558 raft_consensus.cc:3060] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Advancing to term 23
I20260808 01:54:08.610030  9557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" is_pre_election: true
I20260808 01:54:08.643103  9557 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 29.
I20260808 01:54:08.612974  9555 raft_consensus.cc:2955] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Committing config change with OpId 31.19: config changed from index 18 to 19, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:08.644523 10363 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:54:08.646135 20213 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:08.646896 20213 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:54:08.658881 10334 consensus_peers.cc:597] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:08.660566  9524 consensus_peers.cc:597] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:08.664439  9599 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:54:08.666409 20215 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:54:08.666954 20215 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:08.671944 20215 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:54:08.683223  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:54:08.684417  9560 raft_consensus.cc:3060] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Advancing to term 26
I20260808 01:54:08.616685  9631 raft_consensus.cc:2468] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 36.
I20260808 01:54:08.625017  9366 catalog_manager.cc:5697] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index 18 to 19, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 31 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:54:08.689540  9558 raft_consensus.cc:2468] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 23.
I20260808 01:54:08.691190  9599 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:54:08.694738  9366 catalog_manager.cc:5697] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: config changed from index 17 to 18, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 24 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:54:08.698220 20189 raft_consensus.cc:2804] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader election won for term 23
W20260808 01:54:08.703414  9524 leader_election.cc:336] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:54:08.703020 10334 consensus_peers.cc:597] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 26: this message will repeat every 5th retry.
I20260808 01:54:08.703806 20189 raft_consensus.cc:697] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:54:08.706403 20189 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:08.723237  9521 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:08.725438  9560 raft_consensus.cc:2468] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:54:08.727207  9599 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:54:08.728295 20144 raft_consensus.cc:2804] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader election won for term 26
I20260808 01:54:08.729259 20144 raft_consensus.cc:697] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:54:08.730346 20144 consensus_queue.cc:237] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:08.704824 20213 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:08.741457  9601 leader_election.cc:336] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:54:08.745442  9601 leader_election.cc:336] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:08.746721  9601 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:08.748014 20201 raft_consensus.cc:2749] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:54:08.750617 20215 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:08.754611  9560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4"
I20260808 01:54:08.755362  9560 raft_consensus.cc:3060] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:54:08.670008 20186 raft_consensus.cc:515] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:08.780159  9521 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:08.744207 20188 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:54:08.759809 20215 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:08.803911  9521 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:08.808972  9560 raft_consensus.cc:2468] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 30.
I20260808 01:54:08.811713  9521 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:08.824105 20186 leader_election.cc:290] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:08.835821  9638 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260808 01:54:08.840097  9638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:08.841060  9638 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 31.
I20260808 01:54:08.841531  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b1e1cf1fbe745feb69372fed0557ce0" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:54:08.842211  9637 raft_consensus.cc:3060] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Advancing to term 32
I20260808 01:54:08.811406 20180 raft_consensus.cc:2764] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
W20260808 01:54:08.859696  9601 consensus_peers.cc:597] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:08.863813 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 60424.8us
I20260808 01:54:08.817346 20213 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:54:08.819267 20210 raft_consensus.cc:2804] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 36 FOLLOWER]: Leader election won for term 36
W20260808 01:54:08.878815  9601 consensus_peers.cc:597] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:08.880609  9601 leader_election.cc:336] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:08.882781  9601 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:08.887084  9637 raft_consensus.cc:2468] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 32.
I20260808 01:54:08.885161 20210 raft_consensus.cc:697] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 36 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:54:08.892413 20210 consensus_queue.cc:237] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.19, Last appended by leader: 19, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:08.884588 20211 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.013s
W20260808 01:54:08.886979 10334 leader_election.cc:336] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:08.908732 19785 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260808 01:54:08.926025  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 464055us
I20260808 01:54:08.909078  9599 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 
I20260808 01:54:08.931542 19785 raft_consensus.cc:2804] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election won for term 30
I20260808 01:54:08.932188 19785 raft_consensus.cc:697] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
W20260808 01:54:08.909726  9601 leader_election.cc:336] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:08.933219 19785 consensus_queue.cc:237] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:54:08.934639  9601 consensus_peers.cc:597] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:08.936235  9601 consensus_peers.cc:597] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260808 01:54:08.936549 10367 raft_consensus.cc:1275] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 30 index: 24. Preceding OpId from leader: term: 31 index: 25. (index mismatch)
W20260808 01:54:08.938088  9601 consensus_peers.cc:597] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:08.939973  9601 consensus_peers.cc:597] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:08.949651  9601 consensus_peers.cc:597] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:08.951737  9601 leader_election.cc:336] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:54:08.953739  9601 leader_election.cc:336] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:08.964078 10332 leader_election.cc:304] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:08.967180 20202 raft_consensus.cc:2804] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 FOLLOWER]: Leader election won for term 32
I20260808 01:54:08.968351 20202 raft_consensus.cc:697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
W20260808 01:54:08.968611  9524 consensus_peers.cc:597] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:08.969982 20202 consensus_queue.cc:237] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:08.976949  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "54a2156aecd649d391b0acd6dc223843" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:54:08.978374  9637 raft_consensus.cc:3060] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Advancing to term 24
W20260808 01:54:08.975795  9601 consensus_peers.cc:597] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:09.013284  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 135412us
I20260808 01:54:09.015614  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:54:09.015054  9601 leader_election.cc:336] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:54:09.019901  9601 consensus_peers.cc:597] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:09.031057  9601 leader_election.cc:336] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:09.047865  9637 raft_consensus.cc:2468] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 24.
I20260808 01:54:09.028721 20160 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260808 01:54:09.050998 20160 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6e7856bed1be43acaa821290d506ee97)
I20260808 01:54:09.051438 20160 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Advancing to term 32
I20260808 01:54:09.053347  9366 catalog_manager.cc:5697] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 22 to 23, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 23 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:09.056046  9366 catalog_manager.cc:5697] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 25 to 26, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 26 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:09.067996 10332 leader_election.cc:304] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:09.086431  9634 consensus_queue.cc:237] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:09.120728 20168 raft_consensus.cc:2804] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader election won for term 24
I20260808 01:54:09.119450 20211 consensus_queue.cc:1048] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.029s
I20260808 01:54:09.155921 10360 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260808 01:54:09.157806 20199 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
W20260808 01:54:09.158988  9524 consensus_peers.cc:597] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:09.163810  9524 consensus_peers.cc:597] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:09.166087  9550 raft_consensus.cc:1275] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
W20260808 01:54:09.168327 20227 log.cc:927] Time spent T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260808 01:54:09.170445 20199 raft_consensus.cc:2955] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 LEADER]: Committing config change with OpId 31.20: config changed from index 19 to 20, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:09.173435 19785 consensus_queue.cc:1048] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260808 01:54:09.181278  9560 raft_consensus.cc:2955] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Committing config change with OpId 31.20: config changed from index 19 to 20, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:09.195798 10367 raft_consensus.cc:2955] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Committing config change with OpId 31.20: config changed from index 19 to 20, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:09.199321 20160 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:09.214008  9368 catalog_manager.cc:5697] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 31 to 32, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 32 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:54:09.232272 20213 raft_consensus.cc:2955] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Committing config change with OpId 30.17: config changed from index 16 to 17, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:09.249481  9367 catalog_manager.cc:5697] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 35 to 36, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 36 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
W20260808 01:54:09.252822  9524 leader_election.cc:336] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:09.255685  9367 catalog_manager.cc:5697] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 30 to 31, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 31 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:09.277158  9550 raft_consensus.cc:1275] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260808 01:54:09.274253  9368 catalog_manager.cc:5697] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 29 to 30, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 30 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:09.279325 10363 raft_consensus.cc:2955] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Committing config change with OpId 30.17: config changed from index 16 to 17, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:09.278208 20160 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:09.288894  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 3b8e0e8fda6744ad877514b11658b9d9 with cas_config_opid_index 19: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260808 01:54:09.285953  9367 catalog_manager.cc:5697] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 29 to 30, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 30 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:09.284272 20173 consensus_queue.cc:1048] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260808 01:54:09.240127 20168 raft_consensus.cc:697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:54:09.309988 20168 consensus_queue.cc:237] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:09.305789  9601 consensus_peers.cc:597] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260808 01:54:09.332242 20186 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:09.333169 20186 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:09.348315 19785 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:09.349077 19785 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:09.357775 20186 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:09.372709 10334 leader_election.cc:336] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:09.374904 19785 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:09.379364 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:09.380378 10363 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 29.
I20260808 01:54:09.405308  9637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924b85a2ee7e46ecbcfd57257902abef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:54:09.406102  9637 raft_consensus.cc:3060] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Advancing to term 32
I20260808 01:54:09.411216  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:09.412065  9634 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 29.
W20260808 01:54:09.415796  9524 consensus_peers.cc:597] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:09.423743  9631 raft_consensus.cc:1275] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260808 01:54:09.425223  9634 raft_consensus.cc:2955] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Committing config change with OpId 30.17: config changed from index 16 to 17, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:09.426380  9637 raft_consensus.cc:2468] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52ccf4dcca1843f7bb29e9172e2d4ab4 in term 32.
I20260808 01:54:09.434546 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:54:09.440506  9601 consensus_peers.cc:597] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:09.443666  9521 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97
W20260808 01:54:09.450812  9601 consensus_peers.cc:597] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260808 01:54:09.453037  9600 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
W20260808 01:54:09.453866  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 71424.5us
I20260808 01:54:09.455639 20215 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:54:09.456152 20215 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:09.456445 20212 raft_consensus.cc:2804] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Leader election won for term 32
I20260808 01:54:09.457340 20212 raft_consensus.cc:697] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 LEADER]: Becoming Leader. State: Replica: 52ccf4dcca1843f7bb29e9172e2d4ab4, State: Running, Role: LEADER
I20260808 01:54:09.450651  9367 catalog_manager.cc:5697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: config changed from index 16 to 17, 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196) changed from NON_VOTER to VOTER. New cstate: current_term: 30 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:54:09.442458 20131 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260808 01:54:09.456540 20215 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:54:09.460870  9367 catalog_manager.cc:5697] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: config changed from index 19 to 20, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New cstate: current_term: 31 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:09.467352  9638 raft_consensus.cc:1275] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260808 01:54:09.451448 20185 consensus_queue.cc:1048] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
W20260808 01:54:09.491432 10334 consensus_peers.cc:597] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:09.491838 10360 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260808 01:54:09.494059 20199 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260808 01:54:09.496167  9550 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260808 01:54:09.466670 20131 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:09.500092 20131 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Advancing to term 30
I20260808 01:54:09.502416  9368 catalog_manager.cc:5697] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 23 to 24, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 24 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260808 01:54:09.458523 20212 consensus_queue.cc:237] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:09.508282 20173 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.023s
I20260808 01:54:09.560614  9557 raft_consensus.cc:1275] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
W20260808 01:54:09.564049  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 80720.1us
I20260808 01:54:09.608299 20160 consensus_queue.cc:1048] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260808 01:54:09.576758 20144 consensus_queue.cc:1048] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.039s
I20260808 01:54:09.650897  9638 raft_consensus.cc:1275] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 36 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260808 01:54:09.620923 20215 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:09.678960 20131 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:09.737733  9601 consensus_peers.cc:597] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:09.739005  9601 leader_election.cc:336] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:54:09.741760  9601 consensus_peers.cc:597] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260808 01:54:09.771693 20131 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 30 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:09.780319 20215 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:54:09.799465  9601 consensus_peers.cc:597] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260808 01:54:09.806384 20212 consensus_queue.cc:1048] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260808 01:54:09.816213  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:54:09.817075  9634 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in current term 30: Already voted for candidate 259bcb3f0d254e8b9effac3277e654e0 in this term.
I20260808 01:54:09.818281  9632 raft_consensus.cc:1275] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
W20260808 01:54:09.825138  9601 consensus_peers.cc:597] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:09.832379  9601 consensus_peers.cc:597] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:09.834052  9367 catalog_manager.cc:5697] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 reported cstate change: term changed from 31 to 32, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194). New cstate: current_term: 32 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } }
W20260808 01:54:09.844307  9362 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 161821us
W20260808 01:54:09.852576  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 168342us
I20260808 01:54:09.865527 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:54:09.866428 10363 raft_consensus.cc:2393] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 259bcb3f0d254e8b9effac3277e654e0 in current term 30: Already voted for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in this term.
I20260808 01:54:09.890748 20131 consensus_queue.cc:1048] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.005s
W20260808 01:54:09.936930  9601 consensus_peers.cc:597] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:09.868038  9362 catalog_manager.cc:4755] TS 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3b8e0e8fda6744ad877514b11658b9d9: Remote error: Service unavailable: service kudu.tserver.TabletServerAdminService not registered on TabletServer
W20260808 01:54:09.942376  9362 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 74409.8us
W20260808 01:54:09.977587  9601 leader_election.cc:336] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:09.985159  9601 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:54:10.001808  9524 connection.cc:769] client connection to 127.7.165.193:39359 send error: Network error: sendmsg error: Broken pipe (error 32)
I20260808 01:54:10.013845  9632 raft_consensus.cc:1275] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260808 01:54:10.029297 20230 consensus_queue.cc:1048] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260808 01:54:10.046137 20250 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260808 01:54:10.059442  9601 connection.cc:573] client connection to 127.7.165.193:39359 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260808 01:54:10.066977  9601 consensus_peers.cc:597] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:10.068751  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 152685us
W20260808 01:54:10.090925 10334 consensus_peers.cc:597] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:10.186332  9524 consensus_peers.cc:597] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:10.205600  9524 consensus_peers.cc:597] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:10.168035 10334 leader_election.cc:336] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:10.221187 10334 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:10.224395  9557 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 14, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:10.222294 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 54366.2us
W20260808 01:54:10.230967 10334 consensus_peers.cc:597] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:10.231554 10334 consensus_peers.cc:597] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:10.232088 10334 consensus_peers.cc:597] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:10.231945  9631 consensus_queue.cc:237] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:10.234416 10363 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260808 01:54:10.234855 20186 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260808 01:54:10.240377  9557 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260808 01:54:10.242753 20223 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:54:10.244656 20201 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:54:10.247480 10360 raft_consensus.cc:1275] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
W20260808 01:54:10.255478 20263 log.cc:927] Time spent T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0: Append to log took a long time: real 0.086s	user 0.000s	sys 0.004s
W20260808 01:54:10.257767  9524 consensus_peers.cc:597] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:10.264701 20199 consensus_queue.cc:1048] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260808 01:54:10.268002 20201 raft_consensus.cc:2955] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Committing config change with OpId 23.18: config changed from index 15 to 18, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:54:10.270910 20267 log.cc:927] Time spent T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: Append to log took a long time: real 0.091s	user 0.003s	sys 0.000s
I20260808 01:54:10.272542 10360 raft_consensus.cc:2955] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Committing config change with OpId 23.18: config changed from index 15 to 18, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:10.285985 20179 raft_consensus.cc:2955] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Committing config change with OpId 30.18: config changed from index 17 to 18, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:54:10.258565  9524 consensus_peers.cc:597] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260808 01:54:10.295775 10363 raft_consensus.cc:2955] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Committing config change with OpId 30.18: config changed from index 17 to 18, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:10.331353  9366 catalog_manager.cc:5697] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 15 to 18, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New cstate: current_term: 23 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:10.353104  9638 raft_consensus.cc:1275] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260808 01:54:10.418991  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 145c5a9982a24c8fa4e86d0d99cbbdfb with cas_config_opid_index 17: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:54:10.424798  9524 consensus_peers.cc:597] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260808 01:54:10.436846  9366 catalog_manager.cc:5697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: config changed from index 17 to 18, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New cstate: current_term: 30 leader_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:10.445541  9360 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 7c508218361c446898fede7c5682fa9e with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260808 01:54:10.448666 10334 consensus_peers.cc:597] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260808 01:54:10.468799  9632 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Refusing update from remote peer 52ccf4dcca1843f7bb29e9172e2d4ab4: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260808 01:54:10.459884  9554 raft_consensus.cc:2955] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Committing config change with OpId 23.18: config changed from index 15 to 18, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
I20260808 01:54:10.537891 20151 consensus_queue.cc:1048] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.020s
W20260808 01:54:10.593489  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 51242.9us
W20260808 01:54:10.609382  9524 consensus_peers.cc:597] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260808 01:54:10.611001 20259 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.016s
W20260808 01:54:10.625145 10334 consensus_peers.cc:597] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:10.694350 10334 consensus_peers.cc:597] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:10.704625 10334 consensus_peers.cc:597] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:10.775489  9524 consensus_peers.cc:597] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260808 01:54:10.761489  9635 raft_consensus.cc:2955] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Committing config change with OpId 30.18: config changed from index 17 to 18, VOTER 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } } }
W20260808 01:54:10.885651  9601 consensus_peers.cc:597] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:10.909132  9524 consensus_peers.cc:597] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:10.922036  9601 consensus_peers.cc:597] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:10.953714  9362 catalog_manager.cc:4755] TS 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 145c5a9982a24c8fa4e86d0d99cbbdfb: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:11.084666  9524 consensus_peers.cc:597] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:11.141960  9524 consensus_peers.cc:597] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:11.164076  7831 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 9502)
W20260808 01:54:11.198935 10334 consensus_peers.cc:597] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:11.249531  9601 consensus_peers.cc:597] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:11.278713 10334 consensus_peers.cc:597] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:11.285903 19478 tablet_copy_client.cc:1131] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:11.282548  9524 consensus_peers.cc:597] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:11.326597  9524 consensus_peers.cc:597] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:11.373826  9524 consensus_peers.cc:597] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:11.398828  9601 consensus_peers.cc:597] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:11.506572  9601 consensus_peers.cc:597] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:11.513440  9601 consensus_peers.cc:597] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:11.569073  9524 consensus_peers.cc:597] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:11.578636  9601 consensus_peers.cc:597] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:11.581255  9601 consensus_peers.cc:597] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:11.618829  9524 consensus_peers.cc:597] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:11.703285 20279 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:11.704010 20279 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:11.708371 20279 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:11.709949 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:11.710867 10363 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 30.
I20260808 01:54:11.712705  9600 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:54:11.713937 20279 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260808 01:54:11.714521 20279 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:11.715111 20279 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Advancing to term 31
W20260808 01:54:11.720638  9601 leader_election.cc:336] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:11.750603 20279 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:11.753196  9524 consensus_peers.cc:597] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260808 01:54:11.763403 10363 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea7d6b03a61944ea8a8201984886f30a" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:54:11.764096 10363 raft_consensus.cc:3060] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:54:11.773749 10363 raft_consensus.cc:2468] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 31.
I20260808 01:54:11.776059  9600 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:54:11.774339 20279 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 election: Requested vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:11.778450 20279 raft_consensus.cc:2804] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader election won for term 31
I20260808 01:54:11.779134 20279 raft_consensus.cc:697] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 31 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:54:11.784006 20279 consensus_queue.cc:237] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:11.796089  9601 leader_election.cc:336] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:11.827812  9601 consensus_peers.cc:597] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:11.874177 10334 consensus_peers.cc:597] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:11.879608  9601 consensus_peers.cc:597] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:11.889819  9601 consensus_peers.cc:597] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:11.910650  9367 catalog_manager.cc:5697] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 28 to 31, leader changed from 6e7856bed1be43acaa821290d506ee97 (127.7.165.193) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 31 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260808 01:54:11.953332  9601 consensus_peers.cc:597] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:12.152145  9601 consensus_peers.cc:597] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:12.164688  7831 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 9502)
W20260808 01:54:12.190977  9601 consensus_peers.cc:597] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:12.206388 10334 consensus_peers.cc:597] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:12.226119  9524 consensus_peers.cc:597] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:12.251611 10334 consensus_peers.cc:597] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:12.268850  9524 consensus_peers.cc:597] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:12.285701 10334 consensus_peers.cc:597] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:12.317704 10334 consensus_peers.cc:597] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260808 01:54:12.318778 10363 raft_consensus.cc:1275] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260808 01:54:12.321298 20255 consensus_queue.cc:1048] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260808 01:54:12.360868  9601 consensus_peers.cc:597] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:12.381294  9601 consensus_peers.cc:597] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:12.385485  9601 consensus_peers.cc:597] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:12.475013  9601 consensus_peers.cc:597] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:12.493538  9362 catalog_manager.cc:4755] TS 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3b8e0e8fda6744ad877514b11658b9d9: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:12.553054  9524 consensus_peers.cc:597] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:12.556514 10334 consensus_peers.cc:597] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:12.583288  9601 consensus_peers.cc:597] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:12.709314  9524 consensus_peers.cc:597] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:12.973862 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 146797us
W20260808 01:54:12.985523 10334 consensus_peers.cc:597] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:13.056710 10334 consensus_peers.cc:597] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:13.165297  7831 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 9502)
W20260808 01:54:13.220628 10334 consensus_peers.cc:597] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:13.269868  9524 consensus_peers.cc:597] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:13.275164  9523 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 97094.9us
W20260808 01:54:13.301380  9524 consensus_peers.cc:597] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:13.322424 10334 consensus_peers.cc:597] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:13.466041  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 67540.1us
W20260808 01:54:13.524029  9524 consensus_peers.cc:597] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:13.530694 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 252536us
W20260808 01:54:13.558084  9524 consensus_peers.cc:597] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:13.580561  9601 consensus_peers.cc:597] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:13.583194 10334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111) [suppressed 463 similar messages]
W20260808 01:54:13.696758  9524 consensus_peers.cc:597] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:13.719949 10334 consensus_peers.cc:597] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:13.729002  9524 consensus_peers.cc:597] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:13.733911  9601 consensus_peers.cc:597] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:13.767628  9524 consensus_peers.cc:597] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:13.916522  9601 consensus_peers.cc:597] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:14.160282  9524 consensus_peers.cc:597] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:14.166028  7831 thread.cc:527] Waited for 4000ms trying to join with diag-logger (tid 9502)
W20260808 01:54:14.207485  9601 consensus_peers.cc:597] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:14.222308 10334 consensus_peers.cc:597] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:14.224097  9524 consensus_peers.cc:597] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:14.225606  9524 consensus_peers.cc:597] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:14.238585  9524 consensus_peers.cc:597] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:14.239331  9524 consensus_peers.cc:597] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:14.357077  9660 debug-util.cc:398] Leaking SignalData structure 0x7b0800375b40 after lost signal to thread 7834
W20260808 01:54:14.358531  9660 debug-util.cc:398] Leaking SignalData structure 0x7b08003629e0 after lost signal to thread 9412
W20260808 01:54:14.359586  9601 consensus_peers.cc:597] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:14.360232  9660 debug-util.cc:398] Leaking SignalData structure 0x7b08001c49e0 after lost signal to thread 9585
W20260808 01:54:14.361651  9660 debug-util.cc:398] Leaking SignalData structure 0x7b08002ae320 after lost signal to thread 9663
W20260808 01:54:14.363111  9660 debug-util.cc:398] Leaking SignalData structure 0x7b080039d6a0 after lost signal to thread 10393
W20260808 01:54:14.383708  9524 consensus_peers.cc:597] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:14.417758 10334 consensus_peers.cc:597] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:14.468741 10334 consensus_peers.cc:597] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:14.490599  9601 consensus_peers.cc:597] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:14.517051  9601 consensus_peers.cc:597] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:14.526659  9601 consensus_peers.cc:597] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:14.527396  9601 consensus_peers.cc:597] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:14.607725 10334 consensus_peers.cc:597] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:14.687165  9601 consensus_peers.cc:597] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:14.694021  9362 catalog_manager.cc:4755] TS 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3b8e0e8fda6744ad877514b11658b9d9: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:14.717742  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 55849us
W20260808 01:54:14.801023  9524 consensus_peers.cc:597] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:14.835935  9601 consensus_peers.cc:597] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:14.865548 10334 consensus_peers.cc:597] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:14.919806  9601 consensus_peers.cc:597] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:14.993655 10334 consensus_peers.cc:597] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:15.021001  9524 consensus_peers.cc:597] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:15.081054  9601 consensus_peers.cc:597] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:15.145452 10334 consensus_peers.cc:597] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:15.155385  9601 consensus_peers.cc:597] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:15.166622  7831 thread.cc:527] Waited for 5000ms trying to join with diag-logger (tid 9502)
W20260808 01:54:15.190824  9601 consensus_peers.cc:597] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:15.200742  9601 consensus_peers.cc:597] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:15.201243 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 118967us
W20260808 01:54:15.244313  9601 consensus_peers.cc:597] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:15.357173  9524 consensus_peers.cc:597] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:15.484127  9524 consensus_peers.cc:597] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:15.499763 10334 consensus_peers.cc:597] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:15.522011 10334 consensus_peers.cc:597] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:15.762073 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 70377.6us
W20260808 01:54:15.835765 10334 consensus_peers.cc:597] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:15.843051  9524 consensus_peers.cc:597] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:15.957540 10334 consensus_peers.cc:597] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:16.004565 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46191.7us
W20260808 01:54:16.052580  9524 consensus_peers.cc:597] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:16.167265  7831 thread.cc:527] Waited for 6000ms trying to join with diag-logger (tid 9502)
W20260808 01:54:16.197216  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 54096.1us
W20260808 01:54:16.224992  9524 consensus_peers.cc:597] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:16.230006  9524 consensus_peers.cc:597] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:16.230777  9524 consensus_peers.cc:597] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:16.234808  9524 consensus_peers.cc:597] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:16.244987  9601 consensus_peers.cc:597] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:16.248255  9601 consensus_peers.cc:597] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:16.272078  9601 consensus_peers.cc:597] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:16.289501 10334 consensus_peers.cc:597] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:16.315666  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 49432.5us
W20260808 01:54:16.369511  9601 consensus_peers.cc:597] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:16.388849  9601 consensus_peers.cc:597] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:16.467510  9524 consensus_peers.cc:597] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:16.538213  9524 consensus_peers.cc:597] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:16.891083  9601 consensus_peers.cc:597] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:16.935225 10334 consensus_peers.cc:597] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:16.939780  9601 consensus_peers.cc:597] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:17.035373 10334 consensus_peers.cc:597] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:17.138254 10334 consensus_peers.cc:597] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260808 01:54:17.167896  7831 thread.cc:527] Waited for 7000ms trying to join with diag-logger (tid 9502)
W20260808 01:54:17.328424  9601 consensus_peers.cc:597] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:17.383390  9601 consensus_peers.cc:597] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:17.384207  9601 consensus_peers.cc:597] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:17.384893  9601 consensus_peers.cc:597] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:17.394538  9601 consensus_peers.cc:597] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:17.441392  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 47301.3us
W20260808 01:54:17.462174 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 73303.2us
W20260808 01:54:17.519049 10334 consensus_peers.cc:597] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:17.519927 10334 consensus_peers.cc:597] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:17.722363 10334 consensus_peers.cc:597] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:17.728609  9599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:36095, user_credentials={real_user=slave}} blocked reactor thread for 100823us
W20260808 01:54:17.774910  9524 consensus_peers.cc:597] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:17.775823  9524 consensus_peers.cc:597] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:17.776476  9524 consensus_peers.cc:597] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:17.789610  9524 consensus_peers.cc:597] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:17.790354  9524 consensus_peers.cc:597] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:17.822877  9524 consensus_peers.cc:597] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:17.835181 10334 consensus_peers.cc:597] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:17.972074  9524 consensus_peers.cc:597] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:18.016288 10334 consensus_peers.cc:597] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:18.103384 10334 consensus_peers.cc:597] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260808 01:54:18.155373  9601 consensus_peers.cc:597] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:18.224131 10334 consensus_peers.cc:597] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260808 01:54:18.236554  7831 tablet_server.cc:196] TabletServer@127.7.165.193:0 shutdown complete.
W20260808 01:54:18.256871 10334 consensus_peers.cc:597] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:18.385418  9601 consensus_peers.cc:597] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:18.386971  9601 consensus_peers.cc:597] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:18.388582  9601 consensus_peers.cc:597] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:18.442363  9601 consensus_peers.cc:597] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:18.448423  9601 consensus_peers.cc:597] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:18.583789  9524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111) [suppressed 434 similar messages]
W20260808 01:54:18.590591  9524 consensus_peers.cc:597] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:18.684518  9601 consensus_peers.cc:597] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:18.765477  9524 consensus_peers.cc:597] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:18.895249  9524 consensus_peers.cc:597] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:18.925926  9362 catalog_manager.cc:4755] TS 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3b8e0e8fda6744ad877514b11658b9d9: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:18.915851  9524 consensus_peers.cc:597] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:18.959790 10332 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 62977.8us
W20260808 01:54:18.989576  9601 consensus_peers.cc:597] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:19.011162  9601 consensus_peers.cc:597] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:19.011883  9601 consensus_peers.cc:597] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:19.017861 10334 consensus_peers.cc:597] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260808 01:54:19.058015  9524 consensus_peers.cc:597] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260808 01:54:19.059537  7831 tablet_server.cc:179] TabletServer@127.7.165.194:0 shutting down...
W20260808 01:54:19.082095  9599 consensus_peers.cc:597] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.091954  9599 consensus_peers.cc:597] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.099239  9601 consensus_peers.cc:597] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:19.112735 10332 consensus_peers.cc:597] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.134363  9599 consensus_peers.cc:597] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.139245 10332 consensus_peers.cc:597] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.143612  9524 consensus_peers.cc:597] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:19.149214  9599 consensus_peers.cc:597] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.164887  9599 consensus_peers.cc:597] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.188664  9601 consensus_peers.cc:597] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:19.189368 10334 consensus_peers.cc:597] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260808 01:54:19.196009  9599 consensus_peers.cc:597] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.196009 10332 consensus_peers.cc:597] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.204596 10332 consensus_peers.cc:597] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.207284  9599 consensus_peers.cc:597] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.220041  9599 consensus_peers.cc:597] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.226586 10332 consensus_peers.cc:597] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.232786  9524 consensus_peers.cc:597] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:19.247845  9599 consensus_peers.cc:597] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.251550  9599 consensus_peers.cc:597] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.264478  9599 consensus_peers.cc:597] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.284489  9599 consensus_peers.cc:597] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.291263  9599 consensus_peers.cc:597] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:19.337431  7831 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260808 01:54:19.339250  7831 tablet_replica.cc:333] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:19.340607  7831 raft_consensus.cc:2243] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:19.342018  7831 raft_consensus.cc:2272] T 3759209338e14437aff5f92a2fff36da P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:19.346241  7831 tablet_replica.cc:333] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:19.350608  7831 raft_consensus.cc:2243] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 LEADER]: Raft consensus shutting down.
W20260808 01:54:19.351432  9599 consensus_peers.cc:597] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:19.352336  7831 raft_consensus.cc:2272] T 460e1059f5504707b891c4b0e0ac6a09 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:19.355111  9599 consensus_peers.cc:597] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.368858 10332 consensus_peers.cc:597] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.369045  9599 consensus_peers.cc:597] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:19.370113  7831 tablet_replica.cc:333] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:19.371399  7831 raft_consensus.cc:2243] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:19.373080  7831 raft_consensus.cc:2272] T 0b4331dfdd5b43448a400f2eb7b1be70 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:19.378202  7831 tablet_replica.cc:333] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
W20260808 01:54:19.383806 10332 consensus_peers.cc:597] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:19.389942  7831 raft_consensus.cc:2243] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:19.390998  7831 raft_consensus.cc:2272] T 7d3a904863b545ffa227b964ee3a3f18 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:19.408215  9599 consensus_peers.cc:597] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:19.408418  7831 tablet_replica.cc:333] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:19.423671  7831 raft_consensus.cc:2243] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:19.426432  7831 raft_consensus.cc:2272] T 2f740cd3515641a8b2c4add8f809dd82 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 22 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:19.427554  9599 consensus_peers.cc:597] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:19.431366  7831 tablet_replica.cc:333] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:19.432345  7831 raft_consensus.cc:2243] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:19.450541  7831 raft_consensus.cc:2272] T 5cf7e7eadc2347aa84c98413849ebfc9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:19.454172  7831 tablet_replica.cc:333] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:19.456229  7831 raft_consensus.cc:2243] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Raft consensus shutting down.
W20260808 01:54:19.457036  9599 consensus_peers.cc:597] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:19.462302  7831 raft_consensus.cc:2272] T 9bf65b9272854b479e73f2cea21e93f2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:19.477978  7831 tablet_replica.cc:333] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:19.480820  7831 raft_consensus.cc:2243] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 LEADER]: Raft consensus shutting down.
I20260808 01:54:19.488430  7831 raft_consensus.cc:2272] T 17e220e6a25747939bdf18f26b732407 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:19.493719  7831 tablet_replica.cc:333] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:19.497532  7831 raft_consensus.cc:2243] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 LEADER]: Raft consensus shutting down.
W20260808 01:54:19.506060  9599 consensus_peers.cc:597] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:19.506604  7831 raft_consensus.cc:2272] T d0db7b84c3b040c788e0a3f66657d912 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:19.513823  7831 tablet_replica.cc:333] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
W20260808 01:54:19.517624  9599 consensus_peers.cc:597] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.541349 10334 consensus_peers.cc:597] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:19.522224  7831 raft_consensus.cc:2243] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:19.550302  7831 raft_consensus.cc:2272] T e57f5706d264426eb123cb26c6f3b89c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:19.554008  9524 consensus_peers.cc:597] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260808 01:54:19.562422  7831 tablet_replica.cc:333] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
W20260808 01:54:19.562948  9599 consensus_peers.cc:597] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:19.565703  9599 consensus_peers.cc:597] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:19.563738  7831 raft_consensus.cc:2243] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 LEADER]: Raft consensus shutting down.
I20260808 01:54:19.572849  7831 raft_consensus.cc:2272] T d2b2a88358244f93970835ea3120c380 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:19.590837  7831 tablet_replica.cc:333] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:19.592051  7831 raft_consensus.cc:2243] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:19.595249  7831 raft_consensus.cc:2272] T 0ea307d9adc442539683547d0176a05b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:19.599344  7831 tablet_replica.cc:333] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
W20260808 01:54:19.632272  9524 consensus_peers.cc:597] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260808 01:54:19.633698  7831 raft_consensus.cc:2243] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Raft consensus shutting down.
I20260808 01:54:19.636437  7831 raft_consensus.cc:2272] T b335bc0170854e26a77dbec13dbe9eac P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:19.640503  7831 tablet_replica.cc:333] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:19.643460  7831 raft_consensus.cc:2243] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Raft consensus shutting down.
W20260808 01:54:19.659973  9601 consensus_peers.cc:597] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:19.698931  9524 consensus_peers.cc:597] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:19.719326  7831 raft_consensus.cc:2272] T df6982ec010444a8854140f7b64e985d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:19.763886  9601 consensus_peers.cc:597] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:19.781790  9524 consensus_peers.cc:597] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260808 01:54:19.786167  7831 tablet_replica.cc:333] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:19.794342  7831 raft_consensus.cc:2243] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Raft consensus shutting down.
W20260808 01:54:19.815594  9601 consensus_peers.cc:597] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:19.800033  9524 consensus_peers.cc:597] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260808 01:54:19.831266  7831 raft_consensus.cc:2272] T affdd24b03ae4ee795a90703613305f7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:19.860746  9601 consensus_peers.cc:597] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260808 01:54:19.861371  7831 tablet_replica.cc:333] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:19.875947  7831 raft_consensus.cc:2243] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:19.879925  7831 raft_consensus.cc:2272] T e3814b146ccd472f9304a0be09ad3738 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:19.894251  7831 tablet_replica.cc:333] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:19.896014  7831 raft_consensus.cc:2243] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:19.899353  7831 raft_consensus.cc:2272] T 90465c2f160641a7901ab0815e723ce1 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:19.915768  9601 consensus_peers.cc:597] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:19.928407 10334 consensus_peers.cc:597] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260808 01:54:19.953961  7831 tablet_replica.cc:333] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:19.960461  7831 raft_consensus.cc:2243] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 36 LEADER]: Raft consensus shutting down.
I20260808 01:54:20.019179  7831 raft_consensus.cc:2272] T 2c617cb141a34007b6c1b8af0d911923 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 36 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:19.989384 10334 consensus_peers.cc:597] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260808 01:54:20.028874  7831 tablet_replica.cc:333] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.046625  7831 raft_consensus.cc:2243] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Raft consensus shutting down.
I20260808 01:54:20.055830  7831 raft_consensus.cc:2272] T 030eacff61b14455978d2f0de262a089 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:20.081444  9524 consensus_peers.cc:597] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:20.098987  9601 consensus_peers.cc:597] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260808 01:54:20.102807 20330 raft_consensus.cc:493] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:54:20.117004 20343 raft_consensus.cc:493] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:20.120426 20343 raft_consensus.cc:515] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:20.124663 20343 leader_election.cc:290] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:20.120426 20330 raft_consensus.cc:515] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:20.128626  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82043e5428784a1eb53a98811ecbbbb4" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:20.129801 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fca874793205463dbb922993f57bf7f5" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:20.132119 20330 leader_election.cc:290] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:20.136292 20330 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:20.136988 20330 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:20.128317  9524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 46938.5us
W20260808 01:54:20.147053 10334 consensus_peers.cc:597] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:20.148479 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:20.149011 20330 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:20.150954  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2069a355aad47b48173a35b4efc6f9d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:20.167111 20330 raft_consensus.cc:493] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:54:20.167758 20330 raft_consensus.cc:515] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:20.170027 20330 leader_election.cc:290] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:20.174757 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32724e13cbcb49e991b11d18699e899a" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:54:20.184146 10334 consensus_peers.cc:597] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260808 01:54:20.159804  7831 tablet_replica.cc:333] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.194741  7831 raft_consensus.cc:2243] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 LEADER]: Raft consensus shutting down.
I20260808 01:54:20.195623 20330 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:20.197079 20330 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:20.197711  7831 raft_consensus.cc:2272] T 4f04078e3beb42eda250da45f7485537 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:20.195597  9524 leader_election.cc:336] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:20.208627  9524 leader_election.cc:304] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:20.201218 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:54:20.209925  9524 leader_election.cc:336] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:20.210402  9524 leader_election.cc:304] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:20.201462 20330 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:20.213490 20330 raft_consensus.cc:2749] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:54:20.207943 20343 raft_consensus.cc:493] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:54:20.214319 20343 raft_consensus.cc:515] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:20.222999 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cab93716156d46d3b0bda8d98d1fc76c" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:20.223841 20343 leader_election.cc:290] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:20.231369 20343 raft_consensus.cc:493] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:20.231980 20343 raft_consensus.cc:515] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:20.203758  9521 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:20.240694 20330 raft_consensus.cc:493] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:20.242492 20330 raft_consensus.cc:515] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:20.247532 20351 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:54:20.252158 20343 leader_election.cc:290] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:20.254849 20350 raft_consensus.cc:2749] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260808 01:54:20.255661  9521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:34565, user_credentials={real_user=slave}} blocked reactor thread for 54612.1us
I20260808 01:54:20.258841 20350 raft_consensus.cc:493] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:20.254328 20330 leader_election.cc:290] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:20.255141 20343 raft_consensus.cc:493] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:20.259558 20350 raft_consensus.cc:515] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:20.261166  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b8e0e8fda6744ad877514b11658b9d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:20.260223 20343 raft_consensus.cc:515] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:20.264345  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "103b913d6c014692b6dea1780af2aa20" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:20.253470  7831 tablet_replica.cc:333] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.267665  7831 raft_consensus.cc:2243] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Raft consensus shutting down.
I20260808 01:54:20.268288  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "487114a18943467b8d3a6db94b17a84d" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:20.269490  7831 raft_consensus.cc:2272] T 8cbead77343f44dd82199bebcfb2def7 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.272624 20330 raft_consensus.cc:493] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:20.273334 20330 raft_consensus.cc:515] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:20.284761  7831 tablet_replica.cc:333] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.293627  7831 raft_consensus.cc:2243] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:20.285641 20350 leader_election.cc:290] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:20.296243  7831 raft_consensus.cc:2272] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.300091  7831 tablet_replica.cc:333] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.301383  7831 raft_consensus.cc:2243] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:20.303547  7831 raft_consensus.cc:2272] T 4bcbe89d624b472f8be5abc6b34c9026 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.306954  7831 tablet_replica.cc:333] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.307917  7831 raft_consensus.cc:2243] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Raft consensus shutting down.
W20260808 01:54:20.308723  9601 consensus_peers.cc:597] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:20.288134 20330 leader_election.cc:290] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:20.291745  9521 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:20.292382 20343 leader_election.cc:290] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:20.318801  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c99678e59ec845b28293f811d76f24e8" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:54:20.322835  9521 raft_consensus.cc:2686] T 3b8e0e8fda6744ad877514b11658b9d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260808 01:54:20.323387  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa45ab4daad84ee78514f98cb3738f23" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 32 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:20.326308 20351 raft_consensus.cc:493] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:20.327050 20351 raft_consensus.cc:515] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:20.328428  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cacb9622c47c4737beff104125e98b77" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:20.330451 20351 leader_election.cc:290] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:20.333976  7831 raft_consensus.cc:2272] T 2973a134fcef4b8da54331bcd12f1173 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.327509 20350 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:20.338382 20350 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:20.341053  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c85ff498df484fc6ac0913aa57aa868b" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:20.337769  7831 tablet_replica.cc:333] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.343631 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:54:20.344874  9524 leader_election.cc:336] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:20.344331  7831 raft_consensus.cc:2243] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:20.345536  9524 leader_election.cc:304] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:20.346048  7831 raft_consensus.cc:2272] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.346562 20330 raft_consensus.cc:2749] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260808 01:54:20.348107  9524 leader_election.cc:336] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:20.348789  9524 leader_election.cc:304] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:20.349400  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f358d8403bf440059a6753dc4ed0be38" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:20.350193 20330 raft_consensus.cc:2749] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:20.351706  9521 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:20.352877 20330 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260808 01:54:20.349653  7831 tablet_replica.cc:333] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.343039 20350 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:54:20.357668  9524 leader_election.cc:336] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:20.359138  7831 raft_consensus.cc:2243] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:20.360085  7831 raft_consensus.cc:2272] T 8c49fd321f8b4f89868246239530e78b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.358338  9524 leader_election.cc:304] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:20.361876 20350 raft_consensus.cc:2749] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260808 01:54:20.363283  9524 leader_election.cc:336] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:20.363857  9524 leader_election.cc:304] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:20.364727 20330 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:20.365371 20330 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:20.365967 20350 raft_consensus.cc:2749] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:54:20.368815  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:20.370255 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c508218361c446898fede7c5682fa9e" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:20.370826 20330 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:20.372313  9523 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:20.372313  7831 tablet_replica.cc:333] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.374879 20330 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260808 01:54:20.375907  7831 raft_consensus.cc:2243] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:20.379151  7831 raft_consensus.cc:2272] T f358d8403bf440059a6753dc4ed0be38 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.382547  7831 tablet_replica.cc:333] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.384253  7831 raft_consensus.cc:2243] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Raft consensus shutting down.
W20260808 01:54:20.390406  9524 leader_election.cc:336] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:20.392663  9524 leader_election.cc:304] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:20.393684 20330 raft_consensus.cc:2749] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260808 01:54:20.395205  9524 leader_election.cc:336] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:20.395877  9524 leader_election.cc:304] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:20.396795 20330 raft_consensus.cc:2749] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:54:20.399107  7831 raft_consensus.cc:2272] T 103b913d6c014692b6dea1780af2aa20 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:20.399166  9524 leader_election.cc:336] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:20.399933  9524 leader_election.cc:304] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:20.403599  7831 tablet_replica.cc:333] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.404666 20330 raft_consensus.cc:2749] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:54:20.405136  7831 raft_consensus.cc:2243] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:20.406092  7831 raft_consensus.cc:2272] T cab93716156d46d3b0bda8d98d1fc76c P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.409772  7831 tablet_replica.cc:333] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.414252 20350 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:20.416631 20330 raft_consensus.cc:493] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:20.415100 20350 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:20.417873 20330 raft_consensus.cc:515] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:20.420599 20330 leader_election.cc:290] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:20.421454  9524 leader_election.cc:336] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:20.422343  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "407cb6b113ce432587849bc840ba6872" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:20.423189  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:20.422472 20350 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:20.423417  7831 raft_consensus.cc:2243] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 LEADER]: Raft consensus shutting down.
I20260808 01:54:20.423838  9524 leader_election.cc:304] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
W20260808 01:54:20.424790  9524 raft_consensus.cc:2686] T c85ff498df484fc6ac0913aa57aa868b P 52ccf4dcca1843f7bb29e9172e2d4ab4: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260808 01:54:20.437397  9524 leader_election.cc:336] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:20.438062  9524 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:20.439002 20350 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:54:20.425612  7831 raft_consensus.cc:2272] T 244e080726394210b5467bf688813cf0 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.425468 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c485e8f4266462e8413079adcf8faef" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:54:20.439957  9524 leader_election.cc:336] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:20.428349  9601 consensus_peers.cc:597] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260808 01:54:20.440626  9524 leader_election.cc:304] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:20.442968 20350 raft_consensus.cc:2749] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260808 01:54:20.451713 10334 consensus_peers.cc:597] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260808 01:54:20.467357  7831 tablet_replica.cc:333] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.468698  7831 raft_consensus.cc:2243] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:20.469655  7831 raft_consensus.cc:2272] T 32724e13cbcb49e991b11d18699e899a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.478256 20350 raft_consensus.cc:493] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:20.479079 20350 raft_consensus.cc:515] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:20.490702 20330 raft_consensus.cc:493] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:54:20.491398 20330 raft_consensus.cc:515] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:20.493052  7831 tablet_replica.cc:333] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.494092  7831 raft_consensus.cc:2243] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:20.495534 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11c99c174e234329801ee9b1ceae68d9" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:20.500577 20330 leader_election.cc:290] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:20.501247  9631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed168d59c43d4bfaa07d104032ad8977" candidate_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:54:20.501647  9601 consensus_peers.cc:597] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:20.509235  9524 leader_election.cc:336] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:20.510123  9524 leader_election.cc:304] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 6e7856bed1be43acaa821290d506ee97, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:20.508118 20350 leader_election.cc:290] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:20.509872  7831 raft_consensus.cc:2272] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:20.511523  9524 leader_election.cc:336] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:20.535348  9524 leader_election.cc:304] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52ccf4dcca1843f7bb29e9172e2d4ab4; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:20.536352 20350 raft_consensus.cc:2749] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260808 01:54:20.537195 20351 raft_consensus.cc:2706] T 11c99c174e234329801ee9b1ceae68d9 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Received pre-election callback for term 28 while not running: Illegal state: RaftConsensus is not running: State = Stopped
I20260808 01:54:20.515830 20349 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:20.539713 20349 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:20.542183 20349 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:20.546056  7831 tablet_replica.cc:333] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.547185  7831 raft_consensus.cc:2243] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 LEADER]: Raft consensus shutting down.
I20260808 01:54:20.550179  7831 raft_consensus.cc:2272] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 21 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:20.570039  9601 leader_election.cc:336] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:20.573230  9599 leader_election.cc:336] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:54:20.583364 10334 consensus_peers.cc:597] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260808 01:54:20.574631  9599 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:20.588583 20346 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260808 01:54:20.588882  7831 tablet_replica.cc:333] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.589879  7831 raft_consensus.cc:2243] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 LEADER]: Raft consensus shutting down.
I20260808 01:54:20.591378  7831 raft_consensus.cc:2272] T 439040d168824496acc076b1c280bde3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.596269  7831 tablet_replica.cc:333] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.611481  7831 raft_consensus.cc:2243] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 LEADER]: Raft consensus shutting down.
I20260808 01:54:20.612802  7831 raft_consensus.cc:2272] T 7b5ea129398a42fb9981fca652027cd6 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.616403  7831 tablet_replica.cc:333] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.618124  7831 raft_consensus.cc:2243] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:20.619143  7831 raft_consensus.cc:2272] T ed168d59c43d4bfaa07d104032ad8977 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:20.621227  9601 consensus_peers.cc:597] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260808 01:54:20.625746  7831 tablet_replica.cc:333] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.632272  7831 raft_consensus.cc:2243] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:20.633859  7831 raft_consensus.cc:2272] T 7c508218361c446898fede7c5682fa9e P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 23 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:20.639638  9600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:33455, user_credentials={real_user=slave}} blocked reactor thread for 51251.5us
I20260808 01:54:20.648001  7831 tablet_replica.cc:333] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.649248  7831 raft_consensus.cc:2243] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Raft consensus shutting down.
I20260808 01:54:20.650805  7831 raft_consensus.cc:2272] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.654296 20346 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:20.655115 20346 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:20.657564 20346 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:20.654381  7831 tablet_replica.cc:333] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.663304  7831 raft_consensus.cc:2243] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:20.664211  7831 raft_consensus.cc:2272] T c2069a355aad47b48173a35b4efc6f9d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.667657  7831 tablet_replica.cc:333] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.668658  7831 raft_consensus.cc:2243] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:20.670111  7831 raft_consensus.cc:2272] T 4c485e8f4266462e8413079adcf8faef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.684965  7831 tablet_replica.cc:333] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.686020  7831 raft_consensus.cc:2243] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 LEADER]: Raft consensus shutting down.
I20260808 01:54:20.687984  7831 raft_consensus.cc:2272] T 924b85a2ee7e46ecbcfd57257902abef P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.691382  7831 tablet_replica.cc:333] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.692272  7831 raft_consensus.cc:2243] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:20.693363  7831 raft_consensus.cc:2272] T c99678e59ec845b28293f811d76f24e8 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:20.695958 10334 consensus_peers.cc:597] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260808 01:54:20.702057  7831 tablet_replica.cc:333] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
W20260808 01:54:20.708379  9599 leader_election.cc:336] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260808 01:54:20.711454  9601 consensus_peers.cc:597] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260808 01:54:20.712818  7831 raft_consensus.cc:2243] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:20.713850  7831 raft_consensus.cc:2272] T fa45ab4daad84ee78514f98cb3738f23 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.717356  7831 tablet_replica.cc:333] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
W20260808 01:54:20.712826  9601 leader_election.cc:336] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:20.718101  9601 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:20.718592  7831 raft_consensus.cc:2243] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:20.719172 20346 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:54:20.719508  7831 raft_consensus.cc:2272] T 487114a18943467b8d3a6db94b17a84d P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.723048  7831 tablet_replica.cc:333] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
W20260808 01:54:20.730469 10334 consensus_peers.cc:597] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260808 01:54:20.732144  7831 raft_consensus.cc:2243] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 LEADER]: Raft consensus shutting down.
I20260808 01:54:20.733891  7831 raft_consensus.cc:2272] T 50fd024764ea49c7a1f16e992e7599f3 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.745343  7831 tablet_replica.cc:333] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.747231  7831 raft_consensus.cc:2243] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:20.748201  7831 raft_consensus.cc:2272] T cacb9622c47c4737beff104125e98b77 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.751708  7831 tablet_replica.cc:333] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.763342  7831 raft_consensus.cc:2243] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 LEADER]: Raft consensus shutting down.
I20260808 01:54:20.764964  7831 raft_consensus.cc:2272] T 9949664c54584858ae8fc74c9dc07a4a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.768705  7831 tablet_replica.cc:333] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.769830  7831 raft_consensus.cc:2243] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:20.770292 20346 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:20.770967  7831 raft_consensus.cc:2272] T 82043e5428784a1eb53a98811ecbbbb4 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.771088 20346 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:20.774861  7831 tablet_replica.cc:333] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.779829  7831 raft_consensus.cc:2243] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:20.781813  7831 raft_consensus.cc:2272] T 407cb6b113ce432587849bc840ba6872 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:20.784919  9599 leader_election.cc:336] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:20.786005  7831 tablet_replica.cc:333] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.792891  7831 raft_consensus.cc:2243] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 LEADER]: Raft consensus shutting down.
I20260808 01:54:20.793267 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:20.798751  7831 raft_consensus.cc:2272] T 4bce6ea2a8244811bf007638181a417a P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.799258  9600 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:20.802225  7831 tablet_replica.cc:333] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4: stopping tablet replica
I20260808 01:54:20.803108  7831 raft_consensus.cc:2243] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:20.803854  7831 raft_consensus.cc:2272] T fca874793205463dbb922993f57bf7f5 P 52ccf4dcca1843f7bb29e9172e2d4ab4 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:20.802147 20349 raft_consensus.cc:2749] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260808 01:54:20.795264 20346 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:20.828490 20346 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:20.829218 20346 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:54:20.837088  9599 leader_election.cc:336] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:20.833422 20346 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:20.840061  9601 leader_election.cc:336] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:20.840647  9601 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:20.841892 20349 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:54:20.841977 20347 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:20.842864 20347 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:20.845670 20347 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:20.851568 10332 leader_election.cc:336] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:20.854959  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:20.855829  9634 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 26.
I20260808 01:54:20.859513 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:20.860955 20347 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:54:20.861217 20346 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:20.861450 20347 raft_consensus.cc:493] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:20.862119 20347 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Advancing to term 27
I20260808 01:54:20.862164 20346 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:54:20.869787  9599 leader_election.cc:336] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:20.867115 20346 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:20.879532 20349 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:20.880179 20349 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:20.882478 20349 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:20.890950  9601 leader_election.cc:336] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:20.893045  9601 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:20.896306 20349 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260808 01:54:20.900581 20347 raft_consensus.cc:515] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:20.901851  9601 leader_election.cc:336] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:20.920425  9599 leader_election.cc:336] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:20.922564  9599 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:20.935758 20346 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260808 01:54:20.934798 10334 consensus_peers.cc:597] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260808 01:54:20.955090  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bf65b9272854b479e73f2cea21e93f2" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:54:20.963070  9634 raft_consensus.cc:3060] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
W20260808 01:54:20.957644 10332 leader_election.cc:336] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:20.952843 20347 leader_election.cc:290] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:20.978480  9634 raft_consensus.cc:2468] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 27.
W20260808 01:54:21.002339  9601 consensus_peers.cc:597] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260808 01:54:21.005565 10332 leader_election.cc:304] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:21.008590 20353 raft_consensus.cc:2804] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader election won for term 27
I20260808 01:54:21.010870 20353 raft_consensus.cc:697] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:54:21.012097 20353 consensus_queue.cc:237] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:21.017061 20336 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:21.017649 20336 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:21.028805 10332 leader_election.cc:336] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260808 01:54:21.030459 20336 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:21.031288 10334 leader_election.cc:336] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:21.031858 10334 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:21.033247 20336 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260808 01:54:21.083966  9601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 45686.7us
I20260808 01:54:21.125800  9366 catalog_manager.cc:5697] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 26 to 27, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 27 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 01:54:21.148959  9599 connection.cc:769] client connection to 127.7.165.194:36095 send error: Network error: sendmsg error: Broken pipe (error 32)
W20260808 01:54:21.270713  9601 consensus_peers.cc:597] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:21.321731  9599 consensus_peers.cc:597] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:21.332589  9601 consensus_peers.cc:597] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260808 01:54:21.336905 20353 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:21.337710 20353 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:21.340306 20353 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:21.345741  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
W20260808 01:54:21.354943 10332 leader_election.cc:336] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:21.356428 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:21.366618 20353 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260808 01:54:21.373401  9601 consensus_peers.cc:597] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260808 01:54:21.392046 20352 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:21.392680 20352 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:21.396373 20352 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:21.399307  9601 leader_election.cc:336] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:21.401999  9599 leader_election.cc:336] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:21.403261  9599 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:21.404448 20352 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260808 01:54:21.406060 10334 consensus_peers.cc:597] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260808 01:54:21.411445 20352 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:21.412139 20352 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:21.415441 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:21.420663 20352 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:54:21.429718 10332 consensus_peers.cc:597] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:21.435129  9601 consensus_peers.cc:597] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:21.436333  9599 leader_election.cc:336] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:21.438259  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:54:21.439595  9599 consensus_peers.cc:597] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:21.441150 20352 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:54:21.457583 20352 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:21.458392 20352 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:21.462033 20352 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:21.468099  9601 leader_election.cc:336] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:21.471222  9599 leader_election.cc:336] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:21.471771  9599 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:21.473901 20349 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260808 01:54:21.486814 10332 consensus_peers.cc:597] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:21.529075  9634 raft_consensus.cc:1275] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260808 01:54:21.531491 20336 consensus_queue.cc:1048] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260808 01:54:21.540778  9599 consensus_peers.cc:597] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:21.546082  9601 consensus_peers.cc:597] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260808 01:54:21.566140 10332 consensus_peers.cc:597] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:21.573266  9599 consensus_peers.cc:597] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:21.625142 10332 consensus_peers.cc:597] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:21.653316  9599 consensus_peers.cc:597] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:21.674655 20336 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:21.675398 20336 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:21.677793 20336 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:21.682092  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:21.683077  9634 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 21.
I20260808 01:54:21.684851 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:54:21.688113 20336 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260808 01:54:21.688591 20336 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:21.688977 20336 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:54:21.693738 20349 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:21.694397 20349 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:21.698290 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:21.699041 10367 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 26.
I20260808 01:54:21.702075  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:54:21.702996 20352 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260808 01:54:21.703457 20352 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:21.703909 20352 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Advancing to term 27
W20260808 01:54:21.714216  9599 leader_election.cc:336] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:21.716370 10332 leader_election.cc:336] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:21.719074 20352 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:21.730909 20349 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:54:21.733026  9601 consensus_peers.cc:597] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:21.733584  9599 consensus_peers.cc:597] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:21.737612 20352 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:21.749400 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f04078e3beb42eda250da45f7485537" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a"
I20260808 01:54:21.750125 10367 raft_consensus.cc:3060] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Advancing to term 27
W20260808 01:54:21.754638 10334 consensus_peers.cc:597] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260808 01:54:21.779412 20352 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:21.780097 20352 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:54:21.782017 10332 consensus_peers.cc:597] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:21.782704 10332 consensus_peers.cc:597] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:21.796794  9601 leader_election.cc:336] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:21.808977  9599 consensus_peers.cc:597] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:21.811609 10332 consensus_peers.cc:597] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:21.817934 20347 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:21.818348 20349 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:21.818924 20347 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:21.818986 20349 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:21.822449 20352 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:21.823207 20336 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:21.824263 20349 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:21.825865 20347 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:21.826112 10367 raft_consensus.cc:2468] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 259bcb3f0d254e8b9effac3277e654e0 in term 27.
I20260808 01:54:21.830668  9600 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:54:21.831569 20349 raft_consensus.cc:2804] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader election won for term 27
W20260808 01:54:21.832154 10332 leader_election.cc:336] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:21.832238 20349 raft_consensus.cc:697] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Becoming Leader. State: Replica: 259bcb3f0d254e8b9effac3277e654e0, State: Running, Role: LEADER
I20260808 01:54:21.827648 20336 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:21.832968 10332 leader_election.cc:336] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:21.834491 20349 consensus_queue.cc:237] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:21.836658 10334 leader_election.cc:336] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:21.845674 10334 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:21.850276 20336 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:54:21.841145  9601 leader_election.cc:336] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:21.812942  9599 consensus_peers.cc:597] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:21.841310  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
W20260808 01:54:21.851608  9599 leader_election.cc:336] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:21.851660  9634 raft_consensus.cc:3060] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 21 FOLLOWER]: Advancing to term 22
I20260808 01:54:21.851822 20364 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:21.852557 20364 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:21.855474 20364 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:21.861021  9634 raft_consensus.cc:2468] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 22.
W20260808 01:54:21.867043  9599 consensus_peers.cc:597] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:21.871930 20349 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:21.876094 20349 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:21.878600 20349 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
W20260808 01:54:21.879294  9599 consensus_peers.cc:597] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:21.879908 10367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
W20260808 01:54:21.874806  9601 consensus_peers.cc:597] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260808 01:54:21.874226  9367 catalog_manager.cc:5697] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 reported cstate change: term changed from 26 to 27, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195). New cstate: current_term: 27 leader_uuid: "259bcb3f0d254e8b9effac3277e654e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260808 01:54:21.881145  9601 leader_election.cc:336] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:21.882241 10332 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:21.883250 20336 raft_consensus.cc:2804] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Leader election won for term 22
I20260808 01:54:21.883812 20336 raft_consensus.cc:697] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:54:21.884835 20336 consensus_queue.cc:237] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:21.894557  9599 consensus_peers.cc:597] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:21.908210  9599 leader_election.cc:336] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:21.908483 20349 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:21.908656  9599 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:21.909142 20349 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:21.909721 20352 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260808 01:54:21.909999  9599 leader_election.cc:336] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:21.910521  9599 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:21.911171 20352 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:21.911513 20349 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:21.911756 20352 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:21.913941 20352 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:21.918439  9599 consensus_peers.cc:597] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:21.919338  9599 leader_election.cc:336] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:21.919874  9599 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:21.923983 20364 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260808 01:54:21.923144  9601 leader_election.cc:336] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:21.929075  9601 leader_election.cc:336] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:21.934051 20364 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260808 01:54:21.958838  9599 consensus_peers.cc:597] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:21.959384  9599 consensus_peers.cc:597] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:21.959803  9599 leader_election.cc:336] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:21.960192  9599 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:21.963495 20352 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:54:21.973640  9366 catalog_manager.cc:5697] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 21 to 22, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 22 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 01:54:21.984419  9599 consensus_peers.cc:597] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:21.985152  9599 leader_election.cc:336] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:21.986083  9599 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:21.986943 20364 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260808 01:54:21.988405  9599 leader_election.cc:336] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:21.988932  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:21.989856 20364 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260808 01:54:21.991230  9599 consensus_peers.cc:597] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:22.000814  9599 consensus_peers.cc:597] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:22.001354  9599 consensus_peers.cc:597] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:22.003571 10332 consensus_peers.cc:597] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:22.006256 20336 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:22.006932 20336 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:22.009554 20336 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:22.014230  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0" is_pre_election: true
I20260808 01:54:22.015223  9634 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 30.
W20260808 01:54:22.024060  9599 consensus_peers.cc:597] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:22.027104 10332 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 
I20260808 01:54:22.028827 20353 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260808 01:54:22.029310 20353 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Starting leader election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:22.029712 20353 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Advancing to term 31
I20260808 01:54:22.038442 20364 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:22.039137 20364 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:22.041708 20364 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:22.046150 10332 leader_election.cc:336] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:22.053301  9599 leader_election.cc:336] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:22.055467 20353 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting leader election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:22.064747 10332 leader_election.cc:336] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:22.066740  9601 leader_election.cc:336] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:22.067294  9601 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:22.068023 20364 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260808 01:54:22.075323  9634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "259bcb3f0d254e8b9effac3277e654e0"
I20260808 01:54:22.076021  9634 raft_consensus.cc:3060] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Advancing to term 31
W20260808 01:54:22.077800  9599 consensus_peers.cc:597] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:22.081709  9599 consensus_peers.cc:597] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:22.092844 10334 consensus_peers.cc:597] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260808 01:54:22.096735  9634 raft_consensus.cc:2468] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7201605bb2f54c949a4ebe6e9b3b4b2a in term 31.
I20260808 01:54:22.097695 20353 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 31 election: Requested vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:22.100315 10332 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0, 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:22.102993 20336 raft_consensus.cc:2804] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Leader election won for term 31
I20260808 01:54:22.103668 20336 raft_consensus.cc:697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 LEADER]: Becoming Leader. State: Replica: 7201605bb2f54c949a4ebe6e9b3b4b2a, State: Running, Role: LEADER
I20260808 01:54:22.104846 20336 consensus_queue.cc:237] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:22.128261  9599 consensus_peers.cc:597] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:22.138236 20364 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:22.139053 20352 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:22.139672 20352 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:22.142011 20352 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:22.138998 20364 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:22.153528 20364 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:22.152818  9366 catalog_manager.cc:5697] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a reported cstate change: term changed from 30 to 31, leader changed from 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194) to 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196). New cstate: current_term: 31 leader_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260808 01:54:22.160472  7831 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 9582)
W20260808 01:54:22.174755  9599 leader_election.cc:336] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:22.180022  9599 leader_election.cc:336] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:22.204321  9601 leader_election.cc:336] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:22.222393  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:22.225605 20372 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260808 01:54:22.226219  9601 consensus_peers.cc:597] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:22.226974  9601 leader_election.cc:336] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:22.229979  9601 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:22.236946 20372 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260808 01:54:22.270500  9599 consensus_peers.cc:597] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:22.301434  9601 consensus_peers.cc:597] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:22.298213  9599 consensus_peers.cc:597] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:22.315922 10364 raft_consensus.cc:1275] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Refusing update from remote peer 259bcb3f0d254e8b9effac3277e654e0: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260808 01:54:22.318591 20364 consensus_queue.cc:1048] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260808 01:54:22.379518  9601 consensus_peers.cc:597] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260808 01:54:22.389317  9634 raft_consensus.cc:1275] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
W20260808 01:54:22.400204 10332 consensus_peers.cc:597] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:22.419281 20336 consensus_queue.cc:1048] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260808 01:54:22.455696 20372 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:22.456561 20372 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:22.462571 20372 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:22.469288  9599 leader_election.cc:336] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:22.473397  9601 leader_election.cc:336] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:22.473906  9601 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:22.475396 20346 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:54:22.492257 20372 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:22.492969 20372 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:22.497407 20372 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:22.501999  9601 leader_election.cc:336] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:22.517510  9599 leader_election.cc:336] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:22.518178  9599 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:22.519121 20372 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260808 01:54:22.684903  9601 consensus_peers.cc:597] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:22.704406  9601 consensus_peers.cc:597] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260808 01:54:22.707494 20364 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:22.708209 20364 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:22.710588 20364 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:22.727731 10332 consensus_peers.cc:597] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:22.735644 20347 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:22.736302 20347 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:22.744191 20347 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:22.769028  9599 leader_election.cc:336] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:22.772079 10332 leader_election.cc:336] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:22.779507  9634 raft_consensus.cc:1275] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Refusing update from remote peer 7201605bb2f54c949a4ebe6e9b3b4b2a: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
W20260808 01:54:22.779681  9601 leader_election.cc:336] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:22.780269  9601 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:22.787349 20372 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260808 01:54:22.805131 20347 consensus_queue.cc:1048] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
W20260808 01:54:22.826050  9601 consensus_peers.cc:597] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260808 01:54:22.843541 20372 raft_consensus.cc:493] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:22.844122 20372 raft_consensus.cc:515] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:22.846612 20372 leader_election.cc:290] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:22.855031  9601 consensus_peers.cc:597] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:22.871701  9601 leader_election.cc:336] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:22.878633  9599 leader_election.cc:336] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:22.884727  9599 leader_election.cc:304] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:22.885593 20346 raft_consensus.cc:2749] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260808 01:54:22.889722  9601 consensus_peers.cc:597] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:22.943343  9601 consensus_peers.cc:597] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:22.956900  9601 consensus_peers.cc:597] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:23.050050  9601 consensus_peers.cc:597] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260808 01:54:23.071410 20346 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:23.072362 20346 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:23.075824 20346 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:23.079407  9601 leader_election.cc:336] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:23.079977  9599 leader_election.cc:336] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:23.080614  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:23.081728 20346 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260808 01:54:23.161310  7831 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 9582)
W20260808 01:54:23.204778 10334 consensus_peers.cc:597] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260808 01:54:23.205848 10334 consensus_peers.cc:597] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260808 01:54:23.206570 10334 leader_election.cc:336] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:23.207165 10334 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:23.208009 20336 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260808 01:54:23.211730 10334 consensus_peers.cc:597] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260808 01:54:23.227962 10334 consensus_peers.cc:597] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260808 01:54:23.376910  9601 consensus_peers.cc:597] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260808 01:54:23.378419 20364 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:23.379012 20364 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:23.381539 20364 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:23.404894  9599 leader_election.cc:336] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:23.409042  9601 leader_election.cc:336] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:23.409606  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:23.410401 20364 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260808 01:54:23.455336 20346 raft_consensus.cc:493] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:23.456104 20346 raft_consensus.cc:515] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:54:23.473672  9599 leader_election.cc:336] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:23.475756  9601 leader_election.cc:336] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:23.476816  9601 leader_election.cc:304] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:23.474622 20346 leader_election.cc:290] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:23.477988 20364 raft_consensus.cc:2749] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:54:23.506887 20346 raft_consensus.cc:493] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:23.507826 20346 raft_consensus.cc:515] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:23.510285 20346 leader_election.cc:290] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:23.512741 10334 consensus_peers.cc:597] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260808 01:54:23.513538 10334 consensus_peers.cc:597] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260808 01:54:23.524978  9599 leader_election.cc:336] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:23.526199  9601 leader_election.cc:336] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:23.526787  9601 leader_election.cc:304] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:23.527650 20346 raft_consensus.cc:2749] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260808 01:54:23.529217 20346 raft_consensus.cc:493] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:23.529947 20346 raft_consensus.cc:515] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:23.532593 20346 leader_election.cc:290] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:23.546217  9599 leader_election.cc:336] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:23.551189  9601 leader_election.cc:336] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:23.551860  9601 leader_election.cc:304] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:23.552891 20346 raft_consensus.cc:2749] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260808 01:54:23.584235  9601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111) [suppressed 717 similar messages]
I20260808 01:54:23.608973 20346 raft_consensus.cc:493] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:23.617504 20346 raft_consensus.cc:515] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:23.622272 20346 leader_election.cc:290] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:23.631026  9599 leader_election.cc:336] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:23.659930  9601 leader_election.cc:336] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:23.660526  9601 leader_election.cc:304] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:23.661362 20346 raft_consensus.cc:2749] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260808 01:54:23.752759 10334 consensus_peers.cc:597] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260808 01:54:23.787675  9599 consensus_peers.cc:597] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:23.801167  9601 consensus_peers.cc:597] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260808 01:54:23.857613 20379 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:23.858295 20379 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:23.863003 20379 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:23.871004 10334 leader_election.cc:336] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:23.871881 10332 leader_election.cc:336] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:23.872316 10332 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:23.875186 20379 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:54:23.878508  9601 consensus_peers.cc:597] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260808 01:54:23.886160  9599 consensus_peers.cc:597] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:23.890050 10332 consensus_peers.cc:597] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:23.909472  9601 consensus_peers.cc:597] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:23.941284  9601 consensus_peers.cc:597] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:23.966698  9599 consensus_peers.cc:597] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:23.980108 20380 raft_consensus.cc:493] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:23.980868 20380 raft_consensus.cc:515] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:23.983865 20380 leader_election.cc:290] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:23.990988  9599 leader_election.cc:336] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:23.992004  9601 leader_election.cc:336] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:23.992578  9601 leader_election.cc:304] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:23.993515 20380 raft_consensus.cc:2749] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260808 01:54:24.053701 10334 consensus_peers.cc:597] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260808 01:54:24.054409 10334 consensus_peers.cc:597] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260808 01:54:24.055547 10332 consensus_peers.cc:597] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.059401  9599 consensus_peers.cc:597] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:24.082212 20380 raft_consensus.cc:493] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:24.083046 20380 raft_consensus.cc:515] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:24.089659 20372 raft_consensus.cc:493] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:24.090395 20372 raft_consensus.cc:515] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:24.092738 20372 leader_election.cc:290] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:24.094606  9601 leader_election.cc:336] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:24.096305 20380 leader_election.cc:290] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:24.097282  9599 leader_election.cc:336] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:24.100390  9599 leader_election.cc:304] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:24.101305 20380 raft_consensus.cc:2749] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:54:24.106186 20380 raft_consensus.cc:493] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:24.106820 20380 raft_consensus.cc:515] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
W20260808 01:54:24.108163  9599 leader_election.cc:336] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:24.113508 20380 leader_election.cc:290] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:24.114570  9601 leader_election.cc:336] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:24.116315  9601 leader_election.cc:304] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:24.117269 20380 raft_consensus.cc:2749] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260808 01:54:24.130658 10332 consensus_peers.cc:597] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.133244  9601 leader_election.cc:336] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:24.135171  9599 consensus_peers.cc:597] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.145087  9599 consensus_peers.cc:597] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.145949  9599 leader_election.cc:336] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:24.146462  9599 leader_election.cc:304] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:24.149086 20380 raft_consensus.cc:2749] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260808 01:54:24.151186  9599 consensus_peers.cc:597] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.154317  9601 consensus_peers.cc:597] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260808 01:54:24.169410  9599 consensus_peers.cc:597] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.181474  9601 consensus_peers.cc:597] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260808 01:54:24.251072 20380 raft_consensus.cc:493] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:24.251693 20380 raft_consensus.cc:515] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:24.254060 20380 leader_election.cc:290] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:24.258518  7831 tablet_server.cc:196] TabletServer@127.7.165.194:0 shutdown complete.
I20260808 01:54:24.259521 20346 raft_consensus.cc:493] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:24.260110 20346 raft_consensus.cc:515] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:24.262243 20346 leader_election.cc:290] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:24.264447 10332 consensus_peers.cc:597] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.265883  9599 consensus_peers.cc:597] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.271093  9599 leader_election.cc:336] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:24.283797  9601 leader_election.cc:336] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:24.284447  9601 leader_election.cc:304] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:24.285374 20380 raft_consensus.cc:2749] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260808 01:54:24.286636  9601 leader_election.cc:336] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:24.288136  9599 leader_election.cc:336] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:24.288676  9599 leader_election.cc:304] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:24.289623 20380 raft_consensus.cc:2749] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260808 01:54:24.290757  9599 consensus_peers.cc:597] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.320147  9599 consensus_peers.cc:597] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.338306  9599 consensus_peers.cc:597] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.379065 10332 consensus_peers.cc:597] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.383173  9599 consensus_peers.cc:597] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.408506 10332 consensus_peers.cc:597] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.422808  9599 consensus_peers.cc:597] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.437171  9599 consensus_peers.cc:597] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.447981  9599 consensus_peers.cc:597] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.452342 10334 consensus_peers.cc:597] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260808 01:54:24.452454 10332 consensus_peers.cc:597] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.454041  9599 consensus_peers.cc:597] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.455780  9599 consensus_peers.cc:597] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.461848  9599 consensus_peers.cc:597] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.462503  9599 consensus_peers.cc:597] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.468777 10332 consensus_peers.cc:597] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:24.496177 10334 consensus_peers.cc:597] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260808 01:54:24.497026 20364 raft_consensus.cc:493] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:24.497660 20364 raft_consensus.cc:515] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:24.500089 20364 leader_election.cc:290] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:24.513595  9601 leader_election.cc:336] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:24.516621  9599 leader_election.cc:336] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:24.517695  9599 leader_election.cc:304] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:24.519920 20364 raft_consensus.cc:2749] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260808 01:54:24.557816  9599 consensus_peers.cc:597] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.561915  9601 consensus_peers.cc:597] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260808 01:54:24.563328 20364 raft_consensus.cc:493] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:24.563985 20364 raft_consensus.cc:515] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:24.566838 20364 leader_election.cc:290] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:24.585232  9601 leader_election.cc:336] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:24.592694  9599 leader_election.cc:336] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:24.593828  9599 leader_election.cc:304] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:24.600564 20364 raft_consensus.cc:2749] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260808 01:54:24.608649  9599 consensus_peers.cc:597] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:24.643103  9599 consensus_peers.cc:597] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:24.713392  9601 consensus_peers.cc:597] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260808 01:54:24.727617  7831 tablet_server.cc:179] TabletServer@127.7.165.195:0 shutting down...
W20260808 01:54:24.802079 10332 consensus_peers.cc:597] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:24.811034  9601 consensus_peers.cc:597] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260808 01:54:24.812071  9601 consensus_peers.cc:597] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260808 01:54:24.832739 10332 consensus_peers.cc:597] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:24.848999  9599 consensus_peers.cc:597] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260808 01:54:24.880558 20380 raft_consensus.cc:493] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:54:24.881067 10332 consensus_peers.cc:597] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:24.881171 20380 raft_consensus.cc:515] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:24.887619 20380 leader_election.cc:290] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:24.900532  9601 leader_election.cc:336] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:24.904680  9599 leader_election.cc:336] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:24.913568 10332 consensus_peers.cc:597] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:24.931533  9599 leader_election.cc:304] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:24.932716 20364 raft_consensus.cc:2749] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260808 01:54:24.995067 10332 consensus_peers.cc:597] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:25.005755  7831 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260808 01:54:25.007611 10332 consensus_peers.cc:597] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:25.021520  7831 tablet_replica.cc:333] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.024459  7831 raft_consensus.cc:2243] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.026221  7831 raft_consensus.cc:2272] T 0b4331dfdd5b43448a400f2eb7b1be70 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.030197  7831 tablet_replica.cc:333] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.031307  7831 raft_consensus.cc:2243] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.032282  7831 raft_consensus.cc:2272] T 03b17f21031f4ff4828c85b7332fcc2d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.035576  7831 tablet_replica.cc:333] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.048758  7831 raft_consensus.cc:2243] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.050230  7831 raft_consensus.cc:2272] T 7d3a904863b545ffa227b964ee3a3f18 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.053188  7831 tablet_replica.cc:333] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.054133  7831 raft_consensus.cc:2243] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.055528  7831 raft_consensus.cc:2272] T 3759209338e14437aff5f92a2fff36da P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.058665  7831 tablet_replica.cc:333] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.059707  7831 raft_consensus.cc:2243] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.060465  7831 raft_consensus.cc:2272] T 460e1059f5504707b891c4b0e0ac6a09 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.063434  7831 tablet_replica.cc:333] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.064410  7831 raft_consensus.cc:2243] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.065094  7831 raft_consensus.cc:2272] T 9bf65b9272854b479e73f2cea21e93f2 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.068331  7831 tablet_replica.cc:333] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.075142  7831 raft_consensus.cc:2243] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 31 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.076663  7831 raft_consensus.cc:2272] T ea7d6b03a61944ea8a8201984886f30a P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.080068  7831 tablet_replica.cc:333] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.080991  7831 raft_consensus.cc:2243] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.082581  7831 raft_consensus.cc:2272] T 2f740cd3515641a8b2c4add8f809dd82 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.096444  7831 tablet_replica.cc:333] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.097394  7831 raft_consensus.cc:2243] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.098809  7831 raft_consensus.cc:2272] T 20a4f92418934341a6ac4927db61dc87 P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.101809  7831 tablet_replica.cc:333] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
W20260808 01:54:25.119102 10332 consensus_peers.cc:597] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:25.128360 10332 consensus_peers.cc:597] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:25.129375  7831 raft_consensus.cc:2243] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.130884  7831 raft_consensus.cc:2272] T 0ea307d9adc442539683547d0176a05b P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:25.140892 10332 consensus_peers.cc:597] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:25.147471  7831 tablet_replica.cc:333] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.148738  7831 raft_consensus.cc:2243] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.149641  7831 raft_consensus.cc:2272] T b335bc0170854e26a77dbec13dbe9eac P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:25.357672 10334 consensus_peers.cc:597] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260808 01:54:25.516508 10334 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:39359, user_credentials={real_user=slave}} blocked reactor thread for 47904.7us
I20260808 01:54:25.559595 20382 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:25.560232 20382 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:25.562251 10332 consensus_peers.cc:597] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:25.563330 20382 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:25.578485 10334 consensus_peers.cc:597] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:25.588529 10334 consensus_peers.cc:597] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260808 01:54:25.592978 10334 leader_election.cc:336] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:25.595597 10332 leader_election.cc:336] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:25.596835 10332 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:25.597592 20379 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260808 01:54:25.612416 20364 raft_consensus.cc:493] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:25.613188 20364 raft_consensus.cc:515] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } }
I20260808 01:54:25.615522 20364 leader_election.cc:290] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:25.706916  7831 tablet_replica.cc:333] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.719844  7831 raft_consensus.cc:2243] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.723269  7831 raft_consensus.cc:2272] T 3354328ed023401e83a2361925b02e81 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:25.721660 10334 consensus_peers.cc:597] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260808 01:54:25.730664  7831 tablet_replica.cc:333] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.731714  7831 raft_consensus.cc:2243] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.732868  7831 raft_consensus.cc:2272] T 360b0cf59e7749058ca972bb487cbd10 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.743100  7831 tablet_replica.cc:333] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.744067  7831 raft_consensus.cc:2243] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.745258  7831 raft_consensus.cc:2272] T 17e220e6a25747939bdf18f26b732407 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.748661  7831 tablet_replica.cc:333] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.749711  7831 raft_consensus.cc:2243] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.763684  7831 raft_consensus.cc:2272] T d0db7b84c3b040c788e0a3f66657d912 P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.767079  7831 tablet_replica.cc:333] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.768252  7831 raft_consensus.cc:2243] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.769158  7831 raft_consensus.cc:2272] T affdd24b03ae4ee795a90703613305f7 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:25.770629  9601 consensus_peers.cc:597] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260808 01:54:25.771346  9601 consensus_peers.cc:597] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:25.771397  9599 leader_election.cc:336] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:25.772104  9601 consensus_peers.cc:597] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260808 01:54:25.772624  9601 consensus_peers.cc:597] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260808 01:54:25.773108  9601 leader_election.cc:336] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:25.774534  9601 leader_election.cc:304] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
W20260808 01:54:25.775614  9601 consensus_peers.cc:597] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260808 01:54:25.775807 20390 raft_consensus.cc:2749] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260808 01:54:25.776151  9601 consensus_peers.cc:597] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260808 01:54:25.782253 10332 consensus_peers.cc:597] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:25.782574  7831 tablet_replica.cc:333] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
W20260808 01:54:25.783115 10332 consensus_peers.cc:597] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:25.783850  7831 raft_consensus.cc:2243] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.785185  7831 raft_consensus.cc:2272] T df6982ec010444a8854140f7b64e985d P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.788286  7831 tablet_replica.cc:333] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.789215  7831 raft_consensus.cc:2243] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.790537  7831 raft_consensus.cc:2272] T e3814b146ccd472f9304a0be09ad3738 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.793784  7831 tablet_replica.cc:333] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.794984  7831 raft_consensus.cc:2243] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.796264  7831 raft_consensus.cc:2272] T 90465c2f160641a7901ab0815e723ce1 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.799336  7831 tablet_replica.cc:333] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.800196  7831 raft_consensus.cc:2243] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.801424  7831 raft_consensus.cc:2272] T 7b1e1cf1fbe745feb69372fed0557ce0 P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.803335 20390 raft_consensus.cc:493] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:54:25.803884 20390 raft_consensus.cc:515] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:25.805181  7831 tablet_replica.cc:333] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.806420 20390 leader_election.cc:290] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:25.806653  7831 raft_consensus.cc:2243] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 36 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.808022  7831 raft_consensus.cc:2272] T 2c617cb141a34007b6c1b8af0d911923 P 259bcb3f0d254e8b9effac3277e654e0 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.808533 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cbead77343f44dd82199bebcfb2def7" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:25.811478  7831 tablet_replica.cc:333] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.812511  7831 raft_consensus.cc:2243] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.813930  7831 raft_consensus.cc:2272] T 030eacff61b14455978d2f0de262a089 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:25.814922  9599 leader_election.cc:336] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:25.815785  9599 leader_election.cc:304] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
I20260808 01:54:25.816622 20390 raft_consensus.cc:2749] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260808 01:54:25.816932  7831 tablet_replica.cc:333] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.818004  7831 raft_consensus.cc:2243] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.819494  7831 raft_consensus.cc:2272] T 103b913d6c014692b6dea1780af2aa20 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.822552  7831 tablet_replica.cc:333] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.823472  7831 raft_consensus.cc:2243] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.824369  7831 raft_consensus.cc:2272] T 244e080726394210b5467bf688813cf0 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.827692  7831 tablet_replica.cc:333] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.828605  7831 raft_consensus.cc:2243] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.830374  7831 raft_consensus.cc:2272] T c85ff498df484fc6ac0913aa57aa868b P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.833038  7831 tablet_replica.cc:333] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.834024  7831 raft_consensus.cc:2243] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.834052 20390 raft_consensus.cc:493] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 7201605bb2f54c949a4ebe6e9b3b4b2a)
I20260808 01:54:25.834990  7831 raft_consensus.cc:2272] T 2973a134fcef4b8da54331bcd12f1173 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.834896 20390 raft_consensus.cc:515] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:25.838209  7831 tablet_replica.cc:333] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.838837 10364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "145c5a9982a24c8fa4e86d0d99cbbdfb" candidate_uuid: "259bcb3f0d254e8b9effac3277e654e0" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" is_pre_election: true
I20260808 01:54:25.839218  7831 raft_consensus.cc:2243] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.840564  7831 raft_consensus.cc:2272] T 4bcbe89d624b472f8be5abc6b34c9026 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.841022 20390 leader_election.cc:290] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 7201605bb2f54c949a4ebe6e9b3b4b2a (127.7.165.196:33455)
I20260808 01:54:25.846442  7831 tablet_replica.cc:333] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.847566  7831 raft_consensus.cc:2243] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.848284  7831 raft_consensus.cc:2272] T 439040d168824496acc076b1c280bde3 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.851301  7831 tablet_replica.cc:333] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.852173  7831 raft_consensus.cc:2243] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.852826  7831 raft_consensus.cc:2272] T 7b5ea129398a42fb9981fca652027cd6 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.856519  7831 tablet_replica.cc:333] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
W20260808 01:54:25.857324  9599 leader_election.cc:336] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:25.857475  7831 raft_consensus.cc:2243] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.858466 20379 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:25.859282  7831 raft_consensus.cc:2272] T 487114a18943467b8d3a6db94b17a84d P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.859110 20379 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:25.861588 20379 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:25.863771  7831 tablet_replica.cc:333] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.857937  9599 leader_election.cc:304] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 259bcb3f0d254e8b9effac3277e654e0; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 7201605bb2f54c949a4ebe6e9b3b4b2a
W20260808 01:54:25.866623 10332 leader_election.cc:336] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:25.864835  7831 raft_consensus.cc:2243] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.867292 20390 raft_consensus.cc:2749] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260808 01:54:25.867607 10334 leader_election.cc:336] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:25.868130 10334 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:25.868654  7831 raft_consensus.cc:2272] T fa45ab4daad84ee78514f98cb3738f23 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.869824 20379 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260808 01:54:25.873756  7831 tablet_replica.cc:333] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.876358  7831 raft_consensus.cc:2243] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.877919  7831 raft_consensus.cc:2272] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.881201  7831 tablet_replica.cc:333] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.882161  7831 raft_consensus.cc:2243] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 24 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.884100  7831 raft_consensus.cc:2272] T 4c485e8f4266462e8413079adcf8faef P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.887288  7831 tablet_replica.cc:333] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.888324  7831 raft_consensus.cc:2243] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.889575  7831 raft_consensus.cc:2272] T c2069a355aad47b48173a35b4efc6f9d P 259bcb3f0d254e8b9effac3277e654e0 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.892608  7831 tablet_replica.cc:333] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.893517  7831 raft_consensus.cc:2243] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.894879  7831 raft_consensus.cc:2272] T 3b8e0e8fda6744ad877514b11658b9d9 P 259bcb3f0d254e8b9effac3277e654e0 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.897789  7831 tablet_replica.cc:333] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.898631  7831 raft_consensus.cc:2243] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.899361  7831 raft_consensus.cc:2272] T 8cbead77343f44dd82199bebcfb2def7 P 259bcb3f0d254e8b9effac3277e654e0 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.902238  7831 tablet_replica.cc:333] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.903169  7831 raft_consensus.cc:2243] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.904397  7831 raft_consensus.cc:2272] T 4f04078e3beb42eda250da45f7485537 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.907413  7831 tablet_replica.cc:333] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.908252  7831 raft_consensus.cc:2243] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.908895  7831 raft_consensus.cc:2272] T 924b85a2ee7e46ecbcfd57257902abef P 259bcb3f0d254e8b9effac3277e654e0 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.912016  7831 tablet_replica.cc:333] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.912938  7831 raft_consensus.cc:2243] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 23 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.914160  7831 raft_consensus.cc:2272] T 7c508218361c446898fede7c5682fa9e P 259bcb3f0d254e8b9effac3277e654e0 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.917212  7831 tablet_replica.cc:333] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.918294  7831 raft_consensus.cc:2243] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.919876  7831 raft_consensus.cc:2272] T ed168d59c43d4bfaa07d104032ad8977 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.923094  7831 tablet_replica.cc:333] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.924110  7831 raft_consensus.cc:2243] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.925793  7831 raft_consensus.cc:2272] T cacb9622c47c4737beff104125e98b77 P 259bcb3f0d254e8b9effac3277e654e0 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.929227  7831 tablet_replica.cc:333] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.930255  7831 raft_consensus.cc:2243] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.930917  7831 raft_consensus.cc:2272] T 50fd024764ea49c7a1f16e992e7599f3 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.933853  7831 tablet_replica.cc:333] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.934669  7831 raft_consensus.cc:2243] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.935736  7831 raft_consensus.cc:2272] T 54a2156aecd649d391b0acd6dc223843 P 259bcb3f0d254e8b9effac3277e654e0 [term 24 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:25.947846 10334 consensus_peers.cc:597] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260808 01:54:25.950260  7831 tablet_replica.cc:333] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.951495  7831 raft_consensus.cc:2243] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.952231  7831 raft_consensus.cc:2272] T 9949664c54584858ae8fc74c9dc07a4a P 259bcb3f0d254e8b9effac3277e654e0 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.955353  7831 tablet_replica.cc:333] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.956212  7831 raft_consensus.cc:2243] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.956983  7831 raft_consensus.cc:2272] T c094b8e92d5e449aa2641c359d46d622 P 259bcb3f0d254e8b9effac3277e654e0 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.960196  7831 tablet_replica.cc:333] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.961030  7831 raft_consensus.cc:2243] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.962584  7831 raft_consensus.cc:2272] T c99678e59ec845b28293f811d76f24e8 P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.965939  7831 tablet_replica.cc:333] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.966774  7831 raft_consensus.cc:2243] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.967952  7831 raft_consensus.cc:2272] T f358d8403bf440059a6753dc4ed0be38 P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.973757  7831 tablet_replica.cc:333] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.974619  7831 raft_consensus.cc:2243] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.975523  7831 raft_consensus.cc:2272] T 8c49fd321f8b4f89868246239530e78b P 259bcb3f0d254e8b9effac3277e654e0 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.978629  7831 tablet_replica.cc:333] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.979406  7831 raft_consensus.cc:2243] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:25.980042  7831 raft_consensus.cc:2272] T 4bce6ea2a8244811bf007638181a417a P 259bcb3f0d254e8b9effac3277e654e0 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.983110  7831 tablet_replica.cc:333] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.983821  7831 raft_consensus.cc:2243] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.985096  7831 raft_consensus.cc:2272] T 82043e5428784a1eb53a98811ecbbbb4 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:25.988596  7831 tablet_replica.cc:333] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0: stopping tablet replica
I20260808 01:54:25.989288  7831 raft_consensus.cc:2243] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 LEADER]: Raft consensus shutting down.
I20260808 01:54:25.990543  7831 raft_consensus.cc:2272] T 407cb6b113ce432587849bc840ba6872 P 259bcb3f0d254e8b9effac3277e654e0 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:26.004654 10334 consensus_peers.cc:597] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260808 01:54:26.111949 10332 connection.cc:769] client connection to 127.7.165.195:34565 send error: Network error: sendmsg error: Broken pipe (error 32)
W20260808 01:54:26.177629 10334 consensus_peers.cc:597] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260808 01:54:26.253962 20379 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:26.254608 20379 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:26.256778 20379 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:26.261641 10332 leader_election.cc:336] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111)
W20260808 01:54:26.262307 10334 leader_election.cc:336] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:26.262790 10334 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:26.263475 20379 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260808 01:54:26.384513 20336 raft_consensus.cc:493] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:26.385108 20336 raft_consensus.cc:515] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:26.387200 20336 leader_election.cc:290] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:26.397864 10334 leader_election.cc:336] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:26.402416 10332 leader_election.cc:336] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111)
I20260808 01:54:26.402937 10332 leader_election.cc:304] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:26.403676 20383 raft_consensus.cc:2749] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260808 01:54:26.453848 10332 consensus_peers.cc:597] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:26.551100 10332 consensus_peers.cc:597] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:26.585306 10334 consensus_peers.cc:597] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260808 01:54:26.610042 10332 consensus_peers.cc:597] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:26.631564  9660 debug-util.cc:398] Leaking SignalData structure 0x7b08003ef840 after lost signal to thread 7834
W20260808 01:54:26.632958  9660 debug-util.cc:398] Leaking SignalData structure 0x7b080036bec0 after lost signal to thread 9412
W20260808 01:54:26.633626  9660 debug-util.cc:398] Leaking SignalData structure 0x7b0800374b20 after lost signal to thread 9663
W20260808 01:54:26.635027  9660 debug-util.cc:398] Leaking SignalData structure 0x7b08003621e0 after lost signal to thread 10393
W20260808 01:54:26.721762 10334 consensus_peers.cc:597] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260808 01:54:26.722590 10334 consensus_peers.cc:597] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260808 01:54:26.744840 10334 consensus_peers.cc:597] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260808 01:54:26.766753 10332 consensus_peers.cc:597] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:26.894882 10332 consensus_peers.cc:597] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:26.910051 10332 consensus_peers.cc:597] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:27.050865 10332 consensus_peers.cc:597] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:27.079118 10334 consensus_peers.cc:597] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260808 01:54:27.178290 20336 raft_consensus.cc:493] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:27.178936 20336 raft_consensus.cc:515] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:27.181089 20336 leader_election.cc:290] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:27.194806 20336 raft_consensus.cc:493] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:27.195382 20336 raft_consensus.cc:515] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:27.197162 20336 leader_election.cc:290] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:27.204173 20336 raft_consensus.cc:493] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:27.204875 20336 raft_consensus.cc:515] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:27.206471 10332 leader_election.cc:336] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:27.207351 20336 leader_election.cc:290] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:27.211419 20336 raft_consensus.cc:493] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
W20260808 01:54:27.211535 10332 leader_election.cc:336] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111)
I20260808 01:54:27.212250 20336 raft_consensus.cc:515] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:27.212427 10332 leader_election.cc:304] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:27.214282 20393 raft_consensus.cc:2749] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260808 01:54:27.215036 20336 leader_election.cc:290] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:27.217803 20336 raft_consensus.cc:493] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:27.218353 20336 raft_consensus.cc:515] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:27.220814 20336 leader_election.cc:290] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:27.223560  9362 catalog_manager.cc:4755] TS 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3b8e0e8fda6744ad877514b11658b9d9: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:27.227859 20336 raft_consensus.cc:493] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:27.228466 20336 raft_consensus.cc:515] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: true } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:27.231200 20336 leader_election.cc:290] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:27.236290 10334 leader_election.cc:336] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:27.237252 10332 leader_election.cc:336] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:27.244804 10332 leader_election.cc:336] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111)
I20260808 01:54:27.245407 10332 leader_election.cc:304] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:27.246202 20336 raft_consensus.cc:2749] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260808 01:54:27.253049 20336 raft_consensus.cc:493] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:27.253610 20336 raft_consensus.cc:515] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
W20260808 01:54:27.254086 10332 leader_election.cc:336] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:27.255517 10332 leader_election.cc:336] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:27.255635 20336 leader_election.cc:290] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
W20260808 01:54:27.256050 10332 leader_election.cc:336] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:27.256443 20336 raft_consensus.cc:493] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
W20260808 01:54:27.257193 10332 leader_election.cc:336] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111)
I20260808 01:54:27.257011 20336 raft_consensus.cc:515] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:27.259368 20393 raft_consensus.cc:493] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:27.259537 20336 leader_election.cc:290] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
I20260808 01:54:27.259995 20393 raft_consensus.cc:515] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } attrs { replace: false } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:27.262147 20393 leader_election.cc:290] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:27.257731 10332 leader_election.cc:304] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:27.277485 20396 raft_consensus.cc:2749] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260808 01:54:27.278751 10332 leader_election.cc:336] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111)
I20260808 01:54:27.279323 10332 leader_election.cc:304] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:27.280165 20396 raft_consensus.cc:2749] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260808 01:54:27.281268 10332 leader_election.cc:336] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111)
I20260808 01:54:27.281800 10332 leader_election.cc:304] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:27.282626 20396 raft_consensus.cc:2749] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260808 01:54:27.290455 10334 leader_election.cc:336] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:27.293800 10332 leader_election.cc:336] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:27.306890 10332 leader_election.cc:304] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:27.307825 20396 raft_consensus.cc:2749] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260808 01:54:27.308130 10332 consensus_peers.cc:597] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260808 01:54:27.309134 10332 leader_election.cc:336] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111)
W20260808 01:54:27.313109 10332 leader_election.cc:336] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:27.313763 10332 consensus_peers.cc:597] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:27.314265 10332 leader_election.cc:336] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:27.314704 10332 leader_election.cc:304] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:27.315514 20396 raft_consensus.cc:2749] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260808 01:54:27.315737 10332 leader_election.cc:336] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
W20260808 01:54:27.318212 10332 leader_election.cc:336] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111)
I20260808 01:54:27.318715 10332 leader_election.cc:304] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:27.319551 20396 raft_consensus.cc:2749] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260808 01:54:27.320667 10332 leader_election.cc:336] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111)
I20260808 01:54:27.321288 10332 leader_election.cc:304] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 52ccf4dcca1843f7bb29e9172e2d4ab4
I20260808 01:54:27.322129 20396 raft_consensus.cc:2749] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260808 01:54:27.388762 10332 consensus_peers.cc:597] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:27.456534 10332 consensus_peers.cc:597] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:27.457201 10332 consensus_peers.cc:597] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:27.483546 20396 raft_consensus.cc:493] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 52ccf4dcca1843f7bb29e9172e2d4ab4)
I20260808 01:54:27.484153 20396 raft_consensus.cc:515] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:27.486579 20396 leader_election.cc:290] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095)
W20260808 01:54:27.492792 10334 leader_election.cc:336] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
W20260808 01:54:27.505311 10332 leader_election.cc:336] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111)
I20260808 01:54:27.506062 10332 leader_election.cc:304] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 52ccf4dcca1843f7bb29e9172e2d4ab4, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:27.507045 20396 raft_consensus.cc:2749] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260808 01:54:27.556531 10332 consensus_peers.cc:597] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 52ccf4dcca1843f7bb29e9172e2d4ab4 (127.7.165.194:36095): Couldn't send request to peer 52ccf4dcca1843f7bb29e9172e2d4ab4. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.194:36095: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260808 01:54:27.615653 10332 consensus_peers.cc:597] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:27.618405 10332 consensus_peers.cc:597] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:27.669950 10332 consensus_peers.cc:597] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:27.736734 10332 consensus_peers.cc:597] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:27.785755 10332 consensus_peers.cc:597] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260808 01:54:27.860040 10334 consensus_peers.cc:597] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260808 01:54:27.864166 10332 consensus_peers.cc:597] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:27.906311  7831 tablet_server.cc:196] TabletServer@127.7.165.195:0 shutdown complete.
W20260808 01:54:28.013558 10334 consensus_peers.cc:597] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260808 01:54:28.057320 10334 consensus_peers.cc:597] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260808 01:54:28.133242  7831 tablet_server.cc:179] TabletServer@127.7.165.196:0 shutting down...
W20260808 01:54:28.156188 10334 consensus_peers.cc:597] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Couldn't send request to peer 6e7856bed1be43acaa821290d506ee97. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260808 01:54:28.185297 10332 consensus_peers.cc:597] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a -> Peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Couldn't send request to peer 259bcb3f0d254e8b9effac3277e654e0. Status: Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260808 01:54:28.225924  7831 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260808 01:54:28.227484  7831 tablet_replica.cc:333] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.229079  7831 raft_consensus.cc:2243] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 LEADER]: Raft consensus shutting down.
I20260808 01:54:28.232366  7831 raft_consensus.cc:2272] T 145c5a9982a24c8fa4e86d0d99cbbdfb P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.237538  7831 tablet_replica.cc:333] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.238952  7831 raft_consensus.cc:2243] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:28.240814  7831 raft_consensus.cc:2272] T 4c485e8f4266462e8413079adcf8faef P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.243669 20393 raft_consensus.cc:493] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 259bcb3f0d254e8b9effac3277e654e0)
I20260808 01:54:28.244632 20393 raft_consensus.cc:515] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: false } }
I20260808 01:54:28.245975  7831 tablet_replica.cc:333] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.247390  7831 raft_consensus.cc:2243] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 NON_PARTICIPANT]: Raft consensus shutting down.
I20260808 01:54:28.248523  7831 pending_rounds.cc:70] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a: Trying to abort 1 pending ops.
I20260808 01:54:28.248796  7831 pending_rounds.cc:77] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting op as it isn't in flight: id { term: 28 index: 17 } timestamp: 7316086879562960896 op_type: NO_OP noop_request { }
I20260808 01:54:28.249536  7831 raft_consensus.cc:2889] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 NON_PARTICIPANT]: NO_OP replication failed: Aborted: Op aborted
I20260808 01:54:28.249797 20393 leader_election.cc:290] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359), 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565)
I20260808 01:54:28.250138  7831 raft_consensus.cc:2272] T b335bc0170854e26a77dbec13dbe9eac P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 NON_PARTICIPANT]: Raft consensus is shut down!
W20260808 01:54:28.256151 10334 leader_election.cc:336] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:28.257530  7831 tablet_replica.cc:333] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.258646  7831 raft_consensus.cc:2243] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEARNER]: Raft consensus shutting down.
I20260808 01:54:28.259409  7831 pending_rounds.cc:70] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Trying to abort 1 pending ops.
I20260808 01:54:28.259690  7831 pending_rounds.cc:77] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7316085952844951552 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "affdd24b03ae4ee795a90703613305f7" old_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } } new_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } attrs { replace: true } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } } }
I20260808 01:54:28.261122  7831 raft_consensus.cc:2917] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 LEARNER]: Aborting config change with OpId 1.2: Aborted: Op aborted
W20260808 01:54:28.261229 10332 leader_election.cc:336] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 259bcb3f0d254e8b9effac3277e654e0 (127.7.165.195:34565): Network error: Client connection negotiation failed: client connection to 127.7.165.195:34565: connect: Connection refused (error 111)
I20260808 01:54:28.262022  7831 raft_consensus.cc:2889] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 NON_PARTICIPANT]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260808 01:54:28.262008 10332 leader_election.cc:304] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7201605bb2f54c949a4ebe6e9b3b4b2a; no voters: 259bcb3f0d254e8b9effac3277e654e0, 6e7856bed1be43acaa821290d506ee97
I20260808 01:54:28.262660  7831 raft_consensus.cc:2272] T affdd24b03ae4ee795a90703613305f7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 2 NON_PARTICIPANT]: Raft consensus is shut down!
I20260808 01:54:28.263305 20393 raft_consensus.cc:2749] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260808 01:54:28.267201  7831 tablet_replica.cc:333] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.268671  7831 raft_consensus.cc:2243] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 LEADER]: Raft consensus shutting down.
I20260808 01:54:28.270463  7831 raft_consensus.cc:2272] T 03b17f21031f4ff4828c85b7332fcc2d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.275094  7831 tablet_replica.cc:333] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.276041  7831 raft_consensus.cc:2243] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEARNER]: Raft consensus shutting down.
I20260808 01:54:28.276724  7831 pending_rounds.cc:70] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a: Trying to abort 2 pending ops.
I20260808 01:54:28.276949  7831 pending_rounds.cc:77] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting op as it isn't in flight: id { term: 4 index: 4 } timestamp: 7316086041442091008 op_type: NO_OP noop_request { }
I20260808 01:54:28.277381  7831 raft_consensus.cc:2889] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEARNER]: NO_OP replication failed: Aborted: Op aborted
I20260808 01:54:28.277678  7831 pending_rounds.cc:77] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a: Aborting op as it isn't in flight: id { term: 2 index: 3 } timestamp: 7316085980070694912 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "3759209338e14437aff5f92a2fff36da" old_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6e7856bed1be43acaa821290d506ee97" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 39359 } attrs { replace: true } } peers { permanent_uuid: "52ccf4dcca1843f7bb29e9172e2d4ab4" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 36095 } } peers { permanent_uuid: "259bcb3f0d254e8b9effac3277e654e0" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 34565 } } peers { permanent_uuid: "7201605bb2f54c949a4ebe6e9b3b4b2a" member_type: NON_VOTER last_known_addr { host: "127.7.165.196" port: 33455 } attrs { promote: true } } } }
I20260808 01:54:28.278916  7831 raft_consensus.cc:2917] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 LEARNER]: Aborting config change with OpId 2.3: Aborted: Op aborted
I20260808 01:54:28.279587  7831 raft_consensus.cc:2889] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 NON_PARTICIPANT]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260808 01:54:28.279923  7831 raft_consensus.cc:2272] T 3759209338e14437aff5f92a2fff36da P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 4 NON_PARTICIPANT]: Raft consensus is shut down!
I20260808 01:54:28.283803  7831 tablet_replica.cc:333] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.284716  7831 raft_consensus.cc:2243] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 LEADER]: Raft consensus shutting down.
I20260808 01:54:28.286559  7831 raft_consensus.cc:2272] T 9bf65b9272854b479e73f2cea21e93f2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.290822  7831 tablet_replica.cc:333] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.292057  7831 raft_consensus.cc:2243] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:28.295455  7831 raft_consensus.cc:2272] T 7c508218361c446898fede7c5682fa9e P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.299978  7831 tablet_replica.cc:333] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.300794  7831 raft_consensus.cc:2243] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 LEADER]: Raft consensus shutting down.
I20260808 01:54:28.302924  7831 raft_consensus.cc:2272] T c094b8e92d5e449aa2641c359d46d622 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 28 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.306860  7831 tablet_replica.cc:333] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.307921  7831 raft_consensus.cc:2243] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:28.309027  7831 raft_consensus.cc:2272] T d2b2a88358244f93970835ea3120c380 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.312745  7831 tablet_replica.cc:333] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.313679  7831 raft_consensus.cc:2243] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 LEADER]: Raft consensus shutting down.
I20260808 01:54:28.315497  7831 raft_consensus.cc:2272] T 3354328ed023401e83a2361925b02e81 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 26 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.318984  7831 tablet_replica.cc:333] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.319895  7831 raft_consensus.cc:2243] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 LEADER]: Raft consensus shutting down.
I20260808 01:54:28.321635  7831 raft_consensus.cc:2272] T cab93716156d46d3b0bda8d98d1fc76c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.325963  7831 tablet_replica.cc:333] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.326903  7831 raft_consensus.cc:2243] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 33 LEADER]: Raft consensus shutting down.
I20260808 01:54:28.328707  7831 raft_consensus.cc:2272] T 8c49fd321f8b4f89868246239530e78b P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 33 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.332763  7831 tablet_replica.cc:333] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.333716  7831 raft_consensus.cc:2243] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 33 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:28.334868  7831 raft_consensus.cc:2272] T f358d8403bf440059a6753dc4ed0be38 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 33 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.338827  7831 tablet_replica.cc:333] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.339696  7831 raft_consensus.cc:2243] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:28.340934  7831 raft_consensus.cc:2272] T c2069a355aad47b48173a35b4efc6f9d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 25 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.344657  7831 tablet_replica.cc:333] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.345508  7831 raft_consensus.cc:2243] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:28.346597  7831 raft_consensus.cc:2272] T 4f04078e3beb42eda250da45f7485537 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.350411  7831 tablet_replica.cc:333] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.351297  7831 raft_consensus.cc:2243] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 LEADER]: Raft consensus shutting down.
I20260808 01:54:28.353061  7831 raft_consensus.cc:2272] T fca874793205463dbb922993f57bf7f5 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.356652  7831 tablet_replica.cc:333] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.357460  7831 raft_consensus.cc:2243] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 LEADER]: Raft consensus shutting down.
I20260808 01:54:28.359228  7831 raft_consensus.cc:2272] T e57f5706d264426eb123cb26c6f3b89c P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.363005  7831 tablet_replica.cc:333] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.363893  7831 raft_consensus.cc:2243] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:28.365404  7831 raft_consensus.cc:2272] T df6982ec010444a8854140f7b64e985d P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.369001  7831 tablet_replica.cc:333] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.369971  7831 raft_consensus.cc:2243] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Raft consensus shutting down.
I20260808 01:54:28.371800  7831 raft_consensus.cc:2272] T 360b0cf59e7749058ca972bb487cbd10 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.375183  7831 tablet_replica.cc:333] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.376123  7831 raft_consensus.cc:2243] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 LEADER]: Raft consensus shutting down.
I20260808 01:54:28.377826  7831 raft_consensus.cc:2272] T 8cbead77343f44dd82199bebcfb2def7 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 22 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.381232  7831 tablet_replica.cc:333] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.382015  7831 raft_consensus.cc:2243] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:28.382994  7831 raft_consensus.cc:2272] T ea7d6b03a61944ea8a8201984886f30a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.386708  7831 tablet_replica.cc:333] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.387667  7831 raft_consensus.cc:2243] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 LEADER]: Raft consensus shutting down.
I20260808 01:54:28.389726  7831 raft_consensus.cc:2272] T 54a2156aecd649d391b0acd6dc223843 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 24 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.393443  7831 tablet_replica.cc:333] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.394363  7831 raft_consensus.cc:2243] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:28.395305  7831 raft_consensus.cc:2272] T 626249dbd6eb4d66bfa8cce2e716cbe2 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.398818  7831 tablet_replica.cc:333] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.399616  7831 raft_consensus.cc:2243] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 LEADER]: Raft consensus shutting down.
I20260808 01:54:28.401336  7831 raft_consensus.cc:2272] T 7b1e1cf1fbe745feb69372fed0557ce0 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 32 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.404713  7831 tablet_replica.cc:333] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.405467  7831 raft_consensus.cc:2243] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 LEADER]: Raft consensus shutting down.
I20260808 01:54:28.407280  7831 raft_consensus.cc:2272] T 11c99c174e234329801ee9b1ceae68d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 27 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.410746  7831 tablet_replica.cc:333] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.411574  7831 raft_consensus.cc:2243] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:28.412637  7831 raft_consensus.cc:2272] T 20a4f92418934341a6ac4927db61dc87 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 23 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.416131  7831 tablet_replica.cc:333] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.416949  7831 raft_consensus.cc:2243] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 LEADER]: Raft consensus shutting down.
I20260808 01:54:28.418808  7831 raft_consensus.cc:2272] T 5cf7e7eadc2347aa84c98413849ebfc9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 29 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.422299  7831 tablet_replica.cc:333] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.423261  7831 raft_consensus.cc:2243] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:28.425040  7831 raft_consensus.cc:2272] T 3b8e0e8fda6744ad877514b11658b9d9 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 31 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.429675  7831 tablet_replica.cc:333] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.430444  7831 raft_consensus.cc:2243] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Raft consensus shutting down.
I20260808 01:54:28.431517  7831 raft_consensus.cc:2272] T 4bcbe89d624b472f8be5abc6b34c9026 P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 30 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:28.434924  7831 tablet_replica.cc:333] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a: stopping tablet replica
I20260808 01:54:28.435592  7831 raft_consensus.cc:2243] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 LEADER]: Raft consensus shutting down.
I20260808 01:54:28.437150  7831 raft_consensus.cc:2272] T 32724e13cbcb49e991b11d18699e899a P 7201605bb2f54c949a4ebe6e9b3b4b2a [term 19 FOLLOWER]: Raft consensus is shut down!
W20260808 01:54:29.520845  7831 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 10390)
I20260808 01:54:29.808040  7831 tablet_server.cc:196] TabletServer@127.7.165.196:0 shutdown complete.
I20260808 01:54:29.898527  7831 master.cc:562] Master@127.7.165.254:43565 shutting down...
W20260808 01:54:29.949259  9426 auto_rebalancer.cc:342] scheduled replica move failed to complete: Service unavailable: Catalog manager is not initialized. State: Closing
W20260808 01:54:43.658305  9362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111) [suppressed 498 similar messages]
W20260808 01:54:43.662859  9362 catalog_manager.cc:4755] TS 6e7856bed1be43acaa821290d506ee97 (127.7.165.193:39359): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3b8e0e8fda6744ad877514b11658b9d9: Network error: Client connection negotiation failed: client connection to 127.7.165.193:39359: connect: Connection refused (error 111)
I20260808 01:54:46.964038  7831 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [term 1 LEADER]: Raft consensus shutting down.
I20260808 01:54:46.965274  7831 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 01:54:46.965754  7831 tablet_replica.cc:333] T 00000000000000000000000000000000 P 3cd8e8730323483db1c5f22e90bb080c: stopping tablet replica
I20260808 01:54:47.121227  7831 master.cc:584] Master@127.7.165.254:43565 shutdown complete.
[       OK ] LeaderRebalancerTest.AddTserver (350615 ms)
[ RUN      ] LeaderRebalancerTest.RestartTserver
I20260808 01:54:47.255713  7831 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.7.165.254:40793
I20260808 01:54:47.257086  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:54:47.263590 20401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:54:47.264948 20402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:54:47.265553 20404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:54:47.265933  7831 server_base.cc:1061] running on GCE node
I20260808 01:54:47.272675  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:54:47.272931  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:54:47.273113  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786154087273098 us; error 0 us; skew 500 ppm
I20260808 01:54:47.273729  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:54:47.276681  7831 webserver.cc:533] Webserver started at http://127.7.165.254:44339/ using document root <none> and password file <none>
I20260808 01:54:47.277235  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:54:47.277534  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:54:47.277889  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:54:47.279438  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/master-0-root/instance:
uuid: "a981844dfef7462fa98ccf76b69c8f69"
format_stamp: "Formatted at 2026-08-08 01:54:47 on dist-test-slave-8tdl"
I20260808 01:54:47.284629  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260808 01:54:47.288647 20409 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:54:47.289587  7831 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.001s
I20260808 01:54:47.290001  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/master-0-root
uuid: "a981844dfef7462fa98ccf76b69c8f69"
format_stamp: "Formatted at 2026-08-08 01:54:47 on dist-test-slave-8tdl"
I20260808 01:54:47.290443  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:54:47.304968  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:54:47.306962  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:54:47.351719  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.254:40793
I20260808 01:54:47.351862 20461 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.254:40793 every 8 connection(s)
I20260808 01:54:47.356210 20462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:47.369423 20462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69: Bootstrap starting.
I20260808 01:54:47.375252 20462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:47.380228 20462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69: No bootstrap required, opened a new log
I20260808 01:54:47.382838 20462 raft_consensus.cc:359] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a981844dfef7462fa98ccf76b69c8f69" member_type: VOTER }
I20260808 01:54:47.383311 20462 raft_consensus.cc:385] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:47.383601 20462 raft_consensus.cc:740] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a981844dfef7462fa98ccf76b69c8f69, State: Initialized, Role: FOLLOWER
I20260808 01:54:47.384274 20462 consensus_queue.cc:260] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a981844dfef7462fa98ccf76b69c8f69" member_type: VOTER }
I20260808 01:54:47.384765 20462 raft_consensus.cc:399] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 01:54:47.385039 20462 raft_consensus.cc:493] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 01:54:47.385327 20462 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:47.390708 20462 raft_consensus.cc:515] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a981844dfef7462fa98ccf76b69c8f69" member_type: VOTER }
I20260808 01:54:47.391417 20462 leader_election.cc:304] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a981844dfef7462fa98ccf76b69c8f69; no voters: 
I20260808 01:54:47.393056 20462 leader_election.cc:290] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260808 01:54:47.393544 20465 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:47.395125 20465 raft_consensus.cc:697] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69 [term 1 LEADER]: Becoming Leader. State: Replica: a981844dfef7462fa98ccf76b69c8f69, State: Running, Role: LEADER
I20260808 01:54:47.395929 20465 consensus_queue.cc:237] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a981844dfef7462fa98ccf76b69c8f69" member_type: VOTER }
I20260808 01:54:47.396323 20462 sys_catalog.cc:565] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 01:54:47.404347 20466 sys_catalog.cc:455] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a981844dfef7462fa98ccf76b69c8f69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a981844dfef7462fa98ccf76b69c8f69" member_type: VOTER } }
I20260808 01:54:47.404390 20467 sys_catalog.cc:455] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a981844dfef7462fa98ccf76b69c8f69. Latest consensus state: current_term: 1 leader_uuid: "a981844dfef7462fa98ccf76b69c8f69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a981844dfef7462fa98ccf76b69c8f69" member_type: VOTER } }
I20260808 01:54:47.405251 20466 sys_catalog.cc:458] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69 [sys.catalog]: This master's current role is: LEADER
I20260808 01:54:47.405388 20467 sys_catalog.cc:458] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69 [sys.catalog]: This master's current role is: LEADER
I20260808 01:54:47.407689 20473 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260808 01:54:47.416293 20473 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260808 01:54:47.417507  7831 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260808 01:54:47.426851 20473 catalog_manager.cc:1383] Generated new cluster ID: 1077d7b9ffc849fbb008bae828beafc0
I20260808 01:54:47.427238 20473 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260808 01:54:47.463316 20473 catalog_manager.cc:1406] Generated new certificate authority record
I20260808 01:54:47.464787 20473 catalog_manager.cc:1540] Loading token signing keys...
I20260808 01:54:47.482169 20473 catalog_manager.cc:6070] T 00000000000000000000000000000000 P a981844dfef7462fa98ccf76b69c8f69: Generated new TSK 0
I20260808 01:54:47.483034 20473 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260808 01:54:47.549286  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:54:47.557456 20484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:54:47.556612 20483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:54:47.559568 20486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:54:47.562316  7831 server_base.cc:1061] running on GCE node
I20260808 01:54:47.566648  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:54:47.567001  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:54:47.567175  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786154087567161 us; error 0 us; skew 500 ppm
I20260808 01:54:47.567839  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:54:47.570561  7831 webserver.cc:533] Webserver started at http://127.7.165.193:45493/ using document root <none> and password file <none>
I20260808 01:54:47.571201  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:54:47.571410  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:54:47.571677  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:54:47.573089  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-0-root/instance:
uuid: "a80df9f992014374aeef30979cfe393e"
format_stamp: "Formatted at 2026-08-08 01:54:47 on dist-test-slave-8tdl"
I20260808 01:54:47.578920  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260808 01:54:47.582875 20491 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:54:47.583828  7831 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260808 01:54:47.584175  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-0-root
uuid: "a80df9f992014374aeef30979cfe393e"
format_stamp: "Formatted at 2026-08-08 01:54:47 on dist-test-slave-8tdl"
I20260808 01:54:47.584494  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:54:47.625497  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:54:47.626904  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:54:47.628525  7831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:54:47.631264  7831 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:54:47.631505  7831 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:54:47.631764  7831 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:54:47.631950  7831 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:54:47.681047  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.193:34531
I20260808 01:54:47.681254 20555 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.193:34531 every 8 connection(s)
I20260808 01:54:47.687017  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:54:47.695055 20560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:54:47.697348 20561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:54:47.705763 20563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:54:47.708231  7831 server_base.cc:1061] running on GCE node
I20260808 01:54:47.708294 20556 heartbeater.cc:344] Connected to a master server at 127.7.165.254:40793
I20260808 01:54:47.709033 20556 heartbeater.cc:461] Registering TS with master...
I20260808 01:54:47.710320 20556 heartbeater.cc:507] Master 127.7.165.254:40793 requested a full tablet report, sending...
I20260808 01:54:47.713399 20426 ts_manager.cc:194] Registered new tserver with Master: a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:47.713594  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:54:47.713977  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:54:47.714181  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786154087714164 us; error 0 us; skew 500 ppm
I20260808 01:54:47.714949  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:54:47.715739 20426 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48828
I20260808 01:54:47.717952  7831 webserver.cc:533] Webserver started at http://127.7.165.194:46523/ using document root <none> and password file <none>
I20260808 01:54:47.718791  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:54:47.719066  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:54:47.719396  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:54:47.720844  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-1-root/instance:
uuid: "9bfb29168c8947e1b5816f1572d040cc"
format_stamp: "Formatted at 2026-08-08 01:54:47 on dist-test-slave-8tdl"
I20260808 01:54:47.727396  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.000s
I20260808 01:54:47.733259 20569 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:54:47.734225  7831 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20260808 01:54:47.734584  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-1-root
uuid: "9bfb29168c8947e1b5816f1572d040cc"
format_stamp: "Formatted at 2026-08-08 01:54:47 on dist-test-slave-8tdl"
I20260808 01:54:47.735035  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:54:47.770104  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:54:47.771584  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:54:47.773355  7831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:54:47.796329  7831 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:54:47.796804  7831 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
I20260808 01:54:47.797152  7831 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:54:47.797367  7831 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:54:47.853621  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.194:41191
I20260808 01:54:47.853858 20634 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.194:41191 every 8 connection(s)
I20260808 01:54:47.861954  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:54:47.874384 20638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:54:47.876128 20639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:54:47.877993 20641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:54:47.878290  7831 server_base.cc:1061] running on GCE node
I20260808 01:54:47.880503  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:54:47.880930  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:54:47.881173  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786154087881135 us; error 0 us; skew 500 ppm
I20260808 01:54:47.881531 20635 heartbeater.cc:344] Connected to a master server at 127.7.165.254:40793
I20260808 01:54:47.882079 20635 heartbeater.cc:461] Registering TS with master...
I20260808 01:54:47.882155  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:54:47.883278 20635 heartbeater.cc:507] Master 127.7.165.254:40793 requested a full tablet report, sending...
I20260808 01:54:47.886045  7831 webserver.cc:533] Webserver started at http://127.7.165.195:35777/ using document root <none> and password file <none>
I20260808 01:54:47.886926 20426 ts_manager.cc:194] Registered new tserver with Master: 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:47.887099  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:54:47.887606  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:54:47.888131  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:54:47.889828 20426 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48838
I20260808 01:54:47.890799  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-2-root/instance:
uuid: "598db12c38d442c6988b4c4f991874fb"
format_stamp: "Formatted at 2026-08-08 01:54:47 on dist-test-slave-8tdl"
I20260808 01:54:47.898051  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.001s
I20260808 01:54:47.902557 20647 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:54:47.903692  7831 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260808 01:54:47.904119  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-2-root
uuid: "598db12c38d442c6988b4c4f991874fb"
format_stamp: "Formatted at 2026-08-08 01:54:47 on dist-test-slave-8tdl"
I20260808 01:54:47.904500  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:54:47.947911  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:54:47.949503  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:54:47.951311  7831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:54:47.953995  7831 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:54:47.954273  7831 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:54:47.954524  7831 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:54:47.954725  7831 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:54:48.005255  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.195:44325
I20260808 01:54:48.005573 20712 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.195:44325 every 8 connection(s)
I20260808 01:54:48.011155  7831 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260808 01:54:48.019824 20716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 01:54:48.022338 20717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:54:48.026108  7831 server_base.cc:1061] running on GCE node
I20260808 01:54:48.026868 20713 heartbeater.cc:344] Connected to a master server at 127.7.165.254:40793
I20260808 01:54:48.027374 20713 heartbeater.cc:461] Registering TS with master...
W20260808 01:54:48.027293 20719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 01:54:48.028584 20713 heartbeater.cc:507] Master 127.7.165.254:40793 requested a full tablet report, sending...
I20260808 01:54:48.028609  7831 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260808 01:54:48.029007  7831 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260808 01:54:48.029223  7831 hybrid_clock.cc:648] HybridClock initialized: now 1786154088029209 us; error 0 us; skew 500 ppm
I20260808 01:54:48.030146  7831 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 01:54:48.031219 20426 ts_manager.cc:194] Registered new tserver with Master: 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:48.032917 20426 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48846
I20260808 01:54:48.033363  7831 webserver.cc:533] Webserver started at http://127.7.165.196:39113/ using document root <none> and password file <none>
I20260808 01:54:48.034013  7831 fs_manager.cc:362] Metadata directory not provided
I20260808 01:54:48.034274  7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 01:54:48.034623  7831 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 01:54:48.036110  7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-3-root/instance:
uuid: "df82e7f402364d369ed1ec3c5a8772b3"
format_stamp: "Formatted at 2026-08-08 01:54:48 on dist-test-slave-8tdl"
I20260808 01:54:48.041782  7831 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260808 01:54:48.046053 20724 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:54:48.046967  7831 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260808 01:54:48.047322  7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-3-root
uuid: "df82e7f402364d369ed1ec3c5a8772b3"
format_stamp: "Formatted at 2026-08-08 01:54:48 on dist-test-slave-8tdl"
I20260808 01:54:48.047657  7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskKwudOG/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786153669593643-7831-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 01:54:48.062963  7831 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260808 01:54:48.064631  7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 01:54:48.066259  7831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260808 01:54:48.068877  7831 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 01:54:48.069119  7831 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:54:48.069358  7831 ts_tablet_manager.cc:616] Registered 0 tablets
I20260808 01:54:48.069562  7831 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260808 01:54:48.122156  7831 rpc_server.cc:307] RPC server started. Bound to: 127.7.165.196:41117
I20260808 01:54:48.122303 20789 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.165.196:41117 every 8 connection(s)
I20260808 01:54:48.139245 20790 heartbeater.cc:344] Connected to a master server at 127.7.165.254:40793
I20260808 01:54:48.139704 20790 heartbeater.cc:461] Registering TS with master...
I20260808 01:54:48.140625 20790 heartbeater.cc:507] Master 127.7.165.254:40793 requested a full tablet report, sending...
I20260808 01:54:48.142661 20426 ts_manager.cc:194] Registered new tserver with Master: df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:48.142997  7831 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.017138091s
I20260808 01:54:48.144374 20426 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48860
I20260808 01:54:48.177278 20426 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48872:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260808 01:54:48.411566 20475 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:54:48.462181 20519 tablet_service.cc:1511] Processing CreateTablet for tablet 0add41ce95b74822ab26d48b72ccafe0 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260808 01:54:48.464359 20519 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0add41ce95b74822ab26d48b72ccafe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.464632 20518 tablet_service.cc:1511] Processing CreateTablet for tablet 70e7899e1afa4d37a1a8bd26dfa9d803 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260808 01:54:48.466316 20518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e7899e1afa4d37a1a8bd26dfa9d803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.472781 20517 tablet_service.cc:1511] Processing CreateTablet for tablet 022d6236053d4519bff37cbef43a6170 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260808 01:54:48.474474 20517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 022d6236053d4519bff37cbef43a6170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.485200 20516 tablet_service.cc:1511] Processing CreateTablet for tablet 2a9d1eec1a5840328c553a2b78d08f2a (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260808 01:54:48.487056 20516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a9d1eec1a5840328c553a2b78d08f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.492859 20515 tablet_service.cc:1511] Processing CreateTablet for tablet 0590088b74d04c6b96146266db7a6266 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260808 01:54:48.494333 20512 tablet_service.cc:1511] Processing CreateTablet for tablet 054af38e4814474c92c93f3f09db43a3 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260808 01:54:48.496173 20514 tablet_service.cc:1511] Processing CreateTablet for tablet 56010b7f5e3c48ed97bfe024f84ad2ae (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260808 01:54:48.499012 20513 tablet_service.cc:1511] Processing CreateTablet for tablet aade7cc941de4b5781fb940a6625beb3 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260808 01:54:48.499866 20749 tablet_service.cc:1511] Processing CreateTablet for tablet 0590088b74d04c6b96146266db7a6266 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260808 01:54:48.501055 20748 tablet_service.cc:1511] Processing CreateTablet for tablet 56010b7f5e3c48ed97bfe024f84ad2ae (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260808 01:54:48.501977 20752 tablet_service.cc:1511] Processing CreateTablet for tablet 0add41ce95b74822ab26d48b72ccafe0 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260808 01:54:48.504024 20747 tablet_service.cc:1511] Processing CreateTablet for tablet 21214e47c3d34850ae5c4202d6761065 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260808 01:54:48.504521 20751 tablet_service.cc:1511] Processing CreateTablet for tablet 022d6236053d4519bff37cbef43a6170 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260808 01:54:48.505620 20511 tablet_service.cc:1511] Processing CreateTablet for tablet fb4dbfa7c99f46d2a70153ce86f409fb (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260808 01:54:48.507187 20750 tablet_service.cc:1511] Processing CreateTablet for tablet f2b22456199b4e5f95a1eb974dd194fb (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260808 01:54:48.508494 20746 tablet_service.cc:1511] Processing CreateTablet for tablet 054af38e4814474c92c93f3f09db43a3 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260808 01:54:48.509819 20515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0590088b74d04c6b96146266db7a6266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.501610 20749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0590088b74d04c6b96146266db7a6266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.513736 20514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56010b7f5e3c48ed97bfe024f84ad2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.516140 20510 tablet_service.cc:1511] Processing CreateTablet for tablet e3b0638baa0f4f58b2d6146e98b8648b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260808 01:54:48.516724 20748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56010b7f5e3c48ed97bfe024f84ad2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.518261 20752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0add41ce95b74822ab26d48b72ccafe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.517449 20511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb4dbfa7c99f46d2a70153ce86f409fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.519623 20747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21214e47c3d34850ae5c4202d6761065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.520289 20513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aade7cc941de4b5781fb940a6625beb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.523128 20510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3b0638baa0f4f58b2d6146e98b8648b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.523326 20751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 022d6236053d4519bff37cbef43a6170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.523447 20745 tablet_service.cc:1511] Processing CreateTablet for tablet fb4dbfa7c99f46d2a70153ce86f409fb (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260808 01:54:48.525597 20744 tablet_service.cc:1511] Processing CreateTablet for tablet bf6d6021e1384c4eafe94f910b25d517 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260808 01:54:48.526664 20745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb4dbfa7c99f46d2a70153ce86f409fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.527630 20744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf6d6021e1384c4eafe94f910b25d517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.530566 20746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 054af38e4814474c92c93f3f09db43a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.531056 20512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 054af38e4814474c92c93f3f09db43a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.535224 20750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2b22456199b4e5f95a1eb974dd194fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.536661 20743 tablet_service.cc:1511] Processing CreateTablet for tablet 4b877573d5d6413da0a0af3fe2ecb25b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260808 01:54:48.538431 20743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b877573d5d6413da0a0af3fe2ecb25b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.573201 20519 tablet_service.cc:1511] Processing CreateTablet for tablet 4b877573d5d6413da0a0af3fe2ecb25b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260808 01:54:48.574963 20519 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b877573d5d6413da0a0af3fe2ecb25b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.584118 20598 tablet_service.cc:1511] Processing CreateTablet for tablet 022d6236053d4519bff37cbef43a6170 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260808 01:54:48.585909 20598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 022d6236053d4519bff37cbef43a6170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.588125 20597 tablet_service.cc:1511] Processing CreateTablet for tablet f2b22456199b4e5f95a1eb974dd194fb (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260808 01:54:48.589811 20597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2b22456199b4e5f95a1eb974dd194fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.592095 20596 tablet_service.cc:1511] Processing CreateTablet for tablet 2a9d1eec1a5840328c553a2b78d08f2a (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260808 01:54:48.593753 20596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a9d1eec1a5840328c553a2b78d08f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.596140 20595 tablet_service.cc:1511] Processing CreateTablet for tablet 0590088b74d04c6b96146266db7a6266 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260808 01:54:48.597806 20595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0590088b74d04c6b96146266db7a6266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.600117 20594 tablet_service.cc:1511] Processing CreateTablet for tablet 21214e47c3d34850ae5c4202d6761065 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260808 01:54:48.601775 20594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21214e47c3d34850ae5c4202d6761065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.604817 20593 tablet_service.cc:1511] Processing CreateTablet for tablet aade7cc941de4b5781fb940a6625beb3 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260808 01:54:48.606624 20593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aade7cc941de4b5781fb940a6625beb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.581925 20599 tablet_service.cc:1511] Processing CreateTablet for tablet 70e7899e1afa4d37a1a8bd26dfa9d803 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260808 01:54:48.611838 20599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e7899e1afa4d37a1a8bd26dfa9d803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.616770 20592 tablet_service.cc:1511] Processing CreateTablet for tablet 054af38e4814474c92c93f3f09db43a3 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260808 01:54:48.623453 20746 tablet_service.cc:1511] Processing CreateTablet for tablet a1af0d1fa74847a390b02d2313464926 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260808 01:54:48.625029 20746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1af0d1fa74847a390b02d2313464926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.624897 20590 tablet_service.cc:1511] Processing CreateTablet for tablet bf6d6021e1384c4eafe94f910b25d517 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260808 01:54:48.628417 20589 tablet_service.cc:1511] Processing CreateTablet for tablet e3b0638baa0f4f58b2d6146e98b8648b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260808 01:54:48.618491 20592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 054af38e4814474c92c93f3f09db43a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.639196 20590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf6d6021e1384c4eafe94f910b25d517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.640650 20519 tablet_service.cc:1511] Processing CreateTablet for tablet a1af0d1fa74847a390b02d2313464926 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260808 01:54:48.642246 20519 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1af0d1fa74847a390b02d2313464926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.643414 20589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3b0638baa0f4f58b2d6146e98b8648b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.648321 20749 tablet_service.cc:1511] Processing CreateTablet for tablet 8fc615366d544af38cf664c9ad47ab0c (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260808 01:54:48.651427 20749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fc615366d544af38cf664c9ad47ab0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.656435 20743 tablet_service.cc:1511] Processing CreateTablet for tablet 35a1e6afbae44a1db1e2a0e1f743b290 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260808 01:54:48.658106 20743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a1e6afbae44a1db1e2a0e1f743b290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.663744 20810 tablet_bootstrap.cc:492] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:48.670980 20512 tablet_service.cc:1511] Processing CreateTablet for tablet 63fb8c2f494a462686500ce10025a39f (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260808 01:54:48.672626 20512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63fb8c2f494a462686500ce10025a39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.681800 20517 tablet_service.cc:1511] Processing CreateTablet for tablet 35a1e6afbae44a1db1e2a0e1f743b290 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260808 01:54:48.687597 20517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a1e6afbae44a1db1e2a0e1f743b290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.714012 20674 tablet_service.cc:1511] Processing CreateTablet for tablet f2b22456199b4e5f95a1eb974dd194fb (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260808 01:54:48.715704 20674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2b22456199b4e5f95a1eb974dd194fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.720089 20673 tablet_service.cc:1511] Processing CreateTablet for tablet 2a9d1eec1a5840328c553a2b78d08f2a (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260808 01:54:48.721800 20673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a9d1eec1a5840328c553a2b78d08f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.723305 20556 heartbeater.cc:499] Master 127.7.165.254:40793 was elected leader, sending a full tablet report...
I20260808 01:54:48.724857 20514 tablet_service.cc:1511] Processing CreateTablet for tablet 349d2dad09a3492ca13f8784fbdb961b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260808 01:54:48.726459 20514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 349d2dad09a3492ca13f8784fbdb961b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.710485 20676 tablet_service.cc:1511] Processing CreateTablet for tablet 0add41ce95b74822ab26d48b72ccafe0 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260808 01:54:48.732441 20676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0add41ce95b74822ab26d48b72ccafe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.711997 20675 tablet_service.cc:1511] Processing CreateTablet for tablet 70e7899e1afa4d37a1a8bd26dfa9d803 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260808 01:54:48.735410 20744 tablet_service.cc:1511] Processing CreateTablet for tablet 13ab4fe0e17342379d043078e08ca66b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260808 01:54:48.735958 20675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e7899e1afa4d37a1a8bd26dfa9d803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.746065 20671 tablet_service.cc:1511] Processing CreateTablet for tablet 21214e47c3d34850ae5c4202d6761065 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260808 01:54:48.736979 20744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13ab4fe0e17342379d043078e08ca66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.761260 20671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21214e47c3d34850ae5c4202d6761065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.743973 20672 tablet_service.cc:1511] Processing CreateTablet for tablet 56010b7f5e3c48ed97bfe024f84ad2ae (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260808 01:54:48.771643 20672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56010b7f5e3c48ed97bfe024f84ad2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.723364 20750 tablet_service.cc:1511] Processing CreateTablet for tablet 349d2dad09a3492ca13f8784fbdb961b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260808 01:54:48.783578 20750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 349d2dad09a3492ca13f8784fbdb961b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.736502 20515 tablet_service.cc:1511] Processing CreateTablet for tablet 7170feed84a946ba9e383d9bfc614e61 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260808 01:54:48.817354 20515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7170feed84a946ba9e383d9bfc614e61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.820859 20810 tablet_bootstrap.cc:654] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:48.822870 20746 tablet_service.cc:1511] Processing CreateTablet for tablet 611b7203dac24e2e909dee3e0fc5fda7 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260808 01:54:48.824446 20746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 611b7203dac24e2e909dee3e0fc5fda7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.834609 20518 tablet_service.cc:1511] Processing CreateTablet for tablet 611b7203dac24e2e909dee3e0fc5fda7 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260808 01:54:48.836200 20518 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 611b7203dac24e2e909dee3e0fc5fda7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.843771 20747 tablet_service.cc:1511] Processing CreateTablet for tablet aecd7bfdd8a746239184356052321a48 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260808 01:54:48.844568 20670 tablet_service.cc:1511] Processing CreateTablet for tablet aade7cc941de4b5781fb940a6625beb3 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260808 01:54:48.845338 20747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aecd7bfdd8a746239184356052321a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.846174 20670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aade7cc941de4b5781fb940a6625beb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.852757 20669 tablet_service.cc:1511] Processing CreateTablet for tablet fb4dbfa7c99f46d2a70153ce86f409fb (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260808 01:54:48.854555 20669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb4dbfa7c99f46d2a70153ce86f409fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.854635 20516 tablet_service.cc:1511] Processing CreateTablet for tablet aecd7bfdd8a746239184356052321a48 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260808 01:54:48.856314 20516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aecd7bfdd8a746239184356052321a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.869261 20745 tablet_service.cc:1511] Processing CreateTablet for tablet f275a817cc374438aa23f63da2a627b9 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260808 01:54:48.838270 20752 tablet_service.cc:1511] Processing CreateTablet for tablet de62f661f1484ca9af89dc7df85f8b9b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260808 01:54:48.876600 20668 tablet_service.cc:1511] Processing CreateTablet for tablet bf6d6021e1384c4eafe94f910b25d517 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260808 01:54:48.878283 20668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf6d6021e1384c4eafe94f910b25d517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.878551 20512 tablet_service.cc:1511] Processing CreateTablet for tablet 28e98a7c32694002b806eddd7007e550 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260808 01:54:48.880311 20512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28e98a7c32694002b806eddd7007e550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.890144 20811 tablet_bootstrap.cc:492] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:48.895056 20635 heartbeater.cc:499] Master 127.7.165.254:40793 was elected leader, sending a full tablet report...
I20260808 01:54:48.894003 20667 tablet_service.cc:1511] Processing CreateTablet for tablet e3b0638baa0f4f58b2d6146e98b8648b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260808 01:54:48.903851 20667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3b0638baa0f4f58b2d6146e98b8648b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.904074 20745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f275a817cc374438aa23f63da2a627b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.908931 20749 tablet_service.cc:1511] Processing CreateTablet for tablet d681d5f24f674af0a5024867f8a1fb27 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260808 01:54:48.910669 20749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d681d5f24f674af0a5024867f8a1fb27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.919625 20511 tablet_service.cc:1511] Processing CreateTablet for tablet f275a817cc374438aa23f63da2a627b9 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260808 01:54:48.921401 20511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f275a817cc374438aa23f63da2a627b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.927196 20752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de62f661f1484ca9af89dc7df85f8b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.941238 20748 tablet_service.cc:1511] Processing CreateTablet for tablet 27e69fc628bc49cd8d6c653f82ef8c1d (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260808 01:54:48.943132 20748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27e69fc628bc49cd8d6c653f82ef8c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.951135 20513 tablet_service.cc:1511] Processing CreateTablet for tablet d681d5f24f674af0a5024867f8a1fb27 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260808 01:54:48.953245 20513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d681d5f24f674af0a5024867f8a1fb27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.963148 20510 tablet_service.cc:1511] Processing CreateTablet for tablet 711b7cabb802484094416a0e94a14803 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260808 01:54:48.964838 20510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 711b7cabb802484094416a0e94a14803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:48.963136 20811 tablet_bootstrap.cc:654] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:49.001406 20593 tablet_service.cc:1511] Processing CreateTablet for tablet 4b877573d5d6413da0a0af3fe2ecb25b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260808 01:54:49.003109 20593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b877573d5d6413da0a0af3fe2ecb25b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.004218 20590 tablet_service.cc:1511] Processing CreateTablet for tablet 8fc615366d544af38cf664c9ad47ab0c (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260808 01:54:49.006831 20589 tablet_service.cc:1511] Processing CreateTablet for tablet 63fb8c2f494a462686500ce10025a39f (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260808 01:54:49.006098 20590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fc615366d544af38cf664c9ad47ab0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.019922 20810 tablet_bootstrap.cc:492] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:49.020646 20810 ts_tablet_manager.cc:1403] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.357s	user 0.009s	sys 0.010s
I20260808 01:54:49.023990 20592 tablet_service.cc:1511] Processing CreateTablet for tablet 7170feed84a946ba9e383d9bfc614e61 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260808 01:54:49.024333 20810 raft_consensus.cc:359] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:49.025491 20810 raft_consensus.cc:385] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:49.025923 20810 raft_consensus.cc:740] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:49.032578 20676 tablet_service.cc:1511] Processing CreateTablet for tablet 28e98a7c32694002b806eddd7007e550 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260808 01:54:49.033768 20813 tablet_bootstrap.cc:492] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:49.034171 20676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28e98a7c32694002b806eddd7007e550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.036156 20713 heartbeater.cc:499] Master 127.7.165.254:40793 was elected leader, sending a full tablet report...
I20260808 01:54:49.036302 20598 tablet_service.cc:1511] Processing CreateTablet for tablet 35a1e6afbae44a1db1e2a0e1f743b290 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260808 01:54:49.038300 20598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35a1e6afbae44a1db1e2a0e1f743b290. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.046346 20595 tablet_service.cc:1511] Processing CreateTablet for tablet 611b7203dac24e2e909dee3e0fc5fda7 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260808 01:54:49.053609 20671 tablet_service.cc:1511] Processing CreateTablet for tablet 8fc615366d544af38cf664c9ad47ab0c (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260808 01:54:49.056393 20594 tablet_service.cc:1511] Processing CreateTablet for tablet de62f661f1484ca9af89dc7df85f8b9b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260808 01:54:49.056479 20672 tablet_service.cc:1511] Processing CreateTablet for tablet a1af0d1fa74847a390b02d2313464926 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260808 01:54:49.050403 20599 tablet_service.cc:1511] Processing CreateTablet for tablet 13ab4fe0e17342379d043078e08ca66b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260808 01:54:49.064669 20675 tablet_service.cc:1511] Processing CreateTablet for tablet 63fb8c2f494a462686500ce10025a39f (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260808 01:54:49.067405 20673 tablet_service.cc:1511] Processing CreateTablet for tablet 13ab4fe0e17342379d043078e08ca66b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260808 01:54:49.070214 20810 consensus_queue.cc:260] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:49.071345 20743 tablet_service.cc:1511] Processing CreateTablet for tablet e336b4e46ba6416e941f51812c6c38e7 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260808 01:54:49.072916 20743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e336b4e46ba6416e941f51812c6c38e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.072877 20517 tablet_service.cc:1511] Processing CreateTablet for tablet e336b4e46ba6416e941f51812c6c38e7 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260808 01:54:49.075014 20597 tablet_service.cc:1511] Processing CreateTablet for tablet 28e98a7c32694002b806eddd7007e550 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260808 01:54:49.074620 20517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e336b4e46ba6416e941f51812c6c38e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.083468 20589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63fb8c2f494a462686500ce10025a39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.096722 20668 tablet_service.cc:1511] Processing CreateTablet for tablet 349d2dad09a3492ca13f8784fbdb961b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260808 01:54:49.097950 20811 tablet_bootstrap.cc:492] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:49.098855 20811 ts_tablet_manager.cc:1403] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.209s	user 0.016s	sys 0.008s
I20260808 01:54:49.102413 20811 raft_consensus.cc:359] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:49.103586 20811 raft_consensus.cc:385] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:49.104442 20810 ts_tablet_manager.cc:1434] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.083s	user 0.016s	sys 0.009s
I20260808 01:54:49.104836 20811 raft_consensus.cc:740] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:49.107457 20815 tablet_bootstrap.cc:492] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:49.107566 20810 tablet_bootstrap.cc:492] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:49.108429 20811 consensus_queue.cc:260] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:49.115348 20810 tablet_bootstrap.cc:654] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:49.154909 20669 tablet_service.cc:1511] Processing CreateTablet for tablet de62f661f1484ca9af89dc7df85f8b9b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260808 01:54:49.120503 20515 tablet_service.cc:1511] Processing CreateTablet for tablet 20ca72baa7bc4572a51de6b57b4dc3e9 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260808 01:54:49.122341 20670 tablet_service.cc:1511] Processing CreateTablet for tablet 7170feed84a946ba9e383d9bfc614e61 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260808 01:54:49.123495 20594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de62f661f1484ca9af89dc7df85f8b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.123891 20596 tablet_service.cc:1511] Processing CreateTablet for tablet f275a817cc374438aa23f63da2a627b9 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260808 01:54:49.159740 20515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20ca72baa7bc4572a51de6b57b4dc3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.161511 20596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f275a817cc374438aa23f63da2a627b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.163691 20595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 611b7203dac24e2e909dee3e0fc5fda7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.089949 20671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fc615366d544af38cf664c9ad47ab0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.129325 20674 tablet_service.cc:1511] Processing CreateTablet for tablet aecd7bfdd8a746239184356052321a48 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260808 01:54:49.167418 20670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7170feed84a946ba9e383d9bfc614e61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.171103 20674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aecd7bfdd8a746239184356052321a48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.139492 20813 tablet_bootstrap.cc:654] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:49.147974 20790 heartbeater.cc:499] Master 127.7.165.254:40793 was elected leader, sending a full tablet report...
I20260808 01:54:49.173807 20748 tablet_service.cc:1511] Processing CreateTablet for tablet 20ca72baa7bc4572a51de6b57b4dc3e9 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260808 01:54:49.181239 20667 tablet_service.cc:1511] Processing CreateTablet for tablet d681d5f24f674af0a5024867f8a1fb27 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260808 01:54:49.182737 20597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28e98a7c32694002b806eddd7007e550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.186724 20592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7170feed84a946ba9e383d9bfc614e61. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 01:54:49.187619 20557 tablet.cc:2406] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 01:54:49.187858 20749 tablet_service.cc:1511] Processing CreateTablet for tablet fdef2db6e2e04aa795d92ad39538d90b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260808 01:54:49.189483 20749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdef2db6e2e04aa795d92ad39538d90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.191114 20599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13ab4fe0e17342379d043078e08ca66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.194288 20748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20ca72baa7bc4572a51de6b57b4dc3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.208165 20810 tablet_bootstrap.cc:492] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:49.232368 20810 ts_tablet_manager.cc:1403] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.125s	user 0.014s	sys 0.002s
I20260808 01:54:49.205076 20667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d681d5f24f674af0a5024867f8a1fb27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.235417 20675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63fb8c2f494a462686500ce10025a39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.212062 20593 tablet_service.cc:1511] Processing CreateTablet for tablet 711b7cabb802484094416a0e94a14803 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260808 01:54:49.237155 20593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 711b7cabb802484094416a0e94a14803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.237195 20668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 349d2dad09a3492ca13f8784fbdb961b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.217950 20811 ts_tablet_manager.cc:1434] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.119s	user 0.006s	sys 0.005s
I20260808 01:54:49.237321 20810 raft_consensus.cc:359] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:49.240036 20810 raft_consensus.cc:385] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:49.240376 20810 raft_consensus.cc:740] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:49.240177 20811 tablet_bootstrap.cc:492] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:49.241302 20810 consensus_queue.cc:260] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:49.243116 20672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1af0d1fa74847a390b02d2313464926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.248654 20513 tablet_service.cc:1511] Processing CreateTablet for tablet e9201fed5ac044119ff493c519d289a7 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260808 01:54:49.250590 20513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9201fed5ac044119ff493c519d289a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.252076 20811 tablet_bootstrap.cc:654] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:49.255018 20669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de62f661f1484ca9af89dc7df85f8b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.259981 20810 ts_tablet_manager.cc:1434] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.027s	user 0.005s	sys 0.004s
I20260808 01:54:49.261474 20810 tablet_bootstrap.cc:492] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:49.275359 20673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13ab4fe0e17342379d043078e08ca66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.279460 20676 tablet_service.cc:1511] Processing CreateTablet for tablet 27e69fc628bc49cd8d6c653f82ef8c1d (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260808 01:54:49.281035 20676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27e69fc628bc49cd8d6c653f82ef8c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.288913 20671 tablet_service.cc:1511] Processing CreateTablet for tablet 711b7cabb802484094416a0e94a14803 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260808 01:54:49.297468 20813 tablet_bootstrap.cc:492] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:49.298239 20813 ts_tablet_manager.cc:1403] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.265s	user 0.016s	sys 0.000s
I20260808 01:54:49.304677 20667 tablet_service.cc:1511] Processing CreateTablet for tablet 20ca72baa7bc4572a51de6b57b4dc3e9 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260808 01:54:49.191169 20590 tablet_service.cc:1511] Processing CreateTablet for tablet 27e69fc628bc49cd8d6c653f82ef8c1d (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260808 01:54:49.315655 20590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27e69fc628bc49cd8d6c653f82ef8c1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.302207 20813 raft_consensus.cc:359] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:49.317624 20813 raft_consensus.cc:385] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:49.318157 20813 raft_consensus.cc:740] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:49.309638 20743 tablet_service.cc:1511] Processing CreateTablet for tablet bcae3a1b04b44ebfa3776d88ca63cbdb (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260808 01:54:49.319651 20743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcae3a1b04b44ebfa3776d88ca63cbdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.319329 20813 consensus_queue.cc:260] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:49.317653 20515 tablet_service.cc:1511] Processing CreateTablet for tablet bcae3a1b04b44ebfa3776d88ca63cbdb (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260808 01:54:49.323930 20515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcae3a1b04b44ebfa3776d88ca63cbdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.309592 20594 tablet_service.cc:1511] Processing CreateTablet for tablet e336b4e46ba6416e941f51812c6c38e7 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260808 01:54:49.331737 20594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e336b4e46ba6416e941f51812c6c38e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.336882 20671 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 711b7cabb802484094416a0e94a14803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.339823 20667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20ca72baa7bc4572a51de6b57b4dc3e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.342741 20818 raft_consensus.cc:493] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:49.343590 20818 raft_consensus.cc:515] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:49.348596 20818 leader_election.cc:290] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:49.357574 20674 tablet_service.cc:1511] Processing CreateTablet for tablet fdef2db6e2e04aa795d92ad39538d90b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260808 01:54:49.372468 20596 tablet_service.cc:1511] Processing CreateTablet for tablet bcae3a1b04b44ebfa3776d88ca63cbdb (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260808 01:54:49.343457 20811 tablet_bootstrap.cc:492] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:49.360987 20815 tablet_bootstrap.cc:654] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:49.373929 20811 ts_tablet_manager.cc:1403] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.134s	user 0.022s	sys 0.007s
I20260808 01:54:49.374225 20596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcae3a1b04b44ebfa3776d88ca63cbdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.368952 20598 tablet_service.cc:1511] Processing CreateTablet for tablet fdef2db6e2e04aa795d92ad39538d90b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260808 01:54:49.369256 20592 tablet_service.cc:1511] Processing CreateTablet for tablet e9201fed5ac044119ff493c519d289a7 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260808 01:54:49.389041 20674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdef2db6e2e04aa795d92ad39538d90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.403852 20810 tablet_bootstrap.cc:654] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:49.420516 20749 tablet_service.cc:1511] Processing CreateTablet for tablet 73a8186fa08042f1ae22f9b47c77bd46 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260808 01:54:49.422123 20749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73a8186fa08042f1ae22f9b47c77bd46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.422402 20592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9201fed5ac044119ff493c519d289a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.438207 20811 raft_consensus.cc:359] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:49.439121 20811 raft_consensus.cc:385] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:49.439538 20811 raft_consensus.cc:740] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:49.438999 20672 tablet_service.cc:1511] Processing CreateTablet for tablet 73a8186fa08042f1ae22f9b47c77bd46 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
W20260808 01:54:49.439646 20475 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:54:49.435750 20598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdef2db6e2e04aa795d92ad39538d90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.440745 20672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73a8186fa08042f1ae22f9b47c77bd46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.440620 20816 raft_consensus.cc:493] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:49.440382 20811 consensus_queue.cc:260] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:49.477516 20593 tablet_service.cc:1511] Processing CreateTablet for tablet b9a194d9a9eb40d397c76e44235f7aed (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260808 01:54:49.478797 20813 ts_tablet_manager.cc:1434] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.180s	user 0.016s	sys 0.010s
I20260808 01:54:49.482124 20813 tablet_bootstrap.cc:492] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:49.441534 20513 tablet_service.cc:1511] Processing CreateTablet for tablet 73a8186fa08042f1ae22f9b47c77bd46 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260808 01:54:49.445755 20670 tablet_service.cc:1511] Processing CreateTablet for tablet e9201fed5ac044119ff493c519d289a7 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260808 01:54:49.497648 20593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9a194d9a9eb40d397c76e44235f7aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.502440 20669 tablet_service.cc:1511] Processing CreateTablet for tablet b9a194d9a9eb40d397c76e44235f7aed (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260808 01:54:49.471781 20816 raft_consensus.cc:515] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:49.512617 20517 tablet_service.cc:1511] Processing CreateTablet for tablet 45432e629d3b49abab9715ce5c0124d1 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260808 01:54:49.514830 20513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73a8186fa08042f1ae22f9b47c77bd46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.517592 20517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45432e629d3b49abab9715ce5c0124d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.519073 20669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9a194d9a9eb40d397c76e44235f7aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.519317 20810 tablet_bootstrap.cc:492] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:49.519855 20810 ts_tablet_manager.cc:1403] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.259s	user 0.008s	sys 0.012s
I20260808 01:54:49.527422 20810 raft_consensus.cc:359] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:49.529472 20810 raft_consensus.cc:385] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:49.529922 20810 raft_consensus.cc:740] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:49.530892 20810 consensus_queue.cc:260] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:49.528765 20670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9201fed5ac044119ff493c519d289a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.536300 20595 tablet_service.cc:1511] Processing CreateTablet for tablet 45432e629d3b49abab9715ce5c0124d1 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260808 01:54:49.537815 20595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45432e629d3b49abab9715ce5c0124d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.548849 20515 tablet_service.cc:1511] Processing CreateTablet for tablet b8135c0503a8480998454f58362956e5 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260808 01:54:49.550416 20515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8135c0503a8480998454f58362956e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.552563 20748 tablet_service.cc:1511] Processing CreateTablet for tablet b8135c0503a8480998454f58362956e5 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260808 01:54:49.554065 20748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8135c0503a8480998454f58362956e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.528795 20811 ts_tablet_manager.cc:1434] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.154s	user 0.009s	sys 0.000s
I20260808 01:54:49.564667 20675 tablet_service.cc:1511] Processing CreateTablet for tablet 45432e629d3b49abab9715ce5c0124d1 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260808 01:54:49.566256 20675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45432e629d3b49abab9715ce5c0124d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.580084 20594 tablet_service.cc:1511] Processing CreateTablet for tablet b8135c0503a8480998454f58362956e5 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260808 01:54:49.581684 20594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8135c0503a8480998454f58362956e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.595702 20828 raft_consensus.cc:493] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:49.596693 20828 raft_consensus.cc:515] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:49.600625 20815 tablet_bootstrap.cc:492] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:49.604693 20815 ts_tablet_manager.cc:1403] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.498s	user 0.021s	sys 0.012s
I20260808 01:54:49.615868 20743 tablet_service.cc:1511] Processing CreateTablet for tablet b9a194d9a9eb40d397c76e44235f7aed (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260808 01:54:49.618110 20743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9a194d9a9eb40d397c76e44235f7aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.618932 20816 leader_election.cc:290] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:49.622273 20811 tablet_bootstrap.cc:492] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:49.629077 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:54:49.631994 20725 leader_election.cc:343] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:49.674803 20515 tablet_service.cc:1511] Processing CreateTablet for tablet 8c4d9d89ee1743829635451925a89c9e (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260808 01:54:49.680332 20515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c4d9d89ee1743829635451925a89c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.681630 20592 tablet_service.cc:1511] Processing CreateTablet for tablet 48b36b5d19af469885eddc3fa63ce410 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260808 01:54:49.683205 20592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48b36b5d19af469885eddc3fa63ce410. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.700708 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:54:49.702975 20725 leader_election.cc:343] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:49.703562 20725 leader_election.cc:304] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:54:49.705329 20818 raft_consensus.cc:2749] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:49.706569 20815 raft_consensus.cc:359] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:49.707433 20815 raft_consensus.cc:385] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:49.708921 20811 tablet_bootstrap.cc:654] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:49.718340 20744 tablet_service.cc:1511] Processing CreateTablet for tablet 8c4d9d89ee1743829635451925a89c9e (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260808 01:54:49.719478 20828 leader_election.cc:290] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:49.719926 20744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c4d9d89ee1743829635451925a89c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.712430 20815 raft_consensus.cc:740] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:49.728919 20821 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:49.729628 20821 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:49.730392 20810 ts_tablet_manager.cc:1434] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.210s	user 0.024s	sys 0.008s
I20260808 01:54:49.731588 20810 tablet_bootstrap.cc:492] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:49.731271 20510 tablet_service.cc:1511] Processing CreateTablet for tablet f681a9e4ae5a4909afeec36dfb20b490 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260808 01:54:49.742981 20673 tablet_service.cc:1511] Processing CreateTablet for tablet 48b36b5d19af469885eddc3fa63ce410 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260808 01:54:49.743604 20815 consensus_queue.cc:260] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:49.744539 20673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48b36b5d19af469885eddc3fa63ce410. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.732962 20510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f681a9e4ae5a4909afeec36dfb20b490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.777776 20590 tablet_service.cc:1511] Processing CreateTablet for tablet f681a9e4ae5a4909afeec36dfb20b490 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260808 01:54:49.779446 20590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f681a9e4ae5a4909afeec36dfb20b490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.785283 20512 tablet_service.cc:1511] Processing CreateTablet for tablet 6d32821d9c17456394a662fcfd05f6c6 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260808 01:54:49.731359 20667 tablet_service.cc:1511] Processing CreateTablet for tablet 8c4d9d89ee1743829635451925a89c9e (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260808 01:54:49.787372 20667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c4d9d89ee1743829635451925a89c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.787722 20512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d32821d9c17456394a662fcfd05f6c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.779055 20752 tablet_service.cc:1511] Processing CreateTablet for tablet 48b36b5d19af469885eddc3fa63ce410 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260808 01:54:49.789822 20752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48b36b5d19af469885eddc3fa63ce410. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.792104 20745 tablet_service.cc:1511] Processing CreateTablet for tablet 6d32821d9c17456394a662fcfd05f6c6 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260808 01:54:49.794099 20745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d32821d9c17456394a662fcfd05f6c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.810667 20675 tablet_service.cc:1511] Processing CreateTablet for tablet f681a9e4ae5a4909afeec36dfb20b490 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260808 01:54:49.811368 20595 tablet_service.cc:1511] Processing CreateTablet for tablet 6d32821d9c17456394a662fcfd05f6c6 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260808 01:54:49.816202 20595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d32821d9c17456394a662fcfd05f6c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.818567 20821 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:49.812683 20810 tablet_bootstrap.cc:654] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:49.812295 20675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f681a9e4ae5a4909afeec36dfb20b490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.844282 20813 tablet_bootstrap.cc:654] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:49.865082 20670 tablet_service.cc:1511] Processing CreateTablet for tablet 0c59923b4b8949b0ae2f90ac80346e42 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260808 01:54:49.898734 20670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c59923b4b8949b0ae2f90ac80346e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.906029 20749 tablet_service.cc:1511] Processing CreateTablet for tablet 0c59923b4b8949b0ae2f90ac80346e42 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260808 01:54:49.919577 20818 raft_consensus.cc:493] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:49.920258 20818 raft_consensus.cc:515] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:49.922523 20818 leader_election.cc:290] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:49.923671 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:49.924647 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:49.925345 20530 raft_consensus.cc:2468] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 0.
I20260808 01:54:49.926820 20725 leader_election.cc:304] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:49.927474 20815 ts_tablet_manager.cc:1434] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.322s	user 0.020s	sys 0.010s
I20260808 01:54:49.928740 20815 tablet_bootstrap.cc:492] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
W20260808 01:54:49.929775 20492 leader_election.cc:343] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:49.934047 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:54:49.936311 20570 leader_election.cc:343] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:49.938095 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:49.938975 20686 raft_consensus.cc:2468] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:49.940800 20493 leader_election.cc:304] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:49.942000 20832 raft_consensus.cc:2804] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:49.942447 20832 raft_consensus.cc:493] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:49.942835 20832 raft_consensus.cc:3060] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:49.943557 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:54:49.956022 20725 leader_election.cc:343] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:49.955899 20832 raft_consensus.cc:515] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:49.957638 20818 raft_consensus.cc:2804] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:49.958165 20818 raft_consensus.cc:493] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:49.958012 20594 tablet_service.cc:1511] Processing CreateTablet for tablet d3997511f7d7419985d678e8fa59e5d0 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260808 01:54:49.959411 20749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c59923b4b8949b0ae2f90ac80346e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.959988 20594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3997511f7d7419985d678e8fa59e5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:49.965559 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:49.958703 20818 raft_consensus.cc:3060] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:54:49.967751 20571 leader_election.cc:343] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:49.968448 20571 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:54:49.969406 20821 raft_consensus.cc:2749] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:49.973141 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:49.974068 20686 raft_consensus.cc:3060] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:49.975373 20818 raft_consensus.cc:515] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:49.994467 20815 tablet_bootstrap.cc:654] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:49.994722 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
W20260808 01:54:49.996526 20725 leader_election.cc:343] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:49.997231 20818 leader_election.cc:290] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:49.999002 20686 raft_consensus.cc:2468] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:49.999083 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:54:50.000020 20530 raft_consensus.cc:3060] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:50.000624 20493 leader_election.cc:304] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:50.001652 20828 raft_consensus.cc:2804] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:50.002214 20828 raft_consensus.cc:697] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:54:50.004876 20752 tablet_service.cc:1511] Processing CreateTablet for tablet d3997511f7d7419985d678e8fa59e5d0 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260808 01:54:50.006656 20752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3997511f7d7419985d678e8fa59e5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.009554 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:49.976630 20672 tablet_service.cc:1511] Processing CreateTablet for tablet d3997511f7d7419985d678e8fa59e5d0 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260808 01:54:50.011505 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:50.011868 20832 leader_election.cc:290] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:50.012686 20672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3997511f7d7419985d678e8fa59e5d0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 01:54:50.015712 20493 leader_election.cc:343] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:50.016369 20493 leader_election.cc:304] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
W20260808 01:54:50.017781 20493 leader_election.cc:343] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:50.018641 20530 raft_consensus.cc:2468] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:54:50.019622 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:50.020128 20725 leader_election.cc:304] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:54:50.004720 20828 consensus_queue.cc:237] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:54:50.021899 20493 leader_election.cc:343] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:50.023128 20818 raft_consensus.cc:2804] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:50.025702 20832 raft_consensus.cc:2749] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:50.035636 20512 tablet_service.cc:1511] Processing CreateTablet for tablet 4f6c2d3c4da449688d1fca55d391ead2 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260808 01:54:50.037352 20512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f6c2d3c4da449688d1fca55d391ead2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.036996 20513 tablet_service.cc:1511] Processing CreateTablet for tablet 0c59923b4b8949b0ae2f90ac80346e42 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260808 01:54:50.038439 20513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c59923b4b8949b0ae2f90ac80346e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.039563 20590 tablet_service.cc:1511] Processing CreateTablet for tablet 4f6c2d3c4da449688d1fca55d391ead2 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260808 01:54:50.041272 20590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f6c2d3c4da449688d1fca55d391ead2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.051663 20669 tablet_service.cc:1511] Processing CreateTablet for tablet 4f6c2d3c4da449688d1fca55d391ead2 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260808 01:54:50.053427 20669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f6c2d3c4da449688d1fca55d391ead2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.054273 20510 tablet_service.cc:1511] Processing CreateTablet for tablet 51bbd1ceb9fb4045b31d153ff959c663 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260808 01:54:50.055749 20510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51bbd1ceb9fb4045b31d153ff959c663. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.060786 20592 tablet_service.cc:1511] Processing CreateTablet for tablet 51bbd1ceb9fb4045b31d153ff959c663 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260808 01:54:50.061056 20670 tablet_service.cc:1511] Processing CreateTablet for tablet 566d023f94bb449d8cf248744cd5b6ba (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260808 01:54:50.062402 20592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51bbd1ceb9fb4045b31d153ff959c663. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.088393 20810 tablet_bootstrap.cc:492] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:50.062660 20670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 566d023f94bb449d8cf248744cd5b6ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.070130 20811 tablet_bootstrap.cc:492] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:50.076623 20813 tablet_bootstrap.cc:492] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:50.086247 20815 tablet_bootstrap.cc:492] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:50.066381 20744 tablet_service.cc:1511] Processing CreateTablet for tablet 566d023f94bb449d8cf248744cd5b6ba (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260808 01:54:50.120759 20810 ts_tablet_manager.cc:1403] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.389s	user 0.015s	sys 0.013s
I20260808 01:54:50.115876 20818 raft_consensus.cc:697] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:54:50.121819 20811 ts_tablet_manager.cc:1403] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.500s	user 0.017s	sys 0.047s
I20260808 01:54:50.130401 20515 tablet_service.cc:1511] Processing CreateTablet for tablet 566d023f94bb449d8cf248744cd5b6ba (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260808 01:54:50.071838 20752 tablet_service.cc:1511] Processing CreateTablet for tablet 51bbd1ceb9fb4045b31d153ff959c663 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260808 01:54:50.139072 20515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 566d023f94bb449d8cf248744cd5b6ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.127549 20815 ts_tablet_manager.cc:1403] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.199s	user 0.021s	sys 0.009s
I20260808 01:54:50.144174 20815 raft_consensus.cc:359] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:50.144811 20815 raft_consensus.cc:385] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:50.145090 20815 raft_consensus.cc:740] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:50.145848 20815 consensus_queue.cc:260] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:50.147794 20744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 566d023f94bb449d8cf248744cd5b6ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.149448 20752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51bbd1ceb9fb4045b31d153ff959c663. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.149307 20811 raft_consensus.cc:359] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:50.150177 20811 raft_consensus.cc:385] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:50.150466 20811 raft_consensus.cc:740] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:50.151247 20811 consensus_queue.cc:260] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:50.154115 20598 tablet_service.cc:1511] Processing CreateTablet for tablet 62f797fc263e40c0a0380b65f9430154 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260808 01:54:50.155750 20598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f797fc263e40c0a0380b65f9430154. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.169333 20745 tablet_service.cc:1511] Processing CreateTablet for tablet 62f797fc263e40c0a0380b65f9430154 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260808 01:54:50.170277 20517 tablet_service.cc:1511] Processing CreateTablet for tablet 6e71aab2ce9b45369b7eff65e1645f9b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260808 01:54:50.175451 20811 ts_tablet_manager.cc:1434] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.043s	user 0.001s	sys 0.005s
I20260808 01:54:50.176748 20811 tablet_bootstrap.cc:492] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:50.177183 20517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e71aab2ce9b45369b7eff65e1645f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.166867 20818 consensus_queue.cc:237] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:50.180979 20815 ts_tablet_manager.cc:1434] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.040s	user 0.007s	sys 0.000s
I20260808 01:54:50.126294 20813 ts_tablet_manager.cc:1403] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.644s	user 0.035s	sys 0.016s
I20260808 01:54:50.185287 20673 tablet_service.cc:1511] Processing CreateTablet for tablet 6e71aab2ce9b45369b7eff65e1645f9b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260808 01:54:50.186903 20673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e71aab2ce9b45369b7eff65e1645f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.190855 20424 catalog_manager.cc:5697] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:50.194479 20815 tablet_bootstrap.cc:492] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:50.196435 20594 tablet_service.cc:1511] Processing CreateTablet for tablet 6e71aab2ce9b45369b7eff65e1645f9b (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260808 01:54:50.200943 20594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e71aab2ce9b45369b7eff65e1645f9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.198060 20745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f797fc263e40c0a0380b65f9430154. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.202277 20749 tablet_service.cc:1511] Processing CreateTablet for tablet ce51fcffe2ae4bcba55cd864c973bf0d (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260808 01:54:50.171185 20675 tablet_service.cc:1511] Processing CreateTablet for tablet 62f797fc263e40c0a0380b65f9430154 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260808 01:54:50.203969 20675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f797fc263e40c0a0380b65f9430154. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.174031 20810 raft_consensus.cc:359] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:50.207072 20810 raft_consensus.cc:385] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:50.207504 20810 raft_consensus.cc:740] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:50.208439 20810 consensus_queue.cc:260] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:50.227798 20515 tablet_service.cc:1511] Processing CreateTablet for tablet 0bd11a6553c444d488651367f3f0143f (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260808 01:54:50.229516 20515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bd11a6553c444d488651367f3f0143f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.210310 20513 tablet_service.cc:1511] Processing CreateTablet for tablet ce51fcffe2ae4bcba55cd864c973bf0d (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260808 01:54:50.211395 20593 tablet_service.cc:1511] Processing CreateTablet for tablet ce51fcffe2ae4bcba55cd864c973bf0d (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260808 01:54:50.215693 20749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce51fcffe2ae4bcba55cd864c973bf0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.217954 20669 tablet_service.cc:1511] Processing CreateTablet for tablet 0bd11a6553c444d488651367f3f0143f (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260808 01:54:50.231033 20593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce51fcffe2ae4bcba55cd864c973bf0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.232388 20669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bd11a6553c444d488651367f3f0143f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.220683 20815 tablet_bootstrap.cc:654] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:50.222829 20811 tablet_bootstrap.cc:654] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:50.231029 20513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce51fcffe2ae4bcba55cd864c973bf0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.218518 20743 tablet_service.cc:1511] Processing CreateTablet for tablet 0bd11a6553c444d488651367f3f0143f (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260808 01:54:50.239593 20743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bd11a6553c444d488651367f3f0143f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.186184 20813 raft_consensus.cc:359] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:50.241417 20813 raft_consensus.cc:385] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:50.241926 20813 raft_consensus.cc:740] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:50.243191 20813 consensus_queue.cc:260] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:50.259232 20810 ts_tablet_manager.cc:1434] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.133s	user 0.008s	sys 0.002s
I20260808 01:54:50.260677 20810 tablet_bootstrap.cc:492] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:50.273181 20818 raft_consensus.cc:493] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:50.273815 20818 raft_consensus.cc:515] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:50.274291 20590 tablet_service.cc:1511] Processing CreateTablet for tablet 38b9406154bf4f86bf631ea9b73424df (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260808 01:54:50.275856 20590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b9406154bf4f86bf631ea9b73424df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.276512 20818 leader_election.cc:290] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:50.279493 20813 ts_tablet_manager.cc:1434] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.097s	user 0.008s	sys 0.001s
I20260808 01:54:50.280472 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:50.281191 20530 raft_consensus.cc:2468] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 0.
I20260808 01:54:50.281286 20672 tablet_service.cc:1511] Processing CreateTablet for tablet 38b9406154bf4f86bf631ea9b73424df (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260808 01:54:50.282342 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:50.282668 20672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b9406154bf4f86bf631ea9b73424df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.284659 20725 leader_election.cc:304] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
W20260808 01:54:50.285638 20725 leader_election.cc:343] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:50.287592 20818 raft_consensus.cc:2804] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:50.288184 20818 raft_consensus.cc:493] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:50.288692 20818 raft_consensus.cc:3060] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:50.289031 20811 tablet_bootstrap.cc:492] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:50.289642 20811 ts_tablet_manager.cc:1403] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.113s	user 0.012s	sys 0.003s
I20260808 01:54:50.295588 20510 tablet_service.cc:1511] Processing CreateTablet for tablet 1753378f13594e89aee6641e2cb9e6da (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260808 01:54:50.297214 20510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1753378f13594e89aee6641e2cb9e6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.280787 20813 tablet_bootstrap.cc:492] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:50.306408 20592 tablet_service.cc:1511] Processing CreateTablet for tablet 1753378f13594e89aee6641e2cb9e6da (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260808 01:54:50.308215 20592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1753378f13594e89aee6641e2cb9e6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.308557 20811 raft_consensus.cc:359] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:50.287819 20744 tablet_service.cc:1511] Processing CreateTablet for tablet 38b9406154bf4f86bf631ea9b73424df (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260808 01:54:50.309644 20813 tablet_bootstrap.cc:654] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:50.315482 20744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38b9406154bf4f86bf631ea9b73424df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.319136 20818 raft_consensus.cc:515] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:50.318123 20425 catalog_manager.cc:5697] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196). New cstate: current_term: 1 leader_uuid: "df82e7f402364d369ed1ec3c5a8772b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:50.323262 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:54:50.324131 20530 raft_consensus.cc:3060] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:50.326619 20673 tablet_service.cc:1511] Processing CreateTablet for tablet 1753378f13594e89aee6641e2cb9e6da (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260808 01:54:50.328632 20673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1753378f13594e89aee6641e2cb9e6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.336501 20815 tablet_bootstrap.cc:492] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:50.337348 20815 ts_tablet_manager.cc:1403] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.143s	user 0.018s	sys 0.000s
I20260808 01:54:50.338613 20752 tablet_service.cc:1511] Processing CreateTablet for tablet ecc85af5215440be884412c1d240c7af (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260808 01:54:50.340106 20752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecc85af5215440be884412c1d240c7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.309428 20811 raft_consensus.cc:385] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:50.347311 20811 raft_consensus.cc:740] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:50.354406 20512 tablet_service.cc:1511] Processing CreateTablet for tablet ecc85af5215440be884412c1d240c7af (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260808 01:54:50.356200 20512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecc85af5215440be884412c1d240c7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.360639 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
W20260808 01:54:50.362419 20725 leader_election.cc:343] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:50.363287 20818 leader_election.cc:290] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:50.365135 20810 tablet_bootstrap.cc:654] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:50.366250 20815 raft_consensus.cc:359] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:50.380213 20592 tablet_service.cc:1511] Processing CreateTablet for tablet ecc85af5215440be884412c1d240c7af (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260808 01:54:50.381758 20592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecc85af5215440be884412c1d240c7af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.372596 20815 raft_consensus.cc:385] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:50.384886 20815 raft_consensus.cc:740] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:50.387004 20815 consensus_queue.cc:260] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:50.393368 20670 tablet_service.cc:1511] Processing CreateTablet for tablet 07cea569c9ad4d18a8d2f97764e15cc9 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260808 01:54:50.366865 20811 consensus_queue.cc:260] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:50.400493 20815 ts_tablet_manager.cc:1434] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.063s	user 0.008s	sys 0.000s
I20260808 01:54:50.401062 20811 ts_tablet_manager.cc:1434] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.111s	user 0.010s	sys 0.001s
I20260808 01:54:50.401875 20815 tablet_bootstrap.cc:492] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:50.402275 20811 tablet_bootstrap.cc:492] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:50.406592 20828 raft_consensus.cc:493] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:50.408923 20828 raft_consensus.cc:515] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:50.400494 20670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07cea569c9ad4d18a8d2f97764e15cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.407431 20530 raft_consensus.cc:2468] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:54:50.436105 20725 leader_election.cc:304] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:54:50.441684 20818 raft_consensus.cc:2804] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:50.444026 20818 raft_consensus.cc:697] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:54:50.445053 20818 consensus_queue.cc:237] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:50.448606 20828 leader_election.cc:290] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:50.449687 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:50.449857 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:50.450634 20761 raft_consensus.cc:2468] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
W20260808 01:54:50.451684 20492 leader_election.cc:343] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:50.453075 20493 leader_election.cc:304] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:54:50.456187 20828 raft_consensus.cc:2804] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:50.459474 20828 raft_consensus.cc:493] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:50.461058 20828 raft_consensus.cc:3060] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:54:50.462010 20475 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:54:50.456584 20515 tablet_service.cc:1511] Processing CreateTablet for tablet 07cea569c9ad4d18a8d2f97764e15cc9 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260808 01:54:50.464391 20515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07cea569c9ad4d18a8d2f97764e15cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.466192 20749 tablet_service.cc:1511] Processing CreateTablet for tablet 07cea569c9ad4d18a8d2f97764e15cc9 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260808 01:54:50.466706 20815 tablet_bootstrap.cc:654] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:50.469699 20810 tablet_bootstrap.cc:492] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:50.473799 20749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07cea569c9ad4d18a8d2f97764e15cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.477269 20675 tablet_service.cc:1511] Processing CreateTablet for tablet de206cb1608f4bd1b20ae902c82f849f (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260808 01:54:50.478900 20811 tablet_bootstrap.cc:654] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:50.485620 20675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de206cb1608f4bd1b20ae902c82f849f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.486002 20810 ts_tablet_manager.cc:1403] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.226s	user 0.016s	sys 0.012s
I20260808 01:54:50.486101 20593 tablet_service.cc:1511] Processing CreateTablet for tablet de206cb1608f4bd1b20ae902c82f849f (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260808 01:54:50.487874 20593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de206cb1608f4bd1b20ae902c82f849f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.489979 20810 raft_consensus.cc:359] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:50.490857 20810 raft_consensus.cc:385] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:50.491240 20810 raft_consensus.cc:740] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:50.486052 20752 tablet_service.cc:1511] Processing CreateTablet for tablet de206cb1608f4bd1b20ae902c82f849f (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260808 01:54:50.496446 20813 tablet_bootstrap.cc:492] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:50.498111 20673 tablet_service.cc:1511] Processing CreateTablet for tablet 8ac71e66cb4042eb8fe8aafd139af68a (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260808 01:54:50.498800 20752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de206cb1608f4bd1b20ae902c82f849f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.499810 20673 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ac71e66cb4042eb8fe8aafd139af68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.513324 20813 ts_tablet_manager.cc:1403] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.233s	user 0.018s	sys 0.026s
I20260808 01:54:50.499037 20743 tablet_service.cc:1511] Processing CreateTablet for tablet a313d0a0979e4dc8a4205c4f16a24257 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260808 01:54:50.521523 20743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a313d0a0979e4dc8a4205c4f16a24257. 1 dirs total, 0 dirs full, 0 dirs failed
W20260808 01:54:50.522626 20493 consensus_peers.cc:597] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e -> Peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Couldn't send request to peer df82e7f402364d369ed1ec3c5a8772b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:50.522826 20810 consensus_queue.cc:260] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:50.522661 20594 tablet_service.cc:1511] Processing CreateTablet for tablet 8ac71e66cb4042eb8fe8aafd139af68a (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260808 01:54:50.525274 20594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ac71e66cb4042eb8fe8aafd139af68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.531602 20513 tablet_service.cc:1511] Processing CreateTablet for tablet 8ac71e66cb4042eb8fe8aafd139af68a (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260808 01:54:50.533789 20669 tablet_service.cc:1511] Processing CreateTablet for tablet b6d8bf19c324463fb9903b9981068e9d (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260808 01:54:50.535166 20513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ac71e66cb4042eb8fe8aafd139af68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.535810 20744 tablet_service.cc:1511] Processing CreateTablet for tablet b6d8bf19c324463fb9903b9981068e9d (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260808 01:54:50.539109 20744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d8bf19c324463fb9903b9981068e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.540083 20810 ts_tablet_manager.cc:1434] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.053s	user 0.005s	sys 0.002s
I20260808 01:54:50.541308 20810 tablet_bootstrap.cc:492] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:50.543924 20669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d8bf19c324463fb9903b9981068e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.557852 20813 raft_consensus.cc:359] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:50.560727 20590 tablet_service.cc:1511] Processing CreateTablet for tablet a313d0a0979e4dc8a4205c4f16a24257 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260808 01:54:50.562288 20810 tablet_bootstrap.cc:654] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:50.562027 20512 tablet_service.cc:1511] Processing CreateTablet for tablet b6d8bf19c324463fb9903b9981068e9d (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260808 01:54:50.564167 20811 tablet_bootstrap.cc:492] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:50.562300 20590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a313d0a0979e4dc8a4205c4f16a24257. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.566913 20836 raft_consensus.cc:493] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:50.572134 20836 raft_consensus.cc:515] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:50.556787 20423 catalog_manager.cc:5697] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196). New cstate: current_term: 1 leader_uuid: "df82e7f402364d369ed1ec3c5a8772b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:50.574491 20512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d8bf19c324463fb9903b9981068e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.571213 20515 tablet_service.cc:1511] Processing CreateTablet for tablet a313d0a0979e4dc8a4205c4f16a24257 (DEFAULT_TABLE table=test-workload [id=c9e8adababd74210a87cf997fab803f0]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260808 01:54:50.575850 20515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a313d0a0979e4dc8a4205c4f16a24257. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 01:54:50.568601 20813 raft_consensus.cc:385] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:50.579819 20836 leader_election.cc:290] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:50.578467 20811 ts_tablet_manager.cc:1403] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.176s	user 0.019s	sys 0.008s
I20260808 01:54:50.571218 20828 raft_consensus.cc:515] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:50.585464 20813 raft_consensus.cc:740] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:50.603447 20813 consensus_queue.cc:260] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:50.606190 20813 ts_tablet_manager.cc:1434] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.064s	user 0.006s	sys 0.001s
I20260808 01:54:50.608983 20828 leader_election.cc:290] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:50.598593 20811 raft_consensus.cc:359] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:50.619287 20811 raft_consensus.cc:385] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:50.616696 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:50.619989 20761 raft_consensus.cc:3060] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:50.628345 20761 raft_consensus.cc:2468] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:50.629819 20493 leader_election.cc:304] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:50.630692 20836 raft_consensus.cc:493] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:50.630821 20828 raft_consensus.cc:2804] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:50.631385 20836 raft_consensus.cc:515] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:50.631721 20828 raft_consensus.cc:697] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:54:50.632738 20828 consensus_queue.cc:237] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:50.634038 20836 leader_election.cc:290] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:50.636993 20811 raft_consensus.cc:740] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:50.637801 20811 consensus_queue.cc:260] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:50.636765 20813 tablet_bootstrap.cc:492] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:50.640033 20832 raft_consensus.cc:493] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:50.640575 20832 raft_consensus.cc:515] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:50.643426 20832 leader_election.cc:290] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:50.648413 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:50.667819 20815 tablet_bootstrap.cc:492] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:50.668519 20815 ts_tablet_manager.cc:1403] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.267s	user 0.021s	sys 0.002s
W20260808 01:54:50.671082 20493 leader_election.cc:343] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:50.670115 20423 catalog_manager.cc:5697] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:50.672093 20811 ts_tablet_manager.cc:1434] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.077s	user 0.001s	sys 0.006s
I20260808 01:54:50.673385 20811 tablet_bootstrap.cc:492] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:50.680752 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:50.682626 20608 raft_consensus.cc:3060] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:50.688563 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:50.696422 20815 raft_consensus.cc:359] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:50.697530 20815 raft_consensus.cc:385] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260808 01:54:50.701795 20493 leader_election.cc:343] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:50.702392 20493 leader_election.cc:304] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:50.702886 20828 consensus_queue.cc:1048] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:50.703641 20832 raft_consensus.cc:2749] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:54:50.709318 20492 leader_election.cc:343] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:50.710299 20818 raft_consensus.cc:493] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:50.698022 20815 raft_consensus.cc:740] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:50.711723 20818 raft_consensus.cc:515] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:50.712494 20815 consensus_queue.cc:260] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:50.707091 20813 tablet_bootstrap.cc:654] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:50.721913 20815 ts_tablet_manager.cc:1434] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.053s	user 0.012s	sys 0.000s
I20260808 01:54:50.723292 20815 tablet_bootstrap.cc:492] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:50.716450 20818 leader_election.cc:290] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:50.726294 20818 consensus_queue.cc:1048] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:50.727495 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:50.730866 20821 raft_consensus.cc:493] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:50.731501 20821 raft_consensus.cc:515] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:50.733487 20815 tablet_bootstrap.cc:654] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:50.735685 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:50.730362 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:50.736506 20530 raft_consensus.cc:2468] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:54:50.743744 20570 leader_election.cc:304] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:50.739169 20821 leader_election.cc:290] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:50.760092 20818 raft_consensus.cc:493] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:50.761013 20818 raft_consensus.cc:515] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:50.765390 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:50.766847 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:54:50.768119 20725 leader_election.cc:343] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:54:50.769277 20725 leader_election.cc:343] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:50.765594 20821 raft_consensus.cc:2804] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:54:50.771986 20818 leader_election.cc:290] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:50.770999 20821 raft_consensus.cc:493] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:50.778290 20843 consensus_queue.cc:1048] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:50.781612 20821 raft_consensus.cc:3060] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:54:50.803278 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 59685.1us
I20260808 01:54:50.812482 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:50.813027 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:50.813866 20811 tablet_bootstrap.cc:654] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
W20260808 01:54:50.814788 20648 leader_election.cc:343] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:54:50.818838 20649 leader_election.cc:343] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:50.825866 20810 tablet_bootstrap.cc:492] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:50.826807 20815 tablet_bootstrap.cc:492] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:50.828480 20810 ts_tablet_manager.cc:1403] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.287s	user 0.036s	sys 0.022s
I20260808 01:54:50.835304 20815 ts_tablet_manager.cc:1403] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.112s	user 0.023s	sys 0.046s
W20260808 01:54:50.845980 20648 leader_election.cc:343] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:50.846830 20648 leader_election.cc:304] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:54:50.847999 20836 raft_consensus.cc:2749] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:50.848006 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:54:50.848562 20648 leader_election.cc:343] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:50.849067 20648 leader_election.cc:304] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:50.849915 20836 raft_consensus.cc:2749] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:50.856907 20821 raft_consensus.cc:515] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:50.859261 20609 raft_consensus.cc:1240] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260808 01:54:50.859577 20813 tablet_bootstrap.cc:492] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
W20260808 01:54:50.860036 20725 leader_election.cc:343] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:50.860482 20821 leader_election.cc:290] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:50.860642 20725 leader_election.cc:304] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:54:50.861747 20843 raft_consensus.cc:2749] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:50.860234 20813 ts_tablet_manager.cc:1403] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.224s	user 0.002s	sys 0.013s
I20260808 01:54:50.861941 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:54:50.864181 20843 consensus_queue.cc:1059] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:50.865626 20813 raft_consensus.cc:359] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:50.866428 20813 raft_consensus.cc:385] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:50.866761 20813 raft_consensus.cc:740] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:50.867614 20813 consensus_queue.cc:260] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:50.869942 20821 raft_consensus.cc:493] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:50.870548 20821 raft_consensus.cc:515] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:50.874717 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:54:50.877019 20726 leader_election.cc:343] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:50.877557 20726 leader_election.cc:304] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:54:50.878705 20813 ts_tablet_manager.cc:1434] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.016s	user 0.000s	sys 0.008s
I20260808 01:54:50.880609 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:50.881387 20813 tablet_bootstrap.cc:492] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:50.883260 20843 raft_consensus.cc:2749] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:50.881350 20686 raft_consensus.cc:2468] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:54:50.888379 20571 leader_election.cc:304] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:54:50.890465 20870 raft_consensus.cc:2804] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:50.891126 20870 raft_consensus.cc:493] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:50.891610 20870 raft_consensus.cc:3060] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:50.894419 20609 raft_consensus.cc:1240] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260808 01:54:50.848528 20815 raft_consensus.cc:359] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:50.895546 20818 raft_consensus.cc:3055] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:54:50.896796 20843 consensus_queue.cc:1059] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:50.897601 20818 raft_consensus.cc:740] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:54:50.898070 20815 raft_consensus.cc:385] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:50.898814 20818 consensus_queue.cc:260] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:50.899732 20815 raft_consensus.cc:740] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:50.906497 20815 consensus_queue.cc:260] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:50.908810 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:50.900235 20818 raft_consensus.cc:3060] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:54:50.909921 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:50.901150 20810 raft_consensus.cc:359] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:50.898658 20821 leader_election.cc:290] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:50.912367 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:50.911530 20810 raft_consensus.cc:385] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:50.915277 20810 raft_consensus.cc:740] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
W20260808 01:54:50.916043 20571 leader_election.cc:343] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:50.916169 20810 consensus_queue.cc:260] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:50.917538 20571 leader_election.cc:304] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:50.914444 20815 ts_tablet_manager.cc:1434] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.079s	user 0.009s	sys 0.001s
I20260808 01:54:50.919482 20810 ts_tablet_manager.cc:1434] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.088s	user 0.010s	sys 0.000s
I20260808 01:54:50.918955 20821 raft_consensus.cc:2749] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:54:50.919692 20811 tablet_bootstrap.cc:492] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:50.920506 20813 tablet_bootstrap.cc:654] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:50.940650 20815 tablet_bootstrap.cc:492] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:50.944343 20832 raft_consensus.cc:493] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:50.944613 20811 ts_tablet_manager.cc:1403] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.271s	user 0.018s	sys 0.019s
I20260808 01:54:50.945227 20832 raft_consensus.cc:515] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:50.940635 20810 tablet_bootstrap.cc:492] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:50.947937 20832 leader_election.cc:290] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:50.949703 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:50.950582 20761 raft_consensus.cc:2468] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:50.927534 20870 raft_consensus.cc:515] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:50.952503 20493 leader_election.cc:304] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:50.953225 20815 tablet_bootstrap.cc:654] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:50.953899 20870 leader_election.cc:290] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:50.954797 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:50.955396 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:50.955761 20686 raft_consensus.cc:3060] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:50.957980 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:50.959858 20810 tablet_bootstrap.cc:654] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
W20260808 01:54:50.960028 20492 leader_election.cc:343] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:54:50.961688 20571 leader_election.cc:343] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:50.964183 20811 raft_consensus.cc:359] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:50.955425 20832 raft_consensus.cc:2804] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:50.965142 20811 raft_consensus.cc:385] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:50.966341 20811 raft_consensus.cc:740] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:50.966604 20813 tablet_bootstrap.cc:492] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:50.967286 20813 ts_tablet_manager.cc:1403] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.086s	user 0.014s	sys 0.012s
I20260808 01:54:50.966053 20832 raft_consensus.cc:493] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:50.967409 20811 consensus_queue.cc:260] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:50.970813 20811 ts_tablet_manager.cc:1434] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.026s	user 0.010s	sys 0.000s
I20260808 01:54:50.971339 20813 raft_consensus.cc:359] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:50.972139 20811 tablet_bootstrap.cc:492] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:50.972311 20813 raft_consensus.cc:385] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:50.972802 20813 raft_consensus.cc:740] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:50.974259 20815 tablet_bootstrap.cc:492] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:50.974895 20815 ts_tablet_manager.cc:1403] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.035s	user 0.016s	sys 0.012s
I20260808 01:54:50.974640 20813 consensus_queue.cc:260] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:50.977658 20813 ts_tablet_manager.cc:1434] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260808 01:54:50.978422 20815 raft_consensus.cc:359] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:50.979440 20815 raft_consensus.cc:385] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:50.979751 20815 raft_consensus.cc:740] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:50.980659 20815 consensus_queue.cc:260] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:50.981638 20811 tablet_bootstrap.cc:654] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:50.984249 20813 tablet_bootstrap.cc:492] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:50.984329 20815 ts_tablet_manager.cc:1434] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.001s
I20260808 01:54:50.985821 20815 tablet_bootstrap.cc:492] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:50.996443 20810 tablet_bootstrap.cc:492] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:50.997138 20810 ts_tablet_manager.cc:1403] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.057s	user 0.014s	sys 0.005s
I20260808 01:54:50.997326 20815 tablet_bootstrap.cc:654] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:50.998656 20843 consensus_queue.cc:1048] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:50.968091 20832 raft_consensus.cc:3060] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:50.981880 20686 raft_consensus.cc:2468] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:54:51.001156 20810 raft_consensus.cc:359] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.002090 20870 raft_consensus.cc:493] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:51.008229 20810 raft_consensus.cc:385] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.008411 20870 raft_consensus.cc:515] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.009040 20810 raft_consensus.cc:740] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.005898 20762 raft_consensus.cc:2410] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260808 01:54:51.011973 20571 leader_election.cc:304] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:51.011826 20810 consensus_queue.cc:260] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.013411 20813 tablet_bootstrap.cc:654] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.013810 20821 raft_consensus.cc:2804] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:51.019143 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:51.027999 20821 raft_consensus.cc:697] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:54:51.029789 20821 consensus_queue.cc:237] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:51.037433 20870 leader_election.cc:290] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:51.016052 20810 ts_tablet_manager.cc:1434] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.019s	user 0.009s	sys 0.004s
W20260808 01:54:51.039283 20570 leader_election.cc:343] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:51.044811 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:54:51.049974 20725 consensus_peers.cc:597] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 -> Peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Couldn't send request to peer 9bfb29168c8947e1b5816f1572d040cc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:51.052501 20571 leader_election.cc:343] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:51.052954 20810 tablet_bootstrap.cc:492] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:51.053298 20571 leader_election.cc:304] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:54:51.054728 20870 raft_consensus.cc:2749] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:51.060271 20815 tablet_bootstrap.cc:492] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:51.060973 20815 ts_tablet_manager.cc:1403] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.075s	user 0.025s	sys 0.022s
I20260808 01:54:51.061332 20811 tablet_bootstrap.cc:492] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:51.062175 20811 ts_tablet_manager.cc:1403] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.090s	user 0.020s	sys 0.023s
I20260808 01:54:51.063665 20813 tablet_bootstrap.cc:492] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:51.064441 20813 ts_tablet_manager.cc:1403] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.080s	user 0.017s	sys 0.000s
I20260808 01:54:51.065132 20810 tablet_bootstrap.cc:654] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.064786 20815 raft_consensus.cc:359] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:51.065691 20815 raft_consensus.cc:385] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.066046 20815 raft_consensus.cc:740] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.066903 20815 consensus_queue.cc:260] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:51.067430 20811 raft_consensus.cc:359] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:51.068459 20811 raft_consensus.cc:385] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.068822 20811 raft_consensus.cc:740] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.069826 20811 consensus_queue.cc:260] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:51.070587 20815 ts_tablet_manager.cc:1434] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.003s
I20260808 01:54:51.071847 20815 tablet_bootstrap.cc:492] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:51.084092 20815 tablet_bootstrap.cc:654] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.083673 20813 raft_consensus.cc:359] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:51.084544 20813 raft_consensus.cc:385] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.084880 20813 raft_consensus.cc:740] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.085796 20813 consensus_queue.cc:260] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:51.093586 20821 raft_consensus.cc:493] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:51.094139 20821 raft_consensus.cc:515] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:51.094592 20832 raft_consensus.cc:515] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:51.096519 20836 raft_consensus.cc:493] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:51.097283 20836 raft_consensus.cc:515] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.098737 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:51.099651 20762 raft_consensus.cc:2393] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:54:51.100121 20813 ts_tablet_manager.cc:1434] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.035s	user 0.007s	sys 0.002s
I20260808 01:54:51.102257 20813 tablet_bootstrap.cc:492] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:51.104679 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:54:51.107388 20648 leader_election.cc:343] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:51.107939 20821 leader_election.cc:290] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:51.108397 20832 leader_election.cc:290] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:51.110442 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:54:51.113061 20648 leader_election.cc:343] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:51.113678 20813 tablet_bootstrap.cc:654] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.113740 20648 leader_election.cc:304] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:54:51.109543 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:51.109650 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:51.110414 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:51.118446 20570 leader_election.cc:304] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:51.118664 20762 raft_consensus.cc:3060] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:54:51.121098 20492 leader_election.cc:343] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:51.120081 20821 raft_consensus.cc:3060] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.108798 20836 leader_election.cc:290] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:51.127364 20836 raft_consensus.cc:2749] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:51.088874 20424 catalog_manager.cc:5697] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194). New cstate: current_term: 1 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:51.136214 20821 raft_consensus.cc:2749] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:51.145084 20832 consensus_queue.cc:1048] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:51.154277 20836 raft_consensus.cc:493] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:51.155110 20836 raft_consensus.cc:515] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:51.168777 20811 ts_tablet_manager.cc:1434] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.106s	user 0.007s	sys 0.000s
I20260808 01:54:51.173408 20828 consensus_queue.cc:1048] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:51.192489 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:51.193300 20761 raft_consensus.cc:2468] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:54:51.196470 20811 tablet_bootstrap.cc:492] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:51.201210 20762 raft_consensus.cc:2468] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:51.203637 20649 leader_election.cc:304] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:51.204823 20493 leader_election.cc:304] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:54:51.189458 20836 leader_election.cc:290] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:51.216805 20828 raft_consensus.cc:2804] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:51.217957 20836 raft_consensus.cc:2804] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:51.218022 20828 raft_consensus.cc:697] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:54:51.217370 20813 tablet_bootstrap.cc:492] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:51.218554 20836 raft_consensus.cc:493] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:51.220367 20836 raft_consensus.cc:3060] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.221067 20828 consensus_queue.cc:237] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:51.223651 20813 ts_tablet_manager.cc:1403] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.122s	user 0.008s	sys 0.032s
I20260808 01:54:51.241760 20821 raft_consensus.cc:493] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:51.241848 20836 raft_consensus.cc:515] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:51.247793 20810 tablet_bootstrap.cc:492] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:51.248492 20810 ts_tablet_manager.cc:1403] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.196s	user 0.053s	sys 0.086s
I20260808 01:54:51.250795 20815 tablet_bootstrap.cc:492] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:51.242612 20821 raft_consensus.cc:515] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:51.251538 20836 leader_election.cc:290] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:51.252228 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:51.252939 20761 raft_consensus.cc:3060] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.253304 20821 leader_election.cc:290] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:51.257332 20811 tablet_bootstrap.cc:654] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.256563 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:51.258370 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:54:51.268348 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 155362us
I20260808 01:54:51.268989 20686 raft_consensus.cc:2393] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 1: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:54:51.268841 20810 raft_consensus.cc:359] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.269649 20810 raft_consensus.cc:385] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.270013 20810 raft_consensus.cc:740] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.270941 20810 consensus_queue.cc:260] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.272286 20571 leader_election.cc:304] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:51.262475 20761 raft_consensus.cc:2468] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:54:51.273244 20810 ts_tablet_manager.cc:1434] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.024s	user 0.005s	sys 0.000s
I20260808 01:54:51.273432 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:51.274185 20608 raft_consensus.cc:3060] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.274031 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:51.239952 20813 raft_consensus.cc:359] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:51.276729 20813 raft_consensus.cc:385] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.277078 20813 raft_consensus.cc:740] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.278368 20813 consensus_queue.cc:260] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:51.279122 20649 leader_election.cc:304] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:51.280337 20836 raft_consensus.cc:2804] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:51.280866 20836 raft_consensus.cc:697] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:54:51.281731 20836 consensus_queue.cc:237] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:51.283318 20813 ts_tablet_manager.cc:1434] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.059s	user 0.004s	sys 0.006s
I20260808 01:54:51.284888 20813 tablet_bootstrap.cc:492] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:51.290781 20608 raft_consensus.cc:2468] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:54:51.291421 20870 raft_consensus.cc:2749] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:51.263759 20815 ts_tablet_manager.cc:1403] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.192s	user 0.027s	sys 0.037s
I20260808 01:54:51.295521 20813 tablet_bootstrap.cc:654] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.301455 20815 raft_consensus.cc:359] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.302353 20815 raft_consensus.cc:385] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.302661 20815 raft_consensus.cc:740] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.303499 20815 consensus_queue.cc:260] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.305186 20815 ts_tablet_manager.cc:1434] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.000s
I20260808 01:54:51.306267 20815 tablet_bootstrap.cc:492] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:51.308840 20810 tablet_bootstrap.cc:492] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:51.323346 20815 tablet_bootstrap.cc:654] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.327929 20810 tablet_bootstrap.cc:654] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.330255 20423 catalog_manager.cc:5697] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:51.346067 20813 tablet_bootstrap.cc:492] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:51.346992 20813 ts_tablet_manager.cc:1403] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.062s	user 0.020s	sys 0.017s
I20260808 01:54:51.352630 20892 raft_consensus.cc:493] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:51.353341 20892 raft_consensus.cc:515] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:51.354899 20813 raft_consensus.cc:359] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:51.355793 20892 leader_election.cc:290] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:51.355937 20813 raft_consensus.cc:385] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.356423 20813 raft_consensus.cc:740] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.357268 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:51.357394 20813 consensus_queue.cc:260] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:51.358142 20529 raft_consensus.cc:2468] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:54:51.359508 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:54:51.361480 20649 leader_election.cc:343] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:51.366406 20813 ts_tablet_manager.cc:1434] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.004s
I20260808 01:54:51.367118 20811 tablet_bootstrap.cc:492] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:51.367604 20813 tablet_bootstrap.cc:492] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:51.368173 20648 leader_election.cc:304] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:51.369187 20836 raft_consensus.cc:2804] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:51.369963 20836 raft_consensus.cc:493] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:51.370266 20843 raft_consensus.cc:493] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:51.370362 20836 raft_consensus.cc:3060] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.367686 20811 ts_tablet_manager.cc:1403] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.171s	user 0.034s	sys 0.026s
I20260808 01:54:51.370922 20843 raft_consensus.cc:515] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:51.373431 20843 leader_election.cc:290] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:51.375448 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:51.376147 20811 raft_consensus.cc:359] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:51.375650 20426 catalog_manager.cc:5697] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 598db12c38d442c6988b4c4f991874fb (127.7.165.195). New cstate: current_term: 1 leader_uuid: "598db12c38d442c6988b4c4f991874fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:51.376930 20811 raft_consensus.cc:385] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.377270 20811 raft_consensus.cc:740] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.369527 20870 raft_consensus.cc:493] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:51.378151 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:51.378451 20870 raft_consensus.cc:515] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:51.379101 20810 tablet_bootstrap.cc:492] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:51.379400 20686 raft_consensus.cc:2393] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 1: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:54:51.379616 20810 ts_tablet_manager.cc:1403] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.071s	user 0.018s	sys 0.003s
I20260808 01:54:51.380162 20811 consensus_queue.cc:260] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:51.381003 20870 leader_election.cc:290] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:51.382406 20811 ts_tablet_manager.cc:1434] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.001s
I20260808 01:54:51.383033 20810 raft_consensus.cc:359] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:51.383800 20810 raft_consensus.cc:385] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.383725 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:51.384238 20810 raft_consensus.cc:740] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.384523 20761 raft_consensus.cc:2468] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:54:51.385015 20810 consensus_queue.cc:260] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:51.381006 20726 leader_election.cc:304] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:54:51.387390 20571 leader_election.cc:304] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:51.388360 20843 raft_consensus.cc:3060] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.389532 20811 tablet_bootstrap.cc:492] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:51.392179 20870 raft_consensus.cc:2804] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:51.392673 20870 raft_consensus.cc:493] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:51.393056 20870 raft_consensus.cc:3060] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.383244 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:51.396149 20686 raft_consensus.cc:2468] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:54:51.399310 20810 ts_tablet_manager.cc:1434] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.000s
I20260808 01:54:51.400516 20810 tablet_bootstrap.cc:492] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:51.404307 20813 tablet_bootstrap.cc:654] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.409291 20836 raft_consensus.cc:515] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:51.409785 20811 tablet_bootstrap.cc:654] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.411972 20836 leader_election.cc:290] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:51.412930 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
W20260808 01:54:51.414904 20649 leader_election.cc:343] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:51.416673 20810 tablet_bootstrap.cc:654] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.416971 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:54:51.417711 20529 raft_consensus.cc:3060] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.417943 20843 raft_consensus.cc:2749] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:51.425282 20815 tablet_bootstrap.cc:492] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:51.426074 20815 ts_tablet_manager.cc:1403] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.120s	user 0.019s	sys 0.015s
I20260808 01:54:51.422134 20870 raft_consensus.cc:515] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:51.433666 20813 tablet_bootstrap.cc:492] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:51.434347 20813 ts_tablet_manager.cc:1403] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.067s	user 0.010s	sys 0.008s
I20260808 01:54:51.430073 20815 raft_consensus.cc:359] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:51.435595 20815 raft_consensus.cc:385] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.435941 20815 raft_consensus.cc:740] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.436442 20811 tablet_bootstrap.cc:492] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:51.436062 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:51.437105 20811 ts_tablet_manager.cc:1403] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.048s	user 0.010s	sys 0.005s
I20260808 01:54:51.437978 20813 raft_consensus.cc:359] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.438917 20815 consensus_queue.cc:260] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:51.439406 20813 raft_consensus.cc:385] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.439731 20813 raft_consensus.cc:740] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.441386 20815 ts_tablet_manager.cc:1434] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.015s	user 0.009s	sys 0.000s
I20260808 01:54:51.441181 20813 consensus_queue.cc:260] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.442608 20815 tablet_bootstrap.cc:492] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:51.442229 20811 raft_consensus.cc:359] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:51.443329 20811 raft_consensus.cc:385] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.443655 20811 raft_consensus.cc:740] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.444087 20813 ts_tablet_manager.cc:1434] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.002s
I20260808 01:54:51.438299 20870 leader_election.cc:290] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:51.444850 20828 raft_consensus.cc:493] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:51.444521 20811 consensus_queue.cc:260] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:51.447371 20811 ts_tablet_manager.cc:1434] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260808 01:54:51.438393 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:51.445497 20828 raft_consensus.cc:515] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:51.447989 20761 raft_consensus.cc:3060] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.449687 20686 raft_consensus.cc:3060] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.450156 20828 leader_election.cc:290] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:51.447518 20810 tablet_bootstrap.cc:492] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:51.452029 20810 ts_tablet_manager.cc:1403] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.052s	user 0.007s	sys 0.009s
I20260808 01:54:51.454020 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:51.455502 20810 raft_consensus.cc:359] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.456348 20810 raft_consensus.cc:385] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.456662 20810 raft_consensus.cc:740] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.457782 20811 tablet_bootstrap.cc:492] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:51.459455 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:54:51.459811 20493 leader_election.cc:343] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:51.460350 20870 consensus_queue.cc:1048] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:51.468382 20685 raft_consensus.cc:2468] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:51.470453 20493 leader_election.cc:304] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:51.472563 20828 raft_consensus.cc:2804] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:51.473042 20828 raft_consensus.cc:493] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:51.473436 20828 raft_consensus.cc:3060] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:54:51.476564 20475 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:54:51.465535 20813 tablet_bootstrap.cc:492] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:51.466110 20810 consensus_queue.cc:260] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.467561 20815 tablet_bootstrap.cc:654] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.487275 20529 raft_consensus.cc:2468] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:54:51.488803 20648 leader_election.cc:304] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:51.489799 20836 raft_consensus.cc:2804] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:51.490403 20836 raft_consensus.cc:697] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:54:51.491818 20836 consensus_queue.cc:237] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:51.499361 20761 raft_consensus.cc:2468] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:54:51.498134 20813 tablet_bootstrap.cc:654] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.501092 20571 leader_election.cc:304] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:51.502904 20870 raft_consensus.cc:2804] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:51.503438 20870 raft_consensus.cc:697] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:54:51.504294 20870 consensus_queue.cc:237] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:51.507429 20896 raft_consensus.cc:493] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:51.508064 20896 raft_consensus.cc:515] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.486195 20810 ts_tablet_manager.cc:1434] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.034s	user 0.007s	sys 0.000s
I20260808 01:54:51.509871 20810 tablet_bootstrap.cc:492] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:51.511353 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:51.512348 20608 raft_consensus.cc:2468] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:51.513747 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:51.514020 20492 leader_election.cc:304] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:51.514638 20685 raft_consensus.cc:2468] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:51.511389 20896 leader_election.cc:290] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:51.515705 20896 raft_consensus.cc:2804] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:51.515719 20811 tablet_bootstrap.cc:654] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.519766 20896 raft_consensus.cc:493] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:51.514849 20686 raft_consensus.cc:2468] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:54:51.520196 20896 raft_consensus.cc:3060] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.525158 20828 raft_consensus.cc:515] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:51.528793 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:51.529719 20686 raft_consensus.cc:3060] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.537901 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:51.539170 20810 tablet_bootstrap.cc:654] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
W20260808 01:54:51.540280 20493 leader_election.cc:343] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:51.542591 20828 leader_election.cc:290] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:51.546862 20762 raft_consensus.cc:3060] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.555197 20870 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:51.556661 20426 catalog_manager.cc:5697] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 598db12c38d442c6988b4c4f991874fb (127.7.165.195). New cstate: current_term: 1 leader_uuid: "598db12c38d442c6988b4c4f991874fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:51.560346 20896 raft_consensus.cc:515] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.568473 20896 leader_election.cc:290] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:51.569803 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:51.570611 20685 raft_consensus.cc:3060] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.575598 20815 tablet_bootstrap.cc:492] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:51.576267 20815 ts_tablet_manager.cc:1403] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.134s	user 0.012s	sys 0.002s
I20260808 01:54:51.584801 20815 raft_consensus.cc:359] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.585902 20815 raft_consensus.cc:385] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.586366 20815 raft_consensus.cc:740] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.587541 20815 consensus_queue.cc:260] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.596398 20815 ts_tablet_manager.cc:1434] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.002s
I20260808 01:54:51.597950 20815 tablet_bootstrap.cc:492] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:51.598397 20810 tablet_bootstrap.cc:492] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:51.617628 20810 ts_tablet_manager.cc:1403] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.108s	user 0.020s	sys 0.003s
I20260808 01:54:51.604454 20685 raft_consensus.cc:2468] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:51.588737 20896 raft_consensus.cc:493] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:51.619822 20813 tablet_bootstrap.cc:492] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:51.620456 20813 ts_tablet_manager.cc:1403] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.155s	user 0.010s	sys 0.005s
I20260808 01:54:51.622128 20828 raft_consensus.cc:493] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:51.622772 20828 raft_consensus.cc:515] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.625056 20810 raft_consensus.cc:359] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.630914 20810 raft_consensus.cc:385] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.631356 20810 raft_consensus.cc:740] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.588944 20870 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.629707 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:51.633330 20684 raft_consensus.cc:2468] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:51.633874 20815 tablet_bootstrap.cc:654] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.625188 20828 leader_election.cc:290] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:51.621235 20896 raft_consensus.cc:515] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.636588 20896 leader_election.cc:290] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:51.637501 20810 consensus_queue.cc:260] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.640159 20493 leader_election.cc:304] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
W20260808 01:54:51.621342 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 71962.1us
I20260808 01:54:51.641222 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63fb8c2f494a462686500ce10025a39f" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:51.642318 20686 raft_consensus.cc:2468] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:51.642953 20915 raft_consensus.cc:493] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:51.643401 20811 tablet_bootstrap.cc:492] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:51.643509 20915 raft_consensus.cc:515] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:51.644023 20811 ts_tablet_manager.cc:1403] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.186s	user 0.011s	sys 0.006s
I20260808 01:54:51.626816 20424 catalog_manager.cc:5697] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194). New cstate: current_term: 1 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:51.645411 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:51.646039 20608 raft_consensus.cc:3060] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.627801 20832 consensus_queue.cc:1048] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260808 01:54:51.649672 20821 consensus_queue.cc:1048] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:51.654348 20811 raft_consensus.cc:359] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:51.655156 20811 raft_consensus.cc:385] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.655441 20811 raft_consensus.cc:740] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.655591 20608 raft_consensus.cc:2468] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:51.656499 20811 consensus_queue.cc:260] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:51.659205 20811 ts_tablet_manager.cc:1434] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.015s	user 0.003s	sys 0.004s
I20260808 01:54:51.660322 20811 tablet_bootstrap.cc:492] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:51.658835 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63fb8c2f494a462686500ce10025a39f" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:51.662047 20896 raft_consensus.cc:2804] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:51.663954 20810 ts_tablet_manager.cc:1434] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.046s	user 0.007s	sys 0.001s
I20260808 01:54:51.664510 20493 leader_election.cc:304] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:51.665835 20896 raft_consensus.cc:697] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:54:51.666723 20915 leader_election.cc:290] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:51.667232 20896 consensus_queue.cc:237] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.668865 20828 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:51.669449 20828 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.671167 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:54:51.671356 20493 leader_election.cc:343] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:51.672288 20493 leader_election.cc:304] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:51.673328 20870 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:51.674117 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:51.674341 20832 raft_consensus.cc:2804] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:51.675072 20832 raft_consensus.cc:493] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:51.675119 20529 raft_consensus.cc:2468] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:54:51.675670 20832 raft_consensus.cc:3060] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.672472 20836 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:51.676849 20836 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.677692 20570 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:51.679738 20810 tablet_bootstrap.cc:492] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:51.671818 20828 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:51.685982 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:51.687320 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:51.688035 20684 raft_consensus.cc:2468] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:54:51.690879 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:51.691151 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:51.691638 20529 raft_consensus.cc:2468] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:54:51.691983 20609 raft_consensus.cc:2468] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:51.661784 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:51.695909 20648 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:51.696939 20892 raft_consensus.cc:2804] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:51.697376 20892 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:51.697957 20892 raft_consensus.cc:3060] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:54:51.697228 20492 leader_election.cc:343] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:51.702863 20492 leader_election.cc:304] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:54:51.704640 20492 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:51.693261 20828 raft_consensus.cc:2804] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:51.694211 20836 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:51.728003 20813 raft_consensus.cc:359] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.729151 20813 raft_consensus.cc:385] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.729501 20813 raft_consensus.cc:740] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.734884 20914 raft_consensus.cc:2804] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:51.735706 20914 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:51.736169 20914 raft_consensus.cc:3060] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.730507 20813 consensus_queue.cc:260] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.753839 20813 ts_tablet_manager.cc:1434] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.123s	user 0.009s	sys 0.001s
I20260808 01:54:51.746616 20843 raft_consensus.cc:493] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:51.755882 20813 tablet_bootstrap.cc:492] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:51.750515 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:51.759420 20843 raft_consensus.cc:515] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:51.761837 20843 leader_election.cc:290] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:51.762807 20922 raft_consensus.cc:2749] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:51.762916 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:51.764321 20684 raft_consensus.cc:2393] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:54:51.772640 20815 tablet_bootstrap.cc:492] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:51.773284 20815 ts_tablet_manager.cc:1403] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.176s	user 0.022s	sys 0.028s
I20260808 01:54:51.776608 20815 raft_consensus.cc:359] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.777350 20815 raft_consensus.cc:385] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.777667 20815 raft_consensus.cc:740] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
W20260808 01:54:51.748659 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 76448.1us
I20260808 01:54:51.779234 20815 consensus_queue.cc:260] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.787163 20813 tablet_bootstrap.cc:654] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.789417 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:51.790266 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:51.791177 20571 leader_election.cc:304] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:51.792776 20810 tablet_bootstrap.cc:654] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.756484 20896 raft_consensus.cc:2804] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:51.798790 20892 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.802942 20870 raft_consensus.cc:2749] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:51.806413 20684 raft_consensus.cc:2393] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 1: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:54:51.784543 20828 raft_consensus.cc:697] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:54:51.800916 20896 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:51.811199 20836 consensus_queue.cc:1048] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:51.813295 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:51.813872 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:54:51.811650 20892 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:51.816434 20815 ts_tablet_manager.cc:1434] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.043s	user 0.007s	sys 0.002s
I20260808 01:54:51.817941 20815 tablet_bootstrap.cc:492] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:51.816335 20896 raft_consensus.cc:3060] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.828943 20606 raft_consensus.cc:3060] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.834434 20832 raft_consensus.cc:515] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.835594 20811 tablet_bootstrap.cc:654] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.820698 20914 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.837730 20608 raft_consensus.cc:2393] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 1: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:54:51.838517 20843 raft_consensus.cc:493] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:51.839277 20843 raft_consensus.cc:515] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:51.841144 20914 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:51.841482 20843 leader_election.cc:290] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:51.842315 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:54:51.843326 20832 leader_election.cc:290] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:54:51.849840 20492 leader_election.cc:343] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:51.852584 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:51.853266 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:51.853600 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:51.853737 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:51.854010 20685 raft_consensus.cc:3060] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.854764 20683 raft_consensus.cc:2393] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 1: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
W20260808 01:54:51.855082 20726 leader_election.cc:343] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:54:51.857538 20725 leader_election.cc:343] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:51.858134 20725 leader_election.cc:304] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:54:51.859633 20843 raft_consensus.cc:2749] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:51.860687 20571 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:54:51.862058 20870 raft_consensus.cc:2749] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:54:51.863709 20685 raft_consensus.cc:2468] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:51.868057 20493 leader_election.cc:304] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:51.871141 20832 raft_consensus.cc:2804] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:51.868024 20928 consensus_queue.cc:1048] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260808 01:54:51.872902 20810 tablet_bootstrap.cc:492] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:51.873520 20810 ts_tablet_manager.cc:1403] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.194s	user 0.019s	sys 0.003s
I20260808 01:54:51.874132 20811 tablet_bootstrap.cc:492] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:51.874312 20832 raft_consensus.cc:697] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:54:51.875685 20832 consensus_queue.cc:237] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.877071 20811 ts_tablet_manager.cc:1403] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.217s	user 0.014s	sys 0.005s
I20260808 01:54:51.878010 20810 raft_consensus.cc:359] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.878839 20810 raft_consensus.cc:385] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.878460 20896 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.879238 20810 raft_consensus.cc:740] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.881088 20530 raft_consensus.cc:2393] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:54:51.880319 20811 raft_consensus.cc:359] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:51.881582 20811 raft_consensus.cc:385] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.881947 20811 raft_consensus.cc:740] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.871814 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:51.884087 20648 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:54:51.848035 20828 consensus_queue.cc:237] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:51.887205 20813 tablet_bootstrap.cc:492] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:51.887887 20813 ts_tablet_manager.cc:1403] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.133s	user 0.006s	sys 0.013s
I20260808 01:54:51.884308 20606 raft_consensus.cc:3060] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:51.887419 20810 consensus_queue.cc:260] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:51.891413 20811 consensus_queue.cc:260] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:51.895037 20815 tablet_bootstrap.cc:654] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.897666 20811 ts_tablet_manager.cc:1434] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.020s	user 0.009s	sys 0.000s
I20260808 01:54:51.886509 20896 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:51.903180 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:51.908655 20811 tablet_bootstrap.cc:492] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:51.910132 20836 consensus_queue.cc:1048] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:51.893206 20892 raft_consensus.cc:2749] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:54:51.917362 20810 ts_tablet_manager.cc:1434] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.043s	user 0.002s	sys 0.006s
I20260808 01:54:51.919023 20810 tablet_bootstrap.cc:492] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:51.921959 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:51.940358 20813 raft_consensus.cc:359] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:51.941756 20813 raft_consensus.cc:385] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:51.943888 20609 raft_consensus.cc:2393] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 1: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:54:51.937667 20685 raft_consensus.cc:2393] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 1: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:54:51.942138 20813 raft_consensus.cc:740] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:51.951182 20811 tablet_bootstrap.cc:654] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.955622 20424 catalog_manager.cc:5697] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:51.959784 20813 consensus_queue.cc:260] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:51.962150 20810 tablet_bootstrap.cc:654] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:51.962718 20870 consensus_queue.cc:1048] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:51.968267 20492 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:54:51.969200 20928 raft_consensus.cc:2749] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:54:51.976128 20813 ts_tablet_manager.cc:1434] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.088s	user 0.011s	sys 0.000s
I20260808 01:54:51.988543 20606 raft_consensus.cc:2468] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:51.996016 20813 tablet_bootstrap.cc:492] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:52.013598 20815 tablet_bootstrap.cc:492] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:52.014474 20815 ts_tablet_manager.cc:1403] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.197s	user 0.007s	sys 0.032s
I20260808 01:54:52.016723 20606 raft_consensus.cc:3060] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.015231 20938 raft_consensus.cc:493] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:52.021628 20938 raft_consensus.cc:515] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:52.024099 20938 leader_election.cc:290] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:52.026012 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:52.027639 20530 raft_consensus.cc:2468] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:54:52.028259 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:52.028517 20870 consensus_queue.cc:1048] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:52.029002 20609 raft_consensus.cc:2468] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:54:52.018565 20815 raft_consensus.cc:359] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:52.033972 20815 raft_consensus.cc:385] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.034718 20815 raft_consensus.cc:740] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.035713 20648 leader_election.cc:304] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:52.037298 20938 raft_consensus.cc:2804] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:52.037632 20725 leader_election.cc:304] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:54:52.046399 20815 consensus_queue.cc:260] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:52.037698 20938 raft_consensus.cc:493] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:52.039565 20843 raft_consensus.cc:3060] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.051867 20938 raft_consensus.cc:3060] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.060313 20815 ts_tablet_manager.cc:1434] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.045s	user 0.009s	sys 0.000s
I20260808 01:54:52.065644 20815 tablet_bootstrap.cc:492] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
W20260808 01:54:52.071108 20649 consensus_peers.cc:597] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb -> Peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Couldn't send request to peer df82e7f402364d369ed1ec3c5a8772b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:52.083256 20811 tablet_bootstrap.cc:492] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:52.083937 20811 ts_tablet_manager.cc:1403] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.176s	user 0.014s	sys 0.010s
I20260808 01:54:52.087293 20811 raft_consensus.cc:359] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:52.088052 20811 raft_consensus.cc:385] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.088341 20811 raft_consensus.cc:740] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.089123 20811 consensus_queue.cc:260] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:52.092242 20813 tablet_bootstrap.cc:654] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.093782 20948 consensus_queue.cc:1048] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:52.107283 20423 catalog_manager.cc:5697] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:52.124929 20813 tablet_bootstrap.cc:492] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:52.125548 20813 ts_tablet_manager.cc:1403] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.130s	user 0.008s	sys 0.012s
I20260808 01:54:52.104281 20843 raft_consensus.cc:2749] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:52.129670 20813 raft_consensus.cc:359] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:52.130524 20813 raft_consensus.cc:385] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.130837 20813 raft_consensus.cc:740] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.133419 20938 raft_consensus.cc:515] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:52.136443 20815 tablet_bootstrap.cc:654] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.137113 20813 consensus_queue.cc:260] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:52.139680 20813 ts_tablet_manager.cc:1434] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.004s
I20260808 01:54:52.141485 20813 tablet_bootstrap.cc:492] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:52.150535 20423 catalog_manager.cc:5697] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:52.151678 20938 leader_election.cc:290] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:52.154981 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:54:52.155416 20813 tablet_bootstrap.cc:654] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.156001 20529 raft_consensus.cc:3060] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.159227 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:52.160185 20606 raft_consensus.cc:3060] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.180646 20606 raft_consensus.cc:2468] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:54:52.184109 20648 leader_election.cc:304] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:54:52.185220 20938 raft_consensus.cc:2804] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:52.185783 20938 raft_consensus.cc:697] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:54:52.186650 20938 consensus_queue.cc:237] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:52.182585 20529 raft_consensus.cc:2468] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:54:52.195190 20810 tablet_bootstrap.cc:492] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:52.195858 20810 ts_tablet_manager.cc:1403] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.277s	user 0.027s	sys 0.035s
I20260808 01:54:52.196723 20836 raft_consensus.cc:493] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:52.197311 20836 raft_consensus.cc:515] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:52.199745 20836 leader_election.cc:290] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:52.201898 20810 raft_consensus.cc:359] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.202478 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:52.204340 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:54:52.206066 20648 leader_election.cc:343] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:54:52.206908 20648 leader_election.cc:343] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:52.207361 20648 leader_election.cc:304] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:54:52.207809 20811 ts_tablet_manager.cc:1434] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.124s	user 0.011s	sys 0.004s
I20260808 01:54:52.209028 20811 tablet_bootstrap.cc:492] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:52.209246 20836 raft_consensus.cc:2749] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:52.202809 20810 raft_consensus.cc:385] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.213786 20810 raft_consensus.cc:740] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.214643 20810 consensus_queue.cc:260] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.216584 20828 consensus_queue.cc:1048] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:52.218082 20954 raft_consensus.cc:493] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:52.218658 20954 raft_consensus.cc:515] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.221025 20954 leader_election.cc:290] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:52.222280 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:52.224339 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:54:52.225644 20725 leader_election.cc:343] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:52.216938 20810 ts_tablet_manager.cc:1434] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.001s
I20260808 01:54:52.227979 20810 tablet_bootstrap.cc:492] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
W20260808 01:54:52.229815 20725 leader_election.cc:343] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:52.230376 20725 leader_election.cc:304] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:54:52.231266 20954 raft_consensus.cc:2749] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:52.234088 20828 consensus_queue.cc:1048] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:52.238654 20815 tablet_bootstrap.cc:492] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:52.239387 20815 ts_tablet_manager.cc:1403] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.178s	user 0.021s	sys 0.016s
I20260808 01:54:52.242937 20815 raft_consensus.cc:359] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.243686 20815 raft_consensus.cc:385] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.243957 20815 raft_consensus.cc:740] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.251119 20815 consensus_queue.cc:260] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.253584 20815 ts_tablet_manager.cc:1434] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260808 01:54:52.257462 20813 tablet_bootstrap.cc:492] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:52.258131 20813 ts_tablet_manager.cc:1403] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.117s	user 0.020s	sys 0.003s
I20260808 01:54:52.269004 20828 consensus_queue.cc:1048] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:52.269713 20811 tablet_bootstrap.cc:654] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.275885 20815 tablet_bootstrap.cc:492] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:52.279742 20424 catalog_manager.cc:5697] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 598db12c38d442c6988b4c4f991874fb (127.7.165.195). New cstate: current_term: 1 leader_uuid: "598db12c38d442c6988b4c4f991874fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:52.285156 20810 tablet_bootstrap.cc:654] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.286657 20813 raft_consensus.cc:359] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:52.289314 20813 raft_consensus.cc:385] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.289638 20813 raft_consensus.cc:740] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.292533 20813 consensus_queue.cc:260] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:52.297444 20813 ts_tablet_manager.cc:1434] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.039s	user 0.004s	sys 0.004s
I20260808 01:54:52.299765 20815 tablet_bootstrap.cc:654] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.310042 20954 raft_consensus.cc:493] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:52.311173 20954 raft_consensus.cc:515] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:52.335506 20915 raft_consensus.cc:493] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:52.336333 20915 raft_consensus.cc:515] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:52.353536 20896 raft_consensus.cc:493] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:54:52.354528 20896 raft_consensus.cc:515] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.371698 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:52.372972 20760 raft_consensus.cc:2468] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:54:52.373834 20832 consensus_queue.cc:1048] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:52.378952 20493 leader_election.cc:304] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:52.380491 20832 raft_consensus.cc:2804] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:54:52.381167 20832 raft_consensus.cc:493] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:54:52.381754 20832 raft_consensus.cc:3060] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:54:52.388183 20813 tablet_bootstrap.cc:492] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:52.395767 20954 leader_election.cc:290] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:52.399824 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:52.400954 20683 raft_consensus.cc:2468] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 0.
I20260808 01:54:52.403273 20726 leader_election.cc:304] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:52.403544 20915 leader_election.cc:290] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:52.404263 20828 raft_consensus.cc:493] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:52.404803 20828 raft_consensus.cc:515] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.408286 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:52.409083 20915 raft_consensus.cc:493] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:52.409700 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:52.409606 20915 raft_consensus.cc:515] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:52.408363 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:54:52.413779 20571 leader_election.cc:343] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:52.414467 20915 leader_election.cc:290] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:52.408311 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:52.417260 20606 raft_consensus.cc:2468] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 0.
W20260808 01:54:52.419137 20493 leader_election.cc:343] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:54:52.414572 20571 leader_election.cc:343] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:52.422434 20811 tablet_bootstrap.cc:492] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:52.423033 20811 ts_tablet_manager.cc:1403] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.214s	user 0.012s	sys 0.037s
I20260808 01:54:52.425436 20810 tablet_bootstrap.cc:492] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:52.426961 20571 leader_election.cc:304] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:52.428854 20810 ts_tablet_manager.cc:1403] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.201s	user 0.028s	sys 0.031s
I20260808 01:54:52.431883 20915 raft_consensus.cc:2749] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:52.431716 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:52.433188 20810 raft_consensus.cc:359] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.433620 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:52.434105 20810 raft_consensus.cc:385] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.434497 20810 raft_consensus.cc:740] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.428150 20815 tablet_bootstrap.cc:492] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:52.431711 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:52.435448 20606 raft_consensus.cc:2393] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 2: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:54:52.435983 20813 tablet_bootstrap.cc:654] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.436535 20815 ts_tablet_manager.cc:1403] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.182s	user 0.026s	sys 0.015s
I20260808 01:54:52.415566 20896 leader_election.cc:290] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:52.426873 20828 leader_election.cc:290] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
W20260808 01:54:52.436831 20492 leader_election.cc:343] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:52.436759 20928 consensus_queue.cc:1048] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:52.426849 20811 raft_consensus.cc:359] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.437580 20492 leader_election.cc:304] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:52.437999 20811 raft_consensus.cc:385] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.438410 20828 raft_consensus.cc:2749] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:52.439172 20811 raft_consensus.cc:740] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.439993 20811 consensus_queue.cc:260] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.440869 20815 raft_consensus.cc:359] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:52.441682 20815 raft_consensus.cc:385] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.442446 20811 ts_tablet_manager.cc:1434] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.019s	user 0.004s	sys 0.002s
I20260808 01:54:52.442593 20815 raft_consensus.cc:740] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.436120 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:52.444355 20683 raft_consensus.cc:2468] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:54:52.442483 20810 consensus_queue.cc:260] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.443625 20811 tablet_bootstrap.cc:492] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:52.446398 20571 leader_election.cc:304] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:54:52.447331 20915 raft_consensus.cc:2804] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:52.447745 20915 raft_consensus.cc:493] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:52.448119 20915 raft_consensus.cc:3060] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.453461 20811 tablet_bootstrap.cc:654] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.420426 20954 raft_consensus.cc:2804] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:52.455334 20760 raft_consensus.cc:2468] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:54:52.457083 20832 raft_consensus.cc:515] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.457552 20915 raft_consensus.cc:515] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:52.457880 20954 raft_consensus.cc:493] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:52.458969 20954 raft_consensus.cc:3060] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.461445 20810 ts_tablet_manager.cc:1434] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.032s	user 0.003s	sys 0.003s
I20260808 01:54:52.462981 20810 tablet_bootstrap.cc:492] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:52.444352 20815 consensus_queue.cc:260] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:52.469053 20815 ts_tablet_manager.cc:1434] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.032s	user 0.006s	sys 0.000s
I20260808 01:54:52.470131 20815 tablet_bootstrap.cc:492] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:52.480958 20954 raft_consensus.cc:515] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:54:52.490300 20475 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:54:52.491833 20896 consensus_queue.cc:1048] T 2a9d1eec1a5840328c553a2b78d08f2a P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:52.503497 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:52.504390 20606 raft_consensus.cc:2393] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 1: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:54:52.512343 20813 tablet_bootstrap.cc:492] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:52.512766 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:52.513267 20813 ts_tablet_manager.cc:1403] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.215s	user 0.001s	sys 0.024s
I20260808 01:54:52.513513 20686 raft_consensus.cc:3060] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.518586 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:52.519564 20813 raft_consensus.cc:359] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.520463 20954 leader_election.cc:290] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:52.494022 20915 leader_election.cc:290] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:52.496797 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:52.500150 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:52.526373 20761 raft_consensus.cc:2393] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 1: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:54:52.528208 20811 tablet_bootstrap.cc:492] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:52.528771 20811 ts_tablet_manager.cc:1403] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.085s	user 0.023s	sys 0.046s
I20260808 01:54:52.529847 20571 leader_election.cc:304] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:52.531175 20915 raft_consensus.cc:2749] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:54:52.531723 20811 raft_consensus.cc:359] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.532467 20811 raft_consensus.cc:385] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.532748 20811 raft_consensus.cc:740] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.533742 20811 consensus_queue.cc:260] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.526548 20686 raft_consensus.cc:2468] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:54:52.536474 20811 ts_tablet_manager.cc:1434] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260808 01:54:52.537254 20726 leader_election.cc:304] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:54:52.538141 20811 tablet_bootstrap.cc:492] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:52.538251 20761 raft_consensus.cc:3060] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.538710 20954 raft_consensus.cc:2804] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:52.539861 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:52.541754 20954 raft_consensus.cc:697] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:54:52.542062 20609 raft_consensus.cc:2393] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 2: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:54:52.542837 20954 consensus_queue.cc:237] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:52.540473 20832 leader_election.cc:290] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:52.540808 20813 raft_consensus.cc:385] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.544834 20813 raft_consensus.cc:740] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.545053 20811 tablet_bootstrap.cc:654] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.545895 20813 consensus_queue.cc:260] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.548777 20813 ts_tablet_manager.cc:1434] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.035s	user 0.011s	sys 0.000s
I20260808 01:54:52.550026 20813 tablet_bootstrap.cc:492] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:52.558085 20813 tablet_bootstrap.cc:654] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.565446 20815 tablet_bootstrap.cc:654] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.572233 20760 raft_consensus.cc:2468] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:54:52.576789 20493 leader_election.cc:304] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:54:52.577579 20810 tablet_bootstrap.cc:654] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.578542 20832 raft_consensus.cc:2804] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:54:52.579298 20832 raft_consensus.cc:697] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 2 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:54:52.580489 20832 consensus_queue.cc:237] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.595935 20811 tablet_bootstrap.cc:492] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:52.596557 20811 ts_tablet_manager.cc:1403] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.059s	user 0.020s	sys 0.017s
I20260808 01:54:52.598414 20836 consensus_queue.cc:1048] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:52.599565 20811 raft_consensus.cc:359] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.600303 20811 raft_consensus.cc:385] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.600579 20811 raft_consensus.cc:740] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.603511 20870 raft_consensus.cc:493] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:52.604075 20870 raft_consensus.cc:515] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.603246 20423 catalog_manager.cc:5697] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196). New cstate: current_term: 1 leader_uuid: "df82e7f402364d369ed1ec3c5a8772b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:52.606211 20811 consensus_queue.cc:260] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.607618 20870 leader_election.cc:290] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:52.608348 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:52.609558 20760 raft_consensus.cc:2468] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:54:52.607836 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:52.614620 20571 leader_election.cc:304] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
W20260808 01:54:52.615449 20570 leader_election.cc:343] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:52.617818 20870 raft_consensus.cc:2804] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:52.620751 20870 raft_consensus.cc:493] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:52.621236 20870 raft_consensus.cc:3060] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.644742 20811 ts_tablet_manager.cc:1434] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.048s	user 0.007s	sys 0.001s
I20260808 01:54:52.647262 20811 tablet_bootstrap.cc:492] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:52.656420 20810 tablet_bootstrap.cc:492] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:52.657028 20810 ts_tablet_manager.cc:1403] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.194s	user 0.011s	sys 0.004s
I20260808 01:54:52.660022 20810 raft_consensus.cc:359] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.660758 20810 raft_consensus.cc:385] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.661026 20810 raft_consensus.cc:740] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.661195 20811 tablet_bootstrap.cc:654] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.662427 20815 tablet_bootstrap.cc:492] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:52.662137 20810 consensus_queue.cc:260] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.663043 20815 ts_tablet_manager.cc:1403] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.193s	user 0.023s	sys 0.018s
I20260808 01:54:52.664384 20810 ts_tablet_manager.cc:1434] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260808 01:54:52.667349 20810 tablet_bootstrap.cc:492] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:52.669216 20815 raft_consensus.cc:359] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.670255 20815 raft_consensus.cc:385] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.673084 20971 raft_consensus.cc:493] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:52.673688 20971 raft_consensus.cc:515] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.670604 20815 raft_consensus.cc:740] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.675611 20815 consensus_queue.cc:260] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.676164 20971 leader_election.cc:290] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:52.683342 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:52.677949 20815 ts_tablet_manager.cc:1434] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260808 01:54:52.679512 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:54:52.686254 20725 leader_election.cc:343] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:52.689432 20938 consensus_queue.cc:1048] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:52.690162 20870 raft_consensus.cc:515] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.695081 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:52.695822 20761 raft_consensus.cc:3060] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.696367 20815 tablet_bootstrap.cc:492] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:52.686391 20425 catalog_manager.cc:5697] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 1 to 2, leader changed from df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196) to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 2 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:52.691574 20811 tablet_bootstrap.cc:492] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:52.704383 20761 raft_consensus.cc:2468] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:54:52.688230 20686 raft_consensus.cc:2468] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 0.
I20260808 01:54:52.724805 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:54:52.725071 20571 leader_election.cc:304] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:52.725531 20726 leader_election.cc:304] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:54:52.726046 20810 tablet_bootstrap.cc:654] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
W20260808 01:54:52.726523 20570 leader_election.cc:343] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:52.726861 20971 raft_consensus.cc:2804] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:52.727272 20971 raft_consensus.cc:493] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:52.727599 20971 raft_consensus.cc:3060] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.714711 20811 ts_tablet_manager.cc:1403] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.068s	user 0.016s	sys 0.005s
I20260808 01:54:52.712648 20813 tablet_bootstrap.cc:492] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:52.731274 20813 ts_tablet_manager.cc:1403] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.181s	user 0.017s	sys 0.000s
I20260808 01:54:52.732481 20811 raft_consensus.cc:359] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.733208 20811 raft_consensus.cc:385] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.733476 20811 raft_consensus.cc:740] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.733718 20813 raft_consensus.cc:359] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:52.734405 20813 raft_consensus.cc:385] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.726539 20914 raft_consensus.cc:2804] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:52.723520 20870 leader_election.cc:290] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:52.735225 20914 raft_consensus.cc:697] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:54:52.736172 20914 consensus_queue.cc:237] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.734254 20811 consensus_queue.cc:260] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.739471 20813 raft_consensus.cc:740] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.741564 20813 consensus_queue.cc:260] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:52.744094 20813 ts_tablet_manager.cc:1434] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260808 01:54:52.745496 20813 tablet_bootstrap.cc:492] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:52.739560 20811 ts_tablet_manager.cc:1434] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260808 01:54:52.748330 20811 tablet_bootstrap.cc:492] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:52.749212 20815 tablet_bootstrap.cc:654] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.748400 20971 raft_consensus.cc:515] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.756093 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:52.756785 20686 raft_consensus.cc:3060] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.765307 20686 raft_consensus.cc:2468] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:54:52.767905 20726 leader_election.cc:304] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:52.769007 20843 raft_consensus.cc:2804] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:52.769565 20843 raft_consensus.cc:697] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:54:52.770489 20896 raft_consensus.cc:493] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:52.770455 20843 consensus_queue.cc:237] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.771147 20896 raft_consensus.cc:515] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.774541 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:52.775645 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
W20260808 01:54:52.776350 20493 leader_election.cc:343] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:52.777287 20810 tablet_bootstrap.cc:492] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
W20260808 01:54:52.777601 20725 leader_election.cc:343] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:52.777895 20810 ts_tablet_manager.cc:1403] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.111s	user 0.014s	sys 0.012s
I20260808 01:54:52.779775 20896 leader_election.cc:290] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:52.780051 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:52.780776 20761 raft_consensus.cc:2468] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:52.781059 20810 raft_consensus.cc:359] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:52.781810 20810 raft_consensus.cc:385] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.782078 20810 raft_consensus.cc:740] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.782516 20493 leader_election.cc:304] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:54:52.783088 20810 consensus_queue.cc:260] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:52.783927 20896 raft_consensus.cc:2804] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:52.784363 20896 raft_consensus.cc:493] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:52.784714 20896 raft_consensus.cc:3060] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.766433 20971 leader_election.cc:290] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:52.785903 20810 ts_tablet_manager.cc:1434] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260808 01:54:52.787164 20810 tablet_bootstrap.cc:492] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:52.794308 20810 tablet_bootstrap.cc:654] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.802392 20896 raft_consensus.cc:515] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.806649 20811 tablet_bootstrap.cc:654] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.807564 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
W20260808 01:54:52.810561 20493 leader_election.cc:343] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:52.811355 20896 leader_election.cc:290] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:52.813014 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:52.813931 20761 raft_consensus.cc:3060] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.817358 20423 catalog_manager.cc:5697] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196). New cstate: current_term: 1 leader_uuid: "df82e7f402364d369ed1ec3c5a8772b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:52.823117 20761 raft_consensus.cc:2468] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:52.828984 20810 tablet_bootstrap.cc:492] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:52.829769 20810 ts_tablet_manager.cc:1403] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.043s	user 0.020s	sys 0.009s
I20260808 01:54:52.830749 20493 leader_election.cc:304] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:54:52.837167 20896 raft_consensus.cc:2804] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:52.837707 20896 raft_consensus.cc:697] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:54:52.838554 20896 consensus_queue.cc:237] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.844760 20810 raft_consensus.cc:359] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.845686 20810 raft_consensus.cc:385] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.846024 20810 raft_consensus.cc:740] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.846938 20813 tablet_bootstrap.cc:654] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.846851 20810 consensus_queue.cc:260] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.847922 20811 tablet_bootstrap.cc:492] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:52.854143 20811 ts_tablet_manager.cc:1403] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.106s	user 0.009s	sys 0.011s
I20260808 01:54:52.857599 20811 raft_consensus.cc:359] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:52.858341 20811 raft_consensus.cc:385] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.858640 20811 raft_consensus.cc:740] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.859373 20811 consensus_queue.cc:260] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:52.861507 20811 ts_tablet_manager.cc:1434] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.003s
I20260808 01:54:52.851114 20843 raft_consensus.cc:493] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:52.849674 20423 catalog_manager.cc:5697] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194). New cstate: current_term: 1 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:52.863054 20843 raft_consensus.cc:515] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.865005 20815 tablet_bootstrap.cc:492] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:52.865638 20815 ts_tablet_manager.cc:1403] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.169s	user 0.023s	sys 0.014s
I20260808 01:54:52.865908 20843 leader_election.cc:290] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:52.862437 20896 raft_consensus.cc:493] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:52.867051 20896 raft_consensus.cc:515] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:52.868436 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:52.869002 20815 raft_consensus.cc:359] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.869689 20815 raft_consensus.cc:385] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.869946 20815 raft_consensus.cc:740] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.849306 20810 ts_tablet_manager.cc:1434] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.019s	user 0.012s	sys 0.000s
I20260808 01:54:52.870633 20815 consensus_queue.cc:260] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.871529 20810 tablet_bootstrap.cc:492] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
W20260808 01:54:52.871815 20725 leader_election.cc:343] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:52.872684 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:52.872983 20815 ts_tablet_manager.cc:1434] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260808 01:54:52.873396 20609 raft_consensus.cc:2468] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:52.873982 20815 tablet_bootstrap.cc:492] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:52.874747 20896 leader_election.cc:290] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:52.881080 20815 tablet_bootstrap.cc:654] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.874984 20492 leader_election.cc:304] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:52.883852 20810 tablet_bootstrap.cc:654] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.884917 20896 raft_consensus.cc:2804] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:52.885763 20896 raft_consensus.cc:493] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:52.886246 20896 raft_consensus.cc:3060] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.887399 20832 raft_consensus.cc:493] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:52.888245 20832 raft_consensus.cc:515] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.890280 20832 leader_election.cc:290] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:52.892671 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:52.893159 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:52.893455 20686 raft_consensus.cc:2468] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:52.893851 20761 raft_consensus.cc:2468] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:52.895803 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:52.896797 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:54:52.898375 20725 leader_election.cc:343] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:52.899099 20725 leader_election.cc:304] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:54:52.898553 20493 leader_election.cc:304] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:52.900496 20843 raft_consensus.cc:2749] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:52.901315 20813 tablet_bootstrap.cc:492] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:52.901576 20928 raft_consensus.cc:2804] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:52.901881 20813 ts_tablet_manager.cc:1403] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.157s	user 0.020s	sys 0.022s
I20260808 01:54:52.903030 20928 raft_consensus.cc:493] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:52.903594 20928 raft_consensus.cc:3060] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.905535 20810 tablet_bootstrap.cc:492] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:52.906072 20810 ts_tablet_manager.cc:1403] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.035s	user 0.010s	sys 0.004s
I20260808 01:54:52.906293 20813 raft_consensus.cc:359] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.907172 20813 raft_consensus.cc:385] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.907455 20813 raft_consensus.cc:740] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.875475 20811 tablet_bootstrap.cc:492] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:52.908216 20813 consensus_queue.cc:260] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.910318 20813 ts_tablet_manager.cc:1434] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260808 01:54:52.911618 20813 tablet_bootstrap.cc:492] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:52.913795 20928 raft_consensus.cc:515] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
W20260808 01:54:52.917797 20493 leader_election.cc:343] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:52.918566 20813 tablet_bootstrap.cc:654] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.919759 20810 raft_consensus.cc:359] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.920960 20896 raft_consensus.cc:515] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:52.921262 20810 raft_consensus.cc:385] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.921996 20810 raft_consensus.cc:740] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.923076 20810 consensus_queue.cc:260] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:52.929257 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:52.924288 20425 catalog_manager.cc:5697] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:52.932365 20813 tablet_bootstrap.cc:492] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:52.932540 20810 ts_tablet_manager.cc:1434] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.026s	user 0.008s	sys 0.000s
I20260808 01:54:52.933058 20813 ts_tablet_manager.cc:1403] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.022s	user 0.011s	sys 0.007s
I20260808 01:54:52.933607 20810 tablet_bootstrap.cc:492] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:52.934988 20815 tablet_bootstrap.cc:492] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:52.935555 20815 ts_tablet_manager.cc:1403] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.062s	user 0.012s	sys 0.000s
I20260808 01:54:52.937198 20811 tablet_bootstrap.cc:654] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.938650 20815 raft_consensus.cc:359] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:52.939404 20815 raft_consensus.cc:385] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.939689 20815 raft_consensus.cc:740] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.936136 20813 raft_consensus.cc:359] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:52.940758 20813 raft_consensus.cc:385] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:52.940579 20815 consensus_queue.cc:260] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:52.941095 20813 raft_consensus.cc:740] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:52.941025 20914 raft_consensus.cc:493] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:54:52.932273 20493 leader_election.cc:343] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:52.941816 20914 raft_consensus.cc:515] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.942483 20813 consensus_queue.cc:260] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:52.944072 20914 leader_election.cc:290] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:52.944784 20813 ts_tablet_manager.cc:1434] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260808 01:54:52.945825 20813 tablet_bootstrap.cc:492] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:52.945871 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:52.946900 20529 raft_consensus.cc:2468] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:54:52.947211 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:52.948351 20570 leader_election.cc:304] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
W20260808 01:54:52.949075 20571 leader_election.cc:343] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:52.951117 20914 raft_consensus.cc:2804] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:52.951505 20896 leader_election.cc:290] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:52.951618 20914 raft_consensus.cc:493] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:52.951987 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:52.952623 20928 leader_election.cc:290] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:52.952822 20810 tablet_bootstrap.cc:654] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.953692 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:52.954314 20685 raft_consensus.cc:3060] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.952844 20761 raft_consensus.cc:3060] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.957353 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:52.958011 20609 raft_consensus.cc:3060] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.952061 20914 raft_consensus.cc:3060] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:52.970883 20813 tablet_bootstrap.cc:654] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:52.972900 20843 consensus_queue.cc:1048] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:52.980569 20685 raft_consensus.cc:2468] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:52.982920 20915 raft_consensus.cc:493] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:52.983539 20915 raft_consensus.cc:515] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:52.987145 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:52.989296 20915 leader_election.cc:290] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:52.989771 20815 ts_tablet_manager.cc:1434] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.054s	user 0.008s	sys 0.000s
I20260808 01:54:52.990738 20493 leader_election.cc:304] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:52.991772 20928 raft_consensus.cc:2804] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:52.992429 20928 raft_consensus.cc:697] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:54:52.993013 20761 raft_consensus.cc:2468] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:52.990995 20815 tablet_bootstrap.cc:492] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:52.995797 20760 raft_consensus.cc:1275] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:54:52.997784 20493 leader_election.cc:304] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:54:52.993341 20928 consensus_queue.cc:237] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:52.999816 20832 raft_consensus.cc:2804] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:53.000564 20832 raft_consensus.cc:697] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:54:53.001677 20832 consensus_queue.cc:237] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:53.002878 20828 consensus_queue.cc:1048] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:54:53.005300 20914 raft_consensus.cc:515] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:53.014422 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:53.015584 20813 tablet_bootstrap.cc:492] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:53.015786 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:54:53.016268 20813 ts_tablet_manager.cc:1403] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.071s	user 0.017s	sys 0.012s
I20260808 01:54:53.016547 20529 raft_consensus.cc:3060] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:53.019526 20813 raft_consensus.cc:359] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.020395 20813 raft_consensus.cc:385] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:53.020772 20813 raft_consensus.cc:740] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
W20260808 01:54:53.019814 20571 leader_election.cc:343] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:53.021569 20813 consensus_queue.cc:260] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.027020 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:53.027916 20914 leader_election.cc:290] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:53.029680 20815 tablet_bootstrap.cc:654] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:53.032806 20813 ts_tablet_manager.cc:1434] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260808 01:54:53.032919 20571 leader_election.cc:304] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:53.034607 20914 raft_consensus.cc:2749] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:54:53.033078 20609 raft_consensus.cc:2468] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:53.042449 20813 tablet_bootstrap.cc:492] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:53.027057 20529 raft_consensus.cc:2468] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:54:53.047458 20570 leader_election.cc:304] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:53.048121 20843 consensus_queue.cc:1048] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:53.048874 20914 raft_consensus.cc:2804] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:53.049525 20914 raft_consensus.cc:697] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:54:53.050427 20914 consensus_queue.cc:237] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:53.052609 20836 raft_consensus.cc:493] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:53.053176 20836 raft_consensus.cc:515] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:53.058277 20813 tablet_bootstrap.cc:654] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:53.059540 20832 raft_consensus.cc:493] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:53.060220 20832 raft_consensus.cc:515] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:53.061702 20836 leader_election.cc:290] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:53.066457 20836 raft_consensus.cc:493] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:53.067822 20836 raft_consensus.cc:515] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:53.072027 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:53.072970 20609 raft_consensus.cc:2468] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:54:53.074666 20648 leader_election.cc:304] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:54:53.075842 20979 raft_consensus.cc:2804] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:53.076390 20979 raft_consensus.cc:493] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:53.076820 20979 raft_consensus.cc:3060] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:53.079802 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:53.081086 20762 raft_consensus.cc:2393] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 1: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:54:53.066872 20843 raft_consensus.cc:493] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:53.085579 20828 raft_consensus.cc:493] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:53.085960 20832 leader_election.cc:290] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:53.086351 20828 raft_consensus.cc:515] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:53.089139 20828 leader_election.cc:290] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:53.083168 20843 raft_consensus.cc:515] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:53.090377 20423 catalog_manager.cc:5697] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:53.091768 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:53.093773 20811 tablet_bootstrap.cc:492] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:53.094491 20811 ts_tablet_manager.cc:1403] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.219s	user 0.034s	sys 0.023s
I20260808 01:54:53.095948 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:54:53.096218 20493 leader_election.cc:343] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:54:53.098577 20725 leader_election.cc:343] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:53.100652 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:53.101612 20685 raft_consensus.cc:2468] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:53.102725 20606 raft_consensus.cc:1275] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:54:53.104125 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:53.106112 20493 leader_election.cc:304] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:53.105659 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:54:53.108664 20649 leader_election.cc:343] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:53.108781 20896 raft_consensus.cc:2804] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:53.109493 20896 raft_consensus.cc:493] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:53.109925 20896 raft_consensus.cc:3060] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:53.111855 20832 consensus_queue.cc:1048] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:53.114161 20423 catalog_manager.cc:5697] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:53.121394 20811 raft_consensus.cc:359] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:53.122540 20811 raft_consensus.cc:385] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:53.122936 20811 raft_consensus.cc:740] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:53.148644 21005 raft_consensus.cc:493] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:53.132915 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:53.159296 20810 tablet_bootstrap.cc:492] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:53.159840 20810 ts_tablet_manager.cc:1403] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.226s	user 0.033s	sys 0.047s
I20260808 01:54:53.152473 20843 leader_election.cc:290] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:53.167577 20811 consensus_queue.cc:260] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:53.169999 20811 ts_tablet_manager.cc:1434] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.075s	user 0.008s	sys 0.003s
I20260808 01:54:53.171074 20811 tablet_bootstrap.cc:492] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:53.172703 20813 tablet_bootstrap.cc:492] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:53.173273 20813 ts_tablet_manager.cc:1403] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.137s	user 0.017s	sys 0.011s
I20260808 01:54:53.175489 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:53.177148 20811 tablet_bootstrap.cc:654] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:53.178936 20813 raft_consensus.cc:359] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:53.179817 20813 raft_consensus.cc:385] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:53.179713 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:53.150970 20492 leader_election.cc:304] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:53.181568 20832 raft_consensus.cc:3060] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:53.182291 20813 raft_consensus.cc:740] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:53.183301 20813 consensus_queue.cc:260] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:53.161908 20836 leader_election.cc:290] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:53.186161 20813 ts_tablet_manager.cc:1434] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20260808 01:54:53.149303 21005 raft_consensus.cc:515] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:53.193250 20813 tablet_bootstrap.cc:492] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:53.201131 20426 catalog_manager.cc:5697] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194). New cstate: current_term: 1 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } }
W20260808 01:54:53.199090 20649 leader_election.cc:343] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:53.180146 20979 raft_consensus.cc:515] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:53.225591 20810 raft_consensus.cc:359] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
W20260808 01:54:53.208230 20726 leader_election.cc:343] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:53.226459 20810 raft_consensus.cc:385] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:53.228291 20810 raft_consensus.cc:740] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:53.232811 20726 leader_election.cc:304] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:54:53.233814 20843 raft_consensus.cc:2749] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:53.235319 20843 consensus_queue.cc:1048] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:53.236835 20979 leader_election.cc:290] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:53.237871 20811 tablet_bootstrap.cc:492] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:53.240458 20815 tablet_bootstrap.cc:492] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:53.241967 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:53.242991 20815 ts_tablet_manager.cc:1403] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.252s	user 0.030s	sys 0.012s
I20260808 01:54:53.243306 20914 consensus_queue.cc:1048] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:53.245724 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:53.246452 20608 raft_consensus.cc:3060] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:53.246469 20815 raft_consensus.cc:359] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.247594 20815 raft_consensus.cc:385] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:53.247889 20815 raft_consensus.cc:740] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:53.248817 20815 consensus_queue.cc:260] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.251366 20815 ts_tablet_manager.cc:1434] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260808 01:54:53.252483 20815 tablet_bootstrap.cc:492] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:53.255167 20811 ts_tablet_manager.cc:1403] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.084s	user 0.017s	sys 0.017s
I20260808 01:54:53.253098 20810 consensus_queue.cc:260] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.257164 20810 ts_tablet_manager.cc:1434] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.097s	user 0.006s	sys 0.001s
I20260808 01:54:53.257321 20896 raft_consensus.cc:515] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:53.258299 20810 tablet_bootstrap.cc:492] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:53.258739 20529 raft_consensus.cc:2393] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:54:53.268815 20813 tablet_bootstrap.cc:654] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
W20260808 01:54:53.267990 20649 leader_election.cc:343] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:53.272706 20832 raft_consensus.cc:2749] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:53.274118 20810 tablet_bootstrap.cc:654] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:53.274652 20832 raft_consensus.cc:493] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:53.275993 20832 raft_consensus.cc:515] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.286568 20896 leader_election.cc:290] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:53.280216 20648 leader_election.cc:304] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:53.288874 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:53.290210 20761 raft_consensus.cc:3060] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:53.290657 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:53.291795 20979 raft_consensus.cc:3060] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:54:53.292680 20492 leader_election.cc:343] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:53.294512 21005 leader_election.cc:290] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:53.296861 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:53.302635 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:53.303563 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:53.304587 20870 consensus_queue.cc:1048] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:53.305433 20760 raft_consensus.cc:2393] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:54:53.307549 20761 raft_consensus.cc:2468] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:53.308815 20493 leader_election.cc:304] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:53.309897 20928 raft_consensus.cc:2804] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:53.310405 20928 raft_consensus.cc:697] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:54:53.310730 20832 leader_election.cc:290] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:53.311414 20928 consensus_queue.cc:237] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:53.311929 20811 raft_consensus.cc:359] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:53.312634 20811 raft_consensus.cc:385] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:53.312785 20979 raft_consensus.cc:2749] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:53.312961 20811 raft_consensus.cc:740] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:53.313766 20811 consensus_queue.cc:260] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:53.314459 20648 leader_election.cc:304] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:53.317287 20979 raft_consensus.cc:3060] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:53.320802 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:53.321098 20608 raft_consensus.cc:2468] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:54:53.321604 20686 raft_consensus.cc:2468] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:53.323143 20493 leader_election.cc:304] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:54:53.329315 20811 ts_tablet_manager.cc:1434] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.068s	user 0.006s	sys 0.000s
I20260808 01:54:53.331830 20915 raft_consensus.cc:493] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:53.332682 20915 raft_consensus.cc:515] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:53.333623 20811 tablet_bootstrap.cc:492] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:53.331796 20815 tablet_bootstrap.cc:654] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:53.341169 20915 leader_election.cc:290] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:53.343394 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:53.343655 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:53.344420 20761 raft_consensus.cc:2468] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
W20260808 01:54:53.345458 20570 leader_election.cc:343] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:53.345556 20915 raft_consensus.cc:493] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:53.346302 20571 leader_election.cc:304] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:54:53.346382 20915 raft_consensus.cc:515] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
W20260808 01:54:53.346839 20725 consensus_peers.cc:597] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 -> Peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Couldn't send request to peer a80df9f992014374aeef30979cfe393e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:53.347374 20870 raft_consensus.cc:2804] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:53.347831 20870 raft_consensus.cc:493] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:53.348186 20870 raft_consensus.cc:3060] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:53.350392 20915 leader_election.cc:290] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:53.325676 20896 raft_consensus.cc:2804] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:53.350556 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:53.352850 20648 leader_election.cc:304] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: df82e7f402364d369ed1ec3c5a8772b3
W20260808 01:54:53.354019 20570 leader_election.cc:343] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:53.351779 20896 raft_consensus.cc:493] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:53.357635 20896 raft_consensus.cc:3060] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:53.357960 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:53.363230 20938 raft_consensus.cc:2804] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election won for term 1
W20260808 01:54:53.364298 20571 leader_election.cc:343] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:53.367877 20571 leader_election.cc:304] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:54:53.364439 20938 raft_consensus.cc:697] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:54:53.366989 20828 consensus_queue.cc:1048] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260808 01:54:53.369961 20938 consensus_queue.cc:237] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:53.369642 20915 raft_consensus.cc:2749] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:53.371951 20914 raft_consensus.cc:493] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:53.375290 20914 raft_consensus.cc:515] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.378432 20914 leader_election.cc:290] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:53.379941 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63fb8c2f494a462686500ce10025a39f" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:53.380357 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63fb8c2f494a462686500ce10025a39f" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:53.380798 20684 raft_consensus.cc:2468] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:54:53.381021 20528 raft_consensus.cc:2468] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:54:53.382546 20570 leader_election.cc:304] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:53.383945 20914 raft_consensus.cc:2804] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:53.384399 20914 raft_consensus.cc:493] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:53.384786 20914 raft_consensus.cc:3060] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:53.385891 20810 tablet_bootstrap.cc:492] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:53.386550 20810 ts_tablet_manager.cc:1403] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.128s	user 0.015s	sys 0.002s
I20260808 01:54:53.371785 20870 raft_consensus.cc:515] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:53.394533 20810 raft_consensus.cc:359] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:53.374185 20685 raft_consensus.cc:2468] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:53.368280 20811 tablet_bootstrap.cc:654] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:53.397964 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:53.404368 20760 raft_consensus.cc:3060] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:53.403283 20810 raft_consensus.cc:385] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:53.408560 20810 raft_consensus.cc:740] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:53.410662 20979 raft_consensus.cc:2749] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:53.403090 20870 leader_election.cc:290] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:53.409411 20810 consensus_queue.cc:260] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:53.414136 20810 ts_tablet_manager.cc:1434] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.027s	user 0.006s	sys 0.000s
I20260808 01:54:53.417093 20896 raft_consensus.cc:515] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.398068 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:54:53.419054 20528 raft_consensus.cc:3060] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:53.419363 20811 tablet_bootstrap.cc:492] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:53.420146 20813 tablet_bootstrap.cc:492] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:53.422137 20813 ts_tablet_manager.cc:1403] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.235s	user 0.019s	sys 0.030s
I20260808 01:54:53.423559 20914 raft_consensus.cc:515] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.425097 20813 raft_consensus.cc:359] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:53.425853 20813 raft_consensus.cc:385] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:53.425963 20914 leader_election.cc:290] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:53.426169 20813 raft_consensus.cc:740] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:53.426977 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63fb8c2f494a462686500ce10025a39f" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:54:53.427716 20529 raft_consensus.cc:3060] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:53.428707 20810 tablet_bootstrap.cc:492] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:53.430604 20760 raft_consensus.cc:2468] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:54:53.431716 20813 consensus_queue.cc:260] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:53.432093 20571 leader_election.cc:304] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:53.432411 20828 consensus_queue.cc:1048] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:53.433316 20870 raft_consensus.cc:2804] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:53.433393 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63fb8c2f494a462686500ce10025a39f" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:53.433997 20870 raft_consensus.cc:697] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:54:53.434159 20813 ts_tablet_manager.cc:1434] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I20260808 01:54:53.434995 20870 consensus_queue.cc:237] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:53.434269 20685 raft_consensus.cc:3060] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:53.440342 20843 raft_consensus.cc:493] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:53.441295 20815 tablet_bootstrap.cc:492] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:53.440989 20843 raft_consensus.cc:515] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:53.443769 20843 leader_election.cc:290] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:53.445356 20815 ts_tablet_manager.cc:1403] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.193s	user 0.012s	sys 0.011s
I20260808 01:54:53.445729 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:53.445637 20832 consensus_queue.cc:1048] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:53.445562 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:53.447161 20530 raft_consensus.cc:2393] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 1: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:54:53.447465 20832 consensus_queue.cc:1048] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:53.448581 20815 raft_consensus.cc:359] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:53.449333 20815 raft_consensus.cc:385] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:53.449640 20815 raft_consensus.cc:740] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:53.450991 20813 tablet_bootstrap.cc:492] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:53.452600 20426 catalog_manager.cc:5697] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:53.452461 20423 catalog_manager.cc:5697] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 598db12c38d442c6988b4c4f991874fb (127.7.165.195). New cstate: current_term: 1 leader_uuid: "598db12c38d442c6988b4c4f991874fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:53.453229 20815 consensus_queue.cc:260] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:53.455564 20815 ts_tablet_manager.cc:1434] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.001s
I20260808 01:54:53.456557 20815 tablet_bootstrap.cc:492] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:53.464391 20810 tablet_bootstrap.cc:654] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:53.467152 20813 tablet_bootstrap.cc:654] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:53.473443 20725 leader_election.cc:304] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:54:53.474547 20843 raft_consensus.cc:3060] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:53.475710 20424 catalog_manager.cc:5697] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194). New cstate: current_term: 1 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:53.484526 20843 raft_consensus.cc:2749] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:53.490226 20815 tablet_bootstrap.cc:654] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:53.503250 20870 raft_consensus.cc:493] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:53.503981 20870 raft_consensus.cc:515] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:53.506641 20870 leader_election.cc:290] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:53.512276 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:53.512935 20685 raft_consensus.cc:2468] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:54:53.514400 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:53.519626 20896 leader_election.cc:290] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:53.427048 20811 ts_tablet_manager.cc:1403] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.094s	user 0.014s	sys 0.002s
I20260808 01:54:53.514869 20529 raft_consensus.cc:2468] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:54:53.515820 20571 leader_election.cc:304] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
W20260808 01:54:53.522753 20571 leader_election.cc:343] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:53.523379 20811 raft_consensus.cc:359] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:53.524139 20811 raft_consensus.cc:385] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:53.524307 20870 raft_consensus.cc:2804] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:53.524446 20914 consensus_queue.cc:1048] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:53.525672 20528 raft_consensus.cc:2468] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:54:53.526998 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:53.528936 20915 consensus_queue.cc:1048] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 01:54:53.527791 20571 leader_election.cc:343] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:53.530192 20571 leader_election.cc:304] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:53.531167 20915 raft_consensus.cc:2749] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:54:53.533828 20492 leader_election.cc:343] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:53.526260 20896 consensus_queue.cc:1048] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260808 01:54:53.535394 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
W20260808 01:54:53.536711 20475 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:54:53.538935 20870 raft_consensus.cc:697] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:54:53.524425 20811 raft_consensus.cc:740] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:53.541281 20684 raft_consensus.cc:3060] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:53.543030 20811 consensus_queue.cc:260] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:53.540970 20870 consensus_queue.cc:237] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.545341 20811 ts_tablet_manager.cc:1434] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.000s
I20260808 01:54:53.560073 20811 tablet_bootstrap.cc:492] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:53.606613 20810 tablet_bootstrap.cc:492] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:53.607290 20810 ts_tablet_manager.cc:1403] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.179s	user 0.018s	sys 0.000s
I20260808 01:54:53.609005 20813 tablet_bootstrap.cc:492] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:53.609591 20813 ts_tablet_manager.cc:1403] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.159s	user 0.014s	sys 0.000s
I20260808 01:54:53.613463 20813 raft_consensus.cc:359] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:53.619336 20813 raft_consensus.cc:385] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:53.619711 20813 raft_consensus.cc:740] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:53.618508 20810 raft_consensus.cc:359] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.621263 20810 raft_consensus.cc:385] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:53.621579 20810 raft_consensus.cc:740] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:53.621556 20813 consensus_queue.cc:260] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:53.633308 20810 consensus_queue.cc:260] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.634277 20684 raft_consensus.cc:2468] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:53.637118 20810 ts_tablet_manager.cc:1434] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.029s	user 0.008s	sys 0.000s
I20260808 01:54:53.637782 20425 catalog_manager.cc:5697] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194). New cstate: current_term: 1 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:53.651818 20810 tablet_bootstrap.cc:492] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
W20260808 01:54:53.635746 20493 consensus_peers.cc:597] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e -> Peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Couldn't send request to peer 598db12c38d442c6988b4c4f991874fb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:53.655197 20813 ts_tablet_manager.cc:1434] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.045s	user 0.006s	sys 0.002s
I20260808 01:54:53.666347 20811 tablet_bootstrap.cc:654] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:53.669543 20813 tablet_bootstrap.cc:492] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:53.677572 20815 tablet_bootstrap.cc:492] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:53.678241 20815 ts_tablet_manager.cc:1403] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.222s	user 0.025s	sys 0.019s
I20260808 01:54:53.683372 20493 leader_election.cc:304] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:54:53.685608 20828 raft_consensus.cc:2804] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:53.695354 20828 raft_consensus.cc:697] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:54:53.696435 20828 consensus_queue.cc:237] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.704108 20813 tablet_bootstrap.cc:654] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:53.712839 20815 raft_consensus.cc:359] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.713550 20815 raft_consensus.cc:385] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:53.713783 20815 raft_consensus.cc:740] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:53.715467 20815 consensus_queue.cc:260] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.719023 20811 tablet_bootstrap.cc:492] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:53.719596 20811 ts_tablet_manager.cc:1403] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.160s	user 0.009s	sys 0.011s
I20260808 01:54:53.722586 20811 raft_consensus.cc:359] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:53.723439 20811 raft_consensus.cc:385] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:53.723730 20811 raft_consensus.cc:740] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:53.729732 20815 ts_tablet_manager.cc:1434] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.051s	user 0.007s	sys 0.002s
I20260808 01:54:53.733208 20815 tablet_bootstrap.cc:492] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:53.734217 20811 consensus_queue.cc:260] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:53.745127 20810 tablet_bootstrap.cc:654] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:53.749212 20815 tablet_bootstrap.cc:654] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:53.751363 20811 ts_tablet_manager.cc:1434] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.001s
I20260808 01:54:53.755232 20811 tablet_bootstrap.cc:492] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:53.768224 20832 consensus_queue.cc:1048] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:53.761855 20815 tablet_bootstrap.cc:492] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:53.776005 20815 ts_tablet_manager.cc:1403] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.043s	user 0.014s	sys 0.001s
I20260808 01:54:53.778358 20979 raft_consensus.cc:493] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:53.779222 20979 raft_consensus.cc:515] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:53.783522 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:53.784263 20811 tablet_bootstrap.cc:654] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
W20260808 01:54:53.785674 20648 leader_election.cc:343] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:53.787060 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:53.764245 20813 tablet_bootstrap.cc:492] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:53.787916 20761 raft_consensus.cc:2468] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:54:53.788179 20813 ts_tablet_manager.cc:1403] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.132s	user 0.012s	sys 0.001s
I20260808 01:54:53.790261 20649 leader_election.cc:304] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:54:53.791324 20938 raft_consensus.cc:2804] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:53.791370 20813 raft_consensus.cc:359] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.791868 20938 raft_consensus.cc:493] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:53.792352 20938 raft_consensus.cc:3060] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:53.783422 20979 leader_election.cc:290] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:53.801002 20815 raft_consensus.cc:359] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:53.802037 20815 raft_consensus.cc:385] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:53.802472 20815 raft_consensus.cc:740] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:53.802151 20938 raft_consensus.cc:515] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:53.803544 20815 consensus_queue.cc:260] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:53.804692 20938 leader_election.cc:290] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:53.805409 20813 raft_consensus.cc:385] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:53.805749 20813 raft_consensus.cc:740] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:53.806208 20815 ts_tablet_manager.cc:1434] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.030s	user 0.007s	sys 0.002s
I20260808 01:54:53.806397 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:53.806490 20813 consensus_queue.cc:260] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.807009 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:53.807327 20815 tablet_bootstrap.cc:492] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:53.807121 20760 raft_consensus.cc:3060] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:54:53.808795 20648 leader_election.cc:343] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:53.813575 20813 ts_tablet_manager.cc:1434] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.000s
I20260808 01:54:53.814802 20813 tablet_bootstrap.cc:492] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:53.816039 20870 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:53.816322 20815 tablet_bootstrap.cc:654] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:53.816617 20870 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.817898 20760 raft_consensus.cc:2468] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:54:53.819687 20649 leader_election.cc:304] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:54:53.820036 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:53.820816 20530 raft_consensus.cc:2468] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:54:53.821106 20979 raft_consensus.cc:2804] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:53.821704 20979 raft_consensus.cc:697] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:54:53.821558 21035 consensus_queue.cc:1048] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:53.822300 20570 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:53.822767 20979 consensus_queue.cc:237] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:53.823282 20914 raft_consensus.cc:2804] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:54:53.823810 20914 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:53.824201 20914 raft_consensus.cc:3060] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:54:53.827607 20870 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:53.828481 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:53.829409 20684 raft_consensus.cc:2468] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:54:53.835402 20914 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.838758 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:54:53.839184 20815 tablet_bootstrap.cc:492] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:53.839435 20530 raft_consensus.cc:3060] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:54:53.839867 20815 ts_tablet_manager.cc:1403] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.033s	user 0.015s	sys 0.000s
I20260808 01:54:53.840245 20424 catalog_manager.cc:5697] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:53.842821 20815 raft_consensus.cc:359] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:53.843564 20815 raft_consensus.cc:385] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:53.843861 20815 raft_consensus.cc:740] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:53.844596 20815 consensus_queue.cc:260] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:53.848112 20810 tablet_bootstrap.cc:492] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:53.848702 20810 ts_tablet_manager.cc:1403] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.209s	user 0.013s	sys 0.051s
I20260808 01:54:53.851298 20815 ts_tablet_manager.cc:1434] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260808 01:54:53.851727 20810 raft_consensus.cc:359] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:53.852614 20815 tablet_bootstrap.cc:492] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:53.853250 20530 raft_consensus.cc:2468] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:54:53.854355 20914 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:53.855792 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:53.857281 20813 tablet_bootstrap.cc:654] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:53.858042 20684 raft_consensus.cc:3060] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:54:53.864303 20815 tablet_bootstrap.cc:654] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:53.857092 20810 raft_consensus.cc:385] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:53.868997 20810 raft_consensus.cc:740] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:53.870012 20810 consensus_queue.cc:260] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:53.872329 20811 tablet_bootstrap.cc:492] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:53.872949 20811 ts_tablet_manager.cc:1403] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.118s	user 0.011s	sys 0.002s
I20260808 01:54:53.876030 20811 raft_consensus.cc:359] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:53.876734 20811 raft_consensus.cc:385] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:53.877010 20811 raft_consensus.cc:740] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:53.877717 20811 consensus_queue.cc:260] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:53.878845 20424 catalog_manager.cc:5697] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 598db12c38d442c6988b4c4f991874fb (127.7.165.195). New cstate: current_term: 1 leader_uuid: "598db12c38d442c6988b4c4f991874fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:53.881263 20570 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:53.884572 20870 raft_consensus.cc:2804] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:54:53.885305 20870 raft_consensus.cc:697] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:54:53.886196 20870 consensus_queue.cc:237] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:53.879810 20811 ts_tablet_manager.cc:1434] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.001s
I20260808 01:54:53.880223 20979 raft_consensus.cc:493] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:53.889235 20979 raft_consensus.cc:515] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:53.893244 20915 consensus_queue.cc:1048] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:53.893846 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:53.895934 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:53.892130 20979 leader_election.cc:290] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:53.898082 20648 leader_election.cc:304] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:53.900164 20759 raft_consensus.cc:3060] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:53.900712 20938 raft_consensus.cc:2749] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:53.910383 20684 raft_consensus.cc:2468] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:54:53.917980 20979 consensus_queue.cc:1048] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:53.924026 20811 tablet_bootstrap.cc:492] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:53.926715 20810 ts_tablet_manager.cc:1434] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.078s	user 0.008s	sys 0.000s
I20260808 01:54:53.928172 20810 tablet_bootstrap.cc:492] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:53.927650 20425 catalog_manager.cc:5697] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 0 to 2, leader changed from <none> to 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194). New cstate: current_term: 2 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:53.957047 20810 tablet_bootstrap.cc:654] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:53.949482 20811 tablet_bootstrap.cc:654] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:53.951036 20979 consensus_queue.cc:1048] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260808 01:54:53.966353 20870 consensus_queue.cc:1048] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260808 01:54:53.970825 20954 raft_consensus.cc:493] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:53.971436 20954 raft_consensus.cc:515] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:53.992287 20954 leader_election.cc:290] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:53.995177 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:53.997051 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:54:53.997591 20725 leader_election.cc:343] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:53.997783 20608 raft_consensus.cc:2468] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 0.
I20260808 01:54:54.001587 20813 tablet_bootstrap.cc:492] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:54.002247 20813 ts_tablet_manager.cc:1403] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.188s	user 0.014s	sys 0.016s
I20260808 01:54:54.002775 20815 tablet_bootstrap.cc:492] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:54.003403 20815 ts_tablet_manager.cc:1403] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.151s	user 0.028s	sys 0.049s
I20260808 01:54:53.984586 20979 raft_consensus.cc:493] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:54.006072 20979 raft_consensus.cc:515] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:54.006458 20815 raft_consensus.cc:359] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:54.007481 20815 raft_consensus.cc:385] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:54.007789 20815 raft_consensus.cc:740] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:54.008963 20979 leader_election.cc:290] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:54.008698 20815 consensus_queue.cc:260] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:54.011526 20815 ts_tablet_manager.cc:1434] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260808 01:54:54.012267 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:54.012691 20815 tablet_bootstrap.cc:492] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:54.013072 20529 raft_consensus.cc:2468] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:54:54.014202 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:54.014986 20606 raft_consensus.cc:2468] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:54:54.015208 20813 raft_consensus.cc:359] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:54.016847 20648 leader_election.cc:304] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:54.017407 20813 raft_consensus.cc:385] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:54.017833 20813 raft_consensus.cc:740] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:54.020351 20813 consensus_queue.cc:260] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:54.022454 20979 raft_consensus.cc:2804] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:54.026044 20979 raft_consensus.cc:493] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:54.025441 20815 tablet_bootstrap.cc:654] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:54.027906 20979 raft_consensus.cc:3060] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:54.029448 20914 consensus_queue.cc:1048] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:54.039618 20725 leader_election.cc:304] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:54:54.046663 20813 ts_tablet_manager.cc:1434] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.044s	user 0.008s	sys 0.000s
I20260808 01:54:54.055457 20813 tablet_bootstrap.cc:492] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:54.057399 20843 raft_consensus.cc:2804] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:54.057932 20843 raft_consensus.cc:493] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:54.058398 20843 raft_consensus.cc:3060] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:54.063203 20810 tablet_bootstrap.cc:492] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:54.067808 21006 raft_consensus.cc:493] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:54.068892 20810 ts_tablet_manager.cc:1403] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.141s	user 0.025s	sys 0.037s
I20260808 01:54:54.081135 20914 consensus_queue.cc:1048] T 63fb8c2f494a462686500ce10025a39f P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:54.068707 21006 raft_consensus.cc:515] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:54.096441 20810 raft_consensus.cc:359] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:54.097463 20810 raft_consensus.cc:385] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:54.097831 20810 raft_consensus.cc:740] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:54.098861 20810 consensus_queue.cc:260] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:54.096887 21006 leader_election.cc:290] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:54.119035 20686 raft_consensus.cc:3060] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:54.124570 20810 ts_tablet_manager.cc:1434] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.055s	user 0.007s	sys 0.001s
I20260808 01:54:54.127529 20813 tablet_bootstrap.cc:654] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:54.129036 20810 tablet_bootstrap.cc:492] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:54.138144 20810 tablet_bootstrap.cc:654] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:54.158174 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:54.162534 20811 tablet_bootstrap.cc:492] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:54.163213 20811 ts_tablet_manager.cc:1403] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.239s	user 0.023s	sys 0.023s
W20260808 01:54:54.163451 20649 leader_election.cc:343] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:54.164386 20813 tablet_bootstrap.cc:492] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:54.165251 20813 ts_tablet_manager.cc:1403] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.115s	user 0.036s	sys 0.002s
I20260808 01:54:54.166355 20811 raft_consensus.cc:359] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:54.167362 20811 raft_consensus.cc:385] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:54.167694 20811 raft_consensus.cc:740] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:54.168514 20813 raft_consensus.cc:359] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:54.168668 20811 consensus_queue.cc:260] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:54.169788 20813 raft_consensus.cc:385] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:54.170262 20813 raft_consensus.cc:740] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:54.170797 20843 raft_consensus.cc:515] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:54.172086 20811 ts_tablet_manager.cc:1434] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260808 01:54:54.171391 20813 consensus_queue.cc:260] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:54.173200 20811 tablet_bootstrap.cc:492] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:54.173635 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:54.175813 20813 ts_tablet_manager.cc:1434] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.003s
I20260808 01:54:54.177704 20813 tablet_bootstrap.cc:492] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:54.178103 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
W20260808 01:54:54.181322 20648 leader_election.cc:343] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:54.184473 20810 tablet_bootstrap.cc:492] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:54.181828 20648 leader_election.cc:304] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:54.185194 20979 raft_consensus.cc:515] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:54.187101 20938 raft_consensus.cc:2749] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:54.185148 20810 ts_tablet_manager.cc:1403] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.058s	user 0.023s	sys 0.000s
W20260808 01:54:54.187532 20725 leader_election.cc:343] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:54.190825 20810 raft_consensus.cc:359] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:54.191643 20810 raft_consensus.cc:385] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:54.191962 20810 raft_consensus.cc:740] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:54.192822 20810 consensus_queue.cc:260] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:54.195851 20811 tablet_bootstrap.cc:654] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:54.202246 21021 consensus_queue.cc:1048] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:54.203958 21055 raft_consensus.cc:493] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:54.204622 21055 raft_consensus.cc:515] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:54.199754 20843 leader_election.cc:290] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:54.206167 20870 raft_consensus.cc:493] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:54.209494 20870 raft_consensus.cc:515] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:54.211838 20870 leader_election.cc:290] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:54.214753 21055 leader_election.cc:290] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:54.217301 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:54.217662 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:54.218407 20683 raft_consensus.cc:2468] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 0.
W20260808 01:54:54.219014 20571 leader_election.cc:343] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:54.217691 21035 raft_consensus.cc:493] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:54.221781 21035 raft_consensus.cc:515] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:54.223780 20813 tablet_bootstrap.cc:654] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:54.224345 20870 raft_consensus.cc:493] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:54.224696 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:54.224902 20870 raft_consensus.cc:515] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:54.225531 21035 leader_election.cc:290] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:54.220191 20726 leader_election.cc:304] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:54.227092 21055 raft_consensus.cc:2804] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:54.227869 21055 raft_consensus.cc:493] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:54.228394 21055 raft_consensus.cc:3060] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:54.229429 20914 raft_consensus.cc:493] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:54.229568 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:54.230242 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:54.231016 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:54.225363 20608 raft_consensus.cc:3060] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:54:54.232189 20571 leader_election.cc:343] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:54.227180 20870 leader_election.cc:290] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:54.233886 20914 raft_consensus.cc:515] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:54.211334 20810 ts_tablet_manager.cc:1434] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.001s
I20260808 01:54:54.240964 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:54.241432 20815 tablet_bootstrap.cc:492] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:54.233634 20493 leader_election.cc:304] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:54.242250 20815 ts_tablet_manager.cc:1403] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.230s	user 0.021s	sys 0.012s
I20260808 01:54:54.244624 20810 tablet_bootstrap.cc:492] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:54.261112 20815 raft_consensus.cc:359] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:54.262920 20815 raft_consensus.cc:385] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:54.263254 20815 raft_consensus.cc:740] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:54.254336 20979 leader_election.cc:290] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:54.265872 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:54.265780 20815 consensus_queue.cc:260] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:54.264441 21021 raft_consensus.cc:2749] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:54.267401 20811 tablet_bootstrap.cc:492] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:54.255432 20606 raft_consensus.cc:2468] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 0.
I20260808 01:54:54.270711 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:54.271072 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:54:54.277087 20528 raft_consensus.cc:3060] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:54.271296 20815 ts_tablet_manager.cc:1434] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.027s	user 0.009s	sys 0.000s
I20260808 01:54:54.280503 20815 tablet_bootstrap.cc:492] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:54.272826 20607 raft_consensus.cc:3060] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:54.282805 20609 raft_consensus.cc:3060] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:54:54.284569 20570 leader_election.cc:343] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:54.285264 20570 leader_election.cc:304] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:54:54.287292 20870 raft_consensus.cc:2749] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:54.288802 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:54.289665 21021 raft_consensus.cc:493] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:54.270052 20811 ts_tablet_manager.cc:1403] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.097s	user 0.013s	sys 0.003s
I20260808 01:54:54.291828 20914 leader_election.cc:290] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:54.306404 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:54.295938 21021 raft_consensus.cc:515] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:54.313331 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:54.339114 20571 leader_election.cc:304] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:54:54.338415 21021 leader_election.cc:290] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:54.344907 20609 raft_consensus.cc:2468] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:54:54.353941 20530 raft_consensus.cc:3060] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:54.345395 20683 raft_consensus.cc:1275] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
W20260808 01:54:54.347198 20570 leader_election.cc:343] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:54:54.347644 20648 consensus_peers.cc:597] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb -> Peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Couldn't send request to peer 9bfb29168c8947e1b5816f1572d040cc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260808 01:54:54.351095 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:54.361882 20870 consensus_queue.cc:1048] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:54.364140 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:54.352977 20915 raft_consensus.cc:2749] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:54.353379 21055 raft_consensus.cc:515] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:54.362666 20648 leader_election.cc:304] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:54:54.351701 20813 tablet_bootstrap.cc:492] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:54.375658 20813 ts_tablet_manager.cc:1403] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.199s	user 0.024s	sys 0.008s
I20260808 01:54:54.379571 21055 leader_election.cc:290] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:54.384691 20810 tablet_bootstrap.cc:654] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:54.384509 20828 consensus_queue.cc:1048] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:54.385404 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:54.388731 20685 raft_consensus.cc:3060] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:54.386022 20938 raft_consensus.cc:2804] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:54.395957 21006 consensus_queue.cc:1048] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:54.372979 20608 raft_consensus.cc:2468] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:54:54.421728 20815 tablet_bootstrap.cc:654] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:54.372334 20570 leader_election.cc:304] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:54.381317 20683 raft_consensus.cc:2468] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:54.430373 20811 raft_consensus.cc:359] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:54.431823 20813 raft_consensus.cc:359] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:54.434134 20811 raft_consensus.cc:385] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:54.434211 20813 raft_consensus.cc:385] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:54.434625 20811 raft_consensus.cc:740] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:54.434737 20813 raft_consensus.cc:740] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:54.437247 20813 consensus_queue.cc:260] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:54.416476 20938 raft_consensus.cc:697] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:54:54.443507 20811 consensus_queue.cc:260] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:54.447491 20813 ts_tablet_manager.cc:1434] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.071s	user 0.009s	sys 0.000s
I20260808 01:54:54.424089 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:54.450465 20813 tablet_bootstrap.cc:492] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:54.445088 20915 raft_consensus.cc:2749] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:54.452061 20954 raft_consensus.cc:493] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:54.457312 20607 raft_consensus.cc:3060] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:54:54.456975 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 109814us
I20260808 01:54:54.481145 20528 raft_consensus.cc:2468] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:54:54.483224 21035 raft_consensus.cc:493] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:54.485271 20938 consensus_queue.cc:237] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:54.478344 20954 raft_consensus.cc:515] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:54.471843 20725 leader_election.cc:304] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:54:54.490598 20685 raft_consensus.cc:2468] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:54:54.601128 20810 tablet_bootstrap.cc:492] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:54.601837 20810 ts_tablet_manager.cc:1403] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.357s	user 0.031s	sys 0.023s
I20260808 01:54:54.613341 20493 leader_election.cc:304] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:54.614478 21068 raft_consensus.cc:2804] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:54.615108 21068 raft_consensus.cc:493] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:54.615602 21068 raft_consensus.cc:3060] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:54.634927 20811 ts_tablet_manager.cc:1434] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.340s	user 0.029s	sys 0.030s
I20260808 01:54:54.636600 20811 tablet_bootstrap.cc:492] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:54.499317 21035 raft_consensus.cc:515] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:54.598040 20760 raft_consensus.cc:2468] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:54.644346 20815 tablet_bootstrap.cc:492] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:54.645182 20815 ts_tablet_manager.cc:1403] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.365s	user 0.031s	sys 0.028s
I20260808 01:54:54.650863 20815 raft_consensus.cc:359] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:54.652243 20815 raft_consensus.cc:385] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:54.653012 20815 raft_consensus.cc:740] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:54.651073 21068 raft_consensus.cc:515] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:54.627312 20843 raft_consensus.cc:2804] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:54.656684 20843 raft_consensus.cc:697] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:54:54.656767 20870 raft_consensus.cc:493] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:54.654891 20815 consensus_queue.cc:260] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:54.669401 20813 tablet_bootstrap.cc:654] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:54.668028 20870 raft_consensus.cc:515] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:54.671255 20528 raft_consensus.cc:1275] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260808 01:54:54.671396 21068 leader_election.cc:290] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:54.672609 20843 consensus_queue.cc:237] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:54.673957 20815 ts_tablet_manager.cc:1434] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.028s	user 0.012s	sys 0.000s
I20260808 01:54:54.674548 20870 leader_election.cc:290] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:54.681394 20954 leader_election.cc:290] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:54.684103 21035 leader_election.cc:290] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:54.689960 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:54.691394 20683 raft_consensus.cc:3060] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:54:54.692451 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 231325us
I20260808 01:54:54.654779 20726 leader_election.cc:304] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
W20260808 01:54:54.707170 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 68761.3us
I20260808 01:54:54.710047 21055 raft_consensus.cc:2804] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:54.710719 21055 raft_consensus.cc:697] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:54:54.711647 21055 consensus_queue.cc:237] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:54.715610 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:54.716418 20682 raft_consensus.cc:2468] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 0.
I20260808 01:54:54.716636 20815 tablet_bootstrap.cc:492] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:54.721320 20726 leader_election.cc:304] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:54.722795 21073 raft_consensus.cc:2804] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:54.723256 21073 raft_consensus.cc:493] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:54.723577 21073 raft_consensus.cc:3060] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:54.727101 20914 consensus_queue.cc:1048] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:54.733574 20810 raft_consensus.cc:359] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:54.739271 20810 raft_consensus.cc:385] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:54.740053 20810 raft_consensus.cc:740] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:54.741835 20607 raft_consensus.cc:2468] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:54:54.745028 20810 consensus_queue.cc:260] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:54.762235 20810 ts_tablet_manager.cc:1434] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.160s	user 0.004s	sys 0.003s
I20260808 01:54:54.765625 20810 tablet_bootstrap.cc:492] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:54.817870 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:54.847548 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:54.835927 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:54.855453 20760 raft_consensus.cc:3060] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:54.857856 20815 tablet_bootstrap.cc:654] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:54.853125 21073 raft_consensus.cc:515] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:54.862174 20810 tablet_bootstrap.cc:654] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:54.864954 21073 leader_election.cc:290] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:54.867704 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:54.894548 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:54.895660 20607 raft_consensus.cc:2393] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 1: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:54:54.897423 20606 raft_consensus.cc:3060] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:54.881491 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:54.876883 20683 raft_consensus.cc:2468] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:54.932641 20493 leader_election.cc:304] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:54.920763 20682 raft_consensus.cc:3060] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:54.940042 20425 catalog_manager.cc:5697] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 598db12c38d442c6988b4c4f991874fb (127.7.165.195). New cstate: current_term: 1 leader_uuid: "598db12c38d442c6988b4c4f991874fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:54.938714 21035 raft_consensus.cc:2804] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:54.949529 21035 raft_consensus.cc:697] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:54:54.950624 21035 consensus_queue.cc:237] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:54.952936 20811 tablet_bootstrap.cc:654] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:54.979523 20813 tablet_bootstrap.cc:492] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:54.980154 20813 ts_tablet_manager.cc:1403] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.530s	user 0.028s	sys 0.030s
I20260808 01:54:54.989032 20938 raft_consensus.cc:493] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:55.010716 20813 raft_consensus.cc:359] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:55.013810 20813 raft_consensus.cc:385] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:55.015008 20813 raft_consensus.cc:740] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:55.021435 20813 consensus_queue.cc:260] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:55.012531 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:55.013224 20760 raft_consensus.cc:2468] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
W20260808 01:54:55.034432 21085 log.cc:927] Time spent T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 0.094s	user 0.001s	sys 0.001s
W20260808 01:54:55.036875 20725 leader_election.cc:343] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:55.046386 20815 tablet_bootstrap.cc:492] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:55.047225 20815 ts_tablet_manager.cc:1403] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.331s	user 0.020s	sys 0.007s
I20260808 01:54:55.030589 20813 ts_tablet_manager.cc:1434] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.050s	user 0.010s	sys 0.000s
I20260808 01:54:55.011835 20938 raft_consensus.cc:515] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:55.053727 20813 tablet_bootstrap.cc:492] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:55.052601 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:55.060956 20810 tablet_bootstrap.cc:492] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:55.061599 20810 ts_tablet_manager.cc:1403] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.297s	user 0.010s	sys 0.014s
I20260808 01:54:55.068957 20938 leader_election.cc:290] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:55.073872 20815 raft_consensus.cc:359] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:55.034412 20492 leader_election.cc:304] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:55.076202 20815 raft_consensus.cc:385] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:55.080114 20815 raft_consensus.cc:740] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:55.084924 20810 raft_consensus.cc:359] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:55.085829 20810 raft_consensus.cc:385] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:55.086143 20810 raft_consensus.cc:740] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:55.087195 20810 consensus_queue.cc:260] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:55.088632 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:55.089402 20607 raft_consensus.cc:2468] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:54:55.090615 20810 ts_tablet_manager.cc:1434] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.000s
I20260808 01:54:55.091167 20648 leader_election.cc:304] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:54:55.095057 20810 tablet_bootstrap.cc:492] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:55.091763 20811 tablet_bootstrap.cc:492] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:55.097488 20811 ts_tablet_manager.cc:1403] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.461s	user 0.012s	sys 0.025s
W20260808 01:54:55.103655 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 69380.3us
I20260808 01:54:55.105441 20682 raft_consensus.cc:2468] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:54:55.106890 21021 raft_consensus.cc:3060] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:54:55.106506 20475 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:54:55.091877 21070 consensus_queue.cc:1048] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260808 01:54:55.111361 21006 consensus_queue.cc:1048] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260808 01:54:55.113785 20811 raft_consensus.cc:359] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
W20260808 01:54:55.098279 20725 leader_election.cc:343] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:55.108930 20938 raft_consensus.cc:2804] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:55.115471 20938 raft_consensus.cc:493] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:55.115897 20938 raft_consensus.cc:3060] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:55.117215 20815 consensus_queue.cc:260] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:55.117995 20726 leader_election.cc:304] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:54:55.093142 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:55.119819 20815 ts_tablet_manager.cc:1434] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.072s	user 0.008s	sys 0.000s
I20260808 01:54:55.120038 21089 raft_consensus.cc:2804] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:55.120669 21089 raft_consensus.cc:697] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:54:55.121006 20815 tablet_bootstrap.cc:492] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:55.121562 21089 consensus_queue.cc:237] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:55.114594 20811 raft_consensus.cc:385] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:55.123056 20811 raft_consensus.cc:740] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:55.123833 20811 consensus_queue.cc:260] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:55.124084 20571 leader_election.cc:304] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:55.125641 20811 ts_tablet_manager.cc:1434] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.028s	user 0.004s	sys 0.002s
W20260808 01:54:55.127378 20648 leader_election.cc:343] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:55.127731 20811 tablet_bootstrap.cc:492] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:55.126667 21029 raft_consensus.cc:2749] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:55.135555 20811 tablet_bootstrap.cc:654] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:55.139781 21078 raft_consensus.cc:493] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:55.142308 21078 raft_consensus.cc:515] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:55.145483 21078 leader_election.cc:290] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:55.165915 20815 tablet_bootstrap.cc:654] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:55.168272 21070 raft_consensus.cc:493] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:55.173151 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:55.179854 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:55.186812 20609 raft_consensus.cc:2468] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:55.182637 21070 raft_consensus.cc:515] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:55.200296 21021 raft_consensus.cc:2749] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:55.198463 20938 raft_consensus.cc:515] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:55.211483 21070 leader_election.cc:290] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:55.213395 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:55.214808 20954 consensus_queue.cc:1048] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:55.214843 20492 leader_election.cc:304] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:55.215342 20529 raft_consensus.cc:2468] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:54:55.216152 21021 raft_consensus.cc:2804] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:55.216518 20683 raft_consensus.cc:2468] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:55.218571 20938 leader_election.cc:290] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:55.216563 21021 raft_consensus.cc:493] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:55.225009 21021 raft_consensus.cc:3060] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:55.227648 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:54:55.231319 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:55.231521 20954 consensus_queue.cc:1048] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:55.234246 20605 raft_consensus.cc:2468] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:54:55.236501 20813 tablet_bootstrap.cc:654] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:55.240283 20954 raft_consensus.cc:493] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:55.240873 20954 raft_consensus.cc:515] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:55.241788 20811 tablet_bootstrap.cc:492] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:55.242372 20811 ts_tablet_manager.cc:1403] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.116s	user 0.017s	sys 0.005s
I20260808 01:54:55.226220 20648 leader_election.cc:304] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
W20260808 01:54:55.246807 20648 leader_election.cc:343] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:55.250816 20426 catalog_manager.cc:5697] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196). New cstate: current_term: 1 leader_uuid: "df82e7f402364d369ed1ec3c5a8772b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:55.254037 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:55.254797 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:55.255571 20606 raft_consensus.cc:2468] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 0.
I20260808 01:54:55.255043 20426 catalog_manager.cc:5697] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196). New cstate: current_term: 1 leader_uuid: "df82e7f402364d369ed1ec3c5a8772b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:55.245206 20954 leader_election.cc:290] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:55.254863 20528 raft_consensus.cc:2468] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 0.
I20260808 01:54:55.269887 20725 leader_election.cc:304] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:55.271622 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:55.272006 21089 raft_consensus.cc:2804] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:55.272387 20607 raft_consensus.cc:3060] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:55.274629 20810 tablet_bootstrap.cc:654] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:55.258946 20938 raft_consensus.cc:2804] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:55.282181 20938 raft_consensus.cc:493] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:55.286312 20938 raft_consensus.cc:3060] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:55.272521 21089 raft_consensus.cc:493] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:55.293294 21089 raft_consensus.cc:3060] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:55.302562 20815 tablet_bootstrap.cc:492] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:55.303440 20424 catalog_manager.cc:5697] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:55.303457 20815 ts_tablet_manager.cc:1403] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.183s	user 0.019s	sys 0.003s
I20260808 01:54:55.311201 20815 raft_consensus.cc:359] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:55.312052 20815 raft_consensus.cc:385] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:55.312582 20815 raft_consensus.cc:740] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:55.316215 20811 raft_consensus.cc:359] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:55.317728 20811 raft_consensus.cc:385] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:55.318065 20811 raft_consensus.cc:740] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:55.318617 20843 consensus_queue.cc:1048] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:55.318817 20811 consensus_queue.cc:260] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:55.321202 20811 ts_tablet_manager.cc:1434] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.079s	user 0.007s	sys 0.000s
I20260808 01:54:55.322216 20811 tablet_bootstrap.cc:492] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:55.331770 20607 raft_consensus.cc:2468] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:54:55.333649 20648 leader_election.cc:304] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:54:55.335261 20979 raft_consensus.cc:2804] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:55.335795 20979 raft_consensus.cc:697] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:54:55.336614 20979 consensus_queue.cc:237] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:55.342156 20815 consensus_queue.cc:260] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:55.351740 20815 ts_tablet_manager.cc:1434] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.043s	user 0.007s	sys 0.000s
I20260808 01:54:55.352792 20815 tablet_bootstrap.cc:492] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:55.363126 20810 tablet_bootstrap.cc:492] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:55.354408 21021 raft_consensus.cc:515] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:55.364347 20810 ts_tablet_manager.cc:1403] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.270s	user 0.021s	sys 0.006s
I20260808 01:54:55.372498 20811 tablet_bootstrap.cc:654] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:55.377801 20813 tablet_bootstrap.cc:492] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:55.386507 20813 ts_tablet_manager.cc:1403] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.333s	user 0.026s	sys 0.028s
I20260808 01:54:55.381691 20810 raft_consensus.cc:359] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:55.387570 20810 raft_consensus.cc:385] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:55.387905 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:55.388126 20810 raft_consensus.cc:740] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:55.389855 20607 raft_consensus.cc:3060] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:55.389408 21070 raft_consensus.cc:493] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:55.391057 21070 raft_consensus.cc:515] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:55.392369 21021 leader_election.cc:290] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:55.394667 21070 leader_election.cc:290] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:55.395123 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:55.396718 20810 consensus_queue.cc:260] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:55.399257 20810 ts_tablet_manager.cc:1434] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.035s	user 0.004s	sys 0.004s
I20260808 01:54:55.400681 20813 raft_consensus.cc:359] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:55.401425 20813 raft_consensus.cc:385] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:55.396148 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:55.401680 20813 raft_consensus.cc:740] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:55.406935 20810 tablet_bootstrap.cc:492] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
W20260808 01:54:55.407238 20649 leader_election.cc:343] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:55.412928 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:55.417807 21088 consensus_queue.cc:1048] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20260808 01:54:55.422664 20648 leader_election.cc:343] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:55.423549 20648 leader_election.cc:304] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:55.427659 21070 raft_consensus.cc:2749] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:55.422842 20813 consensus_queue.cc:260] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:55.445094 21089 raft_consensus.cc:515] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:55.454020 20815 tablet_bootstrap.cc:654] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:55.455168 20813 ts_tablet_manager.cc:1434] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.068s	user 0.007s	sys 0.000s
I20260808 01:54:55.456961 20813 tablet_bootstrap.cc:492] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:55.448945 20938 raft_consensus.cc:515] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:55.467077 20682 raft_consensus.cc:2393] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 1: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:54:55.473788 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:54:55.478199 20529 raft_consensus.cc:3060] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:55.478554 20810 tablet_bootstrap.cc:654] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:55.488371 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:55.489168 20609 raft_consensus.cc:3060] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:55.491480 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:54:55.492509 20528 raft_consensus.cc:2393] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:54:55.502303 20529 raft_consensus.cc:2468] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:54:55.484598 21089 leader_election.cc:290] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:55.504557 20725 leader_election.cc:304] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:55.506109 21089 raft_consensus.cc:2804] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:55.507513 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:55.506942 21089 raft_consensus.cc:697] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:54:55.509931 21021 consensus_queue.cc:1048] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:55.510216 20609 raft_consensus.cc:2468] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:54:55.510066 21089 consensus_queue.cc:237] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:55.512725 20648 leader_election.cc:304] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:54:55.504313 20426 catalog_manager.cc:5697] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196). New cstate: current_term: 1 leader_uuid: "df82e7f402364d369ed1ec3c5a8772b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:55.515246 21035 raft_consensus.cc:493] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:55.515913 21035 raft_consensus.cc:515] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:55.518236 21073 consensus_queue.cc:1048] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:55.520202 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:55.520974 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:54:55.522176 20492 leader_election.cc:343] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:55.513996 21115 raft_consensus.cc:2749] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:54:55.518513 21035 leader_election.cc:290] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
W20260808 01:54:55.523583 20493 leader_election.cc:343] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:55.531265 20493 leader_election.cc:304] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:55.508606 20938 leader_election.cc:290] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:55.534722 21035 raft_consensus.cc:2749] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:55.535540 21021 consensus_queue.cc:1048] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260808 01:54:55.508415 20607 raft_consensus.cc:2468] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:55.544683 21073 raft_consensus.cc:493] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:55.544001 20425 catalog_manager.cc:5697] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 598db12c38d442c6988b4c4f991874fb (127.7.165.195). New cstate: current_term: 1 leader_uuid: "598db12c38d442c6988b4c4f991874fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: HEALTHY } } }
I20260808 01:54:55.546693 20811 tablet_bootstrap.cc:492] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:55.555228 20811 ts_tablet_manager.cc:1403] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.233s	user 0.017s	sys 0.000s
I20260808 01:54:55.548519 21073 raft_consensus.cc:515] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:55.570137 21073 leader_election.cc:290] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:55.574519 20813 tablet_bootstrap.cc:654] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:55.581240 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:54:55.583707 20725 leader_election.cc:343] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:55.584497 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:54:55.589453 20726 leader_election.cc:343] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:55.589965 20726 leader_election.cc:304] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:54:55.591889 20954 raft_consensus.cc:2749] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:55.598435 20492 leader_election.cc:304] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:54:55.600708 21079 raft_consensus.cc:2804] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:55.602790 20810 tablet_bootstrap.cc:492] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:55.603354 20810 ts_tablet_manager.cc:1403] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.197s	user 0.022s	sys 0.001s
I20260808 01:54:55.601569 20811 raft_consensus.cc:359] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:55.604168 20811 raft_consensus.cc:385] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:55.604517 20811 raft_consensus.cc:740] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:55.605749 20811 consensus_queue.cc:260] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:55.605748 21079 raft_consensus.cc:697] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:54:55.604326 20914 raft_consensus.cc:493] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:55.611094 20914 raft_consensus.cc:515] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:55.612062 20607 raft_consensus.cc:3060] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:55.613389 20914 leader_election.cc:290] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:55.616142 20811 ts_tablet_manager.cc:1434] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.060s	user 0.009s	sys 0.000s
I20260808 01:54:55.617914 20811 tablet_bootstrap.cc:492] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:55.622339 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:55.624171 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:55.629094 20571 leader_election.cc:304] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:55.610090 21079 consensus_queue.cc:237] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:55.635996 20426 catalog_manager.cc:5697] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196). New cstate: current_term: 1 leader_uuid: "df82e7f402364d369ed1ec3c5a8772b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:55.643661 20815 tablet_bootstrap.cc:492] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:55.644240 20815 ts_tablet_manager.cc:1403] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.292s	user 0.033s	sys 0.013s
I20260808 01:54:55.649092 20810 raft_consensus.cc:359] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:55.650313 20810 raft_consensus.cc:385] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:55.650763 20810 raft_consensus.cc:740] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:55.651816 20810 consensus_queue.cc:260] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:55.677237 20813 tablet_bootstrap.cc:492] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:55.677918 20813 ts_tablet_manager.cc:1403] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.221s	user 0.004s	sys 0.014s
I20260808 01:54:55.683293 20813 raft_consensus.cc:359] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:55.693581 20815 raft_consensus.cc:359] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:55.694667 20815 raft_consensus.cc:385] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:55.695039 20815 raft_consensus.cc:740] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:55.695837 20815 consensus_queue.cc:260] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:55.702649 20810 ts_tablet_manager.cc:1434] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.099s	user 0.009s	sys 0.000s
I20260808 01:54:55.704217 20810 tablet_bootstrap.cc:492] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:55.684221 20813 raft_consensus.cc:385] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:55.706523 20813 raft_consensus.cc:740] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:55.707346 20813 consensus_queue.cc:260] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:55.716243 20813 ts_tablet_manager.cc:1434] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.038s	user 0.005s	sys 0.003s
I20260808 01:54:55.715847 20426 catalog_manager.cc:5697] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:55.718022 20813 tablet_bootstrap.cc:492] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:55.716957 20815 ts_tablet_manager.cc:1434] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.072s	user 0.010s	sys 0.001s
I20260808 01:54:55.729470 20815 tablet_bootstrap.cc:492] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:55.729600 20810 tablet_bootstrap.cc:654] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:55.740648 20815 tablet_bootstrap.cc:654] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:55.742825 20914 raft_consensus.cc:2749] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:55.756233 20813 tablet_bootstrap.cc:654] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:55.756948 21089 consensus_queue.cc:1048] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:55.747773 20811 tablet_bootstrap.cc:654] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:55.769881 21115 consensus_queue.cc:1048] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:55.773219 20810 tablet_bootstrap.cc:492] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:55.773993 20810 ts_tablet_manager.cc:1403] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.070s	user 0.011s	sys 0.000s
I20260808 01:54:55.774143 20813 tablet_bootstrap.cc:492] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:55.774955 20813 ts_tablet_manager.cc:1403] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.057s	user 0.014s	sys 0.000s
I20260808 01:54:55.778116 20813 raft_consensus.cc:359] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:55.779675 20813 raft_consensus.cc:385] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:55.778275 20810 raft_consensus.cc:359] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:55.780002 20813 raft_consensus.cc:740] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:55.781194 20810 raft_consensus.cc:385] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:55.781576 20810 raft_consensus.cc:740] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:55.782052 20813 consensus_queue.cc:260] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:55.784485 20813 ts_tablet_manager.cc:1434] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260808 01:54:55.783711 20810 consensus_queue.cc:260] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:55.785560 20813 tablet_bootstrap.cc:492] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:55.812677 20914 raft_consensus.cc:493] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:55.813529 20914 raft_consensus.cc:515] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:55.817344 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:55.818059 20914 leader_election.cc:290] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:55.818472 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:55.819366 20810 ts_tablet_manager.cc:1434] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.045s	user 0.012s	sys 0.000s
W20260808 01:54:55.820322 20571 leader_election.cc:343] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:55.818107 20682 raft_consensus.cc:2468] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:54:55.821520 20811 tablet_bootstrap.cc:492] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:55.822037 20811 ts_tablet_manager.cc:1403] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.204s	user 0.015s	sys 0.012s
I20260808 01:54:55.822839 20571 leader_election.cc:304] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:55.823616 20914 raft_consensus.cc:2804] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:55.823992 20914 raft_consensus.cc:493] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:55.824806 20914 raft_consensus.cc:3060] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:55.825822 20810 tablet_bootstrap.cc:492] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:55.827456 20828 raft_consensus.cc:493] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:55.827958 20828 raft_consensus.cc:515] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:55.828835 20815 tablet_bootstrap.cc:492] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:55.829389 20815 ts_tablet_manager.cc:1403] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.100s	user 0.021s	sys 0.024s
I20260808 01:54:55.830196 20828 leader_election.cc:290] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:55.832729 20815 raft_consensus.cc:359] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:55.833535 20815 raft_consensus.cc:385] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:55.833825 20815 raft_consensus.cc:740] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:55.836884 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:55.837906 20760 raft_consensus.cc:2468] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:55.834520 20815 consensus_queue.cc:260] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:55.840605 20493 leader_election.cc:304] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:55.841348 20815 ts_tablet_manager.cc:1434] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260808 01:54:55.841789 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:55.842320 20813 tablet_bootstrap.cc:654] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:55.842447 20815 tablet_bootstrap.cc:492] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:55.832631 21115 raft_consensus.cc:493] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:55.843241 21115 raft_consensus.cc:515] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:55.845369 21115 leader_election.cc:290] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:55.846048 20682 raft_consensus.cc:2468] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:55.846154 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:55.846730 20828 raft_consensus.cc:2804] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:55.848464 20760 raft_consensus.cc:2468] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:54:55.850026 20810 tablet_bootstrap.cc:654] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:55.850966 20649 leader_election.cc:304] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:55.851972 21006 raft_consensus.cc:2804] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:55.852348 21006 raft_consensus.cc:493] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:55.849020 20828 raft_consensus.cc:493] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:55.852679 21006 raft_consensus.cc:3060] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:55.857477 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:55.856940 20828 raft_consensus.cc:3060] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:55.868937 20811 raft_consensus.cc:359] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:55.870362 20811 raft_consensus.cc:385] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:55.870747 20811 raft_consensus.cc:740] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:55.871430 20529 raft_consensus.cc:3060] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:55.871699 20811 consensus_queue.cc:260] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:55.881973 20811 ts_tablet_manager.cc:1434] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.060s	user 0.007s	sys 0.000s
I20260808 01:54:55.883513 20811 tablet_bootstrap.cc:492] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:55.894177 20815 tablet_bootstrap.cc:654] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:55.895812 21006 raft_consensus.cc:515] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:55.901028 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:55.900846 21006 leader_election.cc:290] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:55.901774 20760 raft_consensus.cc:3060] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:55.903644 20813 tablet_bootstrap.cc:492] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:55.905339 21006 raft_consensus.cc:493] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:55.905902 21006 raft_consensus.cc:515] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:55.908838 21006 leader_election.cc:290] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:55.904264 20813 ts_tablet_manager.cc:1403] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.119s	user 0.019s	sys 0.009s
I20260808 01:54:55.910007 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:54:55.914270 20649 leader_election.cc:343] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:55.914479 20813 raft_consensus.cc:359] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:55.915292 20813 raft_consensus.cc:385] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:55.915599 20813 raft_consensus.cc:740] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:55.889302 20828 raft_consensus.cc:515] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:55.890329 21115 consensus_queue.cc:1048] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:55.916702 20528 raft_consensus.cc:2393] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:54:55.910007 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:55.922212 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:55.919245 20828 leader_election.cc:290] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:55.920902 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:55.924813 20683 raft_consensus.cc:2393] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 1: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:54:55.879563 20914 raft_consensus.cc:515] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:54:55.925680 20648 leader_election.cc:343] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:55.926226 20648 leader_election.cc:304] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:55.927809 20811 tablet_bootstrap.cc:654] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:55.929965 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:55.931613 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:54:55.932557 20529 raft_consensus.cc:2393] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:54:55.943248 20493 leader_election.cc:304] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:55.949271 20813 consensus_queue.cc:260] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:55.950471 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:55.951272 20762 raft_consensus.cc:3060] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:55.947469 21006 raft_consensus.cc:2749] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:55.939751 20914 leader_election.cc:290] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:55.948971 21078 raft_consensus.cc:2749] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:54:55.976234 20683 raft_consensus.cc:3060] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:55.976840 20810 tablet_bootstrap.cc:492] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:55.977382 20810 ts_tablet_manager.cc:1403] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.152s	user 0.014s	sys 0.004s
I20260808 01:54:55.947324 20954 consensus_queue.cc:1048] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20260808 01:54:55.996716 20810 raft_consensus.cc:359] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:55.994904 20683 raft_consensus.cc:2468] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:54:55.999181 20760 raft_consensus.cc:2468] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:54:56.006237 20810 raft_consensus.cc:385] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.008858 20810 raft_consensus.cc:740] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.009649 20810 consensus_queue.cc:260] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:56.007133 20813 ts_tablet_manager.cc:1434] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.096s	user 0.004s	sys 0.004s
I20260808 01:54:56.011456 20813 tablet_bootstrap.cc:492] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:56.000265 20762 raft_consensus.cc:2468] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:54:56.015877 20571 leader_election.cc:304] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:56.017179 21029 raft_consensus.cc:2804] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:56.017755 21029 raft_consensus.cc:697] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:54:56.018777 21029 consensus_queue.cc:237] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:56.020741 20811 tablet_bootstrap.cc:492] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:56.021368 20811 ts_tablet_manager.cc:1403] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.138s	user 0.018s	sys 0.015s
I20260808 01:54:56.032783 20810 ts_tablet_manager.cc:1434] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.055s	user 0.005s	sys 0.004s
I20260808 01:54:56.034149 20810 tablet_bootstrap.cc:492] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:56.037254 20813 tablet_bootstrap.cc:654] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:56.038944 20649 leader_election.cc:304] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:54:56.040185 20979 raft_consensus.cc:2804] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:56.040750 20979 raft_consensus.cc:697] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:54:56.041677 20979 consensus_queue.cc:237] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:56.043273 20811 raft_consensus.cc:359] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:56.048631 20811 raft_consensus.cc:385] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.049048 20811 raft_consensus.cc:740] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.059363 20811 consensus_queue.cc:260] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:56.080199 21089 consensus_queue.cc:1048] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:56.085245 20815 tablet_bootstrap.cc:492] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:56.085875 20815 ts_tablet_manager.cc:1403] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.244s	user 0.027s	sys 0.025s
I20260808 01:54:56.088800 20811 ts_tablet_manager.cc:1434] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.067s	user 0.004s	sys 0.004s
I20260808 01:54:56.095417 20811 tablet_bootstrap.cc:492] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:56.105533 21073 raft_consensus.cc:493] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:56.106202 21073 raft_consensus.cc:515] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:56.107729 21140 raft_consensus.cc:493] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:56.108521 21073 leader_election.cc:290] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:56.118324 20828 consensus_queue.cc:1048] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:56.128113 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:56.128947 20607 raft_consensus.cc:2468] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 0.
I20260808 01:54:56.123243 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:56.132308 20528 raft_consensus.cc:2468] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 0.
I20260808 01:54:56.135208 20815 raft_consensus.cc:359] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:56.136024 20815 raft_consensus.cc:385] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.133939 20725 leader_election.cc:304] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:56.136369 20815 raft_consensus.cc:740] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.142614 20815 consensus_queue.cc:260] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:56.108330 21140 raft_consensus.cc:515] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:56.141075 21089 raft_consensus.cc:2804] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:56.152048 20424 catalog_manager.cc:5697] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 598db12c38d442c6988b4c4f991874fb (127.7.165.195). New cstate: current_term: 1 leader_uuid: "598db12c38d442c6988b4c4f991874fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:56.154000 20815 ts_tablet_manager.cc:1434] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.068s	user 0.008s	sys 0.000s
I20260808 01:54:56.162842 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:56.153103 21089 raft_consensus.cc:493] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:56.163792 21089 raft_consensus.cc:3060] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:56.163961 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:56.164742 20760 raft_consensus.cc:2468] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:54:56.166455 20571 leader_election.cc:304] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:56.166816 21006 raft_consensus.cc:493] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:56.167408 21006 raft_consensus.cc:515] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:56.167905 21142 raft_consensus.cc:2804] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:56.168345 21142 raft_consensus.cc:493] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:56.168735 21142 raft_consensus.cc:3060] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:56.169724 21140 leader_election.cc:290] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:54:56.167464 20571 leader_election.cc:343] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:56.175335 20810 tablet_bootstrap.cc:654] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:56.181231 21089 raft_consensus.cc:515] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:56.182003 21006 leader_election.cc:290] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:56.184139 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:56.185055 20528 raft_consensus.cc:2468] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:54:56.187176 20648 leader_election.cc:304] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:56.156791 20426 catalog_manager.cc:5697] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194). New cstate: current_term: 1 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:56.192085 20813 tablet_bootstrap.cc:492] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
W20260808 01:54:56.233660 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 81147.4us
I20260808 01:54:56.234423 20813 ts_tablet_manager.cc:1403] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.223s	user 0.010s	sys 0.010s
I20260808 01:54:56.236370 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:54:56.237099 20529 raft_consensus.cc:3060] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:56.237584 20813 raft_consensus.cc:359] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.238343 20813 raft_consensus.cc:385] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.193002 20828 consensus_queue.cc:1048] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:56.193922 20979 raft_consensus.cc:2804] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:56.186376 21142 raft_consensus.cc:515] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:56.196714 20811 tablet_bootstrap.cc:654] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:56.241503 20979 raft_consensus.cc:493] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:56.242005 20979 raft_consensus.cc:3060] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:56.238610 20813 raft_consensus.cc:740] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.243185 21142 leader_election.cc:290] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:56.197530 21089 leader_election.cc:290] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:56.221769 20810 tablet_bootstrap.cc:492] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:56.243983 20813 consensus_queue.cc:260] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.244972 20810 ts_tablet_manager.cc:1403] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.211s	user 0.010s	sys 0.004s
I20260808 01:54:56.252846 20810 raft_consensus.cc:359] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.246989 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:56.247236 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:56.261965 20760 raft_consensus.cc:3060] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:54:56.263518 20571 leader_election.cc:343] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:56.263135 20810 raft_consensus.cc:385] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.265913 20810 raft_consensus.cc:740] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.246722 20813 ts_tablet_manager.cc:1434] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.001s
I20260808 01:54:56.267575 20813 tablet_bootstrap.cc:492] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:56.267043 20810 consensus_queue.cc:260] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.197574 20815 tablet_bootstrap.cc:492] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:56.274109 20810 ts_tablet_manager.cc:1434] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.000s
I20260808 01:54:56.272454 20828 raft_consensus.cc:493] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:56.275482 20810 tablet_bootstrap.cc:492] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:56.275393 20828 raft_consensus.cc:515] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.278484 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:56.279186 20607 raft_consensus.cc:2468] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:56.282060 20492 leader_election.cc:304] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:56.282503 20529 raft_consensus.cc:2468] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:54:56.288902 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:56.289579 20607 raft_consensus.cc:3060] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:56.289624 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:54:56.291273 20493 leader_election.cc:343] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:56.292299 20828 leader_election.cc:290] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:56.293473 20760 raft_consensus.cc:2468] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:54:56.295581 20810 tablet_bootstrap.cc:654] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:56.297888 20571 leader_election.cc:304] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:54:56.299134 21142 raft_consensus.cc:2804] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:56.299628 21142 raft_consensus.cc:697] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:54:56.300312 21142 consensus_queue.cc:237] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:56.302052 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:56.302569 20815 tablet_bootstrap.cc:654] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:56.283056 21079 raft_consensus.cc:2804] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:56.303896 21079 raft_consensus.cc:493] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:56.304265 21079 raft_consensus.cc:3060] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:56.325856 20979 raft_consensus.cc:515] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:56.336483 20815 tablet_bootstrap.cc:492] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:56.337147 20815 ts_tablet_manager.cc:1403] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.140s	user 0.013s	sys 0.020s
W20260808 01:54:56.337714 20475 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:54:56.339568 20607 raft_consensus.cc:2468] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:54:56.340229 20815 raft_consensus.cc:359] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.354184 20815 raft_consensus.cc:385] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.354521 20815 raft_consensus.cc:740] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.354877 20811 tablet_bootstrap.cc:492] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:56.355414 20811 ts_tablet_manager.cc:1403] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.266s	user 0.019s	sys 0.002s
I20260808 01:54:56.355340 20815 consensus_queue.cc:260] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.355926 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:54:56.356552 20528 raft_consensus.cc:3060] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:56.357878 20815 ts_tablet_manager.cc:1434] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.001s
I20260808 01:54:56.342285 20813 tablet_bootstrap.cc:654] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:56.343475 21079 raft_consensus.cc:515] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.344946 20810 tablet_bootstrap.cc:492] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:56.358589 20811 raft_consensus.cc:359] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:56.360504 20811 raft_consensus.cc:385] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.365509 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:56.366227 20607 raft_consensus.cc:3060] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:56.366762 21079 leader_election.cc:290] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:54:56.350070 20649 leader_election.cc:343] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:56.352226 20979 leader_election.cc:290] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:56.343308 20725 leader_election.cc:304] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:56.367552 20810 ts_tablet_manager.cc:1403] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.092s	user 0.015s	sys 0.003s
I20260808 01:54:56.360877 20811 raft_consensus.cc:740] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.376099 20811 consensus_queue.cc:260] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:56.376267 20810 raft_consensus.cc:359] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:56.377368 20810 raft_consensus.cc:385] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.377602 20607 raft_consensus.cc:2468] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:56.358933 20815 tablet_bootstrap.cc:492] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:56.379089 20492 leader_election.cc:304] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:56.379879 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:56.380048 21079 raft_consensus.cc:2804] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:56.377779 20810 raft_consensus.cc:740] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.380683 21079 raft_consensus.cc:697] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:54:56.381351 20810 consensus_queue.cc:260] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:56.381672 21079 consensus_queue.cc:237] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.391826 20810 ts_tablet_manager.cc:1434] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.001s
I20260808 01:54:56.394331 20810 tablet_bootstrap.cc:492] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:56.382917 20954 raft_consensus.cc:2804] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:56.386798 20815 tablet_bootstrap.cc:654] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:56.395735 20954 raft_consensus.cc:697] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:54:56.396325 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:56.397136 20760 raft_consensus.cc:3060] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:56.396829 20954 consensus_queue.cc:237] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
W20260808 01:54:56.382910 20493 leader_election.cc:343] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:56.386651 20811 ts_tablet_manager.cc:1434] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.031s	user 0.008s	sys 0.000s
I20260808 01:54:56.424594 20811 tablet_bootstrap.cc:492] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:56.397019 20426 catalog_manager.cc:5697] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194). New cstate: current_term: 1 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:56.430091 20979 raft_consensus.cc:493] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:56.430806 20979 raft_consensus.cc:515] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.433996 20979 leader_election.cc:290] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:56.436714 20811 tablet_bootstrap.cc:654] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:56.438768 20813 tablet_bootstrap.cc:492] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:56.439430 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:56.439896 20813 ts_tablet_manager.cc:1403] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.173s	user 0.010s	sys 0.009s
I20260808 01:54:56.444429 20528 raft_consensus.cc:2468] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
W20260808 01:54:56.445874 20648 leader_election.cc:343] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:56.451522 20648 leader_election.cc:304] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:56.454975 20979 raft_consensus.cc:2804] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:56.455314 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:56.456404 20979 raft_consensus.cc:697] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:54:56.457458 20979 consensus_queue.cc:237] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
W20260808 01:54:56.463035 20648 leader_election.cc:343] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:56.463603 20648 leader_election.cc:304] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:54:56.464854 21006 raft_consensus.cc:2749] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:56.466725 20810 tablet_bootstrap.cc:654] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:56.463289 20813 raft_consensus.cc:359] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:56.472703 20813 raft_consensus.cc:385] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.473133 20813 raft_consensus.cc:740] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.468040 20760 raft_consensus.cc:2468] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:54:56.475054 20424 catalog_manager.cc:5697] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:56.479350 21142 consensus_queue.cc:1048] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:56.474107 20813 consensus_queue.cc:260] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:56.488618 20815 tablet_bootstrap.cc:492] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:56.489233 20815 ts_tablet_manager.cc:1403] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.131s	user 0.014s	sys 0.020s
I20260808 01:54:56.495048 20815 raft_consensus.cc:359] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:56.495633 20815 raft_consensus.cc:385] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.496048 20815 raft_consensus.cc:740] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.497567 20810 tablet_bootstrap.cc:492] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:56.498219 20810 ts_tablet_manager.cc:1403] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.104s	user 0.012s	sys 0.003s
I20260808 01:54:56.499253 20815 consensus_queue.cc:260] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:56.500110 20813 ts_tablet_manager.cc:1434] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.059s	user 0.008s	sys 0.000s
I20260808 01:54:56.501293 20813 tablet_bootstrap.cc:492] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:56.507962 20815 ts_tablet_manager.cc:1434] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.003s
I20260808 01:54:56.509161 20815 tablet_bootstrap.cc:492] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:56.510568 20810 raft_consensus.cc:359] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.511381 20810 raft_consensus.cc:385] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.511650 20810 raft_consensus.cc:740] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.512369 20810 consensus_queue.cc:260] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.515733 20426 catalog_manager.cc:5697] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196). New cstate: current_term: 1 leader_uuid: "df82e7f402364d369ed1ec3c5a8772b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:56.517655 21083 raft_consensus.cc:493] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:56.518191 21083 raft_consensus.cc:515] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.520670 21083 leader_election.cc:290] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:56.521325 20810 ts_tablet_manager.cc:1434] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.000s
I20260808 01:54:56.522471 20810 tablet_bootstrap.cc:492] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:56.522938 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:56.523609 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:56.523692 20682 raft_consensus.cc:2468] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:56.523557 20813 tablet_bootstrap.cc:654] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
W20260808 01:54:56.525462 20492 leader_election.cc:343] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:56.526089 20815 tablet_bootstrap.cc:654] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:56.526638 20493 leader_election.cc:304] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:54:56.528237 21083 raft_consensus.cc:2804] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:56.528702 21083 raft_consensus.cc:493] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:56.529093 21083 raft_consensus.cc:3060] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:56.532747 20810 tablet_bootstrap.cc:654] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:56.540131 20813 tablet_bootstrap.cc:492] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:56.540432 20815 tablet_bootstrap.cc:492] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:56.540683 20813 ts_tablet_manager.cc:1403] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.040s	user 0.005s	sys 0.005s
I20260808 01:54:56.541015 20815 ts_tablet_manager.cc:1403] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.032s	user 0.010s	sys 0.000s
I20260808 01:54:56.543797 20813 raft_consensus.cc:359] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:56.546092 20813 raft_consensus.cc:385] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.546380 20813 raft_consensus.cc:740] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.547971 20815 raft_consensus.cc:359] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.549018 20815 raft_consensus.cc:385] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.549294 20815 raft_consensus.cc:740] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.550184 20815 consensus_queue.cc:260] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.551271 20810 tablet_bootstrap.cc:492] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:56.552424 20810 ts_tablet_manager.cc:1403] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.030s	user 0.011s	sys 0.000s
I20260808 01:54:56.550184 20813 consensus_queue.cc:260] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:56.554991 20813 ts_tablet_manager.cc:1434] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.014s	user 0.009s	sys 0.000s
I20260808 01:54:56.552470 20815 ts_tablet_manager.cc:1434] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260808 01:54:56.558418 21079 raft_consensus.cc:493] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:56.571048 21079 raft_consensus.cc:515] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:56.573354 21079 leader_election.cc:290] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:56.556154 21029 consensus_queue.cc:1048] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:56.576581 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d32821d9c17456394a662fcfd05f6c6" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:56.563100 20813 tablet_bootstrap.cc:492] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:56.568693 20979 consensus_queue.cc:1048] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:56.577601 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d32821d9c17456394a662fcfd05f6c6" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:56.578306 20607 raft_consensus.cc:2468] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:56.579681 20492 leader_election.cc:304] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:56.555680 20810 raft_consensus.cc:359] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:56.581229 20810 raft_consensus.cc:385] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.580839 20828 raft_consensus.cc:2804] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:56.581871 20828 raft_consensus.cc:493] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:56.582217 20828 raft_consensus.cc:3060] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:56.581532 20810 raft_consensus.cc:740] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.584157 20810 consensus_queue.cc:260] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:56.588972 20813 tablet_bootstrap.cc:654] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:56.591662 20810 ts_tablet_manager.cc:1434] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.039s	user 0.008s	sys 0.001s
I20260808 01:54:56.592074 20424 catalog_manager.cc:5697] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 598db12c38d442c6988b4c4f991874fb (127.7.165.195). New cstate: current_term: 1 leader_uuid: "598db12c38d442c6988b4c4f991874fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:56.593066 20810 tablet_bootstrap.cc:492] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:56.601152 20813 tablet_bootstrap.cc:492] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:56.601756 20813 ts_tablet_manager.cc:1403] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.046s	user 0.015s	sys 0.003s
I20260808 01:54:56.604676 20813 raft_consensus.cc:359] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.605434 20813 raft_consensus.cc:385] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.605741 20813 raft_consensus.cc:740] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
W20260808 01:54:56.601177 20493 leader_election.cc:343] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:56.610136 20815 tablet_bootstrap.cc:492] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:56.606515 20813 consensus_queue.cc:260] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.612612 20813 ts_tablet_manager.cc:1434] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260808 01:54:56.613732 20813 tablet_bootstrap.cc:492] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:56.615651 20810 tablet_bootstrap.cc:654] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:56.529173 20811 tablet_bootstrap.cc:492] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:56.613786 20828 raft_consensus.cc:515] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:56.620597 20813 tablet_bootstrap.cc:654] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:56.621332 20828 leader_election.cc:290] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:56.622444 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d32821d9c17456394a662fcfd05f6c6" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:56.623873 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d32821d9c17456394a662fcfd05f6c6" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
W20260808 01:54:56.623956 20493 leader_election.cc:343] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:56.624536 20815 tablet_bootstrap.cc:654] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:56.624622 20607 raft_consensus.cc:3060] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:56.644011 20815 tablet_bootstrap.cc:492] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:56.644795 20815 ts_tablet_manager.cc:1403] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.065s	user 0.014s	sys 0.002s
I20260808 01:54:56.646765 20811 ts_tablet_manager.cc:1403] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.222s	user 0.016s	sys 0.004s
I20260808 01:54:56.649586 20607 raft_consensus.cc:2468] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:56.651858 20815 raft_consensus.cc:359] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:56.642737 21083 raft_consensus.cc:515] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.652751 20815 raft_consensus.cc:385] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.653132 20815 raft_consensus.cc:740] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.653605 20811 raft_consensus.cc:359] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:56.653928 20815 consensus_queue.cc:260] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:56.654749 20811 raft_consensus.cc:385] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.655663 20811 raft_consensus.cc:740] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.656376 20815 ts_tablet_manager.cc:1434] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.001s
I20260808 01:54:56.657402 20815 tablet_bootstrap.cc:492] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:56.659871 21083 leader_election.cc:290] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:54:56.657087 20811 consensus_queue.cc:260] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:56.671357 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:56.672055 20683 raft_consensus.cc:3060] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:56.673508 20811 ts_tablet_manager.cc:1434] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.026s	user 0.010s	sys 0.000s
I20260808 01:54:56.675346 20811 tablet_bootstrap.cc:492] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:56.683015 20492 leader_election.cc:304] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:56.687359 20813 tablet_bootstrap.cc:492] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:56.688081 20813 ts_tablet_manager.cc:1403] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.075s	user 0.015s	sys 0.019s
I20260808 01:54:56.691164 20828 raft_consensus.cc:493] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:56.691733 20828 raft_consensus.cc:515] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:56.693142 20813 raft_consensus.cc:359] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:56.693933 20813 raft_consensus.cc:385] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.694204 20813 raft_consensus.cc:740] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.694957 20813 consensus_queue.cc:260] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:56.695331 20828 leader_election.cc:290] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:56.687491 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:56.699532 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:56.700331 20606 raft_consensus.cc:2468] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:56.696233 21083 raft_consensus.cc:2804] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:56.701201 21083 raft_consensus.cc:697] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:54:56.702128 21083 consensus_queue.cc:237] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:56.706768 20492 leader_election.cc:304] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:56.709066 21021 raft_consensus.cc:2804] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:56.706121 20813 ts_tablet_manager.cc:1434] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.000s
I20260808 01:54:56.709494 21021 raft_consensus.cc:493] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:56.710098 21021 raft_consensus.cc:3060] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:56.711171 20979 consensus_queue.cc:1048] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:56.707762 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:54:56.720059 20493 leader_election.cc:343] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:56.721535 20811 tablet_bootstrap.cc:654] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:56.732935 20813 tablet_bootstrap.cc:492] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:56.724865 20607 raft_consensus.cc:3060] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:56.743005 20810 tablet_bootstrap.cc:492] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:56.744696 20813 tablet_bootstrap.cc:654] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:56.752123 20683 raft_consensus.cc:2468] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:56.758219 20493 leader_election.cc:304] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:54:56.760320 20828 raft_consensus.cc:2804] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:56.760840 20828 raft_consensus.cc:697] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:54:56.761754 20828 consensus_queue.cc:237] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.762651 20815 tablet_bootstrap.cc:654] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:56.763998 20810 ts_tablet_manager.cc:1403] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.171s	user 0.020s	sys 0.012s
I20260808 01:54:56.781108 20810 raft_consensus.cc:359] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:56.782051 20810 raft_consensus.cc:385] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.782457 20810 raft_consensus.cc:740] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.799950 20683 raft_consensus.cc:3060] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:56.803385 21115 raft_consensus.cc:493] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:56.804386 21115 raft_consensus.cc:515] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:56.809494 21021 raft_consensus.cc:515] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:56.811818 21138 consensus_queue.cc:1048] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:56.786307 20810 consensus_queue.cc:260] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:56.828568 21115 leader_election.cc:290] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:56.832255 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:56.833370 20606 raft_consensus.cc:2393] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:54:56.857617 20813 tablet_bootstrap.cc:492] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:56.860740 20813 ts_tablet_manager.cc:1403] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.128s	user 0.019s	sys 0.016s
W20260808 01:54:56.867676 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 51550.9us
I20260808 01:54:56.882753 20810 ts_tablet_manager.cc:1434] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.118s	user 0.020s	sys 0.001s
I20260808 01:54:56.879308 20607 raft_consensus.cc:2468] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:56.898666 21084 consensus_queue.cc:1048] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.045s
W20260808 01:54:56.904389 21180 log.cc:927] Time spent T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260808 01:54:56.917794 21084 raft_consensus.cc:493] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:56.918730 21084 raft_consensus.cc:515] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:56.910056 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:56.921149 20423 catalog_manager.cc:5697] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:56.895161 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:54:56.925851 21021 leader_election.cc:290] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:54:56.925959 20609 raft_consensus.cc:3060] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:56.928293 20813 raft_consensus.cc:359] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:56.929217 20813 raft_consensus.cc:385] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.929522 20813 raft_consensus.cc:740] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.930244 20813 consensus_queue.cc:260] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:56.923308 21084 leader_election.cc:290] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:56.934909 20811 tablet_bootstrap.cc:492] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:56.939128 20815 tablet_bootstrap.cc:492] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:56.939770 20815 ts_tablet_manager.cc:1403] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.283s	user 0.009s	sys 0.027s
I20260808 01:54:56.940156 20811 ts_tablet_manager.cc:1403] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.265s	user 0.039s	sys 0.022s
I20260808 01:54:56.943835 20813 ts_tablet_manager.cc:1434] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.076s	user 0.007s	sys 0.000s
I20260808 01:54:56.945570 20813 tablet_bootstrap.cc:492] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:56.950826 20815 raft_consensus.cc:359] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:56.951624 20815 raft_consensus.cc:385] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.951933 20815 raft_consensus.cc:740] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.952886 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:56.957361 20648 leader_election.cc:304] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:54:56.958176 21115 raft_consensus.cc:3060] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:56.961210 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:56.961756 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:56.964458 20762 raft_consensus.cc:2468] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:54:56.964900 21078 consensus_queue.cc:1048] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260808 01:54:56.962994 20493 leader_election.cc:343] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:56.967417 21073 consensus_queue.cc:1048] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:54:56.962118 20815 consensus_queue.cc:260] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:56.969444 20811 raft_consensus.cc:359] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:56.970252 20811 raft_consensus.cc:385] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:56.962015 20682 raft_consensus.cc:2468] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:54:56.972371 21115 raft_consensus.cc:2749] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:56.970542 20811 raft_consensus.cc:740] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:56.974413 20811 consensus_queue.cc:260] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:56.975528 21006 raft_consensus.cc:493] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:56.978310 21006 raft_consensus.cc:515] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:56.979285 20815 ts_tablet_manager.cc:1434] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.039s	user 0.006s	sys 0.001s
I20260808 01:54:56.984283 20571 leader_election.cc:304] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:56.999552 20811 ts_tablet_manager.cc:1434] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.059s	user 0.008s	sys 0.000s
I20260808 01:54:57.001209 20811 tablet_bootstrap.cc:492] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:56.992533 21084 raft_consensus.cc:2804] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:57.012133 21084 raft_consensus.cc:493] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:57.014101 21142 raft_consensus.cc:493] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:57.014520 21084 raft_consensus.cc:3060] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:57.014775 21142 raft_consensus.cc:515] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:57.016968 21142 leader_election.cc:290] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:57.017591 20813 tablet_bootstrap.cc:654] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:56.995888 20979 raft_consensus.cc:493] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:57.020064 20979 raft_consensus.cc:515] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:57.020951 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:57.021842 20609 raft_consensus.cc:2468] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:57.022277 20828 raft_consensus.cc:493] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:57.023329 20979 leader_election.cc:290] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:54:57.028725 20571 leader_election.cc:343] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:57.030412 20828 raft_consensus.cc:515] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:57.036516 20810 tablet_bootstrap.cc:492] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:57.045140 20828 leader_election.cc:290] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:57.048684 20492 leader_election.cc:304] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:57.030486 21147 consensus_queue.cc:1048] T b8135c0503a8480998454f58362956e5 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260808 01:54:57.046054 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:57.060654 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:57.061506 20760 raft_consensus.cc:2468] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:57.064062 20815 tablet_bootstrap.cc:492] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:57.055075 21083 raft_consensus.cc:2804] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:57.062207 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:57.043663 20979 consensus_queue.cc:1048] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260808 01:54:57.073670 20493 leader_election.cc:304] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:54:57.089118 21078 consensus_queue.cc:1048] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:57.071142 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:57.080081 21083 raft_consensus.cc:697] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:54:57.086730 21181 raft_consensus.cc:2804] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:54:57.094126 21189 consensus_queue.cc:1048] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260808 01:54:57.099468 21084 raft_consensus.cc:515] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:57.100104 20811 tablet_bootstrap.cc:654] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:57.094376 21083 consensus_queue.cc:237] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
W20260808 01:54:57.092859 20571 leader_election.cc:343] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:57.104431 20571 leader_election.cc:304] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:57.096007 21181 raft_consensus.cc:493] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:54:57.110500 21181 raft_consensus.cc:3060] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:57.109229 21029 raft_consensus.cc:2749] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:57.117705 20648 leader_election.cc:304] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:57.126827 21115 raft_consensus.cc:2749] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:54:57.130034 21006 leader_election.cc:290] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:57.133160 20810 tablet_bootstrap.cc:654] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
W20260808 01:54:57.135381 21188 log.cc:927] Time spent T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260808 01:54:57.143810 21084 leader_election.cc:290] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:57.145267 20815 tablet_bootstrap.cc:654] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:57.160296 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:57.161098 20757 raft_consensus.cc:3060] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:57.163206 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:54:57.164331 20685 raft_consensus.cc:2468] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:54:57.213827 20813 tablet_bootstrap.cc:492] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:57.214533 20813 ts_tablet_manager.cc:1403] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.269s	user 0.017s	sys 0.037s
I20260808 01:54:57.203986 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:54:57.235292 20758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:54:57.243418 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:57.244243 20686 raft_consensus.cc:3060] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:57.252722 20607 raft_consensus.cc:2393] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 1: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:54:57.264380 21181 raft_consensus.cc:515] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:57.273496 20426 catalog_manager.cc:5697] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } }
I20260808 01:54:57.293081 20813 raft_consensus.cc:359] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:57.293999 20813 raft_consensus.cc:385] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:57.294442 20813 raft_consensus.cc:740] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:57.283277 21078 consensus_queue.cc:1048] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260808 01:54:57.307345 20649 leader_election.cc:304] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:54:57.300100 21181 leader_election.cc:290] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:57.310201 20811 tablet_bootstrap.cc:492] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:57.321915 20686 raft_consensus.cc:2468] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:54:57.325742 20811 ts_tablet_manager.cc:1403] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.325s	user 0.018s	sys 0.007s
I20260808 01:54:57.326153 21192 raft_consensus.cc:2749] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:54:57.298846 20813 consensus_queue.cc:260] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:57.330946 20813 ts_tablet_manager.cc:1434] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.113s	user 0.009s	sys 0.000s
I20260808 01:54:57.332345 20813 tablet_bootstrap.cc:492] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:57.332945 20811 raft_consensus.cc:359] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:57.333793 20811 raft_consensus.cc:385] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:57.334136 20811 raft_consensus.cc:740] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:57.335469 20811 consensus_queue.cc:260] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:57.336962 20979 raft_consensus.cc:493] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:57.337827 20811 ts_tablet_manager.cc:1434] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I20260808 01:54:57.337563 20979 raft_consensus.cc:515] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:57.338969 20811 tablet_bootstrap.cc:492] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:57.342587 20979 leader_election.cc:290] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:57.317862 20815 tablet_bootstrap.cc:492] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:54:57.345155 20815 ts_tablet_manager.cc:1403] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.281s	user 0.011s	sys 0.013s
I20260808 01:54:57.318722 20757 raft_consensus.cc:2468] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:54:57.324518 21079 consensus_queue.cc:1048] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:57.345498 20571 leader_election.cc:304] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:54:57.365839 20810 tablet_bootstrap.cc:492] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:54:57.366485 20810 ts_tablet_manager.cc:1403] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.330s	user 0.031s	sys 0.022s
I20260808 01:54:57.370486 20810 raft_consensus.cc:359] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:57.371363 20810 raft_consensus.cc:385] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:57.371729 20810 raft_consensus.cc:740] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:54:57.350234 21138 raft_consensus.cc:2804] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:54:57.373657 21138 raft_consensus.cc:697] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:54:57.375042 21138 consensus_queue.cc:237] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:57.387334 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:54:57.387981 20493 consensus_peers.cc:597] T 6d32821d9c17456394a662fcfd05f6c6 P a80df9f992014374aeef30979cfe393e -> Peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Couldn't send request to peer df82e7f402364d369ed1ec3c5a8772b3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260808 01:54:57.388437 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 102976us
I20260808 01:54:57.381633 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:54:57.394156 20758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:54:57.395215 20758 raft_consensus.cc:3060] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:54:57.397096 20649 leader_election.cc:343] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:54:57.401904 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:54:57.402791 20682 raft_consensus.cc:3060] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:54:57.406087 20758 raft_consensus.cc:2468] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:57.420311 20811 tablet_bootstrap.cc:654] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:57.447247 21078 consensus_queue.cc:1048] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:54:57.510730 20813 tablet_bootstrap.cc:654] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:57.477313 20815 raft_consensus.cc:359] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:57.517619 20815 raft_consensus.cc:385] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:57.518170 20815 raft_consensus.cc:740] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:54:57.474620 20682 raft_consensus.cc:2468] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:54:57.522248 20815 consensus_queue.cc:260] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:54:57.523021 20811 tablet_bootstrap.cc:492] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:54:57.523761 20811 ts_tablet_manager.cc:1403] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.185s	user 0.015s	sys 0.006s
I20260808 01:54:57.525465 20815 ts_tablet_manager.cc:1434] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.180s	user 0.008s	sys 0.000s
I20260808 01:54:57.527570 20815 tablet_bootstrap.cc:492] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:54:57.529276 20811 raft_consensus.cc:359] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:57.530350 20811 raft_consensus.cc:385] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:57.530701 20811 raft_consensus.cc:740] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:54:57.530943 20810 consensus_queue.cc:260] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:57.532146 20811 consensus_queue.cc:260] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:54:57.532810 20530 raft_consensus.cc:2468] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:54:57.534034 20810 ts_tablet_manager.cc:1434] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.167s	user 0.011s	sys 0.000s
I20260808 01:54:57.534291 21035 raft_consensus.cc:493] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:57.535320 20810 tablet_bootstrap.cc:492] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:54:57.534883 21035 raft_consensus.cc:515] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:57.534531 20811 ts_tablet_manager.cc:1434] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260808 01:54:57.541669 21035 leader_election.cc:290] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:54:57.544545 20815 tablet_bootstrap.cc:654] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:57.546947 20811 tablet_bootstrap.cc:492] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:54:57.565068 20811 tablet_bootstrap.cc:654] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:57.569120 20813 tablet_bootstrap.cc:492] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:54:57.569969 20813 ts_tablet_manager.cc:1403] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.238s	user 0.014s	sys 0.000s
I20260808 01:54:57.575093 20813 raft_consensus.cc:359] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:57.576354 20813 raft_consensus.cc:385] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:54:57.576766 20813 raft_consensus.cc:740] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:54:57.577790 20813 consensus_queue.cc:260] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:54:57.585628 20810 tablet_bootstrap.cc:654] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:54:57.581101 20813 ts_tablet_manager.cc:1434] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.004s
I20260808 01:54:57.595275 20813 tablet_bootstrap.cc:492] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:54:57.607887 21073 raft_consensus.cc:493] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:54:57.608646 21073 raft_consensus.cc:515] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
W20260808 01:54:58.620723 20552 debug-util.cc:398] Leaking SignalData structure 0x7b080032e100 after lost signal to thread 7834
I20260808 01:54:58.960263 21218 raft_consensus.cc:493] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:54:58.961414 21218 raft_consensus.cc:515] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:54:58.964395 21218 leader_election.cc:290] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:54:59.840406 21220 raft_consensus.cc:493] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.095690 21215 raft_consensus.cc:493] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.101778 21215 raft_consensus.cc:515] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:00.118621 21192 raft_consensus.cc:493] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:00.120098 21192 raft_consensus.cc:515] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:00.149899 21192 leader_election.cc:290] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:00.287494 21192 raft_consensus.cc:493] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.287619 21151 raft_consensus.cc:493] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:00.288693 21192 raft_consensus.cc:515] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:00.288741 21151 raft_consensus.cc:515] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:00.385989 21192 leader_election.cc:290] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:00.391240 21192 raft_consensus.cc:493] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:54:59.842592 21220 raft_consensus.cc:515] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:00.393858 21192 raft_consensus.cc:515] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:00.395779 21021 raft_consensus.cc:493] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:00.397179 21192 leader_election.cc:290] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:00.391887 21215 leader_election.cc:290] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:00.397053 21021 raft_consensus.cc:515] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:00.425559 21192 raft_consensus.cc:493] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:00.432821 21073 leader_election.cc:290] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:00.430830 21192 raft_consensus.cc:515] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:00.429093 21224 raft_consensus.cc:493] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:00.435171 21222 raft_consensus.cc:493] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:00.432732 21021 leader_election.cc:290] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:00.441062 21021 raft_consensus.cc:493] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:00.438946 21192 leader_election.cc:290] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:00.442000 21021 raft_consensus.cc:515] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:00.445837 21222 raft_consensus.cc:515] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:00.445490 21021 leader_election.cc:290] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:00.459448 21073 raft_consensus.cc:493] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:00.460673 21073 raft_consensus.cc:515] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:00.439364 21224 raft_consensus.cc:515] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:00.456496 21222 leader_election.cc:290] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:00.435428 21220 leader_election.cc:290] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:00.465502 21222 raft_consensus.cc:493] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:00.429777 21151 leader_election.cc:290] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:00.481240 21224 leader_election.cc:290] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:00.496835 21223 raft_consensus.cc:493] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:00.491050 21222 raft_consensus.cc:515] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:00.498276 21225 raft_consensus.cc:493] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.504246 21225 raft_consensus.cc:515] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:00.499775 21228 raft_consensus.cc:493] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:00.500720 21215 raft_consensus.cc:493] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.502810 21230 raft_consensus.cc:493] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:00.498548 21223 raft_consensus.cc:515] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:00.650247 21230 raft_consensus.cc:515] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:00.501165 21222 leader_election.cc:290] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:00.502890 21073 leader_election.cc:290] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:00.645305 21021 raft_consensus.cc:493] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:00.502266 21229 raft_consensus.cc:493] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:00.646720 21228 raft_consensus.cc:515] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:00.655678 21229 raft_consensus.cc:515] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:00.647672 21215 raft_consensus.cc:515] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:00.654367 21021 raft_consensus.cc:515] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:00.659160 21222 raft_consensus.cc:493] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:00.662561 21224 raft_consensus.cc:493] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:00.673787 21224 raft_consensus.cc:515] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:00.674438 21215 leader_election.cc:290] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:00.663573 21073 raft_consensus.cc:493] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:00.678378 21073 raft_consensus.cc:515] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:00.679389 21021 leader_election.cc:290] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:00.680298 21222 raft_consensus.cc:515] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:00.681036 21073 leader_election.cc:290] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:00.683585 21223 leader_election.cc:290] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:00.682520 21228 leader_election.cc:290] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:00.685491 21151 raft_consensus.cc:493] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.676697 21225 leader_election.cc:290] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:00.686501 21151 raft_consensus.cc:515] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:00.687002 21228 raft_consensus.cc:493] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:00.687288 21229 leader_election.cc:290] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:00.687983 21228 raft_consensus.cc:515] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:00.689100 21151 leader_election.cc:290] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:00.689400 21224 leader_election.cc:290] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:00.690434 21230 leader_election.cc:290] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:00.677448 21227 raft_consensus.cc:493] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:00.691694 21227 raft_consensus.cc:515] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:00.694474 21227 leader_election.cc:290] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:00.700650 21225 raft_consensus.cc:493] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:00.701198 21223 raft_consensus.cc:493] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:00.701880 21222 leader_election.cc:290] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:00.697592 21073 raft_consensus.cc:493] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:00.703434 21230 raft_consensus.cc:493] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:00.696108 21228 leader_election.cc:290] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:00.708148 21233 raft_consensus.cc:493] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:00.708277 21235 raft_consensus.cc:493] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.709137 21073 raft_consensus.cc:515] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:00.709398 21233 raft_consensus.cc:515] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:00.709625 21235 raft_consensus.cc:515] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:00.710942 21223 raft_consensus.cc:515] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:00.711961 21073 leader_election.cc:290] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:00.712657 21235 leader_election.cc:290] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
W20260808 01:55:00.713177 20725 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.063s	user 0.002s	sys 0.017s
I20260808 01:55:00.713570 21223 leader_election.cc:290] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:00.714715 21225 raft_consensus.cc:515] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:00.717494 21225 leader_election.cc:290] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:00.718395 21230 raft_consensus.cc:515] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
W20260808 01:55:00.713779 20725 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.063s	user 0.002s	sys 0.017s
I20260808 01:55:00.723328 21230 leader_election.cc:290] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:00.727891 21029 raft_consensus.cc:493] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:00.730127 21029 raft_consensus.cc:515] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:00.726404 21225 raft_consensus.cc:493] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:00.733568 21029 leader_election.cc:290] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:00.724539 21073 raft_consensus.cc:493] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:00.733794 21225 raft_consensus.cc:515] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:00.729148 21233 leader_election.cc:290] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:00.734629 20813 tablet_bootstrap.cc:654] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:00.737188 21215 raft_consensus.cc:493] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:00.745086 20815 tablet_bootstrap.cc:492] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:55:00.744942 21215 raft_consensus.cc:515] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:00.746061 20815 ts_tablet_manager.cc:1403] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 3.220s	user 0.021s	sys 0.037s
I20260808 01:55:00.738732 21228 raft_consensus.cc:493] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.754794 21228 raft_consensus.cc:515] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:00.738279 21236 raft_consensus.cc:493] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.741789 21078 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:00.758051 21228 leader_election.cc:290] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:00.757725 21078 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:00.741341 21222 raft_consensus.cc:493] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:00.758877 21223 raft_consensus.cc:493] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.759145 21222 raft_consensus.cc:515] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:00.759866 21223 raft_consensus.cc:515] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:00.760598 21078 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:00.739221 21151 raft_consensus.cc:493] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.757472 21236 raft_consensus.cc:515] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:00.737448 21225 leader_election.cc:290] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:00.789495 21242 raft_consensus.cc:493] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:00.791594 21240 raft_consensus.cc:493] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.792762 21240 raft_consensus.cc:515] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:00.796315 21240 leader_election.cc:290] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:00.797151 21240 raft_consensus.cc:493] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
W20260808 01:55:00.739876 20493 connection.cc:256] RPC call timeout handler was delayed by -1.8890728779952042s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1525s for a response
W20260808 01:55:00.798431 20493 connection.cc:256] RPC call timeout handler was delayed by -1.8890728779952042s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1525s for a response
I20260808 01:55:00.735534 21073 raft_consensus.cc:515] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:00.737716 21029 raft_consensus.cc:493] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:00.761579 21228 raft_consensus.cc:493] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.814102 21236 leader_election.cc:290] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:00.814971 21084 raft_consensus.cc:493] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:00.815290 21233 raft_consensus.cc:493] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.814886 21029 raft_consensus.cc:515] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:00.801959 21222 leader_election.cc:290] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:00.816982 20493 leader_election.cc:304] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:00.818638 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:00.820258 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:00.782661 21225 raft_consensus.cc:493] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:00.825522 21244 raft_consensus.cc:493] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:00.826073 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:00.833968 20684 raft_consensus.cc:2468] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:00.834990 21225 raft_consensus.cc:515] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:00.835913 20683 raft_consensus.cc:2468] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:00.827412 21223 leader_election.cc:290] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:00.839015 21223 raft_consensus.cc:493] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:00.824795 21215 leader_election.cc:290] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:00.800050 21151 raft_consensus.cc:515] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:00.829167 20979 raft_consensus.cc:493] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:00.832999 20682 raft_consensus.cc:2468] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:00.839191 21244 raft_consensus.cc:515] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:00.846768 21151 leader_election.cc:290] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:00.863000 21244 leader_election.cc:290] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:00.863497 20726 leader_election.cc:304] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:00.828037 21228 raft_consensus.cc:515] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:00.847159 21233 raft_consensus.cc:515] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:00.848212 20979 raft_consensus.cc:515] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:00.850908 20810 tablet_bootstrap.cc:492] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:55:00.867805 20810 ts_tablet_manager.cc:1403] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 3.333s	user 0.092s	sys 0.127s
I20260808 01:55:00.868759 20979 leader_election.cc:290] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:00.869495 20979 raft_consensus.cc:493] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:00.853699 21189 raft_consensus.cc:493] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.854292 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:00.852751 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:00.871778 20686 raft_consensus.cc:2468] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:00.873315 20426 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48846 (request call id 76) took 3359 ms. Trace:
I20260808 01:55:00.892292 20426 rpcz_store.cc:276] 0808 01:54:57.513509 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:54:57.513745 (+   236us) service_pool.cc:224] Handling call
0808 01:55:00.873019 (+3359274us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20311,"reported_tablets":1}
I20260808 01:55:00.895277 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:55:00.859454 20552 debug-util.cc:398] Leaking SignalData structure 0x7b080030d780 after lost signal to thread 20461
I20260808 01:55:00.839864 21223 raft_consensus.cc:515] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:00.899780 21147 raft_consensus.cc:493] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:00.900658 21147 raft_consensus.cc:515] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:00.900928 21223 leader_election.cc:290] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:00.901093 21138 raft_consensus.cc:493] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.859973 21235 raft_consensus.cc:493] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.860524 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:00.901892 21138 raft_consensus.cc:515] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:00.866302 21233 leader_election.cc:290] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:00.859115 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:00.870797 21189 raft_consensus.cc:515] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:00.866295 21228 leader_election.cc:290] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:00.870249 20979 raft_consensus.cc:515] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:00.857694 21218 raft_consensus.cc:493] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:00.876847 21073 leader_election.cc:290] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:00.877583 21084 raft_consensus.cc:515] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:00.879686 20726 leader_election.cc:304] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:00.880620 21151 raft_consensus.cc:2804] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:00.805917 21242 raft_consensus.cc:515] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:00.958130 21216 raft_consensus.cc:493] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
W20260808 01:55:00.959206 20530 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41410 (request call id 67) took 3612 ms (client timeout 2087 ms). Trace:
W20260808 01:55:00.959870 20530 rpcz_store.cc:269] 0808 01:54:57.360133 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:54:57.381143 (+ 21010us) service_pool.cc:224] Handling call
0808 01:55:00.958615 (+3577472us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b5400632c54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:00.958911 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":329847680}
I20260808 01:55:00.960932 21242 leader_election.cc:290] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:00.885017 20813 tablet_bootstrap.cc:492] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:55:00.962262 20813 ts_tablet_manager.cc:1403] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 3.367s	user 0.030s	sys 0.056s
I20260808 01:55:00.887332 21078 raft_consensus.cc:2804] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
W20260808 01:55:00.963150 20648 connection.cc:256] RPC call timeout handler was delayed by -1.4257692589890212s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20870000000000002s for a response
I20260808 01:55:00.963725 21078 raft_consensus.cc:697] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:00.965404 21078 consensus_queue.cc:237] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:00.966756 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:00.967705 20685 raft_consensus.cc:2468] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
W20260808 01:55:00.968971 20811 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 3.400s	user 0.019s	sys 0.043s
W20260808 01:55:00.969544 20811 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 3.401s	user 0.019s	sys 0.043s
W20260808 01:55:00.887941 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 71219.9us
I20260808 01:55:00.969616 21243 raft_consensus.cc:493] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:00.970569 21243 raft_consensus.cc:515] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:00.971868 21147 leader_election.cc:290] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:00.973012 20811 tablet_bootstrap.cc:492] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:55:00.863674 21244 raft_consensus.cc:493] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.855398 21238 raft_consensus.cc:493] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.902591 20680 raft_consensus.cc:2468] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:00.838541 21225 leader_election.cc:290] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:00.973400 21243 leader_election.cc:290] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:00.895450 20954 raft_consensus.cc:493] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:00.902513 21235 raft_consensus.cc:515] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:00.973784 21218 raft_consensus.cc:515] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:00.902895 21029 leader_election.cc:290] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
W20260808 01:55:00.882429 20475 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:55:00.904143 20425 catalog_manager.cc:5697] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194). New cstate: current_term: 1 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:01.017859 20810 raft_consensus.cc:359] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:00.948792 21223 raft_consensus.cc:493] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:00.965418 21237 raft_consensus.cc:493] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:01.004801 21244 raft_consensus.cc:515] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:01.006435 21138 leader_election.cc:290] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:01.008693 21084 leader_election.cc:290] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:01.011405 20424 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48860 (request call id 71) took 3468 ms. Trace:
I20260808 01:55:01.012148 20813 raft_consensus.cc:359] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:01.033706 20813 raft_consensus.cc:385] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:01.033594 20424 rpcz_store.cc:276] 0808 01:54:57.543108 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:54:57.544693 (+  1585us) service_pool.cc:224] Handling call
0808 01:55:01.011195 (+3466502us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3288025,"reported_tablets":1}
I20260808 01:55:01.034242 20813 raft_consensus.cc:740] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:55:00.959221 21216 raft_consensus.cc:515] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:01.034789 21223 raft_consensus.cc:515] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:01.035281 20813 consensus_queue.cc:260] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:00.958230 21151 raft_consensus.cc:493] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:01.013029 20815 raft_consensus.cc:359] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:01.015654 20979 leader_election.cc:290] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:01.039359 21151 raft_consensus.cc:3060] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:01.039384 20815 raft_consensus.cc:385] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:01.040107 20815 raft_consensus.cc:740] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:55:01.040920 20815 consensus_queue.cc:260] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:01.042920 21243 raft_consensus.cc:493] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:01.016738 21218 leader_election.cc:290] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:00.903551 21215 raft_consensus.cc:493] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:00.946992 21228 raft_consensus.cc:493] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:00.902860 21236 raft_consensus.cc:493] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:01.044054 21243 raft_consensus.cc:515] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:01.037302 21216 leader_election.cc:290] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:01.017745 21029 raft_consensus.cc:493] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:01.019995 20810 raft_consensus.cc:385] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:01.004930 20954 raft_consensus.cc:515] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:01.068073 21244 leader_election.cc:290] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:01.018900 21147 raft_consensus.cc:2804] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:01.081532 20810 raft_consensus.cc:740] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:55:00.842325 21240 raft_consensus.cc:515] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
W20260808 01:55:01.001242 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 119393us
I20260808 01:55:01.038236 21223 leader_election.cc:290] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:00.997026 21242 raft_consensus.cc:493] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:01.075470 21236 raft_consensus.cc:515] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:01.027105 21225 raft_consensus.cc:493] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:01.075204 21215 raft_consensus.cc:515] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:01.085748 20815 ts_tablet_manager.cc:1434] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.339s	user 0.008s	sys 0.010s
I20260808 01:55:01.065905 20683 raft_consensus.cc:2468] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:01.096472 20813 ts_tablet_manager.cc:1434] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.134s	user 0.007s	sys 0.002s
I20260808 01:55:01.098846 20811 ts_tablet_manager.cc:1403] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 3.552s	user 0.035s	sys 0.046s
I20260808 01:55:01.085809 21147 raft_consensus.cc:493] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
W20260808 01:55:01.093987 20552 debug-util.cc:398] Leaking SignalData structure 0x7b0800188c00 after lost signal to thread 20555
W20260808 01:55:01.086761 20648 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.193:34531 timed out after 2.087s (SENT) [suppressed 2 similar messages]
I20260808 01:55:01.101193 20810 consensus_queue.cc:260] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
W20260808 01:55:01.110419 20552 debug-util.cc:398] Leaking SignalData structure 0x7b08002b9f00 after lost signal to thread 20556
I20260808 01:55:01.124986 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d32821d9c17456394a662fcfd05f6c6" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.125788 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.126302 21236 leader_election.cc:290] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:01.127081 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.127334 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.127436 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.127696 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.128288 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.128672 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.142957 20813 tablet_bootstrap.cc:492] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:55:01.139843 21220 raft_consensus.cc:493] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:01.144263 21220 raft_consensus.cc:515] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:01.144833 20525 raft_consensus.cc:2468] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:01.146571 20810 ts_tablet_manager.cc:1434] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.278s	user 0.010s	sys 0.001s
I20260808 01:55:01.132277 20726 leader_election.cc:304] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:01.136679 20423 catalog_manager.cc:5697] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:01.148252 20810 tablet_bootstrap.cc:492] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:55:01.103634 21238 raft_consensus.cc:515] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:01.150878 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.102386 21073 raft_consensus.cc:493] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:01.152194 21238 leader_election.cc:290] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:01.109843 21029 raft_consensus.cc:515] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:01.153569 21243 leader_election.cc:290] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:01.131119 21235 leader_election.cc:290] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:01.123338 21242 raft_consensus.cc:515] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:01.131850 20954 leader_election.cc:290] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:01.108368 21147 raft_consensus.cc:3060] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:01.157068 21242 leader_election.cc:290] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:01.132368 21150 raft_consensus.cc:493] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:01.103360 21084 raft_consensus.cc:493] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:01.095806 21228 raft_consensus.cc:515] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:01.141438 21233 raft_consensus.cc:493] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:01.154132 21151 raft_consensus.cc:515] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:01.164054 21233 raft_consensus.cc:515] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:01.164885 21150 raft_consensus.cc:515] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:01.123128 21225 raft_consensus.cc:515] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:01.168045 20815 tablet_bootstrap.cc:492] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:55:01.167912 21245 raft_consensus.cc:493] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
W20260808 01:55:01.168344 20552 debug-util.cc:398] Leaking SignalData structure 0x7b08000b5f60 after lost signal to thread 20634
I20260808 01:55:01.169005 21245 raft_consensus.cc:515] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:01.170660 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.171190 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.154479 21250 raft_consensus.cc:493] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:01.173086 21252 raft_consensus.cc:493] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:01.172688 21250 raft_consensus.cc:515] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:01.173838 21252 raft_consensus.cc:515] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:01.172679 21242 raft_consensus.cc:493] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:01.175388 21242 raft_consensus.cc:515] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:01.176499 20914 raft_consensus.cc:493] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:01.177194 20914 raft_consensus.cc:515] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:01.105625 21237 raft_consensus.cc:515] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:01.152197 21073 raft_consensus.cc:515] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:01.177385 21243 raft_consensus.cc:493] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:01.154747 21029 leader_election.cc:290] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:01.168999 20552 debug-util.cc:398] Leaking SignalData structure 0x7b08001b8e60 after lost signal to thread 20635
I20260808 01:55:01.213411 20954 raft_consensus.cc:2804] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:01.215950 21249 raft_consensus.cc:493] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:01.217569 21238 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:01.222522 20954 raft_consensus.cc:493] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:01.224576 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.228379 20810 tablet_bootstrap.cc:654] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:01.224537 20954 raft_consensus.cc:3060] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:01.236608 21233 leader_election.cc:290] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:01.215363 21235 raft_consensus.cc:493] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:01.224373 21238 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:01.235389 21236 raft_consensus.cc:493] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:01.214741 21215 leader_election.cc:290] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
W20260808 01:55:01.239264 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 156923us
I20260808 01:55:01.237524 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.243670 21244 raft_consensus.cc:493] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:01.224352 21249 raft_consensus.cc:515] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:01.222491 21084 raft_consensus.cc:515] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:01.244844 21235 raft_consensus.cc:515] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:01.246546 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.246023 21236 raft_consensus.cc:515] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:01.247063 21073 leader_election.cc:290] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:01.222747 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.222952 21189 leader_election.cc:290] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:01.299352 21240 leader_election.cc:290] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:01.301191 21235 leader_election.cc:290] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:01.244385 21242 leader_election.cc:290] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:01.255148 20552 debug-util.cc:398] Leaking SignalData structure 0x7b080028fde0 after lost signal to thread 20712
I20260808 01:55:01.304006 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:55:01.304716 20552 debug-util.cc:398] Leaking SignalData structure 0x7b08001e4580 after lost signal to thread 20713
I20260808 01:55:01.305670 21245 leader_election.cc:290] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:55:01.306411 20552 debug-util.cc:398] Leaking SignalData structure 0x7b0800241740 after lost signal to thread 20789
I20260808 01:55:01.306742 21225 leader_election.cc:290] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:01.297686 21237 leader_election.cc:290] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:01.298444 21250 leader_election.cc:290] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:01.309643 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.302084 20602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63fb8c2f494a462686500ce10025a39f" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.311343 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.312546 21084 leader_election.cc:290] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:01.313639 21252 leader_election.cc:290] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:01.314931 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.315312 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.316309 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.301446 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e71aab2ce9b45369b7eff65e1645f9b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.317391 20603 raft_consensus.cc:2468] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:55:01.321673 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.250522 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.250519 21238 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:01.323137 20762 raft_consensus.cc:2468] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:01.243742 21220 leader_election.cc:290] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:01.246047 21150 leader_election.cc:290] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:01.244446 20914 leader_election.cc:290] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:01.272354 20811 raft_consensus.cc:359] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:01.325090 20811 raft_consensus.cc:385] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:01.325464 20811 raft_consensus.cc:740] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:55:01.326098 21147 raft_consensus.cc:515] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:01.275410 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.275547 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63fb8c2f494a462686500ce10025a39f" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.336514 20759 raft_consensus.cc:2468] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:01.337409 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.338009 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.339186 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1753378f13594e89aee6641e2cb9e6da" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.339958 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.340693 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.226848 21243 raft_consensus.cc:515] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:01.278313 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.279925 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.263212 21244 raft_consensus.cc:515] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:01.282764 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.283346 21228 leader_election.cc:290] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:01.281241 21151 leader_election.cc:290] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:01.285604 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.286755 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.287482 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.288314 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.289085 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.352725 20608 raft_consensus.cc:2468] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:01.353099 20815 tablet_bootstrap.cc:654] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:01.289988 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.353775 21244 leader_election.cc:290] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:01.354053 20520 raft_consensus.cc:2468] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:01.290418 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.291335 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.363952 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:01.364267 20570 leader_election.cc:343] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:01.365219 20570 leader_election.cc:304] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:01.365483 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.366828 20914 raft_consensus.cc:2804] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:01.367285 20914 raft_consensus.cc:493] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:01.367676 20914 raft_consensus.cc:3060] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:01.369009 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.373713 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.292637 21249 leader_election.cc:290] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:01.293856 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.294781 21236 leader_election.cc:290] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:01.295166 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:01.256083 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 135308us
W20260808 01:55:01.318063 20648 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.7.165.254:40793, user_credentials={real_user=slave}} blocked reactor thread for 50042.7us
I20260808 01:55:01.326546 20811 consensus_queue.cc:260] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:01.273377 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e71aab2ce9b45369b7eff65e1645f9b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.328192 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.329357 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.316359 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.331030 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.328181 20813 tablet_bootstrap.cc:654] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:01.340689 20759 raft_consensus.cc:2468] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:55:01.350452 21243 leader_election.cc:290] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:01.350812 20755 raft_consensus.cc:2468] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:01.360587 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.362280 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.409569 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.390713 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:01.377099 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.410501 20681 raft_consensus.cc:2468] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:55:01.397599 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d32821d9c17456394a662fcfd05f6c6" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.431386 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.432478 20686 raft_consensus.cc:2468] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:01.437822 21147 leader_election.cc:290] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:01.438908 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.439221 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.440140 20760 raft_consensus.cc:2468] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:01.445216 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.446198 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.447058 20759 raft_consensus.cc:2468] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:01.446198 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.448261 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.449323 20756 raft_consensus.cc:2468] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:01.449657 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.450860 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.451810 20761 raft_consensus.cc:2468] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:01.452790 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d32821d9c17456394a662fcfd05f6c6" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.455483 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.455418 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.457744 20754 raft_consensus.cc:2393] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 2: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:01.462018 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.471426 20726 leader_election.cc:304] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:01.473060 21147 raft_consensus.cc:2804] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:01.473441 20726 leader_election.cc:304] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:01.473557 21147 raft_consensus.cc:493] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:01.474852 21249 raft_consensus.cc:2804] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:01.475307 21249 raft_consensus.cc:493] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:01.475672 21249 raft_consensus.cc:3060] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:01.476420 20811 ts_tablet_manager.cc:1434] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.368s	user 0.008s	sys 0.004s
I20260808 01:55:01.478659 20811 tablet_bootstrap.cc:492] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:55:01.489821 21147 raft_consensus.cc:3060] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:01.479609 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.491972 20680 raft_consensus.cc:2468] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:01.396934 20523 raft_consensus.cc:2468] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:55:01.483994 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.500123 20954 raft_consensus.cc:515] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:01.487866 20762 raft_consensus.cc:2468] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:55:01.488884 20600 raft_consensus.cc:2393] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:01.485137 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1753378f13594e89aee6641e2cb9e6da" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.501778 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.502995 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.504171 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.505546 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.507467 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.509860 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.511601 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.512467 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.513267 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.515112 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.515004 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.516259 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63fb8c2f494a462686500ce10025a39f" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.517342 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.518597 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.518818 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.520538 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.526782 20726 leader_election.cc:304] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:01.528331 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.530051 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.533355 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.556279 20609 raft_consensus.cc:2393] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:01.557869 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.562619 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.564004 20677 raft_consensus.cc:2468] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:01.572347 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.572628 20758 raft_consensus.cc:1240] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:01.575868 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.573421 20756 raft_consensus.cc:1240] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:01.567304 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.565033 20682 raft_consensus.cc:2468] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:55:01.567130 20726 leader_election.cc:304] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:01.568706 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.570570 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.580363 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.580804 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.567219 21228 raft_consensus.cc:2804] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:01.576107 20954 leader_election.cc:290] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:01.580873 21236 raft_consensus.cc:2804] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:01.551802 20683 raft_consensus.cc:2468] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:01.585021 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.585995 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.581084 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.586583 21228 raft_consensus.cc:493] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:01.591004 21236 raft_consensus.cc:493] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:01.594655 20757 raft_consensus.cc:2468] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:55:01.603338 21236 raft_consensus.cc:3060] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:01.603516 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.603776 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.604285 20755 raft_consensus.cc:2468] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:01.604797 20758 raft_consensus.cc:3060] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:01.597383 21228 raft_consensus.cc:3060] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:01.608145 20815 tablet_bootstrap.cc:492] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:55:01.608701 20815 ts_tablet_manager.cc:1403] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.441s	user 0.049s	sys 0.067s
I20260808 01:55:01.609529 21249 raft_consensus.cc:515] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:01.611370 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.612124 20608 raft_consensus.cc:2468] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:01.612383 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.613256 20811 tablet_bootstrap.cc:654] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:01.613991 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.615705 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.617308 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.619071 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.594749 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.601922 20810 tablet_bootstrap.cc:492] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:55:01.620805 20810 ts_tablet_manager.cc:1403] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.473s	user 0.026s	sys 0.014s
I20260808 01:55:01.618971 20602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.613188 20603 raft_consensus.cc:2468] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:01.620584 20600 raft_consensus.cc:2468] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:01.619760 21147 raft_consensus.cc:515] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:01.602869 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.632468 21147 leader_election.cc:290] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:01.633759 20753 raft_consensus.cc:1240] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:01.635238 20760 raft_consensus.cc:2393] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 2: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:01.635500 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.636303 20684 raft_consensus.cc:2468] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:01.649837 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.650790 20760 raft_consensus.cc:2468] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:01.651942 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.652812 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.653497 20759 raft_consensus.cc:1240] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:01.659286 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.660142 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.646754 21249 leader_election.cc:290] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:01.669329 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.671298 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.671486 20602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.672897 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.674381 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.652918 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.677502 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.679499 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.682569 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:01.667119 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63fb8c2f494a462686500ce10025a39f" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.686055 20649 leader_election.cc:304] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:01.687623 20649 leader_election.cc:304] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:01.688843 21238 raft_consensus.cc:2804] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:01.688922 20649 leader_election.cc:304] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:01.689504 21238 raft_consensus.cc:493] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:01.690088 21238 raft_consensus.cc:3060] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:01.690160 21237 raft_consensus.cc:2804] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:01.690728 21237 raft_consensus.cc:493] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:01.664121 20815 raft_consensus.cc:359] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:01.691560 20815 raft_consensus.cc:385] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:01.691833 20815 raft_consensus.cc:740] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:55:01.692241 21244 raft_consensus.cc:2804] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:01.692665 21244 raft_consensus.cc:493] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:01.693086 21244 raft_consensus.cc:3060] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:01.692787 20815 consensus_queue.cc:260] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:01.691246 21237 raft_consensus.cc:3060] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:01.645975 20810 raft_consensus.cc:359] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:01.706735 20810 raft_consensus.cc:385] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:01.707181 20810 raft_consensus.cc:740] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:55:01.722183 20725 leader_election.cc:304] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:01.720026 20810 consensus_queue.cc:260] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:01.664927 20762 raft_consensus.cc:3060] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:01.749384 20813 tablet_bootstrap.cc:492] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:55:01.750164 20813 ts_tablet_manager.cc:1403] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.608s	user 0.018s	sys 0.028s
I20260808 01:55:01.755728 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.757081 20759 raft_consensus.cc:1240] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:01.742187 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.640470 20914 raft_consensus.cc:515] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:01.718955 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:01.741750 21228 raft_consensus.cc:515] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:01.742653 21147 raft_consensus.cc:2749] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:01.718816 20649 leader_election.cc:304] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
W20260808 01:55:01.775571 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 91980.5us
I20260808 01:55:01.771107 20603 raft_consensus.cc:2468] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:55:01.777449 20493 leader_election.cc:304] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:01.780520 20811 tablet_bootstrap.cc:492] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:55:01.781198 20811 ts_tablet_manager.cc:1403] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.303s	user 0.030s	sys 0.056s
I20260808 01:55:01.786258 20493 leader_election.cc:304] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:01.764585 21236 raft_consensus.cc:515] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:01.786761 21220 raft_consensus.cc:2804] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260808 01:55:01.784758 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 69854.9us
I20260808 01:55:01.795125 20815 ts_tablet_manager.cc:1434] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.186s	user 0.006s	sys 0.004s
I20260808 01:55:01.795764 20810 ts_tablet_manager.cc:1434] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.175s	user 0.009s	sys 0.000s
I20260808 01:55:01.796315 21078 consensus_queue.cc:1059] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:01.797381 21228 leader_election.cc:290] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:01.783272 20609 raft_consensus.cc:1240] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:01.807468 21078 raft_consensus.cc:2804] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:01.807879 20648 leader_election.cc:304] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:01.800879 21220 raft_consensus.cc:493] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:01.809013 21220 raft_consensus.cc:3060] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:01.810096 21078 raft_consensus.cc:493] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:01.810822 21078 raft_consensus.cc:3060] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:01.815562 20815 tablet_bootstrap.cc:492] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:55:01.793588 20754 raft_consensus.cc:1240] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:01.799268 21222 raft_consensus.cc:2804] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:55:01.802451 20914 leader_election.cc:290] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:01.827879 20813 raft_consensus.cc:359] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:01.820806 21240 raft_consensus.cc:2804] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:01.830286 21240 raft_consensus.cc:493] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:01.831331 21240 raft_consensus.cc:3060] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:01.831331 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.830178 21266 raft_consensus.cc:3055] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:01.829909 20813 raft_consensus.cc:385] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:01.830776 20571 leader_election.cc:304] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:01.827369 21222 raft_consensus.cc:493] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:01.835883 20813 raft_consensus.cc:740] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:55:01.842789 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.834558 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:01.845590 20811 raft_consensus.cc:359] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:01.845819 20685 raft_consensus.cc:2468] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:01.846436 20811 raft_consensus.cc:385] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:01.847986 20811 raft_consensus.cc:740] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
W20260808 01:55:01.847386 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 71286.8us
W20260808 01:55:01.849599 20571 leader_election.cc:343] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:01.850328 20571 leader_election.cc:304] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:01.830590 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.858402 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.858845 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.851796 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:01.860190 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.861896 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:01.848249 21222 raft_consensus.cc:3060] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:01.849721 20914 raft_consensus.cc:2749] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:01.870357 21236 leader_election.cc:290] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:01.857689 20425 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48838 (request call id 80) took 4270 ms. Trace:
I20260808 01:55:01.884778 20810 tablet_bootstrap.cc:492] T ce51fcffe2ae4bcba55cd864c973bf0d P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:55:01.835654 21266 raft_consensus.cc:740] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:01.886421 20811 consensus_queue.cc:260] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:02.067574 21266 consensus_queue.cc:260] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:02.068843 20811 ts_tablet_manager.cc:1434] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.287s	user 0.003s	sys 0.005s
I20260808 01:55:02.069010 21266 raft_consensus.cc:3060] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.071149 20811 tablet_bootstrap.cc:492] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:55:01.886458 21238 raft_consensus.cc:515] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:01.909544 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:02.255890 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:01.903501 20726 leader_election.cc:304] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:01.911119 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:55:02.262593 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 359238us
W20260808 01:55:01.884090 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 65824.9us
I20260808 01:55:01.905931 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:01.927724 21269 consensus_queue.cc:1059] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260808 01:55:02.295673 21269 raft_consensus.cc:3055] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:02.296123 21269 raft_consensus.cc:740] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:02.297169 21269 consensus_queue.cc:260] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:01.995950 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 145663us
I20260808 01:55:02.065423 20425 rpcz_store.cc:276] 0808 01:54:57.587557 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:54:57.587735 (+   178us) service_pool.cc:224] Handling call
0808 01:55:01.857599 (+4269864us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:55:01.301349 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:55:01.302123 (+   774us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:55:01.302194 (+    71us) write_op.cc:435] Acquired schema lock
0808 01:55:01.302238 (+    44us) tablet.cc:662] Decoding operations
0808 01:55:01.302949 (+   711us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:55:01.303106 (+   157us) write_op.cc:641] Partition lock acquired for write op
0808 01:55:01.303145 (+    39us) tablet.cc:685] Acquiring locks for 1 operations
0808 01:55:01.303555 (+   410us) tablet.cc:701] Row locks acquired
0808 01:55:01.303615 (+    60us) write_op.cc:260] PREPARE: finished
0808 01:55:01.303804 (+   189us) write_op.cc:270] Start()
0808 01:55:01.303965 (+   161us) write_op.cc:276] Timestamp: P: 1786154101303747 usec, L: 0
0808 01:55:01.304006 (+    41us) op_driver.cc:348] REPLICATION: starting
0808 01:55:01.330663 (+ 26657us) log.cc:844] Serialized 540 byte log entry
0808 01:55:01.754483 (+423820us) op_driver.cc:464] REPLICATION: finished
0808 01:55:01.851701 (+ 97218us) write_op.cc:301] APPLY: starting
0808 01:55:01.851797 (+    96us) tablet.cc:1367] starting BulkCheckPresence
0808 01:55:01.851952 (+   155us) tablet.cc:1370] finished BulkCheckPresence
0808 01:55:01.852121 (+   169us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:55:01.852466 (+   345us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:55:01.852756 (+   290us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 01:55:01.852819 (+    63us) write_op.cc:312] APPLY: finished
0808 01:55:01.853123 (+   304us) log.cc:844] Serialized 22 byte log entry
0808 01:55:01.853341 (+   218us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:55:01.853426 (+    85us) write_op.cc:454] Released schema lock
0808 01:55:01.853483 (+    57us) write_op.cc:341] FINISH: Updating metrics
0808 01:55:01.854010 (+   527us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:55:01.854041 (+    31us) write_op.cc:454] Released schema lock
0808 01:55:01.933745 (+ 79704us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:55:01.934118 (+   373us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3284653,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":96906,"apply.run_cpu_time_us":2294,"apply.run_wall_time_us":2295,"num_ops":1,"prepare.queue_time_us":122406,"prepare.run_cpu_time_us":42904,"prepare.run_wall_time_us":117692,"raft.queue_time_us":210328,"raft.run_cpu_time_us":28724,"raft.run_wall_time_us":216278,"replication_time_us":447295,"thread_start_us":272955,"threads_started":4,"wal-append.queue_time_us":211956}]]}
I20260808 01:55:02.083863 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:02.075029 20810 tablet_bootstrap.cc:654] T ce51fcffe2ae4bcba55cd864c973bf0d P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:02.080010 20811 tablet_bootstrap.cc:654] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:01.927138 20813 consensus_queue.cc:260] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:01.995998 20914 raft_consensus.cc:2804] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:01.935708 21078 raft_consensus.cc:515] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:01.921221 20815 tablet_bootstrap.cc:654] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:01.935995 21222 raft_consensus.cc:515] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:01.934015 21220 raft_consensus.cc:515] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:01.931384 21237 raft_consensus.cc:515] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:01.945513 21240 raft_consensus.cc:515] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:02.074393 21238 leader_election.cc:290] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:01.931366 21244 raft_consensus.cc:515] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:02.357887 21240 leader_election.cc:290] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:02.358744 20677 raft_consensus.cc:1240] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:02.359676 21244 leader_election.cc:290] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:02.164786 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:02.166873 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:55:02.174582 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 346591us
I20260808 01:55:02.201504 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:02.358785 20680 raft_consensus.cc:2393] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 2: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:02.362936 20681 raft_consensus.cc:2393] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 2: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:02.233665 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:02.234856 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:02.364063 21266 consensus_queue.cc:1048] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:55:02.264101 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:02.264999 21228 raft_consensus.cc:2749] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:02.266047 20726 leader_election.cc:304] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:02.366288 21266 consensus_queue.cc:1059] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260808 01:55:02.366631 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 100675us
I20260808 01:55:02.368103 20726 leader_election.cc:304] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
W20260808 01:55:02.275233 20494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:02.368400 21271 raft_consensus.cc:3055] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:02.368789 21236 raft_consensus.cc:2749] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:02.369160 21228 raft_consensus.cc:493] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:02.369448 21271 raft_consensus.cc:740] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:02.369848 21228 raft_consensus.cc:515] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:02.372491 21228 leader_election.cc:290] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:02.371553 21271 consensus_queue.cc:260] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:02.375030 20726 leader_election.cc:304] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:02.376752 21236 raft_consensus.cc:2804] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:02.377205 21236 raft_consensus.cc:493] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:02.377559 21236 raft_consensus.cc:3060] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:55:02.379415 20649 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.010s	sys 0.003s
I20260808 01:55:02.298657 21269 raft_consensus.cc:3060] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.299243 20571 leader_election.cc:304] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
W20260808 01:55:02.384231 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 85091.3us
I20260808 01:55:02.385208 20571 leader_election.cc:304] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:02.385486 21151 raft_consensus.cc:2749] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:02.317854 20914 raft_consensus.cc:493] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:02.386262 21151 consensus_queue.cc:1059] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:55:02.386546 20914 raft_consensus.cc:3060] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.386879 21245 raft_consensus.cc:2804] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:02.387362 21245 raft_consensus.cc:493] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:02.387691 21228 raft_consensus.cc:3055] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:02.387885 21252 raft_consensus.cc:2749] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:02.388139 21228 raft_consensus.cc:740] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:02.387730 21245 raft_consensus.cc:3060] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.317871 20813 ts_tablet_manager.cc:1434] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.567s	user 0.006s	sys 0.008s
W20260808 01:55:02.394910 20813 ts_tablet_manager.cc:1467] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc: Tablet startup took 644ms
W20260808 01:55:02.395296 20813 ts_tablet_manager.cc:1469] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc: Trace:
0808 01:54:50.210186 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:54:50.210455 (+   269us) service_pool.cc:224] Handling call
0808 01:54:50.230462 (+ 20007us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0808 01:54:50.230563 (+   101us) ts_tablet_manager.cc:682] Creating new metadata...
0808 01:54:50.311193 (+ 80630us) tablet_metadata.cc:665] Metadata flushed
0808 01:54:50.438966 (+127773us) tablet_replica.cc:206] Creating consensus instance
0808 01:54:50.460313 (+ 21347us) inbound_call.cc:177] Queueing success response
0808 01:55:01.138407 (+10678094us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0808 01:55:01.750527 (+612120us) ts_tablet_manager.cc:1435] Starting tablet replica
0808 01:55:01.791974 (+ 41447us) tablet_replica.cc:261] Starting instrumentation
0808 01:55:01.792808 (+   834us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":1722,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":143,"tablet-open.queue_time_us":10678120}
I20260808 01:55:02.397899 20813 tablet_bootstrap.cc:492] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:55:02.321075 20523 raft_consensus.cc:1240] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:02.323472 21222 leader_election.cc:290] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:02.323843 20528 raft_consensus.cc:2393] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:02.324924 21220 leader_election.cc:290] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:02.326117 21237 leader_election.cc:290] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:02.333531 20811 tablet_bootstrap.cc:492] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:55:02.404961 20811 ts_tablet_manager.cc:1403] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.334s	user 0.012s	sys 0.002s
I20260808 01:55:02.349613 20810 tablet_bootstrap.cc:492] T ce51fcffe2ae4bcba55cd864c973bf0d P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:55:02.407398 20810 ts_tablet_manager.cc:1403] T ce51fcffe2ae4bcba55cd864c973bf0d P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.582s	user 0.012s	sys 0.014s
I20260808 01:55:02.407215 21228 consensus_queue.cc:260] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:02.408764 21228 raft_consensus.cc:3060] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.356702 20815 tablet_bootstrap.cc:492] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:55:02.411146 20815 ts_tablet_manager.cc:1403] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.596s	user 0.016s	sys 0.009s
I20260808 01:55:02.414954 20914 raft_consensus.cc:515] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:02.416505 21236 raft_consensus.cc:515] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:02.417081 20914 leader_election.cc:290] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:02.418468 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:02.419070 20815 raft_consensus.cc:359] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:02.421377 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:02.425786 20684 raft_consensus.cc:3060] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.398765 20571 leader_election.cc:304] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:02.427249 20571 leader_election.cc:304] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:02.427582 20914 raft_consensus.cc:2804] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:02.428048 21252 raft_consensus.cc:2804] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:02.428174 20914 raft_consensus.cc:493] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:02.428669 21252 raft_consensus.cc:493] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:02.428967 20914 raft_consensus.cc:3060] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.429173 21252 raft_consensus.cc:3060] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.435424 21243 consensus_queue.cc:1059] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:02.437515 21243 raft_consensus.cc:3055] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:02.437958 20762 raft_consensus.cc:1240] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 1. Current term is 2. Ops: []
W20260808 01:55:02.399477 20649 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.011s	sys 0.004s
I20260808 01:55:02.438169 21243 raft_consensus.cc:740] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
W20260808 01:55:02.438942 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 566914us
I20260808 01:55:02.368554 20423 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48828 (request call id 77) took 4905 ms. Trace:
I20260808 01:55:02.293445 20648 leader_election.cc:304] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:02.424610 20815 raft_consensus.cc:385] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:02.416761 20811 raft_consensus.cc:359] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:02.421633 20810 raft_consensus.cc:359] T ce51fcffe2ae4bcba55cd864c973bf0d P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:02.442821 20811 raft_consensus.cc:385] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:02.443038 20810 raft_consensus.cc:385] T ce51fcffe2ae4bcba55cd864c973bf0d P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:02.443177 20811 raft_consensus.cc:740] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:55:02.320827 21078 leader_election.cc:290] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:02.443720 20810 raft_consensus.cc:740] T ce51fcffe2ae4bcba55cd864c973bf0d P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
W20260808 01:55:02.446528 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 153184us
I20260808 01:55:02.442350 20815 raft_consensus.cc:740] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:55:02.446110 20811 consensus_queue.cc:260] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:02.451084 21244 raft_consensus.cc:3060] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.459333 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:02.459595 20811 ts_tablet_manager.cc:1434] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.054s	user 0.005s	sys 0.004s
I20260808 01:55:02.461036 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d32821d9c17456394a662fcfd05f6c6" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:02.440940 20423 rpcz_store.cc:276] 0808 01:54:57.462750 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:54:57.477039 (+ 14289us) service_pool.cc:224] Handling call
0808 01:54:57.670656 (+193617us) spinlock_profiling.cc:243] Waited 29.4 ms on lock 0x7b540069e1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fd470 00007f5d4499c31a 00007f5d4a78c38c 00007f5d4a888bfe 00007f5d48d424a4 00007f5d48d42432 00007f5d48d42361 00007f5d48d422dc 00007f5d48d41592 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:02.368468 (+4697812us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:55:01.561174 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:55:01.658726 (+ 97552us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:55:01.658768 (+    42us) write_op.cc:435] Acquired schema lock
0808 01:55:01.658787 (+    19us) tablet.cc:662] Decoding operations
0808 01:55:01.659177 (+   390us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:55:01.659243 (+    66us) write_op.cc:641] Partition lock acquired for write op
0808 01:55:01.659272 (+    29us) tablet.cc:685] Acquiring locks for 1 operations
0808 01:55:01.659500 (+   228us) tablet.cc:701] Row locks acquired
0808 01:55:01.659514 (+    14us) write_op.cc:260] PREPARE: finished
0808 01:55:01.659698 (+   184us) write_op.cc:270] Start()
0808 01:55:01.659827 (+   129us) write_op.cc:276] Timestamp: P: 1786154101659642 usec, L: 0
0808 01:55:01.659846 (+    19us) op_driver.cc:348] REPLICATION: starting
0808 01:55:01.660546 (+   700us) log.cc:844] Serialized 540 byte log entry
0808 01:55:01.837321 (+176775us) op_driver.cc:464] REPLICATION: finished
0808 01:55:01.854411 (+ 17090us) write_op.cc:301] APPLY: starting
0808 01:55:01.854497 (+    86us) tablet.cc:1367] starting BulkCheckPresence
0808 01:55:01.854597 (+   100us) tablet.cc:1370] finished BulkCheckPresence
0808 01:55:01.883941 (+ 29344us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:55:01.897487 (+ 13546us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:55:01.897686 (+   199us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 01:55:01.897737 (+    51us) write_op.cc:312] APPLY: finished
0808 01:55:01.897952 (+   215us) log.cc:844] Serialized 22 byte log entry
0808 01:55:01.898155 (+   203us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:55:01.898248 (+    93us) write_op.cc:454] Released schema lock
0808 01:55:01.898291 (+    43us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":3255747,"reported_tablets":4,"spinlock_wait_cycles":67669888,"child_traces":[["op",{"apply.queue_time_us":15669,"apply.run_cpu_time_us":1965,"apply.run_wall_time_us":44263,"num_ops":1,"prepare.queue_time_us":392589,"prepare.run_cpu_time_us":3274,"prepare.run_wall_time_us":104992,"replication_time_us":165676,"thread_start_us":82068,"threads_started":1}]]}
I20260808 01:55:02.461503 20810 consensus_queue.cc:260] T ce51fcffe2ae4bcba55cd864c973bf0d P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:02.440886 21243 consensus_queue.cc:260] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:02.456104 20726 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:02.463812 21243 raft_consensus.cc:3060] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.464183 20649 leader_election.cc:304] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:02.464534 20810 ts_tablet_manager.cc:1434] T ce51fcffe2ae4bcba55cd864c973bf0d P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.057s	user 0.008s	sys 0.000s
I20260808 01:55:02.465209 21220 raft_consensus.cc:2804] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:02.465675 21220 raft_consensus.cc:493] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:02.465927 20810 tablet_bootstrap.cc:492] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:55:02.466365 20648 leader_election.cc:304] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:02.467087 20649 leader_election.cc:304] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:02.466933 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:02.468397 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:02.466061 21220 raft_consensus.cc:3060] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.469705 20758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:02.470504 20758 raft_consensus.cc:2468] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:55:02.471645 20815 consensus_queue.cc:260] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:02.473134 21240 raft_consensus.cc:2804] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:55:02.473465 20815 ts_tablet_manager.cc:1434] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.062s	user 0.006s	sys 0.000s
I20260808 01:55:02.473551 21240 raft_consensus.cc:493] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:02.474169 21240 raft_consensus.cc:3060] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:02.474848 21225 raft_consensus.cc:493] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:02.475338 21225 raft_consensus.cc:515] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:02.476194 20811 tablet_bootstrap.cc:492] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:55:02.477383 21225 leader_election.cc:290] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:02.477959 21225 consensus_queue.cc:1059] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:55:02.478849 20815 tablet_bootstrap.cc:492] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:55:02.479727 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:02.485731 20810 tablet_bootstrap.cc:654] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:02.490332 20813 tablet_bootstrap.cc:654] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:02.491199 21270 raft_consensus.cc:3060] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.491710 20649 leader_election.cc:304] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:02.493337 20649 leader_election.cc:304] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:02.494130 21238 raft_consensus.cc:2749] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:02.446866 21236 leader_election.cc:290] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:02.495918 21245 raft_consensus.cc:515] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:02.498181 21245 leader_election.cc:290] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:02.499306 20684 raft_consensus.cc:2468] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:02.504093 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:02.505050 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:02.505645 20914 raft_consensus.cc:515] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:02.508687 20811 tablet_bootstrap.cc:654] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
W20260808 01:55:02.509223 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 145754us
W20260808 01:55:02.511996 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 165940us
W20260808 01:55:02.494234 20649 leader_election.cc:343] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:02.519996 20815 tablet_bootstrap.cc:654] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:02.524467 20648 leader_election.cc:304] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:02.494304 21252 raft_consensus.cc:515] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:02.530611 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 57909.9us
I20260808 01:55:02.544028 20914 leader_election.cc:290] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:02.525394 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:02.536343 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:02.512501 20758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:02.537286 21266 consensus_queue.cc:1059] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:55:02.537791 21220 raft_consensus.cc:515] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:02.538283 21271 raft_consensus.cc:3060] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.523296 21237 raft_consensus.cc:2764] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260808 01:55:02.555614 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:02.556376 20754 raft_consensus.cc:3060] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.540489 21245 raft_consensus.cc:3055] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:02.556901 21244 raft_consensus.cc:2749] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:02.557041 21245 raft_consensus.cc:740] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:02.558229 21245 consensus_queue.cc:260] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:02.559001 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:02.559582 21245 raft_consensus.cc:3060] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:55:02.546370 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 180505us
I20260808 01:55:02.542840 21273 raft_consensus.cc:493] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:02.543826 21281 raft_consensus.cc:493] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:02.545904 21238 raft_consensus.cc:2749] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:02.574734 20648 leader_election.cc:304] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:02.542310 21222 consensus_queue.cc:1059] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260808 01:55:02.550998 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce51fcffe2ae4bcba55cd864c973bf0d" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:02.568095 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:02.579221 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:02.590601 20648 leader_election.cc:304] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:02.595000 20810 tablet_bootstrap.cc:492] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:55:02.596529 20810 ts_tablet_manager.cc:1403] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.131s	user 0.017s	sys 0.015s
I20260808 01:55:02.601639 20813 tablet_bootstrap.cc:492] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:55:02.603302 20813 ts_tablet_manager.cc:1403] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.206s	user 0.012s	sys 0.007s
I20260808 01:55:02.607479 20725 leader_election.cc:304] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:02.567642 20726 leader_election.cc:304] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:02.608349 20759 raft_consensus.cc:3060] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:02.610292 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:02.612061 21240 raft_consensus.cc:515] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:02.613128 21238 raft_consensus.cc:3060] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:02.614316 21240 leader_election.cc:290] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:02.608507 20523 raft_consensus.cc:2468] T ce51fcffe2ae4bcba55cd864c973bf0d P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:55:02.615033 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:02.584151 21220 leader_election.cc:290] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:55:02.610348 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 58819.8us
I20260808 01:55:02.616876 21252 leader_election.cc:290] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:02.577962 21281 raft_consensus.cc:515] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:02.589099 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:02.619174 20758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:02.624100 21249 raft_consensus.cc:2749] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:55:02.628764 20725 leader_election.cc:304] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:02.591387 21287 raft_consensus.cc:2804] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:02.575377 21273 raft_consensus.cc:515] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:02.643774 20813 raft_consensus.cc:359] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:02.660593 20813 raft_consensus.cc:385] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:02.661037 20813 raft_consensus.cc:740] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
W20260808 01:55:02.620546 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 71178.6us
W20260808 01:55:02.646410 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 62227.4us
I20260808 01:55:02.625113 21236 raft_consensus.cc:2804] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election won for term 2
W20260808 01:55:02.644224 20475 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:55:02.657907 20815 tablet_bootstrap.cc:492] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:55:02.663805 20815 ts_tablet_manager.cc:1403] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.189s	user 0.010s	sys 0.016s
I20260808 01:55:02.666146 20571 leader_election.cc:304] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:02.670431 20813 consensus_queue.cc:260] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:02.671644 20571 leader_election.cc:304] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e
W20260808 01:55:02.659803 20494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:02.661767 21287 raft_consensus.cc:493] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:02.675508 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:02.680244 20760 raft_consensus.cc:3060] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.672152 21252 consensus_queue.cc:1048] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260808 01:55:02.683092 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:02.687786 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:02.680629 20571 leader_election.cc:304] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:02.675828 20810 raft_consensus.cc:359] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:02.690200 20810 raft_consensus.cc:385] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:02.690634 21287 raft_consensus.cc:3060] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.690016 20914 raft_consensus.cc:2749] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:02.696568 21273 leader_election.cc:290] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:02.696806 21269 raft_consensus.cc:3055] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:02.697289 21269 raft_consensus.cc:740] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:02.675705 20753 raft_consensus.cc:2393] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 2: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:02.674029 20757 raft_consensus.cc:2393] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 2: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:02.690969 20810 raft_consensus.cc:740] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:55:02.683863 20815 raft_consensus.cc:359] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:02.690030 20811 tablet_bootstrap.cc:492] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:55:02.700227 20815 raft_consensus.cc:385] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:02.700591 20815 raft_consensus.cc:740] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:55:02.700704 20811 ts_tablet_manager.cc:1403] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.225s	user 0.024s	sys 0.017s
I20260808 01:55:02.673874 21225 raft_consensus.cc:2749] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:02.707156 20813 ts_tablet_manager.cc:1434] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.101s	user 0.008s	sys 0.002s
I20260808 01:55:02.689071 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:02.720341 20811 raft_consensus.cc:359] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:02.706102 21078 raft_consensus.cc:493] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:02.719859 21249 raft_consensus.cc:2749] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:55:02.727945 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 50279.4us
I20260808 01:55:02.712661 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:02.723294 20811 raft_consensus.cc:385] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:02.722388 21281 leader_election.cc:290] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:02.720220 21240 raft_consensus.cc:493] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:02.684696 21252 raft_consensus.cc:2804] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:02.785836 21252 raft_consensus.cc:493] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:02.786413 21252 raft_consensus.cc:3060] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.727011 20815 consensus_queue.cc:260] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:02.792711 20815 ts_tablet_manager.cc:1434] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.129s	user 0.006s	sys 0.001s
I20260808 01:55:02.794020 20815 tablet_bootstrap.cc:492] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:55:02.703557 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:02.728820 20811 raft_consensus.cc:740] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:55:02.795605 20811 consensus_queue.cc:260] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:02.678066 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:02.725016 20571 leader_election.cc:304] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:02.687825 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:02.732414 20810 consensus_queue.cc:260] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:02.736778 21078 raft_consensus.cc:515] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:02.739432 20492 leader_election.cc:304] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
W20260808 01:55:02.740422 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 63907.2us
I20260808 01:55:02.742554 21291 consensus_queue.cc:1048] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:55:02.746049 20759 raft_consensus.cc:2468] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:55:02.750397 20684 raft_consensus.cc:1240] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:02.799713 21222 raft_consensus.cc:493] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:02.802050 20681 raft_consensus.cc:2468] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:55:02.758662 20570 leader_election.cc:304] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:02.763198 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:02.805064 21078 leader_election.cc:290] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
W20260808 01:55:02.807111 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 67810.5us
I20260808 01:55:02.808616 20492 leader_election.cc:304] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:02.765132 21240 raft_consensus.cc:515] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:02.765141 21290 raft_consensus.cc:3055] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:02.811053 21290 raft_consensus.cc:740] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:02.812294 20810 ts_tablet_manager.cc:1434] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.210s	user 0.009s	sys 0.000s
I20260808 01:55:02.776014 21270 raft_consensus.cc:2749] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:02.812410 21290 consensus_queue.cc:260] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:02.813779 20810 tablet_bootstrap.cc:492] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:55:02.814934 21281 raft_consensus.cc:2749] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:02.815764 21281 raft_consensus.cc:2749] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:02.814488 21290 raft_consensus.cc:3060] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.706629 21269 consensus_queue.cc:260] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
W20260808 01:55:02.827546 20494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:02.829983 21240 leader_election.cc:290] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:02.832652 20813 tablet_bootstrap.cc:492] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:55:02.703922 21266 raft_consensus.cc:493] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:02.801463 20811 ts_tablet_manager.cc:1434] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.099s	user 0.008s	sys 0.000s
I20260808 01:55:02.915884 20815 tablet_bootstrap.cc:654] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:02.802515 21222 raft_consensus.cc:515] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:02.764660 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e71aab2ce9b45369b7eff65e1645f9b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:02.916687 20530 raft_consensus.cc:3060] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:02.916953 20811 tablet_bootstrap.cc:492] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
W20260808 01:55:02.841195 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 82790us
W20260808 01:55:02.897258 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 268611us
I20260808 01:55:02.920863 20725 leader_election.cc:304] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:02.922101 20725 leader_election.cc:304] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:02.751040 20760 raft_consensus.cc:2468] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:02.907964 21288 raft_consensus.cc:493] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:02.923174 21222 leader_election.cc:290] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:02.854274 20758 raft_consensus.cc:2468] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:02.923476 21288 raft_consensus.cc:515] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:02.925836 21288 leader_election.cc:290] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:02.926123 20725 leader_election.cc:304] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:02.928313 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1753378f13594e89aee6641e2cb9e6da" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:02.928786 21296 raft_consensus.cc:2749] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:02.930619 20725 leader_election.cc:304] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:02.931388 21296 raft_consensus.cc:2804] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:02.931887 21296 raft_consensus.cc:493] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:02.932333 21296 raft_consensus.cc:3060] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.934440 20530 raft_consensus.cc:2468] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
W20260808 01:55:02.899596 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 174667us
I20260808 01:55:02.900756 21287 raft_consensus.cc:515] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:02.943828 21296 raft_consensus.cc:515] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:02.898792 21225 raft_consensus.cc:2804] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:02.946056 21287 leader_election.cc:290] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:02.946290 21225 raft_consensus.cc:493] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:02.946784 21225 raft_consensus.cc:3060] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.876933 21236 raft_consensus.cc:697] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:02.912001 21238 raft_consensus.cc:2749] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:55:02.905357 20754 raft_consensus.cc:2468] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:02.892895 21245 raft_consensus.cc:2804] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260808 01:55:02.914199 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 95822.6us
W20260808 01:55:02.915287 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 97484.3us
I20260808 01:55:02.906056 21252 raft_consensus.cc:515] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:02.898264 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd11a6553c444d488651367f3f0143f" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:02.935264 21228 raft_consensus.cc:2749] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:55:02.935511 20648 leader_election.cc:343] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:02.950974 21245 raft_consensus.cc:493] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:02.896335 21269 raft_consensus.cc:3060] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.957378 21245 raft_consensus.cc:3060] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:02.957512 21252 leader_election.cc:290] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:02.936374 20725 leader_election.cc:304] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:02.959661 20725 leader_election.cc:304] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:02.960233 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:02.960909 20725 leader_election.cc:304] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:02.961719 21151 raft_consensus.cc:2749] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:02.962085 20725 leader_election.cc:304] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:02.951458 20492 leader_election.cc:304] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:02.963127 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:02.964560 21298 raft_consensus.cc:2749] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:02.965158 20492 leader_election.cc:304] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:02.965423 21298 raft_consensus.cc:2804] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:02.965920 21298 raft_consensus.cc:493] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:02.966111 20492 leader_election.cc:304] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:02.966329 21298 raft_consensus.cc:3060] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:02.938129 21266 raft_consensus.cc:515] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:02.911517 20810 tablet_bootstrap.cc:654] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:02.954921 21249 raft_consensus.cc:2749] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:02.962504 21151 raft_consensus.cc:2749] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:02.957521 20648 leader_election.cc:304] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:02.966116 21222 raft_consensus.cc:2749] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:02.963075 20813 tablet_bootstrap.cc:654] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:02.968261 21291 raft_consensus.cc:2804] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260808 01:55:02.950124 20571 leader_election.cc:343] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:02.954797 20523 raft_consensus.cc:2468] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:55:02.937656 20811 tablet_bootstrap.cc:654] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:02.975752 20725 leader_election.cc:304] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:02.989426 21249 raft_consensus.cc:2804] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:02.989914 21249 raft_consensus.cc:493] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:02.990332 21249 raft_consensus.cc:3060] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.991521 21236 consensus_queue.cc:237] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:02.992918 21238 raft_consensus.cc:2804] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:55:02.993462 21238 raft_consensus.cc:697] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:02.994798 21291 raft_consensus.cc:493] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:02.995316 21291 raft_consensus.cc:3060] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:02.971702 21266 leader_election.cc:290] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:02.999316 20571 leader_election.cc:304] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:02.999668 21288 raft_consensus.cc:493] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:02.999802 20648 leader_election.cc:304] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:03.000699 21240 raft_consensus.cc:2804] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:55:03.001133 21240 raft_consensus.cc:493] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:03.001528 21240 raft_consensus.cc:3060] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:03.001994 21252 raft_consensus.cc:2749] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:55:03.000381 21288 raft_consensus.cc:515] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:03.003499 21266 raft_consensus.cc:2749] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:02.994400 21238 consensus_queue.cc:237] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:02.978529 21228 raft_consensus.cc:493] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:03.017011 21228 raft_consensus.cc:515] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:02.946056 21296 leader_election.cc:290] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:02.954250 20493 leader_election.cc:304] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:02.990399 20686 raft_consensus.cc:2468] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:55:03.022461 21222 raft_consensus.cc:2749] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:55:03.029963 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 75783.2us
I20260808 01:55:03.031124 20493 leader_election.cc:304] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:03.031792 21298 raft_consensus.cc:515] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:03.032593 20493 leader_election.cc:304] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:03.036526 21288 leader_election.cc:290] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:03.033228 21222 raft_consensus.cc:2749] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:55:03.038022 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 94039.5us
I20260808 01:55:03.037825 21222 raft_consensus.cc:2749] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.038964 20571 leader_election.cc:304] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:03.039659 20811 tablet_bootstrap.cc:492] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:55:03.039780 21252 raft_consensus.cc:2749] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.040303 20811 ts_tablet_manager.cc:1403] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.124s	user 0.017s	sys 0.004s
W20260808 01:55:03.040879 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 74842.1us
I20260808 01:55:03.044881 21228 leader_election.cc:290] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:02.956713 21225 raft_consensus.cc:515] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:03.047789 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd11a6553c444d488651367f3f0143f" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:55:03.049147 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 73494.1us
I20260808 01:55:03.054996 20649 leader_election.cc:304] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:03.057224 20649 leader_election.cc:304] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:03.059113 20493 leader_election.cc:304] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:03.061538 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:03.062506 20760 raft_consensus.cc:2468] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:55:03.036244 21245 raft_consensus.cc:515] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:03.064108 20810 tablet_bootstrap.cc:492] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:55:03.065157 20810 ts_tablet_manager.cc:1403] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.252s	user 0.021s	sys 0.000s
I20260808 01:55:03.049211 20571 leader_election.cc:304] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e
W20260808 01:55:03.064193 20726 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:03.066617 20649 leader_election.cc:304] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:03.068549 20493 leader_election.cc:304] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:03.070420 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd11a6553c444d488651367f3f0143f" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:03.071159 21269 raft_consensus.cc:2749] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.071257 21291 raft_consensus.cc:515] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:03.070664 20813 tablet_bootstrap.cc:492] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:55:03.073282 20813 ts_tablet_manager.cc:1403] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.273s	user 0.030s	sys 0.014s
I20260808 01:55:03.073364 21273 raft_consensus.cc:2749] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.074033 21291 leader_election.cc:290] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:03.074301 21237 raft_consensus.cc:2749] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.071295 20759 raft_consensus.cc:2468] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:55:03.064096 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:03.075434 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:03.076458 20755 raft_consensus.cc:2393] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 2: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:03.080742 20810 raft_consensus.cc:359] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:03.083940 20649 leader_election.cc:304] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:03.081656 20810 raft_consensus.cc:385] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:03.083779 21240 raft_consensus.cc:515] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:03.084765 20810 raft_consensus.cc:740] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:55:03.084975 21237 raft_consensus.cc:2804] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:03.085681 21237 raft_consensus.cc:697] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 2 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:03.086251 20810 consensus_queue.cc:260] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:03.087253 21240 leader_election.cc:290] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:03.034377 21298 leader_election.cc:290] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:03.088655 20649 leader_election.cc:304] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:03.086771 21237 consensus_queue.cc:237] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:03.084182 21287 raft_consensus.cc:2749] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.077108 20813 raft_consensus.cc:359] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:03.091358 20813 raft_consensus.cc:385] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:03.091044 20571 leader_election.cc:304] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:03.088868 20686 raft_consensus.cc:2393] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 1: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:03.091701 20813 raft_consensus.cc:740] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:55:03.093196 20811 raft_consensus.cc:359] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:03.094134 20811 raft_consensus.cc:385] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:03.094560 20811 raft_consensus.cc:740] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:55:03.094861 20813 consensus_queue.cc:260] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:03.093242 21252 raft_consensus.cc:2804] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:03.096481 21252 raft_consensus.cc:493] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:03.097002 21252 raft_consensus.cc:3060] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:03.097757 20493 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:03.098620 20571 leader_election.cc:304] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:03.097446 21225 leader_election.cc:290] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:03.102531 20811 consensus_queue.cc:260] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:03.103983 20914 raft_consensus.cc:2804] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:03.104413 20914 raft_consensus.cc:493] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:03.104758 20914 raft_consensus.cc:3060] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:03.104946 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce51fcffe2ae4bcba55cd864c973bf0d" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:03.105712 20523 raft_consensus.cc:3060] T ce51fcffe2ae4bcba55cd864c973bf0d P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:03.099742 21243 raft_consensus.cc:2804] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:55:03.111765 21243 raft_consensus.cc:493] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:03.112218 21243 raft_consensus.cc:3060] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:03.115628 20571 leader_election.cc:304] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:03.116443 20810 ts_tablet_manager.cc:1434] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.051s	user 0.010s	sys 0.000s
I20260808 01:55:03.093016 21270 raft_consensus.cc:493] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:03.120704 20649 leader_election.cc:304] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:03.121097 20914 raft_consensus.cc:515] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:03.099768 21078 raft_consensus.cc:2749] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.125686 21220 raft_consensus.cc:2804] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:55:03.126197 21220 raft_consensus.cc:493] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:03.126705 21220 raft_consensus.cc:3060] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:03.115387 21228 raft_consensus.cc:493] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:03.127573 21228 raft_consensus.cc:515] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:03.096264 21288 raft_consensus.cc:2804] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:03.131155 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd11a6553c444d488651367f3f0143f" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:03.132148 20815 tablet_bootstrap.cc:492] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:55:03.136904 20815 ts_tablet_manager.cc:1403] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.343s	user 0.015s	sys 0.009s
I20260808 01:55:03.123459 20423 catalog_manager.cc:5697] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 reported cstate change: term changed from 1 to 2, leader changed from 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194) to df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196). New cstate: current_term: 2 leader_uuid: "df82e7f402364d369ed1ec3c5a8772b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: HEALTHY } } }
I20260808 01:55:03.141426 20813 ts_tablet_manager.cc:1434] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.048s	user 0.009s	sys 0.000s
I20260808 01:55:03.142843 20753 raft_consensus.cc:3060] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:03.132608 21270 raft_consensus.cc:515] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:03.147394 20523 raft_consensus.cc:2468] T ce51fcffe2ae4bcba55cd864c973bf0d P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:03.147683 21225 raft_consensus.cc:2749] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.148958 20570 leader_election.cc:304] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:03.149866 21225 raft_consensus.cc:2804] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:55:03.150386 21225 raft_consensus.cc:697] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:03.151762 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd11a6553c444d488651367f3f0143f" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:03.151903 21225 consensus_queue.cc:237] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:03.152595 21270 leader_election.cc:290] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:03.153970 20813 tablet_bootstrap.cc:492] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:55:03.150766 20528 raft_consensus.cc:1240] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:03.154870 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:03.155769 20527 raft_consensus.cc:2468] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:03.157714 20571 leader_election.cc:304] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:03.118259 21222 raft_consensus.cc:2804] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 01:55:03.156308 20786 debug-util.cc:398] Leaking SignalData structure 0x7b08001ffa20 after lost signal to thread 7834
I20260808 01:55:03.089778 21302 consensus_queue.cc:1059] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20260808 01:55:03.168378 20753 raft_consensus.cc:2468] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:03.100942 21245 leader_election.cc:290] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:03.152623 20530 raft_consensus.cc:3060] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:03.149979 21220 raft_consensus.cc:515] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:03.165728 20786 debug-util.cc:398] Leaking SignalData structure 0x7b0800154860 after lost signal to thread 20461
I20260808 01:55:03.166379 21240 consensus_queue.cc:1059] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:55:03.166996 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:03.167368 20648 leader_election.cc:304] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:03.171778 21298 raft_consensus.cc:3055] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:03.173955 20815 raft_consensus.cc:359] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:03.163400 21222 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:03.200351 20815 raft_consensus.cc:385] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:03.200594 21222 raft_consensus.cc:3060] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:03.200727 20815 raft_consensus.cc:740] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:55:03.201586 20815 consensus_queue.cc:260] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:03.158236 20493 leader_election.cc:304] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:03.202633 21220 leader_election.cc:290] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:55:03.202702 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 44564.3us
W20260808 01:55:03.205269 20786 debug-util.cc:398] Leaking SignalData structure 0x7b08000d9a20 after lost signal to thread 20555
W20260808 01:55:03.205652 20786 debug-util.cc:398] Leaking SignalData structure 0x7b080017abe0 after lost signal to thread 20556
I20260808 01:55:03.203083 21298 raft_consensus.cc:740] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:03.207681 21298 consensus_queue.cc:260] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:03.208293 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:03.208981 21298 raft_consensus.cc:3060] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:03.209952 21222 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:03.203785 21235 raft_consensus.cc:2804] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:03.211095 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:03.211335 21235 raft_consensus.cc:493] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:03.211912 21235 raft_consensus.cc:3060] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:03.212100 20520 raft_consensus.cc:2410] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260808 01:55:03.215018 21222 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:03.181823 20813 tablet_bootstrap.cc:654] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:03.187029 20811 ts_tablet_manager.cc:1434] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.146s	user 0.010s	sys 0.000s
I20260808 01:55:03.204514 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:03.216176 20760 raft_consensus.cc:3060] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:03.216559 20811 tablet_bootstrap.cc:492] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:55:03.203024 21283 raft_consensus.cc:2749] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.207708 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:03.187294 20810 tablet_bootstrap.cc:492] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
W20260808 01:55:03.219045 20786 debug-util.cc:398] Leaking SignalData structure 0x7b08003ea940 after lost signal to thread 20634
I20260808 01:55:03.203615 21240 raft_consensus.cc:3055] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:03.223685 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:03.223870 21240 raft_consensus.cc:740] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:03.162181 21288 raft_consensus.cc:697] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 2 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:03.189500 21228 leader_election.cc:290] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:03.197616 20571 leader_election.cc:304] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:03.214893 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:03.225123 21240 consensus_queue.cc:260] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:03.192071 20914 leader_election.cc:290] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:03.223024 20786 debug-util.cc:398] Leaking SignalData structure 0x7b080038c000 after lost signal to thread 20635
I20260808 01:55:03.238909 21283 raft_consensus.cc:2749] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.239223 20649 leader_election.cc:304] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:03.186102 21243 raft_consensus.cc:515] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:03.242106 20815 ts_tablet_manager.cc:1434] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.105s	user 0.004s	sys 0.002s
I20260808 01:55:03.226414 21249 raft_consensus.cc:515] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:03.250967 20815 tablet_bootstrap.cc:492] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:55:03.252610 21249 leader_election.cc:290] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:03.239737 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:03.256646 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc85af5215440be884412c1d240c7af" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:03.257298 20530 raft_consensus.cc:2468] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:03.257445 20525 raft_consensus.cc:2468] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 0.
I20260808 01:55:03.239974 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:03.240576 21288 consensus_queue.cc:237] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:03.261340 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd11a6553c444d488651367f3f0143f" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:03.262275 20754 raft_consensus.cc:2393] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 1: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:03.243050 21287 raft_consensus.cc:2804] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:55:03.264362 21287 raft_consensus.cc:697] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:03.238739 20571 leader_election.cc:304] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:03.265372 20758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:03.265322 21287 consensus_queue.cc:237] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:03.248442 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:03.267393 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:03.204094 20759 raft_consensus.cc:2393] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 2: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:03.263700 20756 raft_consensus.cc:1240] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:03.248574 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd11a6553c444d488651367f3f0143f" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:03.203266 21269 raft_consensus.cc:2804] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:55:03.294636 20529 raft_consensus.cc:2393] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 1: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:03.236693 20760 raft_consensus.cc:2468] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:03.239393 21240 raft_consensus.cc:3060] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:03.276353 20725 leader_election.cc:304] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:03.248925 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce51fcffe2ae4bcba55cd864c973bf0d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:03.266291 20571 leader_election.cc:304] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
W20260808 01:55:03.240702 20786 debug-util.cc:398] Leaking SignalData structure 0x7b0800150380 after lost signal to thread 20712
I20260808 01:55:03.297919 20520 raft_consensus.cc:2393] T ce51fcffe2ae4bcba55cd864c973bf0d P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 1: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:03.297523 20649 leader_election.cc:304] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:03.248126 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:03.299679 21220 raft_consensus.cc:2804] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:55:03.300094 20571 leader_election.cc:304] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:03.300305 21220 raft_consensus.cc:697] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
W20260808 01:55:03.300405 20494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:03.300701 20725 leader_election.cc:304] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:03.301620 20571 leader_election.cc:304] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
W20260808 01:55:03.299253 20786 debug-util.cc:398] Leaking SignalData structure 0x7b080033a140 after lost signal to thread 20789
I20260808 01:55:03.302175 21245 raft_consensus.cc:2749] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.303449 21269 raft_consensus.cc:697] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:03.304468 21269 consensus_queue.cc:237] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:03.292834 20754 raft_consensus.cc:1240] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:03.310899 21294 raft_consensus.cc:2749] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.311007 21245 consensus_queue.cc:1059] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:03.311910 21294 raft_consensus.cc:2804] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:55:03.312220 21245 raft_consensus.cc:3055] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:03.312780 21245 raft_consensus.cc:740] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:03.313776 21245 consensus_queue.cc:260] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:03.301213 21220 consensus_queue.cc:237] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:03.315624 20813 tablet_bootstrap.cc:492] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:55:03.283339 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc85af5215440be884412c1d240c7af" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:03.316232 20813 ts_tablet_manager.cc:1403] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.163s	user 0.015s	sys 0.000s
I20260808 01:55:03.316303 20757 raft_consensus.cc:2468] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:55:03.317941 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.318189 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.318859 20608 raft_consensus.cc:2393] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 2: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:03.312290 21294 raft_consensus.cc:493] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:03.319356 21294 raft_consensus.cc:3060] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:03.315464 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.296950 20811 tablet_bootstrap.cc:654] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:03.244601 21243 leader_election.cc:290] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:03.318560 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.322479 20606 raft_consensus.cc:2468] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:03.288432 21222 raft_consensus.cc:493] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:03.260212 20753 raft_consensus.cc:1240] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:03.282567 21283 raft_consensus.cc:2749] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.325663 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:55:03.331367 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 127336us
I20260808 01:55:03.330407 20493 leader_election.cc:304] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:03.282459 21252 raft_consensus.cc:515] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:03.335484 20725 leader_election.cc:304] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:03.336935 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd11a6553c444d488651367f3f0143f" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:03.336758 20813 raft_consensus.cc:359] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:03.339020 20608 raft_consensus.cc:2468] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:03.339236 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.339084 20684 raft_consensus.cc:1240] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:03.346412 20725 leader_election.cc:304] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:03.348189 20813 raft_consensus.cc:385] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:03.352828 20813 raft_consensus.cc:740] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:55:03.339969 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.321040 21228 raft_consensus.cc:2749] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.350922 20493 leader_election.cc:304] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:03.356709 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:03.357450 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:03.358451 20605 raft_consensus.cc:2468] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:03.344120 21236 raft_consensus.cc:2749] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.356936 20678 raft_consensus.cc:1240] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:03.339435 20522 raft_consensus.cc:2468] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:03.360363 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.361874 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.356215 20725 leader_election.cc:304] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:03.331477 21245 raft_consensus.cc:3060] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:03.375886 20810 tablet_bootstrap.cc:654] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:03.377127 20815 tablet_bootstrap.cc:654] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:03.368630 20813 consensus_queue.cc:260] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:03.370065 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:03.370884 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.351298 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:03.333315 20524 raft_consensus.cc:2468] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:03.344352 21266 raft_consensus.cc:2804] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:03.382400 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.354619 21298 raft_consensus.cc:2804] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:55:03.392282 21298 raft_consensus.cc:493] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:03.392721 21298 raft_consensus.cc:3060] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:03.360704 21302 raft_consensus.cc:2749] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.396739 20682 raft_consensus.cc:1275] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260808 01:55:03.402757 21302 consensus_queue.cc:1048] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:55:03.384402 21290 consensus_queue.cc:1059] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260808 01:55:03.406631 21290 raft_consensus.cc:3055] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:03.407051 21290 raft_consensus.cc:740] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:03.384419 20493 leader_election.cc:304] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:03.374647 20527 raft_consensus.cc:2468] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:03.416747 20602 raft_consensus.cc:1240] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:03.390421 21294 raft_consensus.cc:515] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:03.391870 20606 raft_consensus.cc:2393] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 2: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:03.368455 21236 raft_consensus.cc:2749] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:55:03.331097 21222 raft_consensus.cc:515] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:03.394308 20609 raft_consensus.cc:2393] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 2: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:03.380436 21289 raft_consensus.cc:2749] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:55:03.397259 20605 raft_consensus.cc:2393] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 1: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:03.394018 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.396634 21266 raft_consensus.cc:697] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [term 2 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:03.437507 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce51fcffe2ae4bcba55cd864c973bf0d" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:03.438354 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:03.437520 20603 raft_consensus.cc:2393] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 2: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:03.438722 21266 consensus_queue.cc:237] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:03.409029 20813 ts_tablet_manager.cc:1434] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.092s	user 0.010s	sys 0.000s
I20260808 01:55:03.422616 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:03.423008 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:03.425711 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.425845 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1753378f13594e89aee6641e2cb9e6da" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.426184 21294 leader_election.cc:290] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:03.427055 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e71aab2ce9b45369b7eff65e1645f9b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:03.455847 20606 raft_consensus.cc:2468] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:55:03.456358 20604 raft_consensus.cc:3060] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:03.456507 21222 leader_election.cc:290] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:03.456985 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.429304 21235 raft_consensus.cc:515] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:03.459488 21252 leader_election.cc:290] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:03.460680 21235 leader_election.cc:290] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:03.376888 20426 catalog_manager.cc:5697] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 598db12c38d442c6988b4c4f991874fb (127.7.165.195). New cstate: current_term: 1 leader_uuid: "598db12c38d442c6988b4c4f991874fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: HEALTHY } } }
I20260808 01:55:03.461853 20601 raft_consensus.cc:2393] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 1: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:03.462890 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.463554 20601 raft_consensus.cc:2468] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:03.464471 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:03.442448 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:03.443074 20813 tablet_bootstrap.cc:492] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc: Bootstrap starting.
I20260808 01:55:03.443053 20755 raft_consensus.cc:1240] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:03.465655 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.439327 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:03.444089 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.467147 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:03.467701 20609 raft_consensus.cc:3060] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:03.458325 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce51fcffe2ae4bcba55cd864c973bf0d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.471154 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:03.435097 21269 raft_consensus.cc:2804] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:55:03.452400 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:03.452664 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:03.453619 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.454108 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.455224 20602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc85af5215440be884412c1d240c7af" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.476331 20813 tablet_bootstrap.cc:654] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:03.476193 21078 consensus_queue.cc:1059] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:55:03.482700 20810 tablet_bootstrap.cc:492] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:55:03.483031 21078 raft_consensus.cc:3055] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:03.483299 20810 ts_tablet_manager.cc:1403] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.296s	user 0.018s	sys 0.009s
I20260808 01:55:03.483491 21078 raft_consensus.cc:740] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:03.475004 20607 raft_consensus.cc:3060] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:03.471727 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:03.480718 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:03.438421 20753 raft_consensus.cc:2468] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:55:03.488379 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:03.486012 21289 consensus_queue.cc:1059] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260808 01:55:03.489567 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:03.494164 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce51fcffe2ae4bcba55cd864c973bf0d" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:03.487550 21290 consensus_queue.cc:260] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:03.494714 21289 raft_consensus.cc:3055] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:03.494885 20761 raft_consensus.cc:3060] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:03.495173 21289 raft_consensus.cc:740] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:03.495673 21290 raft_consensus.cc:3060] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:03.496155 21289 consensus_queue.cc:260] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:03.497677 21289 raft_consensus.cc:3060] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:03.498979 20600 raft_consensus.cc:2393] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 2: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:03.484536 21078 consensus_queue.cc:260] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:03.500058 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:03.500341 21288 consensus_queue.cc:1048] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:55:03.500723 20600 raft_consensus.cc:3060] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:03.500756 21252 raft_consensus.cc:493] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:03.501590 21252 raft_consensus.cc:515] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:03.502719 20603 raft_consensus.cc:2393] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 2: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:03.479116 21269 raft_consensus.cc:493] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:03.504062 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:03.489848 20602 raft_consensus.cc:2468] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 0.
I20260808 01:55:03.512679 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:03.513618 20524 raft_consensus.cc:2393] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 2: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:03.515547 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:03.522320 20686 raft_consensus.cc:3060] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:03.485574 21298 raft_consensus.cc:515] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:03.511122 21269 raft_consensus.cc:3060] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:03.510957 21078 raft_consensus.cc:3060] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:55:03.536592 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 171732us
I20260808 01:55:03.565613 20810 raft_consensus.cc:359] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:03.567709 20810 raft_consensus.cc:385] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:03.568027 20810 raft_consensus.cc:740] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:55:03.545595 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:03.568652 21300 consensus_queue.cc:1059] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260808 01:55:03.569027 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:03.570894 21252 leader_election.cc:290] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:03.532112 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1753378f13594e89aee6641e2cb9e6da" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:03.575268 20681 raft_consensus.cc:2468] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:55:03.576495 20811 tablet_bootstrap.cc:492] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:55:03.577129 20811 ts_tablet_manager.cc:1403] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.361s	user 0.025s	sys 0.020s
I20260808 01:55:03.581748 20810 consensus_queue.cc:260] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:03.585196 20761 raft_consensus.cc:2468] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:03.512672 20601 raft_consensus.cc:1240] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:03.530458 20606 raft_consensus.cc:2393] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 2: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:03.599789 20815 tablet_bootstrap.cc:492] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:55:03.599665 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:03.600410 20815 ts_tablet_manager.cc:1403] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.350s	user 0.019s	sys 0.003s
I20260808 01:55:03.600968 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:03.601855 20755 raft_consensus.cc:2393] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 1: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:03.585098 20609 raft_consensus.cc:2468] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:03.540707 20648 leader_election.cc:304] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:03.594796 21295 raft_consensus.cc:3055] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:03.596196 21298 leader_election.cc:290] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:03.596727 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
W20260808 01:55:03.596982 20650 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:03.609217 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63fb8c2f494a462686500ce10025a39f" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.612020 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc85af5215440be884412c1d240c7af" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:03.579370 20603 raft_consensus.cc:2393] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 1: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
W20260808 01:55:03.617455 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 76897.1us
I20260808 01:55:03.598390 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc85af5215440be884412c1d240c7af" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.620843 21240 raft_consensus.cc:493] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:03.695748 21240 raft_consensus.cc:515] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:03.697968 21240 leader_election.cc:290] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:03.624559 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.625351 20725 leader_election.cc:304] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:55:03.615913 20602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.691705 20677 raft_consensus.cc:2468] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:03.692308 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:03.711313 20811 raft_consensus.cc:359] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:03.712284 20811 raft_consensus.cc:385] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:03.686446 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:03.694959 20606 raft_consensus.cc:2468] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:55:03.694993 21294 raft_consensus.cc:493] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
W20260808 01:55:03.699951 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 74680.5us
I20260808 01:55:03.700132 20648 leader_election.cc:304] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:03.623036 20607 raft_consensus.cc:2468] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:03.706610 20810 ts_tablet_manager.cc:1434] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.223s	user 0.010s	sys 0.000s
I20260808 01:55:03.738129 21294 raft_consensus.cc:515] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:03.738464 20725 leader_election.cc:304] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:55:03.739707 20810 tablet_bootstrap.cc:492] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:55:03.740548 21294 leader_election.cc:290] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:03.740670 20648 leader_election.cc:304] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:03.707126 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:03.708154 21249 raft_consensus.cc:2804] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:03.693528 20600 raft_consensus.cc:2468] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:03.747751 20601 raft_consensus.cc:1240] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:03.747864 21249 raft_consensus.cc:697] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:03.748716 20600 raft_consensus.cc:1240] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:03.749300 20601 raft_consensus.cc:1240] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260808 01:55:03.748955 21249 consensus_queue.cc:237] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:03.750841 20601 raft_consensus.cc:1240] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:03.712587 20811 raft_consensus.cc:740] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:55:03.622870 21295 raft_consensus.cc:740] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:03.751688 20600 raft_consensus.cc:1240] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:03.752172 21295 consensus_queue.cc:260] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:03.752633 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1753378f13594e89aee6641e2cb9e6da" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.753109 20600 raft_consensus.cc:1240] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:03.753358 20607 raft_consensus.cc:2468] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:55:03.754588 20605 raft_consensus.cc:2468] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:03.684918 21283 raft_consensus.cc:493] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:03.755684 21283 raft_consensus.cc:515] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:03.757652 21287 raft_consensus.cc:3060] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:03.712785 20602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:03.730423 20759 raft_consensus.cc:2393] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 2: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:03.733690 20604 raft_consensus.cc:2468] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:03.734380 20815 raft_consensus.cc:359] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:03.736829 20571 leader_election.cc:304] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:03.771466 20815 raft_consensus.cc:385] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:03.771862 20815 raft_consensus.cc:740] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:55:03.737051 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:03.628058 21235 raft_consensus.cc:3060] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:03.743876 20679 raft_consensus.cc:1240] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260808 01:55:03.745359 21240 raft_consensus.cc:2804] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:03.745988 21225 consensus_queue.cc:1059] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:55:03.747136 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:03.757946 21283 leader_election.cc:290] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:03.759475 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:03.751902 20811 consensus_queue.cc:260] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:03.705773 20686 raft_consensus.cc:2468] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:03.768119 20609 raft_consensus.cc:2468] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:03.772590 20571 leader_election.cc:304] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:03.776134 20815 consensus_queue.cc:260] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:03.773527 21078 raft_consensus.cc:493] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:03.776351 21228 raft_consensus.cc:2804] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:03.808791 21228 raft_consensus.cc:697] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:03.809784 21228 consensus_queue.cc:237] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:03.776103 21243 raft_consensus.cc:2749] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.806253 20914 raft_consensus.cc:2749] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.806775 20571 leader_election.cc:304] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:03.807735 21078 raft_consensus.cc:515] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:03.781934 21240 raft_consensus.cc:493] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
W20260808 01:55:03.820767 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 80187.3us
I20260808 01:55:03.818372 20571 leader_election.cc:304] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:03.823108 20493 leader_election.cc:304] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:03.825549 20811 ts_tablet_manager.cc:1434] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.248s	user 0.009s	sys 0.003s
I20260808 01:55:03.827684 20914 raft_consensus.cc:2749] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:55:03.834833 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 60601.8us
I20260808 01:55:03.833299 20571 leader_election.cc:304] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:03.780714 21269 raft_consensus.cc:515] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:03.837009 20571 leader_election.cc:304] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:03.837666 20493 leader_election.cc:304] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:03.821942 21240 raft_consensus.cc:3060] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:03.820842 21245 raft_consensus.cc:2749] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.840001 21334 raft_consensus.cc:2804] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:55:03.832221 20757 raft_consensus.cc:1240] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:03.845017 20811 tablet_bootstrap.cc:492] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:55:03.842231 20571 leader_election.cc:304] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:03.843765 20815 ts_tablet_manager.cc:1434] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.243s	user 0.011s	sys 0.008s
I20260808 01:55:03.845583 20813 tablet_bootstrap.cc:492] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc: No bootstrap required, opened a new log
I20260808 01:55:03.846179 20813 ts_tablet_manager.cc:1403] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc: Time spent bootstrapping tablet: real 0.403s	user 0.025s	sys 0.004s
I20260808 01:55:03.846414 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac71e66cb4042eb8fe8aafd139af68a" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:03.840683 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:03.830307 21237 consensus_queue.cc:1059] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260808 01:55:03.846621 20815 tablet_bootstrap.cc:492] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
I20260808 01:55:03.853680 21290 raft_consensus.cc:2804] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:03.854547 21290 raft_consensus.cc:697] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [term 2 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:03.855113 20528 raft_consensus.cc:2376] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2. Re-sending same reply.
I20260808 01:55:03.855284 21237 raft_consensus.cc:3055] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:03.855703 21237 raft_consensus.cc:740] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:03.856664 21237 consensus_queue.cc:260] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:03.858203 21237 raft_consensus.cc:3060] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:03.863261 20810 tablet_bootstrap.cc:654] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
W20260808 01:55:03.863649 20570 leader_election.cc:343] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:03.864562 21245 raft_consensus.cc:2749] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.866233 20524 raft_consensus.cc:2393] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:03.866739 21266 consensus_queue.cc:1059] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:55:03.847636 21283 raft_consensus.cc:2749] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:03.867926 21299 raft_consensus.cc:3055] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:55:03.855718 21290 consensus_queue.cc:237] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:03.868479 21299 raft_consensus.cc:740] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:03.849514 20813 raft_consensus.cc:359] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:03.882390 20813 raft_consensus.cc:385] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:03.882889 20813 raft_consensus.cc:740] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Initialized, Role: FOLLOWER
I20260808 01:55:03.844836 21334 raft_consensus.cc:493] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:03.897446 20813 consensus_queue.cc:260] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:03.884734 20571 leader_election.cc:304] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
W20260808 01:55:03.893854 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 152262us
I20260808 01:55:03.844813 21243 raft_consensus.cc:2749] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.872097 21269 leader_election.cc:290] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
W20260808 01:55:03.916321 21332 log.cc:927] Time spent T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260808 01:55:03.903589 21334 raft_consensus.cc:3060] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:03.924111 20649 leader_election.cc:304] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:03.922575 20530 raft_consensus.cc:3060] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:03.937736 20608 raft_consensus.cc:2468] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:03.939044 20813 ts_tablet_manager.cc:1434] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc: Time spent starting tablet: real 0.093s	user 0.009s	sys 0.000s
I20260808 01:55:03.943703 21295 raft_consensus.cc:3060] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:03.944608 20754 raft_consensus.cc:1240] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:03.896334 21300 raft_consensus.cc:493] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:03.947362 21300 raft_consensus.cc:515] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:03.949746 21300 leader_election.cc:290] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:03.924149 21243 raft_consensus.cc:2749] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.869341 21299 consensus_queue.cc:260] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:03.950981 21299 raft_consensus.cc:3060] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:03.951052 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:03.953109 20527 raft_consensus.cc:2468] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:55:03.929791 21078 leader_election.cc:290] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:03.929940 21235 raft_consensus.cc:2749] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:03.930895 21273 raft_consensus.cc:493] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:03.955575 21273 raft_consensus.cc:515] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:03.956528 20682 raft_consensus.cc:1240] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:03.957921 21273 leader_election.cc:290] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
W20260808 01:55:03.932037 20649 leader_election.cc:343] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:03.966183 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d8bf19c324463fb9903b9981068e9d" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:03.932540 21252 raft_consensus.cc:3055] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:03.948382 21240 raft_consensus.cc:515] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:03.967964 21252 raft_consensus.cc:740] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:03.969019 21252 consensus_queue.cc:260] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:03.969832 21240 leader_election.cc:290] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:03.970407 21252 raft_consensus.cc:3060] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:03.934438 20815 tablet_bootstrap.cc:654] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:03.932598 21220 raft_consensus.cc:2749] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260808 01:55:03.976655 20649 leader_election.cc:343] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:03.981266 20725 leader_election.cc:304] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:55:03.985423 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc85af5215440be884412c1d240c7af" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
W20260808 01:55:03.983716 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 55829.7us
I20260808 01:55:03.986447 20761 raft_consensus.cc:2393] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 1: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
W20260808 01:55:03.986720 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 60697.2us
I20260808 01:55:03.999481 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:55:03.994839 20571 leader_election.cc:343] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:04.002058 20761 raft_consensus.cc:2468] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:04.005139 20493 leader_election.cc:304] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:04.006770 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:04.008826 20530 raft_consensus.cc:1275] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Refusing update from remote peer 598db12c38d442c6988b4c4f991874fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:04.009476 20679 raft_consensus.cc:1240] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:04.010661 20759 raft_consensus.cc:1240] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
W20260808 01:55:04.013672 20493 leader_election.cc:343] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:04.019619 21300 raft_consensus.cc:2804] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:04.020190 21300 raft_consensus.cc:493] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:04.020637 21300 raft_consensus.cc:3060] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:04.021632 21334 raft_consensus.cc:515] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:04.020202 21287 raft_consensus.cc:2749] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:04.024673 20811 tablet_bootstrap.cc:654] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:03.977977 21243 consensus_queue.cc:1048] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
I20260808 01:55:04.024848 20686 raft_consensus.cc:1240] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:03.988685 21236 raft_consensus.cc:2804] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:04.029552 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d8bf19c324463fb9903b9981068e9d" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:04.029757 20754 raft_consensus.cc:1275] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Refusing update from remote peer 598db12c38d442c6988b4c4f991874fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:04.030197 20810 tablet_bootstrap.cc:492] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:55:04.032017 20810 ts_tablet_manager.cc:1403] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.293s	user 0.014s	sys 0.005s
I20260808 01:55:04.034930 21334 leader_election.cc:290] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:04.035250 20757 raft_consensus.cc:1275] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Refusing update from remote peer 598db12c38d442c6988b4c4f991874fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:04.036829 20754 raft_consensus.cc:1275] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:03.988580 21238 consensus_queue.cc:1048] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.043s
I20260808 01:55:04.029628 21236 raft_consensus.cc:697] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:04.046368 20648 leader_election.cc:304] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:04.049582 20492 leader_election.cc:304] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:04.051656 20725 leader_election.cc:304] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:04.053421 20725 leader_election.cc:304] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:04.071995 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:04.074658 20492 leader_election.cc:304] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:04.072834 20725 leader_election.cc:304] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:04.077780 20492 leader_election.cc:304] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:04.081777 20648 leader_election.cc:304] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:04.063479 20424 catalog_manager.cc:5697] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194). New cstate: current_term: 1 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } }
I20260808 01:55:04.063030 21249 raft_consensus.cc:2749] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:55:04.063222 21302 raft_consensus.cc:3060] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:04.094278 20648 leader_election.cc:304] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e
W20260808 01:55:04.092548 20571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:04.073249 21273 raft_consensus.cc:2804] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:55:04.072190 21291 raft_consensus.cc:2749] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:04.094995 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac71e66cb4042eb8fe8aafd139af68a" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:04.083683 21289 raft_consensus.cc:2749] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:04.057761 21078 consensus_queue.cc:1048] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20260808 01:55:04.107194 20571 leader_election.cc:304] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:04.086038 21269 raft_consensus.cc:2749] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:04.068634 21222 consensus_queue.cc:1048] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260808 01:55:04.109280 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:04.109802 20648 leader_election.cc:304] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:04.110033 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:04.142244 20609 raft_consensus.cc:2376] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2. Re-sending same reply.
I20260808 01:55:04.110409 21220 raft_consensus.cc:2804] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:55:04.110951 21240 raft_consensus.cc:2749] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:04.091035 21271 consensus_queue.cc:1059] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260808 01:55:04.039228 21300 raft_consensus.cc:515] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:04.046222 21334 consensus_queue.cc:1059] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
W20260808 01:55:04.103332 20475 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:55:04.107828 20602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc85af5215440be884412c1d240c7af" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:04.099996 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:04.163298 20811 tablet_bootstrap.cc:492] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:55:04.164230 20811 ts_tablet_manager.cc:1403] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.325s	user 0.016s	sys 0.024s
I20260808 01:55:04.165138 21300 leader_election.cc:290] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:04.148182 21238 raft_consensus.cc:2804] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:04.104754 21273 raft_consensus.cc:493] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:04.092176 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:04.166404 21238 raft_consensus.cc:493] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:04.166847 21273 raft_consensus.cc:3060] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:04.104553 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:04.093045 21235 consensus_queue.cc:1048] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260808 01:55:04.125157 20815 tablet_bootstrap.cc:492] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
I20260808 01:55:04.121138 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:04.171260 20815 ts_tablet_manager.cc:1403] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 0.325s	user 0.017s	sys 0.006s
I20260808 01:55:04.127113 21343 consensus_queue.cc:1048] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.034s
I20260808 01:55:04.128119 20522 raft_consensus.cc:1240] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:04.128496 21294 raft_consensus.cc:2749] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:04.128808 20571 leader_election.cc:304] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:04.128917 21236 consensus_queue.cc:237] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:04.099130 21266 consensus_queue.cc:1059] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260808 01:55:04.083071 21338 consensus_queue.cc:1048] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260808 01:55:04.110251 20810 raft_consensus.cc:359] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:04.195189 20810 raft_consensus.cc:385] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:04.195634 20810 raft_consensus.cc:740] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:55:04.133394 21249 consensus_queue.cc:1059] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:04.145531 20648 leader_election.cc:304] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:04.162737 20602 raft_consensus.cc:3060] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:04.143564 21220 raft_consensus.cc:493] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:04.203917 21220 raft_consensus.cc:3060] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:04.207024 20603 raft_consensus.cc:1275] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260808 01:55:04.207713 20603 pending_rounds.cc:85] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc: Aborting all ops after (but not including) 0
I20260808 01:55:04.208073 20603 pending_rounds.cc:107] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260808 01:55:04.208451 20603 raft_consensus.cc:2889] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:55:04.167047 21238 raft_consensus.cc:3060] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:04.210232 20811 raft_consensus.cc:359] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:04.211465 20811 raft_consensus.cc:385] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:04.211875 20811 raft_consensus.cc:740] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:55:04.212805 20811 consensus_queue.cc:260] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:04.123385 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:04.127300 21298 raft_consensus.cc:2749] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260808 01:55:04.197454 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 68717.2us
I20260808 01:55:04.218439 20571 leader_election.cc:304] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:04.220264 20571 leader_election.cc:304] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:04.221957 20571 leader_election.cc:304] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:04.203276 21302 raft_consensus.cc:2749] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:04.245756 20810 consensus_queue.cc:260] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:04.247529 20914 consensus_queue.cc:1059] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260808 01:55:04.224248 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 78808.3us
I20260808 01:55:04.251513 20648 leader_election.cc:304] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:04.224675 20571 leader_election.cc:304] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:04.222564 20815 raft_consensus.cc:359] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:04.253422 20815 raft_consensus.cc:385] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:04.253935 20815 raft_consensus.cc:740] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:55:04.245339 21294 raft_consensus.cc:2749] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:04.261986 20571 leader_election.cc:304] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: df82e7f402364d369ed1ec3c5a8772b3
W20260808 01:55:04.195425 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 72818.5us
I20260808 01:55:04.289875 20571 leader_election.cc:304] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:04.256055 21225 raft_consensus.cc:2749] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:55:04.271884 21245 raft_consensus.cc:2749] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:04.273581 21283 raft_consensus.cc:3055] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:55:04.223234 21252 raft_consensus.cc:2749] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260808 01:55:04.294746 20571 leader_election.cc:343] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:04.310721 20571 leader_election.cc:304] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:04.271430 21240 raft_consensus.cc:2804] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:04.254714 21343 raft_consensus.cc:3060] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:04.288056 20815 consensus_queue.cc:260] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:04.307962 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 221011us
W20260808 01:55:04.308055 20648 leader_election.cc:343] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:04.304673 21237 raft_consensus.cc:493] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:04.319132 20648 leader_election.cc:304] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:04.259120 21235 consensus_queue.cc:1059] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260808 01:55:04.290853 21269 raft_consensus.cc:3055] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:04.289765 21337 raft_consensus.cc:2804] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:55:04.301479 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:04.317562 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1753378f13594e89aee6641e2cb9e6da" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:04.319869 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
W20260808 01:55:04.335929 20492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:04.333082 20810 ts_tablet_manager.cc:1434] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.301s	user 0.009s	sys 0.000s
I20260808 01:55:04.337796 20810 tablet_bootstrap.cc:492] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:55:04.322364 21237 raft_consensus.cc:515] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:04.340018 21078 consensus_queue.cc:1059] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:55:04.332600 21269 raft_consensus.cc:740] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
W20260808 01:55:04.323060 20571 leader_election.cc:343] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:04.309849 21294 raft_consensus.cc:2749] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:04.310420 21283 raft_consensus.cc:740] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:04.339244 21350 raft_consensus.cc:3055] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:04.316020 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:04.369951 20815 ts_tablet_manager.cc:1434] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.198s	user 0.003s	sys 0.008s
I20260808 01:55:04.320935 21225 raft_consensus.cc:2749] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:55:04.310527 21290 consensus_queue.cc:1059] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260808 01:55:04.357626 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1753378f13594e89aee6641e2cb9e6da" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:04.382520 20609 raft_consensus.cc:3060] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:04.353235 20681 raft_consensus.cc:2468] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:04.318349 21288 consensus_queue.cc:1048] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260808 01:55:04.385404 20492 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:04.390223 20492 leader_election.cc:304] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:04.360882 21337 raft_consensus.cc:697] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:04.367713 21291 raft_consensus.cc:3055] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:04.398042 21357 raft_consensus.cc:493] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:04.398319 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:04.398871 21295 raft_consensus.cc:2749] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:04.398993 21357 raft_consensus.cc:515] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:04.400171 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc85af5215440be884412c1d240c7af" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:04.383047 20754 raft_consensus.cc:3060] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:04.383940 21335 raft_consensus.cc:2749] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:55:04.364843 21078 consensus_queue.cc:1059] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260808 01:55:04.387826 21252 raft_consensus.cc:493] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:04.382340 21350 raft_consensus.cc:740] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:04.406733 20678 raft_consensus.cc:1240] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:04.392828 21222 raft_consensus.cc:2749] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:55:04.390122 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:04.413359 21352 raft_consensus.cc:493] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:04.430891 21352 raft_consensus.cc:515] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:04.433312 21352 leader_election.cc:290] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:04.435168 21237 leader_election.cc:290] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:04.408180 21336 raft_consensus.cc:2749] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:04.418076 21252 raft_consensus.cc:515] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:04.439373 21252 leader_election.cc:290] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:04.440600 20686 raft_consensus.cc:1275] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:04.412829 21350 consensus_queue.cc:260] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:04.430198 20810 tablet_bootstrap.cc:654] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
W20260808 01:55:04.442931 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 372202us
I20260808 01:55:04.446085 20725 leader_election.cc:304] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:04.448226 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:04.450464 20681 raft_consensus.cc:1275] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:04.401839 21269 consensus_queue.cc:260] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:04.454183 21358 consensus_queue.cc:1048] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260808 01:55:04.471846 20609 raft_consensus.cc:2468] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:55:04.459714 21350 raft_consensus.cc:3060] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:04.395507 21291 raft_consensus.cc:740] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:04.458808 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:04.478200 20811 ts_tablet_manager.cc:1434] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.314s	user 0.035s	sys 0.019s
I20260808 01:55:04.486311 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:04.488013 20609 raft_consensus.cc:2393] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 2: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
W20260808 01:55:04.487708 20571 leader_election.cc:343] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:04.474090 21237 raft_consensus.cc:493] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:04.502611 20609 raft_consensus.cc:1240] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
W20260808 01:55:04.503189 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 154016us
I20260808 01:55:04.470655 20602 raft_consensus.cc:2468] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:04.495898 20571 leader_election.cc:304] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:04.473079 20759 raft_consensus.cc:2468] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:04.477607 21249 raft_consensus.cc:2804] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:04.460539 21337 consensus_queue.cc:237] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:04.533953 20811 tablet_bootstrap.cc:492] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:55:04.490396 21357 leader_election.cc:290] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:04.534549 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 46882.1us
I20260808 01:55:04.537961 20571 leader_election.cc:304] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
W20260808 01:55:04.544137 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 68339.1us
I20260808 01:55:04.490304 21334 consensus_queue.cc:1059] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260808 01:55:04.545143 20725 leader_election.cc:304] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:04.533404 20678 raft_consensus.cc:1240] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:04.533690 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:04.497985 21273 raft_consensus.cc:515] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:04.543263 21252 raft_consensus.cc:2749] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:55:04.527451 21288 consensus_queue.cc:1048] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20260808 01:55:04.523046 21237 raft_consensus.cc:515] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:04.554178 20493 leader_election.cc:304] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:04.540459 21240 raft_consensus.cc:697] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 2 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:04.548834 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a313d0a0979e4dc8a4205c4f16a24257" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:04.565778 20648 leader_election.cc:304] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:04.570264 20492 leader_election.cc:304] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:04.545081 21249 raft_consensus.cc:697] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:04.574466 20725 leader_election.cc:304] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:04.559298 21236 raft_consensus.cc:2804] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election won for term 2
W20260808 01:55:04.673405 21363 log.cc:927] Time spent T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
W20260808 01:55:04.674161 21354 log.cc:927] Time spent T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260808 01:55:04.675760 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:55:04.676079 20495 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:04.676260 20683 raft_consensus.cc:1275] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:04.677187 20681 raft_consensus.cc:2393] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 1: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:04.680642 21273 leader_election.cc:290] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:04.569000 21291 consensus_queue.cc:260] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:04.682302 21291 raft_consensus.cc:3060] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:04.683890 21220 raft_consensus.cc:515] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:04.684090 21240 consensus_queue.cc:237] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:04.686129 21362 raft_consensus.cc:2749] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:04.595546 21078 raft_consensus.cc:2749] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:55:04.571084 21237 leader_election.cc:290] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:04.649485 20684 raft_consensus.cc:1240] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:04.687120 21249 consensus_queue.cc:237] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:04.596275 21289 raft_consensus.cc:493] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:04.707528 21359 raft_consensus.cc:493] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:04.708022 21289 raft_consensus.cc:515] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:04.689247 21288 raft_consensus.cc:493] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:04.708334 21359 raft_consensus.cc:515] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:04.709101 21288 raft_consensus.cc:515] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:04.573391 21335 consensus_queue.cc:1048] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.116s
I20260808 01:55:04.710850 21228 raft_consensus.cc:493] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:04.711342 21288 leader_election.cc:290] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:04.595758 21299 raft_consensus.cc:2804] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:04.711421 21228 raft_consensus.cc:515] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:04.712152 21299 raft_consensus.cc:493] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:04.712563 21299 raft_consensus.cc:3060] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:04.713707 21228 leader_election.cc:290] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:04.571435 21238 raft_consensus.cc:515] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:04.595458 21358 consensus_queue.cc:1059] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.111s
I20260808 01:55:04.568455 21360 consensus_queue.cc:1048] T ce51fcffe2ae4bcba55cd864c973bf0d P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.129s
I20260808 01:55:04.595765 21351 consensus_queue.cc:1048] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.102s
I20260808 01:55:04.753834 21220 leader_election.cc:290] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:04.701598 20754 raft_consensus.cc:2468] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3.
I20260808 01:55:04.708523 20679 raft_consensus.cc:2393] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 1: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:04.691671 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a313d0a0979e4dc8a4205c4f16a24257" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:55:04.718217 20493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:04.560165 21336 raft_consensus.cc:2749] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:55:04.727731 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 153373us
I20260808 01:55:04.775835 20725 leader_election.cc:304] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:55:04.738538 20811 tablet_bootstrap.cc:654] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:04.710556 21289 leader_election.cc:290] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:04.726707 21359 leader_election.cc:290] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:04.758533 21228 raft_consensus.cc:2749] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:04.782284 20725 leader_election.cc:304] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:04.786203 20493 leader_election.cc:304] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:55:04.757031 21236 raft_consensus.cc:697] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:04.790437 20725 leader_election.cc:304] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:04.794926 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:04.797348 20609 raft_consensus.cc:2468] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
I20260808 01:55:04.781245 21357 raft_consensus.cc:2804] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:04.781626 21343 raft_consensus.cc:2749] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:04.803205 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:04.809551 21369 consensus_queue.cc:1059] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260808 01:55:04.811754 20524 raft_consensus.cc:1275] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:04.775224 21360 consensus_queue.cc:1059] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260808 01:55:04.813249 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:04.792474 21351 raft_consensus.cc:2749] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:04.791358 20606 raft_consensus.cc:3060] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:04.798712 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:04.826642 20609 raft_consensus.cc:2376] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2. Re-sending same reply.
I20260808 01:55:04.800083 21228 raft_consensus.cc:2804] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:55:04.823442 20686 raft_consensus.cc:1240] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:04.800608 21289 raft_consensus.cc:2804] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:04.840643 20492 leader_election.cc:304] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:04.847926 20648 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:04.829200 20604 raft_consensus.cc:1275] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:04.834522 20684 raft_consensus.cc:2393] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 2: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:04.829761 20530 raft_consensus.cc:2468] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:04.833811 21228 raft_consensus.cc:697] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:04.833366 21289 raft_consensus.cc:697] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [term 3 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
W20260808 01:55:04.856835 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 60610.8us
I20260808 01:55:04.851047 20648 leader_election.cc:304] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:04.847254 21249 raft_consensus.cc:493] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:04.869760 20648 leader_election.cc:304] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:04.859109 21295 raft_consensus.cc:2804] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:55:04.872738 21295 raft_consensus.cc:697] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:04.874737 21249 raft_consensus.cc:515] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:04.878038 21249 leader_election.cc:290] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:04.881880 21289 consensus_queue.cc:237] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:04.884732 21228 consensus_queue.cc:237] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:04.888157 20524 raft_consensus.cc:1240] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:04.850173 21078 consensus_queue.cc:1048] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260808 01:55:04.889891 21238 leader_election.cc:290] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:04.863950 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac71e66cb4042eb8fe8aafd139af68a" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:04.874154 21295 consensus_queue.cc:237] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:04.841611 21236 consensus_queue.cc:237] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:04.852480 21240 raft_consensus.cc:2749] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:55:04.910221 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 71575.1us
I20260808 01:55:04.909881 20648 leader_election.cc:304] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:04.857234 21357 raft_consensus.cc:697] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:04.869129 21343 raft_consensus.cc:3060] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:04.916293 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac71e66cb4042eb8fe8aafd139af68a" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:04.918193 20609 raft_consensus.cc:2468] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:55:04.913195 20648 leader_election.cc:304] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:55:04.915215 21291 consensus_queue.cc:1059] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
W20260808 01:55:04.918722 20571 leader_election.cc:343] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:04.919220 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:04.914646 21238 raft_consensus.cc:2749] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:55:04.929986 21240 raft_consensus.cc:2804] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:55:04.930001 21361 raft_consensus.cc:493] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:04.930919 21361 raft_consensus.cc:515] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:04.934243 21361 leader_election.cc:290] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:04.920672 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:04.920341 21370 raft_consensus.cc:2749] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:04.936373 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1753378f13594e89aee6641e2cb9e6da" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:04.944123 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:04.943912 20810 tablet_bootstrap.cc:492] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:55:04.946381 20810 ts_tablet_manager.cc:1403] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 0.609s	user 0.009s	sys 0.019s
I20260808 01:55:04.926604 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:04.944480 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:04.920501 21364 consensus_queue.cc:1048] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.041s
I20260808 01:55:04.939038 21240 raft_consensus.cc:493] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:05.048192 20648 leader_election.cc:304] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:05.045397 21334 raft_consensus.cc:493] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:05.125608 20606 raft_consensus.cc:1275] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:05.071738 20604 raft_consensus.cc:2393] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:05.052893 21240 raft_consensus.cc:3060] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:05.052786 21370 raft_consensus.cc:2749] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:05.132469 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
W20260808 01:55:05.136466 21350 consensus_meta.cc:287] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e: Time spent flushing consensus metadata: real 0.650s	user 0.014s	sys 0.000s
I20260808 01:55:05.136822 21283 consensus_queue.cc:260] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
W20260808 01:55:05.137878 21376 log.cc:927] Time spent T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.223s	user 0.000s	sys 0.001s
I20260808 01:55:05.072264 21351 raft_consensus.cc:493] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:05.071918 21238 raft_consensus.cc:493] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:55:05.160614 21377 log.cc:927] Time spent T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 0.254s	user 0.001s	sys 0.000s
I20260808 01:55:05.161075 21366 raft_consensus.cc:493] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:05.134502 21334 raft_consensus.cc:515] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:05.126121 21358 raft_consensus.cc:493] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:05.156168 21220 raft_consensus.cc:493] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:05.170933 21343 raft_consensus.cc:2749] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:05.174062 20810 raft_consensus.cc:359] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:05.150235 21357 consensus_queue.cc:237] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:05.178860 20811 tablet_bootstrap.cc:492] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:55:05.217648 21225 raft_consensus.cc:493] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:05.158442 21299 raft_consensus.cc:515] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:05.208078 21361 raft_consensus.cc:493] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:05.230566 20810 raft_consensus.cc:385] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:05.179989 20602 raft_consensus.cc:1240] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:05.217912 21337 raft_consensus.cc:493] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:05.232584 20811 ts_tablet_manager.cc:1403] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.704s	user 0.025s	sys 0.016s
I20260808 01:55:05.255237 20810 raft_consensus.cc:740] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
W20260808 01:55:05.257045 20493 leader_election.cc:343] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:05.230207 20601 raft_consensus.cc:2393] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 2: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:05.260084 20810 consensus_queue.cc:260] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
W20260808 01:55:05.273921 20493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:05.231246 21358 raft_consensus.cc:515] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:05.306910 21358 leader_election.cc:290] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:05.260449 21225 raft_consensus.cc:515] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:05.310997 21225 leader_election.cc:290] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:05.311571 21225 raft_consensus.cc:493] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:05.312095 21225 raft_consensus.cc:515] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:05.314229 21225 leader_election.cc:290] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:05.225232 21238 raft_consensus.cc:515] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:05.258286 21291 raft_consensus.cc:493] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:05.221546 21351 raft_consensus.cc:515] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:05.275327 21370 raft_consensus.cc:493] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:05.357079 21238 leader_election.cc:290] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:05.368839 21358 consensus_queue.cc:1048] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260808 01:55:05.232163 21220 raft_consensus.cc:515] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:05.382692 20810 ts_tablet_manager.cc:1434] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.436s	user 0.010s	sys 0.000s
I20260808 01:55:05.361090 20725 leader_election.cc:304] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:05.394318 20725 leader_election.cc:304] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:05.274664 21361 raft_consensus.cc:515] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:05.228701 21366 raft_consensus.cc:515] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:05.378945 21249 raft_consensus.cc:493] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:05.316354 21337 raft_consensus.cc:515] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:05.393131 21238 raft_consensus.cc:493] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:05.400185 21228 raft_consensus.cc:493] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:05.379180 21291 raft_consensus.cc:515] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:05.408401 21388 raft_consensus.cc:3060] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:05.393368 21370 raft_consensus.cc:515] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:05.429888 21225 raft_consensus.cc:493] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:05.429418 21364 raft_consensus.cc:493] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:05.433308 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:05.435360 21228 raft_consensus.cc:515] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:05.440749 21343 raft_consensus.cc:493] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:05.427654 21249 raft_consensus.cc:515] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:05.442471 21358 raft_consensus.cc:493] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:05.436296 21238 raft_consensus.cc:515] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:05.489663 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:05.490906 21334 leader_election.cc:290] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:05.491693 21392 log.cc:927] Time spent T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260808 01:55:05.473145 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:05.452009 21225 raft_consensus.cc:515] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:05.525174 21225 leader_election.cc:290] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:05.503448 21351 leader_election.cc:290] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
W20260808 01:55:05.504570 21394 log.cc:927] Time spent T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260808 01:55:05.493523 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:05.505862 21291 leader_election.cc:290] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:05.498629 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:05.475944 21389 raft_consensus.cc:493] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:05.504595 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:05.536938 20609 raft_consensus.cc:2468] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:05.499152 20756 raft_consensus.cc:3060] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:05.507650 21337 leader_election.cc:290] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:05.456163 21364 raft_consensus.cc:515] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:05.475952 21399 raft_consensus.cc:493] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:05.511350 21379 raft_consensus.cc:493] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:05.492038 21401 raft_consensus.cc:493] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:05.512342 21220 leader_election.cc:290] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:05.535614 20602 raft_consensus.cc:1275] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Refusing update from remote peer 598db12c38d442c6988b4c4f991874fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:05.475225 21343 raft_consensus.cc:515] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:05.521888 20679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46954 (request call id 191) took 1230 ms. Trace:
I20260808 01:55:05.526369 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:05.508630 21299 leader_election.cc:290] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:05.552858 21249 leader_election.cc:290] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:05.554178 21366 leader_election.cc:290] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:05.543303 20678 raft_consensus.cc:3060] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:05.482892 21358 raft_consensus.cc:515] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:05.508687 20811 raft_consensus.cc:359] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:05.536720 21298 raft_consensus.cc:2804] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:05.544924 20754 raft_consensus.cc:2393] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 2: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:05.540499 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:05.549979 20529 raft_consensus.cc:2468] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:05.563326 20609 raft_consensus.cc:2468] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:05.564150 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:05.564647 20609 raft_consensus.cc:2468] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:05.543179 21337 raft_consensus.cc:493] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:05.560927 20492 leader_election.cc:304] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:55:05.561074 20811 raft_consensus.cc:385] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:05.540158 20524 raft_consensus.cc:2468] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:05.565294 21299 consensus_queue.cc:1048] T 611b7203dac24e2e909dee3e0fc5fda7 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260808 01:55:05.567631 20525 raft_consensus.cc:1275] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:05.568162 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:05.568885 20524 raft_consensus.cc:2468] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 0.
I20260808 01:55:05.569125 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:05.548964 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:05.566642 20811 raft_consensus.cc:740] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:55:05.540470 21389 raft_consensus.cc:515] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:05.546519 21399 raft_consensus.cc:515] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:05.576315 21403 raft_consensus.cc:493] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:05.576961 21403 raft_consensus.cc:515] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:05.578164 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:05.572896 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd11a6553c444d488651367f3f0143f" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:05.574844 21334 raft_consensus.cc:493] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:05.579538 20725 leader_election.cc:304] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:05.560206 21302 raft_consensus.cc:3055] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:05.565958 20680 raft_consensus.cc:1240] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:05.583796 21334 raft_consensus.cc:515] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:05.585170 20725 leader_election.cc:304] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:05.571326 20811 consensus_queue.cc:260] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:05.581296 21299 raft_consensus.cc:2804] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:55:05.581754 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:05.584666 21302 raft_consensus.cc:740] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:05.581112 21337 raft_consensus.cc:515] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:05.598026 20602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:05.598894 20602 raft_consensus.cc:2468] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:05.598544 21302 consensus_queue.cc:260] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:05.599804 21302 raft_consensus.cc:3060] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:05.599916 21337 leader_election.cc:290] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:05.592670 21236 consensus_queue.cc:1048] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260808 01:55:05.602488 21399 leader_election.cc:290] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:05.602852 20648 leader_election.cc:336] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.778s (SENT)
I20260808 01:55:05.597028 20759 raft_consensus.cc:2393] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 3: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:05.604071 20602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:05.604635 21358 leader_election.cc:290] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:05.605182 21228 leader_election.cc:290] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:05.548197 21401 raft_consensus.cc:515] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:05.603464 20648 leader_election.cc:304] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:05.607510 21366 raft_consensus.cc:2749] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:05.607630 21403 leader_election.cc:290] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:05.607903 21238 leader_election.cc:290] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:05.551151 21379 raft_consensus.cc:515] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:05.610623 20602 raft_consensus.cc:1240] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:05.584893 21351 raft_consensus.cc:2804] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:05.612809 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:05.595768 21299 raft_consensus.cc:493] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:05.551256 20679 rpcz_store.cc:276] 0808 01:55:04.291852 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:04.293439 (+  1587us) service_pool.cc:224] Handling call
0808 01:55:05.521560 (+1228121us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b54006e8cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:05.521768 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":856584064}
I20260808 01:55:05.622589 20811 ts_tablet_manager.cc:1434] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.364s	user 0.005s	sys 0.004s
I20260808 01:55:05.623258 21351 raft_consensus.cc:493] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:05.625917 21240 raft_consensus.cc:515] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:05.629578 21351 raft_consensus.cc:3060] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:05.627647 21299 raft_consensus.cc:3060] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:05.626472 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:05.614770 21334 leader_election.cc:290] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:55:05.890093 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 86751.5us
I20260808 01:55:05.889703 20492 leader_election.cc:304] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:55:05.714690 20677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46954 (request call id 189) took 1454 ms. Trace:
I20260808 01:55:05.906330 20677 rpcz_store.cc:276] 0808 01:55:04.260400 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:04.271742 (+ 11342us) service_pool.cc:224] Handling call
0808 01:55:05.713437 (+1441695us) spinlock_profiling.cc:243] Waited 4.19 ms on lock 0x7b5800775598. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d4185584c 00007f5d4185540b 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:55:05.713739 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:05.714308 (+   569us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:05.714577 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9634560}
I20260808 01:55:05.907263 20648 leader_election.cc:304] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:55:05.908048 21298 raft_consensus.cc:493] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
W20260808 01:55:05.819231 20811 ts_tablet_manager.cc:1467] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3: Tablet startup took 560ms
W20260808 01:55:05.734216 20725 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:05.640460 20810 tablet_bootstrap.cc:492] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e: Bootstrap starting.
I20260808 01:55:05.715989 20815 tablet_bootstrap.cc:492] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb: Bootstrap starting.
W20260808 01:55:05.683979 20727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:05.650519 20686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46954 (request call id 195) took 1119 ms. Trace:
I20260808 01:55:05.637540 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:55:05.711615 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 120749us
I20260808 01:55:05.642114 20606 raft_consensus.cc:1275] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260808 01:55:05.756507 20573 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:55:05.695051 20570 leader_election.cc:343] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:05.656759 21245 raft_consensus.cc:3055] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:05.636302 21389 leader_election.cc:290] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:05.666368 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:05.754011 21333 raft_consensus.cc:3055] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:05.908357 21235 raft_consensus.cc:2804] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:05.927215 20686 rpcz_store.cc:276] 0808 01:55:04.531069 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:04.531233 (+   164us) service_pool.cc:224] Handling call
0808 01:55:04.531664 (+   431us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:04.828033 (+296369us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:05.650208 (+822175us) spinlock_profiling.cc:243] Waited 291 ms on lock 0x7b540051b154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:05.650406 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":670394112}
I20260808 01:55:05.765060 21300 raft_consensus.cc:3055] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:05.986968 20761 raft_consensus.cc:2468] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 0.
W20260808 01:55:06.041615 20493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:05.647104 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:05.758025 21270 raft_consensus.cc:3055] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:05.650274 20680 raft_consensus.cc:3060] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:05.638195 21401 leader_election.cc:290] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:05.712885 21343 leader_election.cc:290] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:05.637398 21379 leader_election.cc:290] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:05.892412 21078 raft_consensus.cc:2804] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:05.664386 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:06.035019 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:06.067760 20760 raft_consensus.cc:3060] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:05.636520 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44768 (request call id 115) took 1530 ms. Trace:
I20260808 01:55:05.717893 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44780 (request call id 123) took 1365 ms. Trace:
I20260808 01:55:05.782816 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:05.767428 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:06.037079 21245 raft_consensus.cc:740] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:06.103401 21283 raft_consensus.cc:3060] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:06.103323 21245 consensus_queue.cc:260] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:06.104789 21245 raft_consensus.cc:3060] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:55:06.106082 21412 log.cc:927] Time spent T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260808 01:55:06.107292 21290 raft_consensus.cc:493] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:06.107554 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:06.107996 21290 raft_consensus.cc:515] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:06.109375 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:06.111389 20684 raft_consensus.cc:2393] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 2: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:06.113097 20604 raft_consensus.cc:1240] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:06.094919 21366 raft_consensus.cc:493] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.042421 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:05.720956 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53302 (request call id 180) took 1435 ms. Trace:
I20260808 01:55:06.123772 20606 raft_consensus.cc:2468] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:06.129428 20425 catalog_manager.cc:5697] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } }
I20260808 01:55:05.644155 21295 raft_consensus.cc:493] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:05.640992 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:05.635475 21240 leader_election.cc:290] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:05.689635 20605 raft_consensus.cc:1275] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Refusing update from remote peer 598db12c38d442c6988b4c4f991874fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:05.648702 20756 raft_consensus.cc:2468] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:05.646601 20602 raft_consensus.cc:1240] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:06.073990 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd11a6553c444d488651367f3f0143f" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:05.719471 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44780 (request call id 124) took 1367 ms. Trace:
I20260808 01:55:05.716069 21271 raft_consensus.cc:493] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:05.952901 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:05.647689 21364 leader_election.cc:290] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:05.643872 21361 leader_election.cc:290] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:05.640846 21371 raft_consensus.cc:493] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:05.644376 21352 raft_consensus.cc:493] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:06.073642 20600 rpcz_store.cc:276] 0808 01:55:04.352072 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:04.353530 (+  1458us) service_pool.cc:224] Handling call
0808 01:55:05.554997 (+1201467us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:55:05.555258 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:05.584791 (+ 29533us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:05.585855 (+  1064us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:05.590006 (+  4151us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:05.592263 (+  2257us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:05.717542 (+125279us) spinlock_profiling.cc:243] Waited 22.7 ms on lock 0x7b54001d9cd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:05.717617 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:05.717811 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":75861632}
I20260808 01:55:06.178848 21352 raft_consensus.cc:515] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:06.181648 21352 leader_election.cc:290] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:05.695783 21388 raft_consensus.cc:2749] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:55:05.751366 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:06.183030 20601 raft_consensus.cc:3060] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:05.709846 21370 leader_election.cc:290] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:05.640115 21404 raft_consensus.cc:493] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:05.688565 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:05.918855 20811 ts_tablet_manager.cc:1469] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3: Trace:
0808 01:54:50.484245 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:54:50.484959 (+   714us) service_pool.cc:224] Handling call
0808 01:54:50.493001 (+  8042us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0808 01:54:50.493128 (+   127us) ts_tablet_manager.cc:682] Creating new metadata...
0808 01:54:50.544879 (+ 51751us) tablet_metadata.cc:665] Metadata flushed
0808 01:54:50.627198 (+ 82319us) tablet_replica.cc:206] Creating consensus instance
0808 01:54:50.627697 (+   499us) inbound_call.cc:177] Queueing success response
0808 01:55:04.525212 (+13897515us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0808 01:55:05.258957 (+733745us) ts_tablet_manager.cc:1435] Starting tablet replica
0808 01:55:05.456757 (+197800us) tablet_replica.cc:261] Starting instrumentation
0808 01:55:05.474012 (+ 17255us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":344,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":7345,"tablet-open.queue_time_us":13897583}
I20260808 01:55:05.686738 20678 raft_consensus.cc:2468] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3.
I20260808 01:55:05.783766 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:05.909003 21338 consensus_queue.cc:1059] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.148s
I20260808 01:55:05.909240 21350 consensus_queue.cc:1048] T 45432e629d3b49abab9715ce5c0124d1 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.159s
I20260808 01:55:06.039882 21383 raft_consensus.cc:3055] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:06.039763 21334 raft_consensus.cc:493] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.042817 21266 raft_consensus.cc:493] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.044576 21269 raft_consensus.cc:3060] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:05.954205 21238 consensus_queue.cc:1048] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.105s
I20260808 01:55:06.075985 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:06.045928 21235 raft_consensus.cc:493] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:06.054178 21300 raft_consensus.cc:740] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:06.034327 20570 leader_election.cc:304] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:06.182535 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:55:06.078204 20681 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46972 (request call id 100) took 1243 ms (client timeout 1546 ms). Trace:
I20260808 01:55:06.093184 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:55:06.093771 20685 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46954 (request call id 197) took 1558 ms (client timeout 1796 ms). Trace:
I20260808 01:55:06.096469 20682 raft_consensus.cc:2393] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 1: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:06.059118 21336 raft_consensus.cc:493] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.100049 20424 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48838 (request call id 85) took 1673 ms. Trace:
I20260808 01:55:06.101114 21381 raft_consensus.cc:493] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.101841 21398 raft_consensus.cc:493] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
W20260808 01:55:06.125247 20475 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:55:06.054546 21270 raft_consensus.cc:740] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:06.059719 21389 raft_consensus.cc:493] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.057204 21405 raft_consensus.cc:493] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:55:06.126987 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 55081us
I20260808 01:55:05.908818 21360 raft_consensus.cc:493] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.069178 21222 raft_consensus.cc:493] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.069110 21078 raft_consensus.cc:493] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:06.119921 21366 raft_consensus.cc:515] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:05.953236 21298 raft_consensus.cc:3060] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:06.136588 21243 raft_consensus.cc:493] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:06.143760 20760 raft_consensus.cc:2468] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3.
I20260808 01:55:06.137087 21401 raft_consensus.cc:493] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.163393 21382 raft_consensus.cc:2764] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260808 01:55:06.176666 20649 leader_election.cc:304] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:06.191020 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:06.158525 21295 raft_consensus.cc:515] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:06.143625 21291 consensus_queue.cc:1059] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20260808 01:55:06.166910 20607 rpcz_store.cc:276] 0808 01:55:04.352459 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:04.353901 (+  1442us) service_pool.cc:224] Handling call
0808 01:55:05.718375 (+1364474us) spinlock_profiling.cc:243] Waited 9.18 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d4185589e 000000000038d209 00007f5d49914940 00007f5d499dced4 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:55:05.718631 (+   256us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:05.718936 (+   305us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:05.718980 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:05.719062 (+    82us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:05.719118 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:05.719216 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:05.719384 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21123840}
I20260808 01:55:06.173127 21240 raft_consensus.cc:493] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.200019 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:06.150930 20726 leader_election.cc:304] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
W20260808 01:55:06.252144 20682 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46954 (request call id 190) took 1991 ms (client timeout 1539 ms). Trace:
I20260808 01:55:06.043738 21333 raft_consensus.cc:740] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:06.218870 20811 tablet_bootstrap.cc:492] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
W20260808 01:55:06.220230 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 525193us
I20260808 01:55:06.190605 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44780 (request call id 122) took 1839 ms. Trace:
I20260808 01:55:06.070863 20603 rpcz_store.cc:276] 0808 01:55:04.105583 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:04.108489 (+  2906us) service_pool.cc:224] Handling call
0808 01:55:04.108829 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:04.209187 (+100358us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:05.636031 (+1426844us) spinlock_profiling.cc:243] Waited 62.9 ms on lock 0x7b5400363654. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:05.636382 (+   351us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":144710912}
I20260808 01:55:06.172369 21271 raft_consensus.cc:515] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:06.195353 21383 raft_consensus.cc:740] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:06.233315 20424 rpcz_store.cc:276] 0808 01:55:04.426504 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:04.426740 (+   236us) service_pool.cc:224] Handling call
0808 01:55:06.099913 (+1673173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":525,"reported_tablets":3}
I20260808 01:55:06.184825 21371 raft_consensus.cc:515] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:06.217399 21235 raft_consensus.cc:3060] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:55:06.203380 20608 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 120) took 1842 ms (client timeout 1778 ms). Trace:
I20260808 01:55:06.228397 21361 raft_consensus.cc:2749] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:55:06.279711 21243 raft_consensus.cc:515] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:06.242801 20493 leader_election.cc:304] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:06.247911 21078 raft_consensus.cc:3060] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:06.190258 21404 raft_consensus.cc:515] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:06.261919 20648 leader_election.cc:304] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:55:06.267709 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:06.219303 21266 raft_consensus.cc:515] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:06.216938 21334 raft_consensus.cc:515] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:06.266705 20725 leader_election.cc:304] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:06.244350 21338 raft_consensus.cc:493] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:06.286841 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:06.252327 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:55:06.308131 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 131582us
W20260808 01:55:06.275388 21351 consensus_meta.cc:287] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3: Time spent flushing consensus metadata: real 0.645s	user 0.010s	sys 0.006s
I20260808 01:55:06.281273 21424 consensus_queue.cc:1059] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260808 01:55:06.240618 21336 raft_consensus.cc:515] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:06.281849 21403 raft_consensus.cc:493] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.275913 21422 raft_consensus.cc:493] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:06.323285 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:06.245726 21389 raft_consensus.cc:515] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:06.249627 21222 raft_consensus.cc:515] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:06.326081 20570 leader_election.cc:304] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:06.323499 21270 consensus_queue.cc:260] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:06.284309 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:06.290319 21290 leader_election.cc:290] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:06.286427 20758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:06.243310 21398 raft_consensus.cc:515] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:06.367151 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:06.244776 21360 raft_consensus.cc:515] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:06.265122 21418 consensus_queue.cc:1048] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.081s
I20260808 01:55:06.240999 21381 raft_consensus.cc:515] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:06.245361 21405 raft_consensus.cc:515] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:06.285609 21401 raft_consensus.cc:515] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:06.319074 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:06.322499 21240 raft_consensus.cc:515] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:06.230118 20685 rpcz_store.cc:269] 0808 01:55:04.534957 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:04.539529 (+  4572us) service_pool.cc:224] Handling call
0808 01:55:05.767132 (+1227603us) spinlock_profiling.cc:243] Waited 8.69 ms on lock 0x7b5800775598. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:06.093679 (+326547us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19976448}
W20260808 01:55:06.228377 20681 rpcz_store.cc:269] 0808 01:55:04.847173 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:04.850016 (+  2843us) service_pool.cc:224] Handling call
0808 01:55:05.666128 (+816112us) spinlock_profiling.cc:243] Waited 5.72 ms on lock 0x7b5800775598. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:06.073892 (+407764us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13161216}
W20260808 01:55:06.494130 20727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:06.672462 20529 raft_consensus.cc:3060] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:06.138648 20753 rpcz_store.cc:276] 0808 01:55:04.285314 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:04.287660 (+  2346us) service_pool.cc:224] Handling call
0808 01:55:05.719830 (+1432170us) spinlock_profiling.cc:243] Waited 5.01 ms on lock 0x7b5800184e98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:55:05.720148 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:05.720422 (+   274us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:05.720466 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:05.720541 (+    75us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:05.720588 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:05.720678 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:05.720843 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11520896}
I20260808 01:55:06.672946 20760 raft_consensus.cc:3060] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:55:06.371220 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 110351us
I20260808 01:55:06.329561 21379 raft_consensus.cc:3060] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:55:06.685137 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 442427us
W20260808 01:55:06.676980 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 415236us
I20260808 01:55:06.319286 21220 raft_consensus.cc:2749] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:06.328640 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:55:06.583118 21420 log.cc:927] Time spent T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.387s	user 0.000s	sys 0.001s
W20260808 01:55:06.606215 20680 consensus_meta.cc:287] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb: Time spent flushing consensus metadata: real 0.542s	user 0.010s	sys 0.000s
W20260808 01:55:06.486043 21416 log.cc:927] Time spent T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.318s	user 0.000s	sys 0.001s
W20260808 01:55:06.747174 20648 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:06.483131 20815 tablet_bootstrap.cc:654] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:06.746054 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:55:06.777204 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 510653us
I20260808 01:55:06.773875 20686 raft_consensus.cc:2468] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:06.781088 20686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46972 (request call id 106) took 1032 ms. Trace:
I20260808 01:55:06.781414 20686 rpcz_store.cc:276] 0808 01:55:05.748771 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:06.055472 (+306701us) service_pool.cc:224] Handling call
0808 01:55:06.781012 (+725540us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:06.784108 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:06.332293 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:06.786422 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:06.789361 20761 raft_consensus.cc:2468] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:55:06.791093 20609 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44770 (request call id 173) took 1212 ms. Trace:
I20260808 01:55:06.607353 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:06.384380 21388 raft_consensus.cc:493] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:06.480527 21364 raft_consensus.cc:493] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.519207 21283 raft_consensus.cc:493] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.587567 21371 leader_election.cc:290] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:06.610359 21271 leader_election.cc:290] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:06.590746 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:06.445981 21399 raft_consensus.cc:493] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.674217 21386 raft_consensus.cc:493] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.382736 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:06.584757 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:06.601241 21295 leader_election.cc:290] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:06.520107 20526 raft_consensus.cc:2468] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
W20260808 01:55:06.445014 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 260186us
I20260808 01:55:06.372474 21291 raft_consensus.cc:493] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.599359 21366 leader_election.cc:290] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:06.563445 21425 raft_consensus.cc:493] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.791740 20609 rpcz_store.cc:276] 0808 01:55:05.578866 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:06.113229 (+534363us) service_pool.cc:224] Handling call
0808 01:55:06.113661 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:06.746790 (+633129us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:06.746824 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:06.746879 (+    55us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:06.746921 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:06.790739 (+ 43818us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x7b540095abd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:06.790814 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:06.791004 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1455504768}
I20260808 01:55:06.732955 21370 raft_consensus.cc:2804] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:06.807298 20683 raft_consensus.cc:2393] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 1: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:06.731107 21270 raft_consensus.cc:3060] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:06.825917 20760 raft_consensus.cc:2468] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:06.741994 21361 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.748845 21290 raft_consensus.cc:2804] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:06.883064 21290 raft_consensus.cc:697] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 3 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:06.884238 21290 consensus_queue.cc:237] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:06.885398 21398 leader_election.cc:290] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:06.777261 21343 raft_consensus.cc:493] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.887013 21343 raft_consensus.cc:515] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:06.887768 21240 leader_election.cc:290] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:06.888003 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:06.888940 20683 raft_consensus.cc:2468] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:55:06.889575 21343 leader_election.cc:290] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:06.891077 21431 raft_consensus.cc:493] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.830972 21283 raft_consensus.cc:515] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
W20260808 01:55:06.318923 20682 rpcz_store.cc:269] 0808 01:55:04.260779 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:04.271499 (+ 10720us) service_pool.cc:224] Handling call
0808 01:55:05.664118 (+1392619us) spinlock_profiling.cc:243] Waited 3.56 ms on lock 0x7b5800775598. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:06.249373 (+585255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8182656}
I20260808 01:55:06.787206 21350 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.794947 21382 raft_consensus.cc:2804] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:06.723904 21351 raft_consensus.cc:515] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:06.812387 20680 raft_consensus.cc:1275] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:06.919428 20759 raft_consensus.cc:1275] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:06.920001 20680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46954 (request call id 207) took 1339 ms. Trace:
I20260808 01:55:06.920401 20680 rpcz_store.cc:276] 0808 01:55:05.580759 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:05.630962 (+ 50203us) service_pool.cc:224] Handling call
0808 01:55:05.650131 (+ 19169us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:06.919701 (+1269570us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:06.919938 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:06.818831 21388 raft_consensus.cc:515] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:06.922020 20680 raft_consensus.cc:1240] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:06.805110 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44770 (request call id 174) took 1114 ms. Trace:
I20260808 01:55:06.890424 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:55:06.383652 20608 rpcz_store.cc:269] 0808 01:55:04.350569 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:04.353464 (+  2895us) service_pool.cc:224] Handling call
0808 01:55:05.640864 (+1287400us) spinlock_profiling.cc:243] Waited 5.05 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d4185589e 000000000038d209 00007f5d49914940 00007f5d499dced4 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:06.192630 (+551766us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11605888}
I20260808 01:55:06.923288 21401 leader_election.cc:290] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:06.855806 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:06.924005 21401 raft_consensus.cc:493] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
W20260808 01:55:06.924177 20601 consensus_meta.cc:287] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc: Time spent flushing consensus metadata: real 0.740s	user 0.008s	sys 0.005s
I20260808 01:55:06.581995 21300 consensus_queue.cc:260] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:06.839242 21399 raft_consensus.cc:515] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:06.924566 21401 raft_consensus.cc:515] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:06.925323 20601 raft_consensus.cc:2468] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
W20260808 01:55:06.926394 20601 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 142) took 1233 ms (client timeout 1592 ms). Trace:
I20260808 01:55:06.922868 20604 rpcz_store.cc:276] 0808 01:55:05.691153 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:06.114152 (+422999us) service_pool.cc:224] Handling call
0808 01:55:06.114535 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:06.118513 (+  3978us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:06.118556 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:06.119218 (+   662us) log.cc:844] Serialized 26 byte log entry
0808 01:55:06.241406 (+122188us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:06.372142 (+130736us) log.cc:844] Serialized 14 byte log entry
0808 01:55:06.676470 (+304328us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:06.739644 (+ 63174us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:06.743212 (+  3568us) raft_consensus.cc:1685] finished
0808 01:55:06.746276 (+  3064us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:06.797507 (+ 51231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35166,"prepare.queue_time_us":555994,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":5490,"raft.queue_time_us":431654,"raft.run_cpu_time_us":394,"raft.run_wall_time_us":17947,"spinlock_wait_cycles":43118336,"thread_start_us":184710,"threads_started":2,"wal-append.queue_time_us":212141}
I20260808 01:55:06.873553 21366 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:06.322875 20605 rpcz_store.cc:276] 0808 01:55:04.351514 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:04.353584 (+  2070us) service_pool.cc:224] Handling call
0808 01:55:05.689013 (+1335429us) spinlock_profiling.cc:243] Waited 6.92 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:55:05.689245 (+   232us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:06.169990 (+480745us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:06.187941 (+ 17951us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15920896}
I20260808 01:55:06.842761 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:06.910230 21382 raft_consensus.cc:493] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:06.927621 21366 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:06.928170 20761 raft_consensus.cc:2468] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:06.928262 21382 raft_consensus.cc:3060] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:06.929832 21366 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:06.723537 21422 raft_consensus.cc:515] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:06.877924 21295 raft_consensus.cc:493] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:06.869379 21291 raft_consensus.cc:515] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:06.930963 21295 raft_consensus.cc:515] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:06.932492 20683 raft_consensus.cc:1275] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:06.932904 21291 leader_election.cc:290] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:06.878326 21361 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:06.933229 20683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 107) took 1184 ms. Trace:
I20260808 01:55:06.933727 20683 rpcz_store.cc:276] 0808 01:55:05.749116 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:06.931709 (+1182593us) service_pool.cc:224] Handling call
0808 01:55:06.932089 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:06.932949 (+   860us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:06.933169 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:06.934893 21361 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:06.935227 21269 consensus_meta.cc:287] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e: Time spent flushing consensus metadata: real 0.717s	user 0.010s	sys 0.002s
I20260808 01:55:06.936106 21351 leader_election.cc:290] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
W20260808 01:55:06.938362 20648 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.193:34531 timed out after 1.910s (SENT) [suppressed 18 similar messages]
I20260808 01:55:06.871806 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:06.938791 21410 raft_consensus.cc:493] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:06.810793 21424 consensus_queue.cc:1059] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20260808 01:55:06.855808 20528 raft_consensus.cc:1275] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:06.858349 20677 raft_consensus.cc:2468] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:06.914592 21405 leader_election.cc:290] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:06.915650 21389 leader_election.cc:290] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:06.916237 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1753378f13594e89aee6641e2cb9e6da" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:06.867633 21370 raft_consensus.cc:493] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:06.999075 20683 raft_consensus.cc:1240] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:06.999938 20683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46954 (request call id 210) took 1010 ms. Trace:
I20260808 01:55:07.000317 20683 rpcz_store.cc:276] 0808 01:55:05.990047 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:06.939662 (+949615us) service_pool.cc:224] Handling call
0808 01:55:06.998861 (+ 59199us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:06.999637 (+   776us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:06.999872 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:06.611160 21333 consensus_queue.cc:260] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:06.813324 20684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46972 (request call id 110) took 1057 ms. Trace:
I20260808 01:55:06.905287 21343 raft_consensus.cc:493] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:06.892606 20758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:06.739108 21338 raft_consensus.cc:515] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:06.687590 21403 raft_consensus.cc:515] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:06.923336 21334 leader_election.cc:290] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:55:06.942322 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 167719us
W20260808 01:55:06.926942 20601 rpcz_store.cc:269] 0808 01:55:05.693387 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:05.750928 (+ 57541us) service_pool.cc:224] Handling call
0808 01:55:06.926316 (+1175388us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:55:06.947340 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 165776us
I20260808 01:55:06.991457 21401 leader_election.cc:290] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:06.822332 21364 raft_consensus.cc:515] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:06.840826 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:06.924642 21283 leader_election.cc:290] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:06.916364 21266 leader_election.cc:290] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:06.947583 21350 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:06.917704 21404 leader_election.cc:290] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:06.854483 21386 raft_consensus.cc:515] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:06.933135 21295 leader_election.cc:290] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:06.926648 21399 leader_election.cc:290] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
W20260808 01:55:07.009210 20571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:06.894259 21433 raft_consensus.cc:2749] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:06.895691 21398 raft_consensus.cc:493] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:07.012297 20810 tablet_bootstrap.cc:700] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e: Time spent opening tablet: real 0.115s	user 0.000s	sys 0.000s
I20260808 01:55:07.013870 20529 raft_consensus.cc:2468] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:06.834470 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:06.947696 20680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 105) took 1191 ms. Trace:
W20260808 01:55:06.988757 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 212536us
I20260808 01:55:07.176635 20493 leader_election.cc:304] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:06.993258 21222 leader_election.cc:290] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:06.891649 21431 raft_consensus.cc:515] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:07.008248 21370 raft_consensus.cc:3060] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:06.997462 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:07.003625 20762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53306 (request call id 128) took 1730 ms. Trace:
I20260808 01:55:07.056260 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53306 (request call id 123) took 1792 ms. Trace:
W20260808 01:55:07.060746 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 734703us
I20260808 01:55:07.025291 21291 raft_consensus.cc:493] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:07.155030 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.157663 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.248634 20609 raft_consensus.cc:2468] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
W20260808 01:55:07.035746 21299 consensus_meta.cc:287] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e: Time spent flushing consensus metadata: real 1.103s	user 0.008s	sys 0.007s
I20260808 01:55:07.185189 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.170302 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:07.014236 21352 consensus_queue.cc:1059] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260808 01:55:07.032444 21388 leader_election.cc:290] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:07.035406 20758 raft_consensus.cc:2468] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:55:07.187827 20493 leader_election.cc:304] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:07.190562 20810 tablet_bootstrap.cc:654] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:07.011483 20684 rpcz_store.cc:276] 0808 01:55:05.755730 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:06.191476 (+435746us) service_pool.cc:224] Handling call
0808 01:55:06.809637 (+618161us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:07.054281 21351 raft_consensus.cc:493] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:06.991153 21410 raft_consensus.cc:515] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:07.029539 21343 raft_consensus.cc:515] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:06.988090 21078 raft_consensus.cc:515] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:07.180856 21398 raft_consensus.cc:515] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:07.270393 20754 raft_consensus.cc:2468] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:07.317927 20815 tablet_bootstrap.cc:492] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb: No bootstrap required, opened a new log
W20260808 01:55:07.313521 21235 consensus_meta.cc:287] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb: Time spent flushing consensus metadata: real 0.831s	user 0.012s	sys 0.000s
I20260808 01:55:07.253975 21360 leader_election.cc:290] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:07.319376 21235 raft_consensus.cc:515] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:07.246824 20755 rpcz_store.cc:276] 0808 01:55:05.263536 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:05.274859 (+ 11323us) service_pool.cc:224] Handling call
0808 01:55:05.574121 (+299262us) spinlock_profiling.cc:243] Waited 2.04 ms on lock 0x7b5800184e98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:55:05.574350 (+   229us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:05.581325 (+  6975us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:05.582599 (+  1274us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:05.584264 (+  1665us) log.cc:844] Serialized 26 byte log entry
0808 01:55:05.712029 (+127765us) raft_consensus.cc:1645] Marking committed up to 0
0808 01:55:05.712086 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:05.712176 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:07.008083 (+1295907us) raft_consensus.cc:1685] finished
0808 01:55:07.008111 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:07.050446 (+ 42335us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1026,"raft.queue_time_us":4607,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":4320,"spinlock_wait_cycles":4684160,"thread_start_us":127319,"threads_started":1,"wal-append.queue_time_us":538531}
I20260808 01:55:06.872963 21425 raft_consensus.cc:515] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:07.322115 21235 leader_election.cc:290] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:07.054636 20753 raft_consensus.cc:2468] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:07.322971 20753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53306 (request call id 140) took 1068 ms. Trace:
I20260808 01:55:07.250553 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1753378f13594e89aee6641e2cb9e6da" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:07.181803 21238 raft_consensus.cc:493] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:07.323589 21425 leader_election.cc:290] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:07.048156 20678 raft_consensus.cc:2393] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 1: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:07.311429 21401 raft_consensus.cc:493] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:07.323998 21238 raft_consensus.cc:515] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:07.324601 21401 raft_consensus.cc:515] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:07.325577 20678 raft_consensus.cc:1240] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:07.326251 21238 leader_election.cc:290] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:07.188318 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:07.258234 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 70485.4us
I20260808 01:55:07.222364 21399 raft_consensus.cc:493] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:07.270181 20570 leader_election.cc:304] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
W20260808 01:55:07.256630 21298 consensus_meta.cc:287] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3: Time spent flushing consensus metadata: real 0.977s	user 0.012s	sys 0.000s
I20260808 01:55:07.247387 21240 raft_consensus.cc:493] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:07.249745 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.323367 20753 rpcz_store.cc:276] 0808 01:55:06.254361 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:06.824457 (+570096us) service_pool.cc:224] Handling call
0808 01:55:07.322901 (+498444us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:07.300024 20571 leader_election.cc:304] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
W20260808 01:55:07.223054 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 187302us
W20260808 01:55:07.320012 21435 log.cc:927] Time spent T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20260808 01:55:07.320806 21449 consensus_queue.cc:1048] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260808 01:55:07.352527 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 82436.2us
I20260808 01:55:07.353533 20811 tablet_bootstrap.cc:654] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:07.331494 21422 leader_election.cc:290] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:07.331647 21401 leader_election.cc:290] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:07.332616 20677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46954 (request call id 213) took 1299 ms. Trace:
I20260808 01:55:07.193274 21222 raft_consensus.cc:2749] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:07.308300 21351 raft_consensus.cc:515] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:07.192906 21243 leader_election.cc:290] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:07.311725 21445 raft_consensus.cc:493] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:07.327216 20685 raft_consensus.cc:2393] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 2: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
W20260808 01:55:07.328871 21245 consensus_meta.cc:287] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc: Time spent flushing consensus metadata: real 1.224s	user 0.014s	sys 0.000s
I20260808 01:55:07.318809 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53306 (request call id 141) took 1064 ms. Trace:
I20260808 01:55:07.318511 20815 ts_tablet_manager.cc:1403] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb: Time spent bootstrapping tablet: real 1.603s	user 0.038s	sys 0.048s
I20260808 01:55:07.175886 20680 rpcz_store.cc:276] 0808 01:55:05.748438 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:06.921479 (+1173041us) service_pool.cc:224] Handling call
0808 01:55:06.921828 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:06.922443 (+   615us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:06.939601 (+ 17158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:07.378418 20757 rpcz_store.cc:276] 0808 01:55:06.254673 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:06.894550 (+639877us) service_pool.cc:224] Handling call
0808 01:55:07.318712 (+424162us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:07.380445 20680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46968 (request call id 127) took 1205 ms. Trace:
I20260808 01:55:07.381369 21350 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:07.381599 20815 raft_consensus.cc:359] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:07.382368 20815 raft_consensus.cc:385] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:07.382659 20815 raft_consensus.cc:740] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Initialized, Role: FOLLOWER
I20260808 01:55:07.383388 20815 consensus_queue.cc:260] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:07.172171 21437 raft_consensus.cc:493] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:07.347772 21298 raft_consensus.cc:515] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:07.385061 20815 ts_tablet_manager.cc:1434] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.002s
I20260808 01:55:07.388221 21245 raft_consensus.cc:493] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:07.390157 21245 raft_consensus.cc:515] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:07.390501 21437 raft_consensus.cc:515] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:07.392469 21245 leader_election.cc:290] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:07.392747 21386 leader_election.cc:290] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:07.393612 21333 raft_consensus.cc:3060] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:55:07.397925 21379 consensus_meta.cc:287] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3: Time spent flushing consensus metadata: real 0.649s	user 0.009s	sys 0.000s
I20260808 01:55:07.389096 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.401844 20607 raft_consensus.cc:2468] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:55:07.261621 21336 leader_election.cc:290] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:07.270910 21334 raft_consensus.cc:2804] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:07.326406 20678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46968 (request call id 125) took 1152 ms. Trace:
I20260808 01:55:07.326978 21442 raft_consensus.cc:493] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:07.262295 20604 raft_consensus.cc:2468] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:55:07.346423 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.311627 21405 raft_consensus.cc:493] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:07.254963 21291 raft_consensus.cc:515] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
W20260808 01:55:07.223937 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 328874us
I20260808 01:55:07.330331 21441 raft_consensus.cc:493] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:07.414902 21441 raft_consensus.cc:515] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:07.415102 21405 raft_consensus.cc:515] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:07.415896 21078 leader_election.cc:290] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:07.416774 21444 log.cc:927] Time spent T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 0.162s	user 0.002s	sys 0.000s
I20260808 01:55:07.416949 20684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46968 (request call id 123) took 1243 ms. Trace:
I20260808 01:55:07.417276 21441 leader_election.cc:290] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:07.417419 20684 rpcz_store.cc:276] 0808 01:55:06.173531 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:07.314655 (+1141124us) service_pool.cc:224] Handling call
0808 01:55:07.348087 (+ 33432us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:07.361504 (+ 13417us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:07.370337 (+  8833us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:07.372564 (+  2227us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:07.400589 (+ 28025us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:07.406511 (+  5922us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:07.412097 (+  5586us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:07.374929 21451 consensus_queue.cc:1048] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.051s
I20260808 01:55:07.419159 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:07.419860 20684 raft_consensus.cc:2468] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:07.254827 21299 raft_consensus.cc:515] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
W20260808 01:55:07.424444 21447 log.cc:927] Time spent T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260808 01:55:07.425344 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53322 (request call id 144) took 1328 ms. Trace:
I20260808 01:55:07.384895 20528 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41392 (request call id 109) took 1939 ms. Trace:
I20260808 01:55:07.185153 20522 raft_consensus.cc:2468] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:07.426175 20528 rpcz_store.cc:276] 0808 01:55:05.444994 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:05.447081 (+  2087us) service_pool.cc:224] Handling call
0808 01:55:06.028293 (+581212us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b58000d9e98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:55:06.034505 (+  6212us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:06.993518 (+959013us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:07.384474 (+390956us) spinlock_profiling.cc:243] Waited 756 ms on lock 0x7b540016e9d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:07.384755 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1764140544}
I20260808 01:55:07.427079 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.427186 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc85af5215440be884412c1d240c7af" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.427772 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d32821d9c17456394a662fcfd05f6c6" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.427917 20522 raft_consensus.cc:2468] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:07.428076 20523 raft_consensus.cc:2468] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:07.429189 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.430992 20523 raft_consensus.cc:2468] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:07.435189 21364 leader_election.cc:290] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:07.436018 21403 leader_election.cc:290] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:07.425642 21299 leader_election.cc:290] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:07.436537 21343 leader_election.cc:290] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:07.386790 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.439134 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:07.440865 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.346267 21399 raft_consensus.cc:515] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:07.350127 21240 raft_consensus.cc:515] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:07.222961 20762 rpcz_store.cc:276] 0808 01:55:05.265669 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:05.275121 (+  9452us) service_pool.cc:224] Handling call
0808 01:55:05.747244 (+472123us) spinlock_profiling.cc:243] Waited 3.97 ms on lock 0x7b5800184e98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:55:05.747596 (+   352us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:05.748128 (+   532us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:05.748175 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:05.748867 (+   692us) log.cc:844] Serialized 26 byte log entry
0808 01:55:06.062399 (+313532us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:06.064393 (+  1994us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:06.073286 (+  8893us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:06.872101 (+798815us) raft_consensus.cc:1685] finished
0808 01:55:06.874884 (+  2783us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:06.996179 (+121295us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2019,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"spinlock_wait_cycles":9138432,"thread_start_us":295713,"threads_started":1,"wal-append.queue_time_us":414639}
I20260808 01:55:07.377647 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d8bf19c324463fb9903b9981068e9d" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.362145 21283 raft_consensus.cc:2804] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:07.360839 21361 raft_consensus.cc:2804] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:07.406315 21334 raft_consensus.cc:493] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:07.445860 21334 raft_consensus.cc:3060] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:07.444485 21371 raft_consensus.cc:493] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:07.446643 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.447062 21371 raft_consensus.cc:515] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:07.447891 20520 raft_consensus.cc:2468] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:07.407711 20678 rpcz_store.cc:276] 0808 01:55:06.174284 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:07.324970 (+1150686us) service_pool.cc:224] Handling call
0808 01:55:07.325378 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:07.326107 (+   729us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:07.326338 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:07.380666 21388 raft_consensus.cc:493] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:07.450801 21371 leader_election.cc:290] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:07.450523 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:07.444803 21240 leader_election.cc:290] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:07.414189 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.452386 20529 raft_consensus.cc:2468] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:07.453614 21298 leader_election.cc:290] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:07.453778 20529 raft_consensus.cc:1240] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:07.454034 21370 raft_consensus.cc:515] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:07.354895 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 54996.8us
I20260808 01:55:07.422626 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a313d0a0979e4dc8a4205c4f16a24257" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.455331 20600 raft_consensus.cc:2468] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:55:07.359218 20677 rpcz_store.cc:276] 0808 01:55:06.032934 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:07.042701 (+1009767us) service_pool.cc:224] Handling call
0808 01:55:07.223011 (+180310us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:07.253499 (+ 30488us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:07.255296 (+  1797us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:07.257272 (+  1976us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:07.270601 (+ 13329us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:07.310078 (+ 39477us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:07.330592 (+ 20514us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:07.456734 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:07.456817 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.457511 20607 raft_consensus.cc:2468] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:07.457691 20571 leader_election.cc:304] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:07.457954 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:07.423570 20524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41410 (request call id 131) took 1249 ms. Trace:
I20260808 01:55:07.404173 21445 raft_consensus.cc:515] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:07.458918 20682 raft_consensus.cc:1240] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:07.398532 21379 raft_consensus.cc:2749] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:07.460151 20759 raft_consensus.cc:1240] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:07.460588 20682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 104) took 1712 ms. Trace:
I20260808 01:55:07.346622 21360 raft_consensus.cc:493] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:07.461448 20759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53322 (request call id 141) took 1365 ms. Trace:
I20260808 01:55:07.447352 21399 leader_election.cc:290] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:07.461923 21360 raft_consensus.cc:515] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:07.419855 21431 leader_election.cc:290] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:07.458632 20677 raft_consensus.cc:2468] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:07.445724 21361 raft_consensus.cc:493] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:07.463439 21361 raft_consensus.cc:3060] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:07.463845 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:07.464210 21360 leader_election.cc:290] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:07.443460 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.446338 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.465188 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:07.465830 20527 raft_consensus.cc:2410] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:55:07.451241 21388 raft_consensus.cc:515] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:07.412103 21442 raft_consensus.cc:515] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:07.417666 21291 leader_election.cc:290] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:07.413873 21398 leader_election.cc:290] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:07.422171 21351 leader_election.cc:290] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:07.425814 20756 rpcz_store.cc:276] 0808 01:55:06.097267 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:07.424381 (+1327114us) service_pool.cc:224] Handling call
0808 01:55:07.424757 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:07.424986 (+   229us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:07.425015 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:07.425066 (+    51us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:07.425097 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:07.425157 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:07.425284 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:07.458709 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.462107 20759 rpcz_store.cc:276] 0808 01:55:06.096162 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:06.920596 (+824434us) service_pool.cc:224] Handling call
0808 01:55:06.920966 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:07.460605 (+539639us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:07.461209 (+   604us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b54006da1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:07.461387 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1239457536}
I20260808 01:55:07.460013 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.468366 20601 raft_consensus.cc:2468] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:07.459232 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.468880 20604 raft_consensus.cc:2468] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:07.469486 20605 raft_consensus.cc:2468] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 3.
I20260808 01:55:07.460918 21401 raft_consensus.cc:2749] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:07.451593 20678 raft_consensus.cc:2468] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:07.468391 21442 leader_election.cc:290] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:07.461045 20682 rpcz_store.cc:276] 0808 01:55:05.747999 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:06.913408 (+1165409us) service_pool.cc:224] Handling call
0808 01:55:06.913849 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:07.459550 (+545701us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:07.460304 (+   754us) spinlock_profiling.cc:243] Waited 545 ms on lock 0x7b54002f3354. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:07.460499 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1253013888}
I20260808 01:55:07.445641 21283 raft_consensus.cc:493] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:07.471333 21283 raft_consensus.cc:3060] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:07.471993 20571 leader_election.cc:304] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:07.429349 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.447860 20523 raft_consensus.cc:2393] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 3: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:07.440984 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:07.475328 20492 leader_election.cc:304] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: df82e7f402364d369ed1ec3c5a8772b3
W20260808 01:55:07.370653 20570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:55:07.478929 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 140723us
I20260808 01:55:07.484117 20649 leader_election.cc:304] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
W20260808 01:55:07.496259 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 84479.3us
I20260808 01:55:07.497071 20649 leader_election.cc:304] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:07.500273 20570 leader_election.cc:304] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:07.512161 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce51fcffe2ae4bcba55cd864c973bf0d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.513962 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac71e66cb4042eb8fe8aafd139af68a" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:07.514958 20607 raft_consensus.cc:3060] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:55:07.526528 20648 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.017s	sys 0.007s
W20260808 01:55:07.527204 20648 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.017s	sys 0.007s
W20260808 01:55:07.527702 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 589333us
I20260808 01:55:07.532302 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:07.533689 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.466459 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:07.380868 20680 rpcz_store.cc:276] 0808 01:55:06.176264 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:07.379302 (+1203038us) service_pool.cc:224] Handling call
0808 01:55:07.379698 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:07.379914 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:07.379948 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:07.380106 (+   158us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:07.380166 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:07.380244 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:07.380382 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:07.514868 20570 leader_election.cc:304] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:07.442425 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.535696 20492 leader_election.cc:304] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:07.469791 21388 leader_election.cc:290] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:07.540004 20570 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:07.534427 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:07.460956 21445 leader_election.cc:290] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:07.385893 21382 raft_consensus.cc:515] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:07.543511 20609 raft_consensus.cc:3060] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:07.488410 21401 raft_consensus.cc:2749] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:07.505858 21360 raft_consensus.cc:493] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:07.534376 20606 raft_consensus.cc:2468] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3.
I20260808 01:55:07.549338 20493 leader_election.cc:304] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:07.553902 20493 leader_election.cc:304] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
W20260808 01:55:07.545967 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 48991.8us
I20260808 01:55:07.531574 21343 raft_consensus.cc:2804] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:55:07.535027 21431 raft_consensus.cc:2804] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:07.537714 21432 raft_consensus.cc:2804] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:07.544665 21399 raft_consensus.cc:2804] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:07.538767 21078 consensus_queue.cc:1048] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
W20260808 01:55:07.548831 20526 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41392 (request call id 112) took 2096 ms (client timeout 1772 ms). Trace:
I20260808 01:55:07.556763 20493 leader_election.cc:304] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:07.550238 21222 raft_consensus.cc:2804] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:07.556643 20571 leader_election.cc:304] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:55:07.546514 20492 leader_election.cc:304] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:07.566220 20607 raft_consensus.cc:2468] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:55:07.578747 20492 leader_election.cc:304] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:07.542799 20677 raft_consensus.cc:2468] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:07.555891 21431 raft_consensus.cc:493] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:07.568356 20686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46954 (request call id 212) took 1538 ms. Trace:
I20260808 01:55:07.551222 21403 raft_consensus.cc:2804] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:07.550572 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.547432 20528 raft_consensus.cc:2468] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:07.579056 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.585659 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:07.586378 20607 raft_consensus.cc:2468] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:07.587060 20810 tablet_bootstrap.cc:492] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e: No bootstrap required, opened a new log
I20260808 01:55:07.587268 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:07.587643 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.588058 20680 raft_consensus.cc:2468] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:07.588913 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:07.558146 21350 raft_consensus.cc:2749] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:07.589996 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.576709 21388 raft_consensus.cc:493] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:07.590510 21235 raft_consensus.cc:493] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:07.590773 21388 raft_consensus.cc:515] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:07.591197 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.592131 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e71aab2ce9b45369b7eff65e1645f9b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:07.547444 21442 raft_consensus.cc:2749] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:07.581488 20492 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:07.593289 21388 leader_election.cc:290] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:07.593364 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:07.594377 20684 raft_consensus.cc:2393] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 2: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:07.584506 20571 leader_election.cc:304] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:07.592295 20607 raft_consensus.cc:2468] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:07.594728 21370 leader_election.cc:290] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:07.571635 20604 raft_consensus.cc:1275] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:07.577950 21222 raft_consensus.cc:493] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:07.595683 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.598526 20493 leader_election.cc:304] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:07.596151 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.600504 20529 raft_consensus.cc:2410] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260808 01:55:07.587666 20810 ts_tablet_manager.cc:1403] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e: Time spent bootstrapping tablet: real 1.947s	user 0.024s	sys 0.044s
I20260808 01:55:07.601277 21437 leader_election.cc:290] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:07.595568 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.602921 21299 raft_consensus.cc:2749] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:55:07.603346 20607 raft_consensus.cc:2468] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:07.603701 21299 raft_consensus.cc:2804] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:07.604125 21299 raft_consensus.cc:493] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:07.604552 20810 raft_consensus.cc:359] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:07.605348 20810 raft_consensus.cc:385] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:07.605710 20810 raft_consensus.cc:740] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Initialized, Role: FOLLOWER
I20260808 01:55:07.597113 21222 raft_consensus.cc:3060] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:07.607658 21410 leader_election.cc:290] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:07.610093 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.565119 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:07.572348 20526 rpcz_store.cc:269] 0808 01:55:05.446063 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:05.447473 (+  1410us) service_pool.cc:224] Handling call
0808 01:55:05.688454 (+240981us) spinlock_profiling.cc:243] Waited 1.82 ms on lock 0x7b58000d9e98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d4185593c 00007f5d418558a6 000000000038d209 00007f5d49914940 00007f5d499dced4 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:07.537508 (+1849054us) spinlock_profiling.cc:243] Waited 299 ms on lock 0x7b5400023ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:07.542727 (+  5219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":690920576}
I20260808 01:55:07.611045 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.610198 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.611862 20527 raft_consensus.cc:2468] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:07.612075 21398 raft_consensus.cc:2804] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:07.612105 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.591133 21235 raft_consensus.cc:515] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:07.615175 21235 leader_election.cc:290] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:07.559091 21432 raft_consensus.cc:493] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:07.458854 20524 rpcz_store.cc:276] 0808 01:55:06.195414 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:06.203095 (+  7681us) service_pool.cc:224] Handling call
0808 01:55:06.244918 (+ 41823us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:06.483634 (+238716us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:06.509175 (+ 25541us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:06.564095 (+ 54920us) log.cc:844] Serialized 26 byte log entry
0808 01:55:07.003507 (+439412us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:07.008859 (+  5352us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:07.030896 (+ 22037us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:07.423297 (+392401us) raft_consensus.cc:1685] finished
0808 01:55:07.423327 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:07.423491 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":296975,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":191,"thread_start_us":504593,"threads_started":2,"wal-append.queue_time_us":476051}
I20260808 01:55:07.616752 21432 raft_consensus.cc:3060] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:07.604931 21404 consensus_queue.cc:1048] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:07.610606 21382 leader_election.cc:290] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:07.566888 21399 raft_consensus.cc:493] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:07.619835 21399 raft_consensus.cc:3060] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:07.620952 21334 raft_consensus.cc:515] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:07.597399 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.586380 20762 raft_consensus.cc:2468] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:07.623286 20604 raft_consensus.cc:2468] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:07.624173 21401 raft_consensus.cc:493] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:07.551412 21360 raft_consensus.cc:515] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:07.624702 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:07.625139 21441 raft_consensus.cc:2749] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:07.625507 20678 raft_consensus.cc:2468] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:07.625622 21381 leader_election.cc:290] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:07.625985 21441 raft_consensus.cc:493] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:07.563660 21343 raft_consensus.cc:697] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:07.626477 21360 leader_election.cc:290] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:07.626600 21441 raft_consensus.cc:515] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:07.624713 21401 raft_consensus.cc:515] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:07.627219 21343 consensus_queue.cc:237] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:07.627640 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:07.628504 20760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53306 (request call id 138) took 1374 ms. Trace:
I20260808 01:55:07.629065 21441 leader_election.cc:290] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:07.629386 21401 leader_election.cc:290] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:07.629871 20606 raft_consensus.cc:2468] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:07.579608 20609 raft_consensus.cc:2468] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:07.624766 20571 leader_election.cc:304] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:55:07.636848 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d8bf19c324463fb9903b9981068e9d" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:07.637300 20571 leader_election.cc:304] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:07.638509 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:07.639299 20759 raft_consensus.cc:2468] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:07.589085 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac71e66cb4042eb8fe8aafd139af68a" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:07.640056 20680 raft_consensus.cc:2468] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 0.
I20260808 01:55:07.640164 21433 raft_consensus.cc:2804] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:07.640625 20757 raft_consensus.cc:1240] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:07.640731 21433 raft_consensus.cc:493] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:07.641333 21433 raft_consensus.cc:3060] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:07.611706 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63fb8c2f494a462686500ce10025a39f" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.642763 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:07.593408 21350 raft_consensus.cc:2804] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:07.644102 21350 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:07.644558 21350 raft_consensus.cc:3060] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:07.644816 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.645588 20526 raft_consensus.cc:2468] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:07.646628 20570 leader_election.cc:304] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:07.647081 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.647763 20570 leader_election.cc:304] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:07.631053 21431 raft_consensus.cc:3060] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:07.649150 20524 raft_consensus.cc:1240] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:07.646667 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.645190 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac71e66cb4042eb8fe8aafd139af68a" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.650727 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.651134 20528 raft_consensus.cc:2393] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:07.604490 21299 raft_consensus.cc:3060] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:55:07.648365 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 47257.2us
I20260808 01:55:07.584332 21290 raft_consensus.cc:2804] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:07.649932 21360 raft_consensus.cc:2804] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:07.652673 21290 raft_consensus.cc:493] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:07.652784 21360 raft_consensus.cc:493] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:07.653155 21290 raft_consensus.cc:3060] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:07.653378 21360 raft_consensus.cc:3060] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:07.655561 21271 consensus_queue.cc:1048] T 1753378f13594e89aee6641e2cb9e6da P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:55:07.656319 21300 raft_consensus.cc:3060] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:07.560343 21424 raft_consensus.cc:2749] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:07.660117 20685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46968 (request call id 126) took 1485 ms. Trace:
I20260808 01:55:07.660655 20685 rpcz_store.cc:276] 0808 01:55:06.174628 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:06.813207 (+638579us) service_pool.cc:224] Handling call
0808 01:55:07.659857 (+846650us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b54004876d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:07.660040 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":891140992}
W20260808 01:55:07.580047 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 165364us
I20260808 01:55:07.662461 20726 leader_election.cc:304] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:07.663077 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e71aab2ce9b45369b7eff65e1645f9b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.663213 20571 leader_election.cc:304] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:55:07.663385 21379 raft_consensus.cc:2749] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:07.663534 20683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46968 (request call id 122) took 1495 ms. Trace:
I20260808 01:55:07.664260 21336 raft_consensus.cc:2804] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:07.664746 21336 raft_consensus.cc:493] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:07.665201 21336 raft_consensus.cc:3060] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:07.628881 20760 rpcz_store.cc:276] 0808 01:55:06.253754 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:06.892063 (+638309us) service_pool.cc:224] Handling call
0808 01:55:07.627447 (+735384us) spinlock_profiling.cc:243] Waited 1.01 ms on lock 0x7b5800184e98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:07.628433 (+   986us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2321280}
I20260808 01:55:07.647861 20527 raft_consensus.cc:2468] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:07.652134 21441 raft_consensus.cc:2804] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:07.667186 21441 raft_consensus.cc:493] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:07.667594 21441 raft_consensus.cc:3060] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:07.606464 20810 consensus_queue.cc:260] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:07.583056 21440 raft_consensus.cc:2804] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:55:07.668583 21440 raft_consensus.cc:493] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:07.668977 21440 raft_consensus.cc:3060] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:07.669281 20810 ts_tablet_manager.cc:1434] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e: Time spent starting tablet: real 0.068s	user 0.005s	sys 0.000s
I20260808 01:55:07.664024 20683 rpcz_store.cc:276] 0808 01:55:06.167734 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:07.164072 (+996338us) service_pool.cc:224] Handling call
0808 01:55:07.190647 (+ 26575us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:07.391874 (+201227us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:07.663232 (+271358us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b54002f3854. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:07.663453 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":461923712}
I20260808 01:55:07.660742 21222 raft_consensus.cc:515] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:07.643548 20762 raft_consensus.cc:2468] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:07.671717 20681 raft_consensus.cc:1240] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:07.697916 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:07.692937 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce51fcffe2ae4bcba55cd864c973bf0d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.705823 21432 raft_consensus.cc:515] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:07.714208 20570 leader_election.cc:304] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:07.715960 20492 leader_election.cc:304] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:07.667531 20726 leader_election.cc:304] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
W20260808 01:55:07.728904 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 61468us
I20260808 01:55:07.729135 21459 raft_consensus.cc:2804] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:07.729759 21459 raft_consensus.cc:493] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:07.730098 21459 raft_consensus.cc:3060] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:07.730695 21379 consensus_queue.cc:1048] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:55:07.732373 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:07.732570 21283 raft_consensus.cc:515] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:07.733076 20760 raft_consensus.cc:2468] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:07.734123 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:07.734450 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:07.718053 21245 raft_consensus.cc:2764] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260808 01:55:07.734736 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:07.735666 20680 raft_consensus.cc:2468] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:07.736779 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:07.735090 20526 raft_consensus.cc:3060] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 3
I20260808 01:55:07.737627 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:55:07.722333 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 88255.2us
I20260808 01:55:07.741686 20762 raft_consensus.cc:2376] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1. Re-sending same reply.
I20260808 01:55:07.727576 20522 raft_consensus.cc:2468] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:07.739053 20492 leader_election.cc:304] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:07.713671 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
W20260808 01:55:07.757470 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 81425.5us
I20260808 01:55:07.716884 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 103) took 1966 ms. Trace:
I20260808 01:55:07.699005 21361 raft_consensus.cc:515] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:07.827138 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 83834.2us
I20260808 01:55:07.718446 21424 raft_consensus.cc:493] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:07.833382 20649 leader_election.cc:304] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:07.740644 21243 consensus_queue.cc:1059] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.091s
W20260808 01:55:07.842377 20648 leader_election.cc:343] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:07.851234 20648 leader_election.cc:304] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:07.751822 21333 raft_consensus.cc:2804] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:55:07.849043 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.854245 20649 leader_election.cc:304] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:07.855662 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.849109 21424 raft_consensus.cc:515] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:07.856449 20606 raft_consensus.cc:2468] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:07.857654 21361 leader_election.cc:290] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:07.848985 21401 raft_consensus.cc:2749] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:07.749233 21334 leader_election.cc:290] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:07.818428 21404 raft_consensus.cc:2804] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:07.869094 21364 consensus_queue.cc:1059] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:07.880117 20811 tablet_bootstrap.cc:492] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:55:07.880889 20811 ts_tablet_manager.cc:1403] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 1.688s	user 0.034s	sys 0.002s
I20260808 01:55:07.840656 21442 raft_consensus.cc:2804] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:07.874877 20606 raft_consensus.cc:1240] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:07.742954 21379 consensus_queue.cc:1059] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.108s
I20260808 01:55:07.873991 20571 leader_election.cc:304] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:55:07.839036 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:07.829663 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
W20260808 01:55:07.837924 20794 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.IsCreateTableDone -> {remote=127.7.165.254:40793, user_credentials={real_user=slave}} blocked reactor thread for 81262.4us
I20260808 01:55:07.889290 20571 leader_election.cc:304] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:55:07.858428 21424 leader_election.cc:290] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:07.849295 20524 raft_consensus.cc:1240] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:07.818470 20681 rpcz_store.cc:276] 0808 01:55:05.747612 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:07.671019 (+1923407us) service_pool.cc:224] Handling call
0808 01:55:07.671458 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:07.694401 (+ 22943us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:07.714302 (+ 19901us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:55:07.900781 20475 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260808 01:55:07.903937 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 60195.3us
I20260808 01:55:07.868670 21343 raft_consensus.cc:2804] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:07.873510 21404 raft_consensus.cc:493] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:07.882494 21333 raft_consensus.cc:697] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:07.888268 21442 raft_consensus.cc:493] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:07.934396 20570 leader_election.cc:304] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:07.932196 20601 raft_consensus.cc:1240] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:07.918445 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:07.943996 20570 leader_election.cc:304] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:07.909705 21283 leader_election.cc:290] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:07.919970 21240 raft_consensus.cc:2804] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:07.930945 21343 raft_consensus.cc:493] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:07.927798 20685 raft_consensus.cc:1275] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:07.910494 20753 raft_consensus.cc:2376] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1. Re-sending same reply.
I20260808 01:55:07.933336 21404 raft_consensus.cc:3060] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:07.941170 21442 raft_consensus.cc:3060] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:55:07.958102 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 88944.7us
I20260808 01:55:07.991091 20648 leader_election.cc:304] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:07.985507 21432 leader_election.cc:290] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:07.961920 21243 raft_consensus.cc:2804] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:08.001075 21243 raft_consensus.cc:493] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:07.987001 21476 raft_consensus.cc:3055] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:08.001540 21243 raft_consensus.cc:3060] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.001947 21476 raft_consensus.cc:740] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:08.003011 21476 consensus_queue.cc:260] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:08.004343 21476 raft_consensus.cc:3060] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:07.956305 21440 raft_consensus.cc:515] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:07.997217 20648 leader_election.cc:304] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
W20260808 01:55:08.011754 20492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:08.008469 21388 raft_consensus.cc:493] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:08.008620 20726 leader_election.cc:304] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:07.955696 20603 raft_consensus.cc:2468] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
W20260808 01:55:07.998481 21462 log.cc:927] Time spent T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260808 01:55:08.043746 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd11a6553c444d488651367f3f0143f" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:08.045819 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:08.046586 21405 leader_election.cc:290] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
W20260808 01:55:08.047861 21468 log.cc:927] Time spent T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 0.176s	user 0.001s	sys 0.000s
I20260808 01:55:08.048949 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
W20260808 01:55:08.055040 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 208613us
I20260808 01:55:08.057237 20811 raft_consensus.cc:359] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:08.060845 20811 raft_consensus.cc:385] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:08.061192 20811 raft_consensus.cc:740] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
I20260808 01:55:08.009572 21270 raft_consensus.cc:2749] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:07.957729 21343 raft_consensus.cc:3060] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:07.954948 21240 raft_consensus.cc:493] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:07.945405 21383 consensus_queue.cc:260] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
W20260808 01:55:08.063935 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 120030us
I20260808 01:55:08.048969 21403 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:08.020470 21351 raft_consensus.cc:493] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:08.019433 21388 raft_consensus.cc:515] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:08.110123 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:08.095544 21352 consensus_queue.cc:1048] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260808 01:55:08.112135 21475 raft_consensus.cc:2749] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:55:08.115455 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 59357.9us
I20260808 01:55:08.067348 20811 consensus_queue.cc:260] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
W20260808 01:55:08.101083 21461 log.cc:927] Time spent T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 0.260s	user 0.000s	sys 0.001s
I20260808 01:55:08.092056 20493 leader_election.cc:304] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:55:08.067895 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:08.037401 21333 consensus_queue.cc:237] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:08.055344 20527 raft_consensus.cc:3060] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:07.993355 21283 consensus_queue.cc:1048] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.077s
I20260808 01:55:08.055692 21361 raft_consensus.cc:2804] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:08.095508 21240 raft_consensus.cc:3060] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:07.987154 21431 raft_consensus.cc:515] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:08.055630 21478 raft_consensus.cc:2804] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 01:55:08.021971 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 119170us
I20260808 01:55:08.160362 20571 leader_election.cc:304] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e
W20260808 01:55:08.123284 20570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:08.023506 21222 leader_election.cc:290] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:08.101689 21403 raft_consensus.cc:3060] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:08.106040 21383 raft_consensus.cc:3060] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:55:08.136837 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 44920.1us
I20260808 01:55:08.095641 21360 raft_consensus.cc:515] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:08.106114 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.135663 21379 raft_consensus.cc:2804] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:08.139134 20601 raft_consensus.cc:3060] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.151150 21352 raft_consensus.cc:2804] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:08.138999 21334 raft_consensus.cc:493] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:08.106477 21351 raft_consensus.cc:515] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
W20260808 01:55:08.181511 20571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:08.159790 21478 raft_consensus.cc:493] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:08.157505 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd11a6553c444d488651367f3f0143f" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.115216 21433 raft_consensus.cc:515] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
W20260808 01:55:08.194986 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 340585us
I20260808 01:55:08.196532 20525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41392 (request call id 117) took 1452 ms. Trace:
W20260808 01:55:08.199474 20649 leader_election.cc:343] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:08.171065 21475 raft_consensus.cc:2764] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260808 01:55:08.157279 21298 consensus_queue.cc:1048] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20260808 01:55:08.207378 20649 leader_election.cc:304] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:08.188552 20571 leader_election.cc:304] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:55:08.201849 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:08.209822 20649 leader_election.cc:304] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:08.209930 20607 raft_consensus.cc:3060] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.178035 21379 raft_consensus.cc:493] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:08.197969 21482 raft_consensus.cc:493] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
W20260808 01:55:08.215085 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 98600.9us
I20260808 01:55:08.216624 20649 leader_election.cc:304] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:08.221580 20649 leader_election.cc:304] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
W20260808 01:55:08.202975 21300 consensus_meta.cc:287] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e: Time spent flushing consensus metadata: real 0.546s	user 0.014s	sys 0.001s
W20260808 01:55:08.200294 21299 consensus_meta.cc:287] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e: Time spent flushing consensus metadata: real 0.548s	user 0.008s	sys 0.005s
I20260808 01:55:08.224130 20649 leader_election.cc:304] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:08.189477 21352 raft_consensus.cc:493] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:08.199900 21423 raft_consensus.cc:2804] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:08.192852 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:08.230450 20530 raft_consensus.cc:2468] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:08.194094 21478 raft_consensus.cc:3060] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:55:08.192179 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 183667us
I20260808 01:55:08.190176 20680 raft_consensus.cc:2468] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
W20260808 01:55:08.241158 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 52697.3us
I20260808 01:55:08.221730 21379 raft_consensus.cc:3060] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.239284 20607 raft_consensus.cc:2468] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3.
I20260808 01:55:08.232156 21352 raft_consensus.cc:3060] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:08.233736 21271 raft_consensus.cc:493] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:08.249742 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:08.219573 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:08.193115 21334 raft_consensus.cc:515] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:08.216938 21485 raft_consensus.cc:2749] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:55:08.233777 21299 raft_consensus.cc:515] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:08.280892 21299 leader_election.cc:290] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:08.283686 21494 raft_consensus.cc:493] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:08.257491 21271 raft_consensus.cc:515] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:08.284304 21494 raft_consensus.cc:515] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:08.228689 21270 raft_consensus.cc:2749] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:08.255537 21489 consensus_queue.cc:1048] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260808 01:55:08.287648 21497 raft_consensus.cc:493] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:08.252641 21283 raft_consensus.cc:2804] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:08.288491 21283 raft_consensus.cc:493] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:08.289031 21283 raft_consensus.cc:3060] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:08.290848 21491 raft_consensus.cc:493] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:08.291479 21491 raft_consensus.cc:515] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:08.293834 21491 leader_election.cc:290] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:08.233335 21370 raft_consensus.cc:2804] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:08.302440 21370 raft_consensus.cc:493] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:08.228291 21482 raft_consensus.cc:515] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:08.270744 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.272935 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:07.584518 20686 rpcz_store.cc:276] 0808 01:55:06.030420 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:07.407128 (+1376708us) service_pool.cc:224] Handling call
0808 01:55:07.409181 (+  2053us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:07.413146 (+  3965us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:55:07.436836 (+ 23690us) log.cc:844] Serialized 14 byte log entry
0808 01:55:07.566576 (+129740us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:07.566629 (+    53us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:07.568067 (+  1438us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:07.568156 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:07.568295 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40343,"prepare.queue_time_us":23979,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":55,"raft.queue_time_us":575,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":3930,"thread_start_us":129011,"threads_started":1}
I20260808 01:55:08.274966 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.275422 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:08.276733 21492 raft_consensus.cc:493] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:08.303982 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:08.338579 20754 raft_consensus.cc:1275] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:08.339941 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.340035 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:08.340621 20754 raft_consensus.cc:2468] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:08.341522 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:08.341763 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
W20260808 01:55:08.256671 21487 log.cc:927] Time spent T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260808 01:55:08.307593 21431 leader_election.cc:290] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:08.342453 20523 raft_consensus.cc:2393] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 3: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:08.307737 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
W20260808 01:55:08.344488 21459 consensus_meta.cc:287] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3: Time spent flushing consensus metadata: real 0.596s	user 0.011s	sys 0.000s
I20260808 01:55:08.344925 21459 raft_consensus.cc:515] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
W20260808 01:55:08.308174 21495 log.cc:927] Time spent T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260808 01:55:08.346781 20679 raft_consensus.cc:2468] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:08.309300 21334 leader_election.cc:290] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:55:08.349100 21350 consensus_meta.cc:287] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e: Time spent flushing consensus metadata: real 0.704s	user 0.003s	sys 0.009s
I20260808 01:55:08.288334 21497 raft_consensus.cc:515] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:08.208875 20525 rpcz_store.cc:276] 0808 01:55:06.744341 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:06.748276 (+  3935us) service_pool.cc:224] Handling call
0808 01:55:07.667843 (+919567us) spinlock_profiling.cc:243] Waited 8.35 ms on lock 0x7b58000d9e98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d4185589e 000000000038d209 00007f5d49914940 00007f5d499dced4 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:55:07.668048 (+   205us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:07.693875 (+ 25827us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:07.695433 (+  1558us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:07.818041 (+122608us) log.cc:844] Serialized 26 byte log entry
0808 01:55:08.053507 (+235466us) spinlock_profiling.cc:243] Waited 3.61 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4186fa5c 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb
0808 01:55:08.131389 (+ 77882us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:08.131453 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:08.131548 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:08.196207 (+ 64659us) raft_consensus.cc:1685] finished
0808 01:55:08.196242 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:08.196422 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5940,"raft.queue_time_us":23059,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":14667,"spinlock_wait_cycles":27506176,"thread_start_us":238377,"threads_started":1,"wal-append.queue_time_us":325005}
I20260808 01:55:08.352398 21398 raft_consensus.cc:493] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:08.352916 21398 raft_consensus.cc:3060] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.301044 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.313164 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.355123 21388 leader_election.cc:290] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:08.313551 20492 leader_election.cc:304] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
W20260808 01:55:08.355990 20492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:08.314947 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.356415 21351 leader_election.cc:290] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:08.356873 20761 raft_consensus.cc:2468] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:55:08.222780 21364 consensus_queue.cc:1059] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.036s
I20260808 01:55:08.357978 21364 raft_consensus.cc:3055] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:55:08.358317 21364 raft_consensus.cc:740] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
W20260808 01:55:08.358790 21441 consensus_meta.cc:287] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc: Time spent flushing consensus metadata: real 0.691s	user 0.010s	sys 0.003s
I20260808 01:55:08.359200 21364 consensus_queue.cc:260] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:08.360520 21364 raft_consensus.cc:3060] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.246088 21405 raft_consensus.cc:2804] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:08.306310 21243 raft_consensus.cc:515] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:08.231596 21477 raft_consensus.cc:2804] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:08.362597 21338 leader_election.cc:290] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:08.362901 21477 raft_consensus.cc:493] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:08.363379 21477 raft_consensus.cc:3060] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.363521 21243 leader_election.cc:290] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:08.365195 21334 raft_consensus.cc:2804] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:08.365768 21334 raft_consensus.cc:697] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [term 3 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:08.365958 21440 leader_election.cc:290] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:08.316012 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.366954 21449 raft_consensus.cc:3055] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:55:08.366731 21334 consensus_queue.cc:237] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:08.367058 20682 raft_consensus.cc:2468] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:08.367375 21449 raft_consensus.cc:740] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:08.368546 21422 raft_consensus.cc:3055] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:55:08.368961 21422 raft_consensus.cc:740] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:08.369801 21422 consensus_queue.cc:260] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:08.371182 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc85af5215440be884412c1d240c7af" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:08.371879 20525 raft_consensus.cc:3060] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:08.381819 20811 ts_tablet_manager.cc:1434] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.501s	user 0.012s	sys 0.007s
W20260808 01:55:08.382282 20811 ts_tablet_manager.cc:1467] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3: Tablet startup took 501ms
I20260808 01:55:08.382349 20571 leader_election.cc:304] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:08.320323 21370 raft_consensus.cc:3060] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:08.286547 21494 leader_election.cc:290] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:08.332947 21437 raft_consensus.cc:493] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:08.338110 20600 raft_consensus.cc:2468] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:55:08.359258 21441 raft_consensus.cc:515] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:08.390619 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a313d0a0979e4dc8a4205c4f16a24257" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.349686 21350 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:08.385020 21492 raft_consensus.cc:515] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:08.390933 21243 raft_consensus.cc:2804] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 01:55:08.324595 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 104659us
I20260808 01:55:08.385881 20571 leader_election.cc:304] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:08.405393 21441 leader_election.cc:290] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:08.401414 21437 raft_consensus.cc:515] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:08.375753 21360 leader_election.cc:290] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:08.342458 20754 raft_consensus.cc:3060] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:08.366534 21440 raft_consensus.cc:493] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:08.363583 20602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:08.397267 21433 leader_election.cc:290] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:08.398826 20684 raft_consensus.cc:2393] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 2: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:08.399945 21482 leader_election.cc:290] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:08.475841 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:55:08.411875 20492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:08.541668 20760 raft_consensus.cc:2468] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
W20260808 01:55:08.449229 20526 consensus_meta.cc:287] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e: Time spent flushing consensus metadata: real 0.610s	user 0.011s	sys 0.001s
I20260808 01:55:08.447674 20522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41396 (request call id 139) took 1025 ms. Trace:
I20260808 01:55:08.559437 20522 rpcz_store.cc:276] 0808 01:55:07.422056 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:07.428938 (+  6882us) service_pool.cc:224] Handling call
0808 01:55:08.447453 (+1018515us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b54003f2fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:08.447599 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":562411392}
I20260808 01:55:08.451123 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:08.542999 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:08.543891 21440 raft_consensus.cc:515] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:08.572602 20685 raft_consensus.cc:2393] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 2: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:08.573761 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:08.574119 21440 leader_election.cc:290] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:08.574994 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:08.575726 20522 raft_consensus.cc:2468] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:08.576812 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:08.577430 20522 raft_consensus.cc:3060] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.449865 21459 leader_election.cc:290] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:08.545828 20602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 182) took 1389 ms. Trace:
I20260808 01:55:08.459544 21405 raft_consensus.cc:493] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:08.410487 20527 raft_consensus.cc:2468] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:08.557977 20754 raft_consensus.cc:2468] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:08.481002 20680 raft_consensus.cc:1240] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:08.437000 21271 leader_election.cc:290] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:08.457913 21423 raft_consensus.cc:493] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:08.477202 21351 consensus_queue.cc:1059] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20260808 01:55:08.548136 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:08.433846 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:08.587374 21405 raft_consensus.cc:3060] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.496044 20758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.452607 20608 raft_consensus.cc:2468] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:08.416960 21442 raft_consensus.cc:515] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:08.486177 21437 leader_election.cc:290] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:55:08.382756 20811 ts_tablet_manager.cc:1469] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3: Trace:
0808 01:54:50.532536 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:54:50.534796 (+  2260us) service_pool.cc:224] Handling call
0808 01:54:50.536687 (+  1891us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0808 01:54:50.536798 (+   111us) ts_tablet_manager.cc:682] Creating new metadata...
0808 01:54:50.561635 (+ 24837us) tablet_metadata.cc:665] Metadata flushed
0808 01:54:50.616821 (+ 55186us) tablet_replica.cc:206] Creating consensus instance
0808 01:54:50.639208 (+ 22387us) inbound_call.cc:177] Queueing success response
0808 01:55:06.192395 (+15553187us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0808 01:55:07.881260 (+1688865us) ts_tablet_manager.cc:1435] Starting tablet replica
0808 01:55:07.985189 (+103929us) tablet_replica.cc:261] Starting instrumentation
0808 01:55:07.985885 (+   696us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":2288,"raft.queue_time_us":150396,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":9071,"tablet-open.queue_time_us":15551148,"thread_start_us":204003,"threads_started":1}
I20260808 01:55:08.503352 21449 consensus_queue.cc:260] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:08.604302 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.604436 21449 raft_consensus.cc:3060] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.605098 20754 raft_consensus.cc:2468] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
W20260808 01:55:08.606041 20650 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:08.607613 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:08.608640 20608 raft_consensus.cc:2393] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 3: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:08.609762 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:08.450387 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.579592 20602 rpcz_store.cc:276] 0808 01:55:07.156245 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:07.165294 (+  9049us) service_pool.cc:224] Handling call
0808 01:55:08.363398 (+1198104us) spinlock_profiling.cc:243] Waited 2.03 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:08.545752 (+182354us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4662144}
I20260808 01:55:08.611234 20755 raft_consensus.cc:2468] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
W20260808 01:55:08.422201 21399 consensus_meta.cc:287] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc: Time spent flushing consensus metadata: real 0.798s	user 0.012s	sys 0.003s
I20260808 01:55:08.405757 20609 raft_consensus.cc:2393] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 3: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:08.612782 21399 raft_consensus.cc:515] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:08.462431 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a313d0a0979e4dc8a4205c4f16a24257" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:08.614074 20606 raft_consensus.cc:3060] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:08.615018 20607 raft_consensus.cc:2393] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 3: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:08.615494 20811 tablet_bootstrap.cc:492] T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3: Bootstrap starting.
I20260808 01:55:08.449218 20528 raft_consensus.cc:1275] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:55:08.439080 21240 raft_consensus.cc:515] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:08.469339 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.535637 21497 leader_election.cc:290] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:08.435226 20601 raft_consensus.cc:2468] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:08.485852 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:08.641002 20530 raft_consensus.cc:3060] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.417538 21243 raft_consensus.cc:493] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:08.480733 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.479177 21361 raft_consensus.cc:493] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:08.501147 21403 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
W20260808 01:55:08.432896 21336 consensus_meta.cc:287] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc: Time spent flushing consensus metadata: real 0.767s	user 0.012s	sys 0.000s
I20260808 01:55:08.422101 21422 raft_consensus.cc:3060] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.476560 20726 leader_election.cc:304] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:08.400971 21283 raft_consensus.cc:515] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:08.431176 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:08.412544 21350 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:08.432984 21404 raft_consensus.cc:515] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:08.464161 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d32821d9c17456394a662fcfd05f6c6" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.415515 21492 leader_election.cc:290] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
W20260808 01:55:08.469187 21290 consensus_meta.cc:287] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e: Time spent flushing consensus metadata: real 0.815s	user 0.012s	sys 0.000s
I20260808 01:55:08.557119 21439 raft_consensus.cc:493] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:08.500389 20426 catalog_manager.cc:5697] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 reported cstate change: term changed from 1 to 2, leader changed from a80df9f992014374aeef30979cfe393e (127.7.165.193) to df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196). New cstate: current_term: 2 leader_uuid: "df82e7f402364d369ed1ec3c5a8772b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:08.661816 20426 catalog_manager.cc:5697] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 reported cstate change: term changed from 1 to 2, leader changed from a80df9f992014374aeef30979cfe393e (127.7.165.193) to df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196). New cstate: current_term: 2 leader_uuid: "df82e7f402364d369ed1ec3c5a8772b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:08.442406 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:08.557889 20529 raft_consensus.cc:2468] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:08.540552 20725 leader_election.cc:304] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:08.564383 20492 leader_election.cc:304] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:08.538004 20425 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48828 (request call id 78) took 5061 ms. Trace:
W20260808 01:55:08.664137 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 99906.1us
I20260808 01:55:08.509248 21352 raft_consensus.cc:515] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:08.410526 21478 raft_consensus.cc:515] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:08.570585 20682 raft_consensus.cc:3060] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.430895 21343 raft_consensus.cc:515] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:08.455768 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.560914 20526 raft_consensus.cc:2468] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:08.570463 21370 raft_consensus.cc:515] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:08.579049 20570 leader_election.cc:304] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:08.670539 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:55:08.718178 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 139219us
I20260808 01:55:08.574446 20685 raft_consensus.cc:3060] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.718415 21482 raft_consensus.cc:2804] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:08.497551 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.718991 21482 raft_consensus.cc:493] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:08.719337 20681 raft_consensus.cc:2468] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:08.482090 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.719769 20570 leader_election.cc:304] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:08.720485 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.720675 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac71e66cb4042eb8fe8aafd139af68a" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:08.721143 21401 raft_consensus.cc:2804] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:08.721251 20678 raft_consensus.cc:3060] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:08.589452 20762 raft_consensus.cc:1240] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:08.721834 21401 raft_consensus.cc:697] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:08.590242 21423 raft_consensus.cc:3060] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.591444 21450 raft_consensus.cc:3055] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:08.591884 20684 raft_consensus.cc:3060] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.723927 21450 raft_consensus.cc:740] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:08.593851 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.591612 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.616590 20605 raft_consensus.cc:1240] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:08.616600 21442 leader_election.cc:290] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:08.640084 20606 raft_consensus.cc:2468] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:55:08.647684 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.726233 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:08.726778 20754 raft_consensus.cc:2468] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:08.727157 20528 raft_consensus.cc:2393] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 3: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:08.647779 21336 raft_consensus.cc:515] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:08.416671 21379 raft_consensus.cc:515] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:08.729161 21336 leader_election.cc:290] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:08.729547 21379 leader_election.cc:290] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:08.724916 21450 consensus_queue.cc:260] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:08.644722 21243 raft_consensus.cc:3060] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.460438 21477 raft_consensus.cc:515] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:08.730516 21401 consensus_queue.cc:237] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:08.646235 21361 raft_consensus.cc:3060] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.665699 20492 leader_election.cc:304] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:08.657075 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.733789 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d8bf19c324463fb9903b9981068e9d" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.734481 20680 raft_consensus.cc:2468] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:55:08.663378 20648 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:08.735526 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.736294 20680 raft_consensus.cc:2468] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:08.660476 21497 raft_consensus.cc:2804] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:08.447075 21430 raft_consensus.cc:2804] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:08.719429 21482 raft_consensus.cc:3060] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:08.477487 21388 consensus_queue.cc:1048] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20260808 01:55:08.653329 21399 leader_election.cc:290] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:08.756316 21382 consensus_queue.cc:1048] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:08.756366 21398 raft_consensus.cc:515] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:08.757851 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc85af5215440be884412c1d240c7af" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.759023 20758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.759923 20603 raft_consensus.cc:1240] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 01:55:08.760131 20677 raft_consensus.cc:1240] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260808 01:55:08.750311 21430 raft_consensus.cc:493] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:08.760910 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:08.746822 21271 raft_consensus.cc:2804] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:08.761861 21271 raft_consensus.cc:697] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 2 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:08.743788 21298 raft_consensus.cc:493] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:55:08.763283 21504 log.cc:927] Time spent T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260808 01:55:08.764091 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.765234 21370 leader_election.cc:290] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:08.765408 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.766216 20756 raft_consensus.cc:2468] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:08.732170 21477 leader_election.cc:290] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:08.759914 21240 leader_election.cc:290] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:08.767364 20686 raft_consensus.cc:1240] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 1. Current term is 2. Ops: [1.1-1.1]
W20260808 01:55:08.668979 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 128513us
I20260808 01:55:08.763341 21298 raft_consensus.cc:515] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:08.771560 20525 raft_consensus.cc:2468] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:08.773072 21352 leader_election.cc:290] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:55:08.735580 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 72308.3us
I20260808 01:55:08.667218 21290 raft_consensus.cc:515] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:08.773985 21388 consensus_queue.cc:1059] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:55:08.774616 20648 leader_election.cc:304] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:08.775729 21290 leader_election.cc:290] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:08.775909 21388 raft_consensus.cc:3055] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Stepping down as leader of term 1
W20260808 01:55:08.704687 20571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:08.730948 20678 raft_consensus.cc:2468] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:55:08.739104 20570 leader_election.cc:304] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:08.751224 20684 raft_consensus.cc:2468] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3.
I20260808 01:55:08.664076 20425 rpcz_store.cc:276] 0808 01:55:03.476324 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:03.476501 (+   177us) service_pool.cc:224] Handling call
0808 01:55:03.822529 (+346028us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b540069e1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fd470 00007f5d4499c31a 00007f5d4a78c38c 00007f5d4a888bfe 00007f5d48d424a4 00007f5d48d42432 00007f5d48d42361 00007f5d48d422dc 00007f5d48d41592 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:08.537920 (+4715391us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:55:06.596806 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:55:06.892063 (+295257us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:55:06.894660 (+  2597us) write_op.cc:435] Acquired schema lock
0808 01:55:06.910443 (+ 15783us) tablet.cc:662] Decoding operations
0808 01:55:07.032203 (+121760us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:55:07.042982 (+ 10779us) write_op.cc:641] Partition lock acquired for write op
0808 01:55:07.051442 (+  8460us) tablet.cc:685] Acquiring locks for 1 operations
0808 01:55:07.178584 (+127142us) tablet.cc:701] Row locks acquired
0808 01:55:07.183868 (+  5284us) write_op.cc:260] PREPARE: finished
0808 01:55:07.225437 (+ 41569us) write_op.cc:270] Start()
0808 01:55:07.247423 (+ 21986us) write_op.cc:276] Timestamp: P: 1786154107222425 usec, L: 0
0808 01:55:07.250001 (+  2578us) op_driver.cc:348] REPLICATION: starting
0808 01:55:07.352223 (+102222us) log.cc:844] Serialized 540 byte log entry
0808 01:55:08.098439 (+746216us) op_driver.cc:464] REPLICATION: finished
0808 01:55:08.273479 (+175040us) write_op.cc:301] APPLY: starting
0808 01:55:08.273564 (+    85us) tablet.cc:1367] starting BulkCheckPresence
0808 01:55:08.273661 (+    97us) tablet.cc:1370] finished BulkCheckPresence
0808 01:55:08.273792 (+   131us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:55:08.274047 (+   255us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:55:08.274219 (+   172us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0808 01:55:08.274263 (+    44us) write_op.cc:312] APPLY: finished
0808 01:55:08.274442 (+   179us) log.cc:844] Serialized 22 byte log entry
0808 01:55:08.274610 (+   168us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:55:08.274738 (+   128us) write_op.cc:454] Released schema lock
0808 01:55:08.274780 (+    42us) write_op.cc:341] FINISH: Updating metrics
0808 01:55:08.275197 (+   417us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:55:08.275224 (+    27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":438880,"reported_tablets":16,"spinlock_wait_cycles":654621056,"child_traces":[["op",{"apply.queue_time_us":169473,"apply.run_cpu_time_us":1725,"apply.run_wall_time_us":1726,"num_ops":1,"prepare.queue_time_us":214093,"prepare.run_cpu_time_us":16332,"prepare.run_wall_time_us":1085319,"raft.queue_time_us":278583,"raft.run_cpu_time_us":17160,"raft.run_wall_time_us":200311,"replication_time_us":803479,"thread_start_us":613464,"threads_started":4,"wal-append.queue_time_us":351790}]]}
I20260808 01:55:08.657510 20524 raft_consensus.cc:3060] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.799775 21399 raft_consensus.cc:2749] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:08.751151 21497 raft_consensus.cc:493] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:55:08.744632 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 79008.8us
I20260808 01:55:08.717190 20608 raft_consensus.cc:2393] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 4: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:08.809552 21497 raft_consensus.cc:3060] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:55:08.658829 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 182337us
I20260808 01:55:08.776628 21493 raft_consensus.cc:2749] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:08.815037 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.816007 20760 raft_consensus.cc:2393] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 3: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:08.817039 21450 raft_consensus.cc:3060] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:08.817638 20524 raft_consensus.cc:2468] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:08.811790 20492 leader_election.cc:304] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:08.671154 21350 raft_consensus.cc:2804] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:08.749414 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.672662 21492 raft_consensus.cc:2804] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election won for term 3
W20260808 01:55:08.717188 21484 log.cc:927] Time spent T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260808 01:55:08.828725 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:08.670156 21439 raft_consensus.cc:515] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:08.666937 21424 raft_consensus.cc:493] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:08.752714 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.809556 21478 leader_election.cc:290] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:08.762919 21271 consensus_queue.cc:237] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:08.668409 20759 raft_consensus.cc:2468] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:08.721845 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.776232 20648 leader_election.cc:304] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:08.736847 20685 raft_consensus.cc:2468] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3.
I20260808 01:55:08.777081 20725 leader_election.cc:304] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
W20260808 01:55:08.844255 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 67270.7us
I20260808 01:55:08.845062 20725 leader_election.cc:304] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
W20260808 01:55:08.846177 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 70014.1us
I20260808 01:55:08.846879 20725 leader_election.cc:304] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:08.852062 20529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41410 (request call id 148) took 1243 ms. Trace:
I20260808 01:55:08.844830 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.856812 20753 raft_consensus.cc:2410] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc for term 3 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20260808 01:55:08.857831 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 44012.4us
I20260808 01:55:08.836213 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:08.825992 21350 raft_consensus.cc:493] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:08.736943 21442 raft_consensus.cc:2804] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:08.667371 21364 consensus_queue.cc:1048] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.083s
W20260808 01:55:08.797842 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 58820.6us
I20260808 01:55:08.686318 20530 raft_consensus.cc:2468] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:08.745922 20526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41392 (request call id 126) took 1042 ms. Trace:
W20260808 01:55:08.823727 20492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:08.790961 21430 raft_consensus.cc:3060] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:08.896660 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.826411 20761 raft_consensus.cc:2410] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260808 01:55:08.787441 21403 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:08.792500 21388 raft_consensus.cc:740] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:08.783851 21398 leader_election.cc:290] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:08.753816 21343 leader_election.cc:290] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:08.826926 21222 raft_consensus.cc:2804] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:55:08.872691 21243 raft_consensus.cc:515] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:08.846488 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44768 (request call id 131) took 1352 ms. Trace:
I20260808 01:55:08.834941 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:08.873760 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.844542 21424 raft_consensus.cc:515] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:08.863196 21405 raft_consensus.cc:515] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:08.847920 20648 leader_election.cc:304] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:55:08.868839 21449 raft_consensus.cc:2804] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:08.869323 21439 leader_election.cc:290] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:08.838073 21492 raft_consensus.cc:697] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:08.880752 21240 raft_consensus.cc:2804] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:08.862946 21502 raft_consensus.cc:2749] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:08.860765 21351 consensus_queue.cc:1048] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:55:08.858918 21379 raft_consensus.cc:2804] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:08.881825 20811 tablet_bootstrap.cc:654] T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3: Neither blocks nor log segments found. Creating new log.
I20260808 01:55:08.871577 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:08.860399 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:08.862378 20725 leader_election.cc:304] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:08.897914 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44768 (request call id 142) took 1031 ms. Trace:
I20260808 01:55:08.871052 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:55:08.984128 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 121854us
I20260808 01:55:08.870457 21283 leader_election.cc:290] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:08.906473 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.868181 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:08.828320 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.852831 20529 rpcz_store.cc:276] 0808 01:55:07.609068 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:07.610769 (+  1701us) service_pool.cc:224] Handling call
0808 01:55:08.851798 (+1241029us) spinlock_profiling.cc:243] Waited 946 ms on lock 0x7b5400030554. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:08.851969 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2175933312}
I20260808 01:55:08.770278 20522 raft_consensus.cc:2468] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:08.912075 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.903735 21459 raft_consensus.cc:493] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:08.915558 20570 leader_election.cc:304] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:08.913071 21350 raft_consensus.cc:3060] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:08.890161 21482 raft_consensus.cc:515] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:08.760978 21404 leader_election.cc:290] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:08.921984 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.928766 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:08.998656 20761 raft_consensus.cc:2393] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 3: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:08.985342 21451 raft_consensus.cc:2749] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:09.001653 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:08.847884 20679 raft_consensus.cc:2376] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3. Re-sending same reply.
I20260808 01:55:08.913403 21442 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:08.928804 21491 raft_consensus.cc:493] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:08.914577 20682 raft_consensus.cc:2468] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:08.982858 21361 raft_consensus.cc:515] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:08.967553 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 119760us
I20260808 01:55:08.954790 21478 raft_consensus.cc:493] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:08.916371 20526 rpcz_store.cc:276] 0808 01:55:07.701342 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:07.734161 (+ 32819us) service_pool.cc:224] Handling call
0808 01:55:08.743809 (+1009648us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:08.947960 20677 raft_consensus.cc:2468] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:08.957377 21298 leader_election.cc:290] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:08.978664 21388 consensus_queue.cc:260] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:08.962823 21222 raft_consensus.cc:697] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
W20260808 01:55:08.981329 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 134039us
I20260808 01:55:08.986043 20725 leader_election.cc:304] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:08.989967 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:08.976411 21500 raft_consensus.cc:493] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:08.983417 20684 raft_consensus.cc:2393] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 3: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:08.984050 20680 raft_consensus.cc:2393] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 3: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:09.028762 20571 leader_election.cc:304] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:55:09.029263 20725 leader_election.cc:304] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
W20260808 01:55:08.992780 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 77411.8us
I20260808 01:55:08.993256 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:08.976989 21449 raft_consensus.cc:697] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:08.978549 21379 raft_consensus.cc:493] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:09.008921 21491 raft_consensus.cc:515] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:08.982259 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:08.985423 20759 raft_consensus.cc:3060] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.911542 21456 raft_consensus.cc:493] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:08.987188 20678 raft_consensus.cc:3060] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:08.974828 21496 consensus_queue.cc:1059] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260808 01:55:08.976037 21423 raft_consensus.cc:515] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:08.953696 20605 rpcz_store.cc:276] 0808 01:55:07.494132 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:07.505828 (+ 11696us) service_pool.cc:224] Handling call
0808 01:55:08.270643 (+764815us) spinlock_profiling.cc:243] Waited 3.61 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d4185589e 000000000038d209 00007f5d49914940 00007f5d499dced4 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:55:08.616354 (+345711us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:08.725479 (+109125us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:08.834639 (+109160us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b54003912d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:08.843019 (+  8380us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":765360768}
I20260808 01:55:08.921680 20523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41392 (request call id 130) took 1058 ms. Trace:
I20260808 01:55:09.010962 20685 raft_consensus.cc:2410] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:55:09.032949 21499 raft_consensus.cc:2749] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:08.997225 21459 raft_consensus.cc:515] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:09.011814 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:09.009109 21493 raft_consensus.cc:2764] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260808 01:55:09.009820 20648 leader_election.cc:304] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:55:08.999162 21401 raft_consensus.cc:2804] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:08.988305 21494 consensus_queue.cc:1048] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260808 01:55:09.017230 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:09.055222 20683 raft_consensus.cc:2393] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 3: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:08.974311 21240 raft_consensus.cc:493] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:08.991449 21497 raft_consensus.cc:515] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:09.059173 21240 raft_consensus.cc:3060] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:09.059468 21405 leader_election.cc:290] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:09.059995 20755 raft_consensus.cc:2393] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 3: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:09.059082 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:09.061657 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:09.063024 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:09.007302 21442 raft_consensus.cc:3060] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:09.071069 20686 raft_consensus.cc:2393] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 3: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:09.067716 21401 raft_consensus.cc:697] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [term 2 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:09.073359 20762 raft_consensus.cc:2468] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:09.073280 21401 consensus_queue.cc:237] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:09.073638 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:09.074416 20755 raft_consensus.cc:3060] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:09.064450 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:09.075102 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d8bf19c324463fb9903b9981068e9d" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:09.074846 21449 consensus_queue.cc:237] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:09.076695 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a313d0a0979e4dc8a4205c4f16a24257" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:08.984308 20603 rpcz_store.cc:276] 0808 01:55:07.866246 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:08.111311 (+245065us) service_pool.cc:224] Handling call
0808 01:55:08.111717 (+   406us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:08.760494 (+648777us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:08.897579 (+137085us) spinlock_profiling.cc:243] Waited 632 ms on lock 0x7b540005b254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:08.897839 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1453150464}
I20260808 01:55:09.032418 20725 leader_election.cc:304] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
W20260808 01:55:08.995577 20475 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:55:09.077150 20683 raft_consensus.cc:1240] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:09.035290 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
W20260808 01:55:09.081383 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 49043.5us
I20260808 01:55:09.038790 20571 leader_election.cc:304] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:55:09.042383 20726 leader_election.cc:304] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:09.033181 21379 raft_consensus.cc:3060] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:09.054405 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:09.021225 21478 raft_consensus.cc:515] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:09.103134 21398 raft_consensus.cc:2804] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:09.103500 20685 raft_consensus.cc:2393] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 3: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:09.103688 21398 raft_consensus.cc:493] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:09.104110 21398 raft_consensus.cc:3060] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:09.104828 21478 leader_election.cc:290] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:55:09.011447 21514 log.cc:927] Time spent T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260808 01:55:09.081768 21494 raft_consensus.cc:2749] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:09.106308 21494 consensus_queue.cc:1048] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:55:09.106806 20649 leader_election.cc:304] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:09.108387 20649 leader_election.cc:304] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:09.033617 21298 raft_consensus.cc:2749] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:55:09.033804 21245 raft_consensus.cc:2804] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:09.041673 21456 raft_consensus.cc:515] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:09.014065 21492 consensus_queue.cc:237] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:09.051703 20648 leader_election.cc:304] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:55:09.108691 21422 raft_consensus.cc:2749] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:09.037658 21351 consensus_queue.cc:1059] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260808 01:55:09.063877 21243 leader_election.cc:290] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:09.034348 21283 consensus_queue.cc:1059] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
W20260808 01:55:09.024174 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 186649us
I20260808 01:55:09.055263 20757 raft_consensus.cc:2468] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:09.030277 20570 leader_election.cc:304] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:09.053185 21364 raft_consensus.cc:2764] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260808 01:55:09.032835 21500 raft_consensus.cc:515] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:09.030269 20760 raft_consensus.cc:3060] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 3
W20260808 01:55:09.096441 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 57787.8us
I20260808 01:55:09.116087 21493 raft_consensus.cc:2804] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election won for term 1
W20260808 01:55:09.114136 20725 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:55:09.117647 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 66094us
I20260808 01:55:09.038403 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:09.083029 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc85af5215440be884412c1d240c7af" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:09.124254 21477 raft_consensus.cc:2804] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:09.081873 21423 leader_election.cc:290] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:09.079736 21459 leader_election.cc:290] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:09.067528 21222 consensus_queue.cc:237] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:09.132539 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:09.136118 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 105934us
I20260808 01:55:09.171816 20529 raft_consensus.cc:2468] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:09.137228 20762 raft_consensus.cc:2468] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 0.
I20260808 01:55:09.114472 21245 raft_consensus.cc:493] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:09.114042 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:09.172006 20570 leader_election.cc:304] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:09.178180 20570 leader_election.cc:304] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:09.140237 20571 leader_election.cc:304] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
W20260808 01:55:09.109280 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 66989us
I20260808 01:55:09.145494 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a313d0a0979e4dc8a4205c4f16a24257" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:09.088960 21442 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:09.186631 21496 raft_consensus.cc:493] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:09.186829 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:09.187726 20524 raft_consensus.cc:2393] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 3: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:09.187851 21442 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:09.188092 21271 raft_consensus.cc:493] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:09.188740 21271 raft_consensus.cc:515] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:09.176162 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:09.191761 20528 raft_consensus.cc:2393] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 3: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:09.192706 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:09.193274 20528 raft_consensus.cc:3060] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:09.195518 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:09.113933 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:09.166088 21450 raft_consensus.cc:493] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:09.142460 21243 raft_consensus.cc:3055] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:09.139374 21499 raft_consensus.cc:2804] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:09.137373 21482 leader_election.cc:290] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:09.184844 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:09.185111 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:09.166706 21493 raft_consensus.cc:697] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
W20260808 01:55:09.197009 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 56808.6us
I20260808 01:55:09.234215 20571 leader_election.cc:304] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:09.141294 21403 raft_consensus.cc:2804] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:09.180866 20760 raft_consensus.cc:2468] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3.
I20260808 01:55:09.176630 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:09.164803 20648 leader_election.cc:304] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:09.148154 21404 raft_consensus.cc:493] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:09.166486 21477 raft_consensus.cc:493] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:09.223759 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:09.220103 21433 raft_consensus.cc:2749] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:09.245671 21399 raft_consensus.cc:2749] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:09.247490 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:09.261482 20760 raft_consensus.cc:2393] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 3: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:09.248833 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:09.264081 20685 raft_consensus.cc:1275] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:09.250295 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:09.265512 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:09.265785 20753 raft_consensus.cc:2468] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:09.251050 21403 raft_consensus.cc:697] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:09.267217 21494 raft_consensus.cc:3055] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:55:09.267618 21494 raft_consensus.cc:740] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:09.268051 21403 consensus_queue.cc:237] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:09.268478 21494 consensus_queue.cc:260] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:09.269752 21494 raft_consensus.cc:3060] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:09.270802 20684 raft_consensus.cc:2393] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 3: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:09.272086 20649 leader_election.cc:304] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:09.253516 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:09.177311 21361 leader_election.cc:290] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:09.175457 21430 raft_consensus.cc:515] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:09.254220 21404 raft_consensus.cc:515] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:09.264251 20761 raft_consensus.cc:2393] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 3: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:09.253963 21477 raft_consensus.cc:3060] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:09.196499 21283 raft_consensus.cc:2804] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:09.187337 21496 raft_consensus.cc:515] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:09.275015 20600 raft_consensus.cc:1240] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:09.276273 21430 leader_election.cc:290] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:09.279922 20726 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:09.176548 21245 raft_consensus.cc:3060] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:09.280903 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:09.221062 20526 raft_consensus.cc:1240] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:09.201377 21478 raft_consensus.cc:2804] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election won for term 3
W20260808 01:55:09.232637 21518 log.cc:927] Time spent T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.190s	user 0.001s	sys 0.000s
W20260808 01:55:09.234763 20650 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:09.290812 20493 leader_election.cc:304] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:09.227660 21243 raft_consensus.cc:740] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:09.233726 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:09.364995 21271 leader_election.cc:290] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:09.044096 20523 rpcz_store.cc:276] 0808 01:55:07.862971 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:08.343743 (+480772us) service_pool.cc:224] Handling call
0808 01:55:08.344270 (+   527us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:08.344520 (+   250us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:55:08.674868 (+330348us) log.cc:844] Serialized 14 byte log entry
0808 01:55:08.903090 (+228222us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:08.906863 (+  3773us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:08.911233 (+  4370us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:08.911347 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:08.916913 (+  5566us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":255989,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"raft.queue_time_us":32653,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":108,"thread_start_us":409778,"threads_started":2,"wal-append.queue_time_us":240787}
I20260808 01:55:09.249276 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:09.247110 20755 raft_consensus.cc:2468] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:09.287276 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:09.350157 21519 log.cc:927] Time spent T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc: Append to log took a long time: real 0.313s	user 0.000s	sys 0.001s
I20260808 01:55:09.233830 21450 raft_consensus.cc:515] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:09.398605 21491 leader_election.cc:290] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:09.411266 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:09.255774 21493 consensus_queue.cc:237] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:09.303702 21333 consensus_queue.cc:1048] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.119s
I20260808 01:55:09.292650 20525 raft_consensus.cc:2468] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
W20260808 01:55:09.448339 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 283628us
I20260808 01:55:09.452486 20528 raft_consensus.cc:2468] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:09.258384 21502 raft_consensus.cc:3055] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:55:09.222648 20681 raft_consensus.cc:2468] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3.
I20260808 01:55:09.196884 21459 raft_consensus.cc:493] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
W20260808 01:55:09.458616 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 186609us
I20260808 01:55:09.462165 21379 raft_consensus.cc:515] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:09.491813 21478 raft_consensus.cc:697] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:09.478662 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
W20260808 01:55:09.482309 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 248163us
I20260808 01:55:09.542546 20571 leader_election.cc:304] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:09.442629 21361 raft_consensus.cc:493] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:09.479992 20754 raft_consensus.cc:3060] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:09.465016 21364 raft_consensus.cc:2804] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 01:55:09.496383 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 205711us
I20260808 01:55:09.442322 21283 raft_consensus.cc:697] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:09.476492 21388 raft_consensus.cc:3060] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:09.504719 21404 leader_election.cc:290] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:09.466223 20759 raft_consensus.cc:2468] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:09.554579 20571 leader_election.cc:304] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:09.507858 20757 raft_consensus.cc:1240] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 LEADER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:09.509608 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:09.518393 20492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:09.525931 20570 leader_election.cc:304] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:09.484392 20678 raft_consensus.cc:2468] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:09.458360 21425 raft_consensus.cc:2749] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:09.513746 21271 raft_consensus.cc:2764] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260808 01:55:09.553606 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1753378f13594e89aee6641e2cb9e6da" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:09.547281 21364 raft_consensus.cc:493] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:09.535055 21502 raft_consensus.cc:740] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:09.564366 20685 raft_consensus.cc:1240] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260808 01:55:09.564941 20571 leader_election.cc:304] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:09.543296 21456 leader_election.cc:290] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:09.556849 20762 raft_consensus.cc:3060] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:09.550480 21399 raft_consensus.cc:2749] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:09.581913 20571 leader_election.cc:304] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:09.587756 20571 leader_election.cc:304] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:55:09.544895 21459 raft_consensus.cc:515] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:09.597023 20811 tablet_bootstrap.cc:492] T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3: No bootstrap required, opened a new log
I20260808 01:55:09.597898 20811 ts_tablet_manager.cc:1403] T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3: Time spent bootstrapping tablet: real 0.983s	user 0.025s	sys 0.049s
I20260808 01:55:09.602461 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac71e66cb4042eb8fe8aafd139af68a" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:09.603956 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:09.604533 20528 raft_consensus.cc:3060] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:09.606041 20762 raft_consensus.cc:2468] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 3.
I20260808 01:55:09.550318 21361 raft_consensus.cc:515] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:09.572826 21430 raft_consensus.cc:3060] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:55:09.574514 21240 consensus_meta.cc:287] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb: Time spent flushing consensus metadata: real 0.515s	user 0.011s	sys 0.003s
I20260808 01:55:09.570896 21500 leader_election.cc:290] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:09.547268 21243 consensus_queue.cc:260] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:09.615079 21502 consensus_queue.cc:260] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:09.644532 21450 leader_election.cc:290] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:09.644941 21424 leader_election.cc:290] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:09.645216 21524 log.cc:927] Time spent T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260808 01:55:09.577203 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:09.647053 20525 raft_consensus.cc:2376] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1. Re-sending same reply.
I20260808 01:55:09.648072 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:09.619227 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:09.657577 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:09.659188 20679 raft_consensus.cc:1240] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:09.664095 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:09.585431 21364 raft_consensus.cc:3060] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:09.671929 20677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46954 (request call id 232) took 1154 ms. Trace:
I20260808 01:55:09.600862 20523 raft_consensus.cc:2368] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for earlier term 2. Current term is 3.
I20260808 01:55:09.602471 21283 consensus_queue.cc:237] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:09.631378 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:09.632261 20811 raft_consensus.cc:359] T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:09.677022 20811 raft_consensus.cc:385] T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 01:55:09.588472 21526 raft_consensus.cc:2804] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:09.635684 21245 raft_consensus.cc:515] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:09.635946 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:09.585196 21403 raft_consensus.cc:2749] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:09.638568 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:09.642789 21379 leader_election.cc:290] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:09.683606 20686 raft_consensus.cc:2468] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3.
I20260808 01:55:09.643296 21243 raft_consensus.cc:3060] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:09.643664 21361 leader_election.cc:290] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:09.615675 21499 raft_consensus.cc:493] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:09.646377 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d32821d9c17456394a662fcfd05f6c6" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:09.596993 21401 raft_consensus.cc:2749] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:55:09.620311 21496 leader_election.cc:290] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:09.650753 21399 consensus_queue.cc:1175] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "df82e7f402364d369ed1ec3c5a8772b3" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20260808 01:55:09.653985 21497 leader_election.cc:290] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:09.654513 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:09.656229 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:09.600126 21433 raft_consensus.cc:2804] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:09.619690 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:09.588657 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:09.665520 21482 raft_consensus.cc:493] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
W20260808 01:55:09.666124 20728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:09.644548 21502 raft_consensus.cc:3060] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:55:09.667953 21522 log.cc:927] Time spent T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.157s	user 0.002s	sys 0.002s
W20260808 01:55:09.669345 21398 consensus_meta.cc:287] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc: Time spent flushing consensus metadata: real 0.565s	user 0.012s	sys 0.003s
I20260808 01:55:09.629712 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
W20260808 01:55:09.608414 21350 consensus_meta.cc:287] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e: Time spent flushing consensus metadata: real 0.606s	user 0.006s	sys 0.006s
I20260808 01:55:09.678761 20606 raft_consensus.cc:1275] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:09.740589 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac71e66cb4042eb8fe8aafd139af68a" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:09.741379 20606 raft_consensus.cc:2393] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:09.677323 20811 raft_consensus.cc:740] T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Initialized, Role: FOLLOWER
W20260808 01:55:09.690922 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 138255us
I20260808 01:55:09.691915 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d8bf19c324463fb9903b9981068e9d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:09.743055 20686 raft_consensus.cc:2468] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 0.
I20260808 01:55:09.672169 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44770 (request call id 203) took 1251 ms. Trace:
W20260808 01:55:09.694617 20725 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:55:09.696810 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 150050us
I20260808 01:55:09.747251 20493 leader_election.cc:304] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:09.703840 20528 raft_consensus.cc:2468] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 4.
I20260808 01:55:09.715193 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:09.721520 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:09.724072 20522 raft_consensus.cc:2393] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 3: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:09.733518 21245 leader_election.cc:290] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:09.675961 21240 raft_consensus.cc:515] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:09.706319 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:09.612923 21478 consensus_queue.cc:237] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:09.772346 20605 raft_consensus.cc:3060] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:09.735975 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:09.727073 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:09.731710 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:09.707265 21499 raft_consensus.cc:3060] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:09.748885 20648 leader_election.cc:304] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:09.749115 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:09.779641 20528 raft_consensus.cc:3060] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:09.780674 20760 raft_consensus.cc:3055] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:09.781091 20760 raft_consensus.cc:740] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:09.777015 20493 leader_election.cc:304] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:09.750340 20570 leader_election.cc:304] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:09.719213 21423 raft_consensus.cc:493] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:09.693715 21449 consensus_queue.cc:1059] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260808 01:55:09.743657 20811 consensus_queue.cc:260] T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:09.753850 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:09.718734 21433 raft_consensus.cc:493] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:09.754988 20726 leader_election.cc:304] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:09.731756 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:09.759153 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:09.760370 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:09.762279 20602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63fb8c2f494a462686500ce10025a39f" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:09.739606 21398 raft_consensus.cc:515] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:09.763697 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:55:09.805832 21531 log.cc:927] Time spent T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260808 01:55:09.767278 20753 raft_consensus.cc:2393] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 3: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:09.768429 20524 raft_consensus.cc:1240] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260808 01:55:09.768937 20526 raft_consensus.cc:2393] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 4: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:09.736833 21482 raft_consensus.cc:515] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:09.743839 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:09.770845 21497 consensus_queue.cc:1059] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260808 01:55:09.743126 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:09.731873 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:09.771729 21240 leader_election.cc:290] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:09.776074 20761 raft_consensus.cc:2410] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:55:09.843145 20683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46954 (request call id 233) took 1325 ms. Trace:
I20260808 01:55:09.844472 20761 raft_consensus.cc:3055] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:09.843645 20683 rpcz_store.cc:276] 0808 01:55:08.517765 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:09.092463 (+574698us) service_pool.cc:224] Handling call
0808 01:55:09.195997 (+103534us) spinlock_profiling.cc:243] Waited 18.2 ms on lock 0x7b5800775598. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:55:09.196264 (+   267us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:09.196654 (+   390us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:09.196682 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:09.228632 (+ 31950us) log.cc:844] Serialized 26 byte log entry
0808 01:55:09.639676 (+411044us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:09.639734 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:09.639824 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:09.842795 (+202971us) raft_consensus.cc:1685] finished
0808 01:55:09.842828 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:09.843074 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":311,"raft.queue_time_us":4922,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":13407,"spinlock_wait_cycles":41899776,"thread_start_us":191008,"threads_started":1,"wal-append.queue_time_us":454519}
I20260808 01:55:09.799238 20530 raft_consensus.cc:1240] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260808 01:55:09.736411 21496 consensus_queue.cc:1059] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260808 01:55:09.803684 20528 raft_consensus.cc:2468] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 3.
I20260808 01:55:09.845819 21496 raft_consensus.cc:3055] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:55:09.800714 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d32821d9c17456394a662fcfd05f6c6" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:09.846215 21496 raft_consensus.cc:740] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:09.800571 21245 raft_consensus.cc:2804] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:55:09.847512 21245 raft_consensus.cc:697] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 4 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:09.847187 21496 consensus_queue.cc:260] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:09.805320 21430 raft_consensus.cc:2749] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:09.848600 21496 raft_consensus.cc:3060] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:09.766781 20525 raft_consensus.cc:2393] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 3: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:09.849000 21333 raft_consensus.cc:2764] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260808 01:55:09.850224 20525 raft_consensus.cc:1240] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:09.810745 20811 ts_tablet_manager.cc:1434] T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3: Time spent starting tablet: real 0.212s	user 0.007s	sys 0.000s
I20260808 01:55:09.851665 20520 raft_consensus.cc:1240] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:09.800279 20760 consensus_queue.cc:260] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:09.852806 20760 raft_consensus.cc:3060] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:09.822132 21401 raft_consensus.cc:493] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:09.853322 20520 raft_consensus.cc:1275] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:09.853883 21401 raft_consensus.cc:515] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:09.672415 20677 rpcz_store.cc:276] 0808 01:55:08.517388 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:09.078899 (+561511us) service_pool.cc:224] Handling call
0808 01:55:09.082099 (+  3200us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:09.099006 (+ 16907us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:09.099048 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:09.099601 (+   553us) log.cc:844] Serialized 26 byte log entry
0808 01:55:09.256204 (+156603us) spinlock_profiling.cc:243] Waited 3.19 ms on lock 0x7b18001901e0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:55:09.532310 (+276106us) raft_consensus.cc:1645] Marking committed up to 0
0808 01:55:09.532405 (+    95us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:09.536979 (+  4574us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:09.671601 (+134622us) raft_consensus.cc:1685] finished
0808 01:55:09.671635 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:09.671864 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8635,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":130,"spinlock_wait_cycles":7331200,"thread_start_us":260251,"threads_started":1,"wal-append.queue_time_us":513056}
I20260808 01:55:09.855875 21398 leader_election.cc:290] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:09.855993 21403 raft_consensus.cc:493] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:09.803936 20648 leader_election.cc:400] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Vote denied by peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531) with higher term. Message: Invalid argument: T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for earlier term 2. Current term is 3.
I20260808 01:55:09.752095 21350 raft_consensus.cc:515] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:09.797497 21433 raft_consensus.cc:3060] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:09.825057 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:09.857558 20601 raft_consensus.cc:2393] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 3: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:09.858438 21350 leader_election.cc:290] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:09.858562 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:09.815704 20680 raft_consensus.cc:2393] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 3: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:09.816576 21526 raft_consensus.cc:697] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:09.805886 20602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:09.806178 21450 raft_consensus.cc:2804] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260808 01:55:09.832414 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 55497.7us
I20260808 01:55:09.820886 20754 raft_consensus.cc:2468] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:09.840160 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
W20260808 01:55:09.841434 21541 log.cc:927] Time spent T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260808 01:55:09.844909 20761 raft_consensus.cc:740] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:09.783524 21442 raft_consensus.cc:2749] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:55:09.778415 21222 raft_consensus.cc:493] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:09.856771 20677 raft_consensus.cc:1240] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260808 01:55:09.798398 21423 raft_consensus.cc:515] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:09.856794 20648 leader_election.cc:403] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:55:09.786804 21351 raft_consensus.cc:493] T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:55:09.827203 21477 consensus_meta.cc:287] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb: Time spent flushing consensus metadata: real 0.516s	user 0.011s	sys 0.003s
I20260808 01:55:09.751821 20600 rpcz_store.cc:276] 0808 01:55:08.420821 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:08.473263 (+ 52442us) service_pool.cc:224] Handling call
0808 01:55:08.473621 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:09.468788 (+995167us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:09.671898 (+203110us) spinlock_profiling.cc:243] Waited 801 ms on lock 0x7b540005b754. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:09.672105 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1842440576}
I20260808 01:55:09.849436 20683 raft_consensus.cc:1275] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:09.859984 20757 raft_consensus.cc:2468] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:09.819538 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:09.867987 20529 raft_consensus.cc:2393] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 2: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:09.853518 21497 raft_consensus.cc:3055] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:55:09.899850 21497 raft_consensus.cc:740] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:09.900882 21497 consensus_queue.cc:260] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:09.902087 21497 raft_consensus.cc:3060] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:09.870141 20760 raft_consensus.cc:1275] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:09.825891 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:09.867336 20523 raft_consensus.cc:2368] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for earlier term 3. Current term is 4.
I20260808 01:55:09.881178 21450 raft_consensus.cc:493] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:09.874420 20493 leader_election.cc:304] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
W20260808 01:55:09.870390 20728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:09.884922 20606 raft_consensus.cc:2468] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 3.
I20260808 01:55:09.882375 20761 consensus_queue.cc:260] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:09.799873 21538 raft_consensus.cc:2804] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260808 01:55:09.893208 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 108548us
I20260808 01:55:09.917158 20648 leader_election.cc:304] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:09.893920 21442 raft_consensus.cc:2749] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531) with higher term. Message: Invalid argument: T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for earlier term 2. Current term is 3.
I20260808 01:55:09.871555 21398 consensus_queue.cc:1048] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260808 01:55:09.895996 20683 raft_consensus.cc:3055] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:55:09.925897 20683 raft_consensus.cc:740] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:09.905022 20528 raft_consensus.cc:1275] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:09.892257 20759 raft_consensus.cc:2393] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 3: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:09.908885 20493 leader_election.cc:304] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:09.911753 20761 raft_consensus.cc:3060] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:09.897643 21351 raft_consensus.cc:515] T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:09.898241 20601 raft_consensus.cc:2393] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 3: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:09.939105 20761 raft_consensus.cc:1275] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260808 01:55:09.898022 21245 consensus_queue.cc:237] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:09.856694 21403 raft_consensus.cc:515] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:09.894776 21477 raft_consensus.cc:515] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:09.943485 20686 raft_consensus.cc:2393] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 3: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:09.944247 21477 leader_election.cc:290] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:09.926743 20683 consensus_queue.cc:260] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:09.944834 20572 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:09.916349 21538 raft_consensus.cc:493] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:09.945286 20683 raft_consensus.cc:3060] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:09.945709 21538 raft_consensus.cc:3060] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:09.888124 21222 raft_consensus.cc:515] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:09.946940 21401 leader_election.cc:290] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:09.947149 21499 raft_consensus.cc:515] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:09.910318 20679 raft_consensus.cc:2468] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:09.884802 20602 raft_consensus.cc:3060] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:09.948678 21222 leader_election.cc:290] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:09.949291 21430 consensus_queue.cc:1048] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:55:09.910939 21450 raft_consensus.cc:3060] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:09.952968 20605 raft_consensus.cc:2468] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:09.954383 21364 raft_consensus.cc:515] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:09.956976 20604 raft_consensus.cc:1240] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:09.958808 21499 leader_election.cc:290] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:09.931260 20684 raft_consensus.cc:2368] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3.
I20260808 01:55:09.931704 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:09.933175 21482 leader_election.cc:290] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:09.910849 21350 raft_consensus.cc:2749] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:55:09.935793 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46954 (request call id 227) took 1461 ms. Trace:
I20260808 01:55:09.937330 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:09.937600 20526 raft_consensus.cc:1275] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260808 01:55:09.939498 20761 pending_rounds.cc:85] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3: Aborting all ops after (but not including) 0
I20260808 01:55:09.925264 20648 leader_election.cc:304] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:09.939646 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:09.930984 21423 leader_election.cc:290] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:09.929121 21240 raft_consensus.cc:2749] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:09.969444 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:09.970623 20608 raft_consensus.cc:2410] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260808 01:55:09.972036 21364 leader_election.cc:290] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:55:09.973505 20726 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:55:10.030444 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 121645us
I20260808 01:55:10.020594 20761 pending_rounds.cc:107] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260808 01:55:10.008739 20426 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48860 (request call id 74) took 5209 ms. Trace:
I20260808 01:55:10.031656 20761 raft_consensus.cc:2889] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260808 01:55:10.021291 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 96115.2us
I20260808 01:55:10.032670 20648 leader_election.cc:304] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
W20260808 01:55:10.027247 20727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:10.029562 20683 raft_consensus.cc:1275] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:55:10.034772 20726 leader_election.cc:304] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:10.021247 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d32821d9c17456394a662fcfd05f6c6" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:10.032788 20493 leader_election.cc:304] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:09.975236 20605 raft_consensus.cc:1240] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260808 01:55:09.977139 20602 raft_consensus.cc:2468] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 3.
I20260808 01:55:09.980330 20604 raft_consensus.cc:1240] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: []
W20260808 01:55:09.981438 21547 log.cc:927] Time spent T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
W20260808 01:55:09.981571 20728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:55:09.985342 21523 log.cc:927] Time spent T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.482s	user 0.004s	sys 0.000s
I20260808 01:55:09.989228 20680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46968 (request call id 153) took 1448 ms. Trace:
I20260808 01:55:10.080058 20648 leader_election.cc:400] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Vote denied by peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531) with higher term. Message: Invalid argument: T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for earlier term 3. Current term is 4.
I20260808 01:55:10.080494 20648 leader_election.cc:403] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260808 01:55:09.994071 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e71aab2ce9b45369b7eff65e1645f9b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:10.022254 21423 raft_consensus.cc:2749] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:55:10.039438 20726 leader_election.cc:400] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Vote denied by peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325) with higher term. Message: Invalid argument: T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3.
I20260808 01:55:10.083523 20726 leader_election.cc:403] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20260808 01:55:10.084056 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 45361.5us
I20260808 01:55:10.011512 20600 raft_consensus.cc:2368] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for earlier term 3. Current term is 4.
I20260808 01:55:10.003707 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:10.065786 20727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:09.998611 21351 leader_election.cc:290] T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
W20260808 01:55:10.060828 20493 leader_election.cc:343] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
I20260808 01:55:10.021852 20601 raft_consensus.cc:2393] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 3: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:10.031786 20426 rpcz_store.cc:276] 0808 01:55:04.799634 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:04.799837 (+   203us) service_pool.cc:224] Handling call
0808 01:55:10.008656 (+5208819us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:55:09.011827 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:55:09.012453 (+   626us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:55:09.012495 (+    42us) write_op.cc:435] Acquired schema lock
0808 01:55:09.012512 (+    17us) tablet.cc:662] Decoding operations
0808 01:55:09.012951 (+   439us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:55:09.013020 (+    69us) write_op.cc:641] Partition lock acquired for write op
0808 01:55:09.013124 (+   104us) tablet.cc:685] Acquiring locks for 2 operations
0808 01:55:09.013451 (+   327us) tablet.cc:701] Row locks acquired
0808 01:55:09.013469 (+    18us) write_op.cc:260] PREPARE: finished
0808 01:55:09.013623 (+   154us) write_op.cc:270] Start()
0808 01:55:09.015480 (+  1857us) write_op.cc:276] Timestamp: P: 1786154109013572 usec, L: 0
0808 01:55:09.015507 (+    27us) op_driver.cc:348] REPLICATION: starting
0808 01:55:09.052436 (+ 36929us) log.cc:844] Serialized 928 byte log entry
0808 01:55:09.311123 (+258687us) op_driver.cc:464] REPLICATION: finished
0808 01:55:09.591302 (+280179us) write_op.cc:301] APPLY: starting
0808 01:55:09.591420 (+   118us) tablet.cc:1367] starting BulkCheckPresence
0808 01:55:09.591581 (+   161us) tablet.cc:1370] finished BulkCheckPresence
0808 01:55:09.591728 (+   147us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:55:09.592256 (+   528us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:55:09.592494 (+   238us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0808 01:55:09.592561 (+    67us) write_op.cc:312] APPLY: finished
0808 01:55:09.592805 (+   244us) log.cc:844] Serialized 28 byte log entry
0808 01:55:09.593038 (+   233us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:55:09.593160 (+   122us) write_op.cc:454] Released schema lock
0808 01:55:09.593213 (+    53us) write_op.cc:341] FINISH: Updating metrics
0808 01:55:09.594411 (+  1198us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:55:09.594458 (+    47us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":144812,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":147575,"apply.run_cpu_time_us":2383,"apply.run_wall_time_us":3090,"num_ops":2,"prepare.queue_time_us":254190,"prepare.run_cpu_time_us":3473,"prepare.run_wall_time_us":63589,"replication_time_us":265348,"thread_start_us":327088,"threads_started":2}]]}
I20260808 01:55:10.023176 20679 raft_consensus.cc:2368] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3.
I20260808 01:55:10.076205 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
W20260808 01:55:10.012984 20650 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:10.112486 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:10.030627 21496 raft_consensus.cc:493] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:10.033286 21222 raft_consensus.cc:2749] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:55:10.081940 20607 raft_consensus.cc:2468] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:55:10.126101 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:10.026036 21459 leader_election.cc:290] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:10.037891 21350 raft_consensus.cc:493] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:10.041290 21364 raft_consensus.cc:2804] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:10.051403 20571 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:10.092108 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:10.093531 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:10.080192 20680 rpcz_store.cc:276] 0808 01:55:08.541168 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:09.061337 (+520169us) service_pool.cc:224] Handling call
0808 01:55:09.988913 (+927576us) spinlock_profiling.cc:243] Waited 711 ms on lock 0x7b54002d5ad4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:09.989136 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1636064000}
I20260808 01:55:10.075425 21404 raft_consensus.cc:2749] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:10.025946 21526 consensus_queue.cc:237] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:10.096683 21494 raft_consensus.cc:3055] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:55:10.098907 21537 raft_consensus.cc:3060] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:10.138931 20607 raft_consensus.cc:1240] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:10.139012 21494 raft_consensus.cc:740] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:10.141116 21494 consensus_queue.cc:260] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:10.141626 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:10.012629 20681 rpcz_store.cc:276] 0808 01:55:08.474645 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:08.721574 (+246929us) service_pool.cc:224] Handling call
0808 01:55:09.935577 (+1214003us) spinlock_profiling.cc:243] Waited 354 ms on lock 0x7b540051a754. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:09.935722 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":814261632}
I20260808 01:55:10.143150 20682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46972 (request call id 122) took 1255 ms. Trace:
I20260808 01:55:10.143618 20682 rpcz_store.cc:276] 0808 01:55:08.888058 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:09.027087 (+139029us) service_pool.cc:224] Handling call
0808 01:55:10.137729 (+1110642us) spinlock_profiling.cc:243] Waited 578 ms on lock 0x7b540051c054. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:10.143089 (+  5360us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1330011392}
I20260808 01:55:10.145378 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:10.146081 20606 raft_consensus.cc:2468] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:10.079577 21240 consensus_queue.cc:1059] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260808 01:55:10.135079 21245 raft_consensus.cc:2749] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:55:10.104944 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:10.038029 21538 raft_consensus.cc:515] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:10.150342 21538 leader_election.cc:290] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:10.087522 21502 raft_consensus.cc:2749] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325) with higher term. Message: Invalid argument: T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3.
I20260808 01:55:10.151036 21364 raft_consensus.cc:697] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [term 3 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:10.151633 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:10.152719 20609 raft_consensus.cc:2393] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 2: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:10.114198 21398 raft_consensus.cc:493] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:10.048050 21388 raft_consensus.cc:2804] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:10.153344 21398 raft_consensus.cc:515] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:10.153963 21388 raft_consensus.cc:493] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:10.154433 21388 raft_consensus.cc:3060] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:10.136407 20527 raft_consensus.cc:2468] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:10.156242 21398 leader_election.cc:290] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:10.156529 20570 leader_election.cc:304] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:10.157658 21398 raft_consensus.cc:2804] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:10.158174 21398 raft_consensus.cc:493] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:10.158524 21398 raft_consensus.cc:3060] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:10.116979 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:10.120299 20651 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:10.120198 20649 leader_election.cc:304] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:10.161221 20726 leader_election.cc:400] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Vote denied by peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325) with higher term. Message: Invalid argument: T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3.
I20260808 01:55:10.161648 20726 leader_election.cc:403] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20260808 01:55:10.106830 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 46086.3us
I20260808 01:55:10.123198 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:10.162722 20685 raft_consensus.cc:3060] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:10.162792 21502 raft_consensus.cc:3060] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:10.163429 21403 leader_election.cc:290] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:10.121194 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:10.031747 21243 consensus_queue.cc:1048] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.055s
W20260808 01:55:10.124264 20725 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:10.165503 20683 raft_consensus.cc:2393] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 3: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:10.132592 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:10.135143 20530 raft_consensus.cc:2393] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 3: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
W20260808 01:55:10.166828 20493 leader_election.cc:343] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117): Illegal state: must be running to vote when last-logged opid is not known
W20260808 01:55:10.133351 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 82079.6us
I20260808 01:55:10.147450 21240 raft_consensus.cc:3055] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:10.025609 21433 raft_consensus.cc:515] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:10.148109 20605 raft_consensus.cc:2468] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:10.087405 21343 consensus_queue.cc:1059] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260808 01:55:10.152192 21364 consensus_queue.cc:237] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
W20260808 01:55:10.110391 20492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:10.174577 20571 leader_election.cc:304] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:10.093138 21499 consensus_queue.cc:1048] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.080s
I20260808 01:55:10.172784 20493 leader_election.cc:304] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:10.173553 21240 raft_consensus.cc:740] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
W20260808 01:55:10.128940 20475 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:55:10.173920 20526 raft_consensus.cc:2393] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 3: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:10.162789 21535 raft_consensus.cc:2749] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:55:10.188892 20648 leader_election.cc:304] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:10.171733 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:10.179581 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:55:10.202661 20728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:10.202870 20725 leader_election.cc:304] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:10.203831 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:10.204691 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:10.205103 21388 raft_consensus.cc:515] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:10.178084 21450 raft_consensus.cc:515] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:10.136647 21496 raft_consensus.cc:515] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:10.209760 20493 leader_election.cc:304] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:10.136852 21350 raft_consensus.cc:515] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:10.200507 20685 raft_consensus.cc:2468] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:10.192435 21403 raft_consensus.cc:2804] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:10.219264 20492 leader_election.cc:304] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:10.177302 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac71e66cb4042eb8fe8aafd139af68a" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:10.211290 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:10.224979 20725 leader_election.cc:304] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
W20260808 01:55:10.224915 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 62632.5us
I20260808 01:55:10.217533 21423 raft_consensus.cc:2749] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 01:55:10.190048 21333 consensus_queue.cc:1048] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20260808 01:55:10.201179 21222 consensus_queue.cc:1048] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260808 01:55:10.216305 21544 raft_consensus.cc:2804] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:10.222121 20648 leader_election.cc:304] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:10.241487 20725 leader_election.cc:304] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:10.247010 20725 leader_election.cc:304] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:10.221293 21492 raft_consensus.cc:2749] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:10.250478 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:10.222186 20684 raft_consensus.cc:2468] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:10.257294 21497 raft_consensus.cc:2749] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:10.239810 21535 consensus_queue.cc:1048] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260808 01:55:10.245478 21459 raft_consensus.cc:2749] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:55:10.255501 21404 raft_consensus.cc:2804] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:10.243688 20528 raft_consensus.cc:2468] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:10.239902 21271 raft_consensus.cc:2804] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 01:55:10.240293 21403 raft_consensus.cc:697] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:10.270164 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:10.273767 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac71e66cb4042eb8fe8aafd139af68a" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:10.223176 21424 consensus_queue.cc:1059] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260808 01:55:10.258002 21351 raft_consensus.cc:2804] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:10.274881 20683 raft_consensus.cc:2468] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:10.267239 21271 raft_consensus.cc:493] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:10.263602 21544 raft_consensus.cc:697] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [term 3 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:10.236088 21240 consensus_queue.cc:260] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:10.277269 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:10.281191 21404 raft_consensus.cc:697] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [term 2 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:10.285064 21351 raft_consensus.cc:493] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:10.298163 21271 raft_consensus.cc:3060] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:10.297168 21423 raft_consensus.cc:2749] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:10.296722 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
W20260808 01:55:10.307631 21556 log.cc:927] Time spent T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.073s	user 0.000s	sys 0.002s
I20260808 01:55:10.303996 21351 raft_consensus.cc:3060] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:10.308561 21404 consensus_queue.cc:237] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:10.300865 21403 consensus_queue.cc:237] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:10.310935 21544 consensus_queue.cc:237] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:10.312729 21433 leader_election.cc:290] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:10.314901 21350 leader_election.cc:290] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:10.320960 21537 raft_consensus.cc:2749] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531) with higher term. Message: Invalid argument: T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for earlier term 3. Current term is 4.
I20260808 01:55:10.321580 21496 leader_election.cc:290] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:10.319586 20757 raft_consensus.cc:2393] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 4: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:10.326944 21502 raft_consensus.cc:2749] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325) with higher term. Message: Invalid argument: T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3.
I20260808 01:55:10.337981 21398 raft_consensus.cc:515] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:10.338420 21271 raft_consensus.cc:515] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:10.341159 21398 leader_election.cc:290] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:10.342056 21271 leader_election.cc:290] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:55:10.386698 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 154969us
W20260808 01:55:10.392030 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 162907us
W20260808 01:55:10.395337 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 176008us
I20260808 01:55:10.395943 20522 raft_consensus.cc:1240] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:10.401281 20527 raft_consensus.cc:1275] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:10.402263 20571 leader_election.cc:304] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
W20260808 01:55:10.407781 20572 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:55:10.411633 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 189462us
I20260808 01:55:10.412420 20684 raft_consensus.cc:2393] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 3: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:10.412946 20605 raft_consensus.cc:1275] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260808 01:55:10.413512 20648 leader_election.cc:304] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:10.398211 20492 leader_election.cc:304] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
W20260808 01:55:10.415446 20650 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:10.419070 20525 raft_consensus.cc:1240] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:10.421168 20601 raft_consensus.cc:1240] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 LEADER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:10.417394 20529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41396 (request call id 172) took 1234 ms. Trace:
I20260808 01:55:10.437345 20529 rpcz_store.cc:276] 0808 01:55:09.182988 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:09.254528 (+ 71540us) service_pool.cc:224] Handling call
0808 01:55:09.714959 (+460431us) spinlock_profiling.cc:243] Waited 26.6 ms on lock 0x7b58000d9e98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:10.416738 (+701779us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b54002d96d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:10.417077 (+   339us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":315703296}
I20260808 01:55:10.417049 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:10.440300 20725 leader_election.cc:304] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:55:10.440529 20602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:10.424844 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:10.441933 20603 raft_consensus.cc:3060] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:10.442098 20602 raft_consensus.cc:2368] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3.
I20260808 01:55:10.336756 20756 raft_consensus.cc:2468] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
W20260808 01:55:10.443758 20726 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:10.444540 21283 consensus_queue.cc:1059] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:10.446043 20607 raft_consensus.cc:1240] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:10.446882 21336 raft_consensus.cc:3055] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:10.447510 21336 raft_consensus.cc:740] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:10.447764 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:10.447894 21351 raft_consensus.cc:515] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:10.449378 21336 consensus_queue.cc:260] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:10.430842 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:10.432806 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e71aab2ce9b45369b7eff65e1645f9b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:55:10.451228 20758 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53322 (request call id 165) took 1701 ms (client timeout 1860 ms). Trace:
I20260808 01:55:10.451787 20757 raft_consensus.cc:2410] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:55:10.432762 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e71aab2ce9b45369b7eff65e1645f9b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:10.432806 21245 raft_consensus.cc:2804] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:10.453248 21245 raft_consensus.cc:697] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:10.433980 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:10.454504 20522 raft_consensus.cc:3060] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:10.428269 20609 raft_consensus.cc:1240] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [term 2 LEADER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260808 01:55:10.440805 20600 raft_consensus.cc:2393] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 3: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:10.461977 21496 raft_consensus.cc:493] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:10.462357 21499 raft_consensus.cc:2804] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:10.462934 21499 raft_consensus.cc:493] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:10.463454 21499 raft_consensus.cc:3060] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:10.463167 21496 raft_consensus.cc:515] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
W20260808 01:55:10.467670 20728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:10.481303 21564 raft_consensus.cc:2749] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:55:10.451821 20758 rpcz_store.cc:269] 0808 01:55:08.749310 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:08.750642 (+  1332us) service_pool.cc:224] Handling call
0808 01:55:10.450631 (+1699989us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b54001be754. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:10.450971 (+   340us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3337875840}
I20260808 01:55:10.485711 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:10.486881 21351 leader_election.cc:290] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:10.487423 21336 raft_consensus.cc:3060] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:10.487798 21398 consensus_queue.cc:1048] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:10.472801 21496 leader_election.cc:290] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
W20260808 01:55:10.496155 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 56450.3us
W20260808 01:55:10.500376 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 87210.4us
I20260808 01:55:10.506559 20523 raft_consensus.cc:1275] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:55:10.508586 21477 raft_consensus.cc:3055] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:10.521682 21544 raft_consensus.cc:493] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:10.522490 21245 consensus_queue.cc:237] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:10.524312 21538 raft_consensus.cc:3055] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Stepping down as leader of term 1
W20260808 01:55:10.525954 20728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:10.532657 20725 leader_election.cc:304] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:10.534265 20759 raft_consensus.cc:1240] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: []
W20260808 01:55:10.540903 20727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:10.552506 21450 leader_election.cc:290] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:10.553534 21459 raft_consensus.cc:493] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:10.554519 21459 raft_consensus.cc:515] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:10.556193 21494 raft_consensus.cc:3060] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:10.557559 21459 leader_election.cc:290] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:10.541905 20757 raft_consensus.cc:1240] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 3. Current term is 4. Ops: [1.1-3.2]
I20260808 01:55:10.543201 21497 raft_consensus.cc:493] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:10.560683 21497 raft_consensus.cc:515] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:10.540952 21240 raft_consensus.cc:3060] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:10.550730 21538 raft_consensus.cc:740] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:10.547573 21477 raft_consensus.cc:740] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:10.568616 20606 raft_consensus.cc:1240] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:10.566581 21388 leader_election.cc:290] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
W20260808 01:55:10.590986 20651 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:10.591658 21399 consensus_queue.cc:1048] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:55:10.568413 21538 consensus_queue.cc:260] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:10.593410 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:10.594213 21538 raft_consensus.cc:3060] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:10.594374 20680 raft_consensus.cc:3060] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:10.595506 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d8bf19c324463fb9903b9981068e9d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:10.577399 20648 leader_election.cc:304] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:10.601142 20683 raft_consensus.cc:3060] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 0 FOLLOWER]: Advancing to term 1
I20260808 01:55:10.600850 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:10.570765 21271 raft_consensus.cc:2804] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:10.569551 21497 leader_election.cc:290] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:10.609453 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:10.609669 20684 raft_consensus.cc:1240] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:10.595831 20423 catalog_manager.cc:5697] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb reported cstate change: term changed from 1 to 2, leader changed from 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194) to 598db12c38d442c6988b4c4f991874fb (127.7.165.195). New cstate: current_term: 2 leader_uuid: "598db12c38d442c6988b4c4f991874fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:10.604301 21388 raft_consensus.cc:2749] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:55:10.610561 20686 raft_consensus.cc:2468] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:10.608043 21271 raft_consensus.cc:493] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:10.617954 20685 raft_consensus.cc:2468] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:10.574574 21544 raft_consensus.cc:515] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:10.619959 21364 raft_consensus.cc:2749] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260808 01:55:10.623411 20727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:10.615442 21497 raft_consensus.cc:493] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:10.619357 21271 raft_consensus.cc:3060] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:10.619987 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:55:10.627557 20727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:10.576548 21477 consensus_queue.cc:260] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:10.636024 20726 leader_election.cc:304] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:10.640784 21477 raft_consensus.cc:3060] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:10.642267 20679 raft_consensus.cc:2393] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 2: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:10.642845 20570 leader_election.cc:304] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
W20260808 01:55:10.643358 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 66293.3us
I20260808 01:55:10.629669 20423 catalog_manager.cc:5697] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb reported cstate change: term changed from 1 to 2, leader changed from df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196) to 598db12c38d442c6988b4c4f991874fb (127.7.165.195). New cstate: current_term: 2 leader_uuid: "598db12c38d442c6988b4c4f991874fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } }
W20260808 01:55:10.651885 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 45590.8us
W20260808 01:55:10.648954 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 313655us
I20260808 01:55:10.651595 21459 raft_consensus.cc:2804] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:10.661253 21459 raft_consensus.cc:493] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:10.661772 21459 raft_consensus.cc:3060] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:10.663090 20726 leader_election.cc:304] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:10.668067 20423 catalog_manager.cc:5697] T 0bd11a6553c444d488651367f3f0143f P 598db12c38d442c6988b4c4f991874fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 598db12c38d442c6988b4c4f991874fb (127.7.165.195). New cstate: current_term: 1 leader_uuid: "598db12c38d442c6988b4c4f991874fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:10.690495 20683 raft_consensus.cc:2468] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:10.693341 20423 catalog_manager.cc:5697] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 598db12c38d442c6988b4c4f991874fb (127.7.165.195). New cstate: current_term: 1 leader_uuid: "598db12c38d442c6988b4c4f991874fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:10.696295 20604 raft_consensus.cc:1240] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260808 01:55:10.697435 20728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:10.699496 20726 leader_election.cc:304] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:10.702912 21364 consensus_queue.cc:1048] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:55:10.705717 21572 raft_consensus.cc:2804] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 01:55:10.706789 20492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:55:10.709733 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 177412us
I20260808 01:55:10.713251 21388 raft_consensus.cc:2804] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 01:55:10.714061 21388 raft_consensus.cc:697] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:10.715101 20522 raft_consensus.cc:2468] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 4.
I20260808 01:55:10.715310 21388 consensus_queue.cc:237] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:10.717751 20570 leader_election.cc:304] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:10.725728 21336 raft_consensus.cc:2804] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:55:10.726399 21336 raft_consensus.cc:697] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 4 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:10.706254 21572 raft_consensus.cc:493] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:55:10.732705 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 124141us
I20260808 01:55:10.731751 21336 consensus_queue.cc:237] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
W20260808 01:55:10.741158 21567 log.cc:927] Time spent T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
I20260808 01:55:10.743391 20571 leader_election.cc:304] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:10.730908 21572 raft_consensus.cc:3060] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:10.747690 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:10.747927 21499 raft_consensus.cc:515] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:10.749788 20757 raft_consensus.cc:1240] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:10.752084 21544 leader_election.cc:290] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:10.752992 21240 raft_consensus.cc:493] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:10.753386 20725 leader_election.cc:304] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:10.753785 21240 raft_consensus.cc:515] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:10.754666 21500 raft_consensus.cc:2749] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:55:10.756320 20726 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:10.678784 21497 raft_consensus.cc:515] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:10.757949 20571 leader_election.cc:304] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:10.758203 21399 raft_consensus.cc:2749] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:10.758239 21240 leader_election.cc:290] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:10.759014 21403 consensus_queue.cc:1048] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:10.764439 21283 raft_consensus.cc:2804] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:10.765054 21283 raft_consensus.cc:493] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:10.765383 20571 leader_election.cc:304] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:55:10.769690 21283 raft_consensus.cc:3060] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:10.778553 21240 raft_consensus.cc:493] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:10.778810 21364 raft_consensus.cc:493] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:10.779307 21240 raft_consensus.cc:515] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:10.779619 21364 raft_consensus.cc:515] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:10.785234 21499 leader_election.cc:290] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:10.785393 21271 raft_consensus.cc:515] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:10.786625 21364 leader_election.cc:290] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:10.788872 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:10.795740 20758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d8bf19c324463fb9903b9981068e9d" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
W20260808 01:55:10.796516 20493 connection.cc:256] RPC call timeout handler was delayed by -1.1185733700112905s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1796s for a response
W20260808 01:55:10.797031 20493 connection.cc:256] RPC call timeout handler was delayed by -1.0823583629971836s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2191s for a response
W20260808 01:55:10.797636 20492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:10.798082 20725 leader_election.cc:304] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
W20260808 01:55:10.799198 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 127268us
I20260808 01:55:10.800302 20725 leader_election.cc:304] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:10.801067 21494 raft_consensus.cc:2804] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:10.801882 21568 consensus_queue.cc:1059] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:10.792251 21240 leader_election.cc:290] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:10.792781 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:10.804044 20648 leader_election.cc:304] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:10.793887 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:10.792371 20757 raft_consensus.cc:3060] T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3 [term 0 FOLLOWER]: Advancing to term 1
W20260808 01:55:10.803869 20493 connection.cc:256] RPC call timeout handler was delayed by -1.0645733700075652s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18560000000000001s for a response
I20260808 01:55:10.804800 21364 consensus_queue.cc:1048] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:55:10.778018 21459 raft_consensus.cc:515] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:10.806945 20603 raft_consensus.cc:2468] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3.
I20260808 01:55:10.811062 21494 raft_consensus.cc:493] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:10.830523 21494 raft_consensus.cc:3060] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:10.844507 21401 raft_consensus.cc:2804] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:10.845276 21401 raft_consensus.cc:697] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 2 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:10.846441 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:10.847524 20684 raft_consensus.cc:2468] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:10.845567 21497 leader_election.cc:290] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:10.846486 21401 consensus_queue.cc:237] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:10.852298 20680 raft_consensus.cc:2468] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 4.
W20260808 01:55:10.853942 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 53433.8us
I20260808 01:55:10.860308 21478 raft_consensus.cc:493] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:10.861565 21478 raft_consensus.cc:515] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:10.864653 21478 leader_election.cc:290] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:10.816865 21399 consensus_queue.cc:1048] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260808 01:55:10.823792 21388 raft_consensus.cc:2749] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:10.886500 20529 raft_consensus.cc:1275] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:10.860704 21430 raft_consensus.cc:2804] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:10.833503 21271 leader_election.cc:290] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:10.891352 20725 leader_election.cc:304] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
W20260808 01:55:10.875463 20603 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 209) took 1157 ms (client timeout 1514 ms). Trace:
I20260808 01:55:10.899479 21493 raft_consensus.cc:2804] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:10.890462 21283 raft_consensus.cc:515] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:10.890131 21430 raft_consensus.cc:493] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:10.904235 21430 raft_consensus.cc:3060] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:10.910959 20493 leader_election.cc:304] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:10.912140 21336 consensus_queue.cc:1048] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:10.915082 21493 raft_consensus.cc:697] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [term 3 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:10.920598 20571 leader_election.cc:304] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:10.921353 21493 consensus_queue.cc:237] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:10.909667 21572 raft_consensus.cc:515] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:10.935420 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 131570us
I20260808 01:55:10.939167 21271 raft_consensus.cc:2804] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:10.939693 21271 raft_consensus.cc:493] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:10.940078 21271 raft_consensus.cc:3060] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:10.940574 21572 leader_election.cc:290] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:10.932775 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:10.937176 21497 raft_consensus.cc:2749] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:10.942269 20606 raft_consensus.cc:2393] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 2: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
W20260808 01:55:10.943549 20726 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:10.945365 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:10.946065 20683 raft_consensus.cc:3060] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:10.933344 21336 raft_consensus.cc:2804] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:10.975980 20604 raft_consensus.cc:1275] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:10.977502 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:10.978304 20601 raft_consensus.cc:2376] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3. Re-sending same reply.
W20260808 01:55:10.979058 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 68239.6us
I20260808 01:55:10.980537 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:10.981101 21459 leader_election.cc:290] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:10.981895 20493 leader_election.cc:304] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: df82e7f402364d369ed1ec3c5a8772b3
W20260808 01:55:10.983111 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 107901us
I20260808 01:55:10.984184 20725 leader_election.cc:304] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:10.986189 20726 leader_election.cc:304] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:10.987232 20726 leader_election.cc:304] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:10.987879 21283 leader_election.cc:290] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:10.988940 21388 raft_consensus.cc:2749] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:55:10.989811 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:10.979051 21240 raft_consensus.cc:3055] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:10.991226 21572 raft_consensus.cc:2804] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:10.991698 21240 raft_consensus.cc:740] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:10.991822 21572 raft_consensus.cc:493] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:10.992249 21572 raft_consensus.cc:3060] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:10.992885 21240 consensus_queue.cc:260] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:10.994084 21240 raft_consensus.cc:3060] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:10.982425 20607 raft_consensus.cc:1275] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:10.995288 21492 raft_consensus.cc:493] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:10.989879 21388 raft_consensus.cc:2804] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:55:10.996151 21492 raft_consensus.cc:515] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:10.996601 21388 raft_consensus.cc:697] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:10.997445 21534 raft_consensus.cc:2804] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:10.998136 21534 raft_consensus.cc:697] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 3 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:10.997769 21388 consensus_queue.cc:237] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:10.995424 21336 raft_consensus.cc:697] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:10.988878 20602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:10.993093 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:10.998653 21492 leader_election.cc:290] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:11.004936 20680 raft_consensus.cc:3060] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 3
I20260808 01:55:10.999198 21534 consensus_queue.cc:237] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:11.007314 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e71aab2ce9b45369b7eff65e1645f9b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
W20260808 01:55:10.902912 20603 rpcz_store.cc:269] 0808 01:55:09.673321 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:09.676614 (+  3293us) service_pool.cc:224] Handling call
0808 01:55:10.098330 (+421716us) spinlock_profiling.cc:243] Waited 9.68 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:10.422565 (+324235us) spinlock_profiling.cc:243] Waited 7.66 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d4185584c 00007f5d4185540b 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:10.830870 (+408305us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":39882496}
I20260808 01:55:10.999729 21336 consensus_queue.cc:237] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:11.012813 20602 raft_consensus.cc:2468] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:11.004498 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:11.004565 21497 raft_consensus.cc:493] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:11.015138 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:11.015553 21497 raft_consensus.cc:515] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:11.017367 20648 leader_election.cc:400] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Vote denied by peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191) with higher term. Message: Invalid argument: T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for earlier term 3. Current term is 4.
I20260808 01:55:11.003225 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:11.018482 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:55:11.019037 20650 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:11.019521 20608 raft_consensus.cc:2468] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:11.020210 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:11.023641 21497 leader_election.cc:290] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
W20260808 01:55:11.024295 20650 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:11.024886 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:11.029162 20648 leader_election.cc:304] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:11.043467 20493 leader_election.cc:304] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:11.044708 21538 raft_consensus.cc:2749] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:55:11.051327 21240 consensus_queue.cc:1059] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:11.052947 21240 raft_consensus.cc:3055] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:11.053318 21240 raft_consensus.cc:740] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:11.054913 21240 consensus_queue.cc:260] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:11.056631 21240 raft_consensus.cc:3060] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:11.057683 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:11.058604 20684 raft_consensus.cc:2393] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 2: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:11.061025 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:11.061022 20493 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:11.062197 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:11.064098 21538 raft_consensus.cc:2749] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:55:11.069847 20683 raft_consensus.cc:2468] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 4.
I20260808 01:55:11.064635 21544 consensus_queue.cc:1048] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.006s
I20260808 01:55:11.043556 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a313d0a0979e4dc8a4205c4f16a24257" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:11.079913 20726 leader_election.cc:304] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:11.072890 21477 raft_consensus.cc:2764] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260808 01:55:11.095490 20604 raft_consensus.cc:2393] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
W20260808 01:55:11.107280 20725 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:11.074339 21492 raft_consensus.cc:493] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:11.143132 21430 raft_consensus.cc:515] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
W20260808 01:55:11.147688 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 128839us
I20260808 01:55:11.126925 21459 raft_consensus.cc:2804] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:55:11.152318 21459 raft_consensus.cc:697] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:11.155630 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
W20260808 01:55:11.156366 21590 log.cc:927] Time spent T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260808 01:55:11.156641 20529 raft_consensus.cc:3060] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
W20260808 01:55:11.158161 20727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:11.150420 20648 leader_election.cc:304] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:11.106097 21497 raft_consensus.cc:493] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:55:11.165540 20726 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:11.166761 20683 raft_consensus.cc:1275] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:11.094074 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:11.136533 21499 raft_consensus.cc:493] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:55:11.182273 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 117591us
I20260808 01:55:11.155630 21271 raft_consensus.cc:515] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:11.161445 21430 leader_election.cc:290] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:11.168519 21477 raft_consensus.cc:2749] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260808 01:55:11.171754 20728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:11.196923 20756 raft_consensus.cc:1275] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260808 01:55:11.204116 21499 raft_consensus.cc:515] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:11.204572 21459 consensus_queue.cc:237] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:11.206233 21586 log.cc:927] Time spent T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc: Append to log took a long time: real 0.112s	user 0.001s	sys 0.002s
W20260808 01:55:11.207345 21587 log.cc:927] Time spent T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.113s	user 0.002s	sys 0.000s
I20260808 01:55:11.170452 21492 raft_consensus.cc:515] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:11.218469 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 68275.6us
I20260808 01:55:11.212332 20756 pending_rounds.cc:85] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3: Aborting all ops after (but not including) 2
I20260808 01:55:11.187989 21497 raft_consensus.cc:515] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:11.186867 21572 raft_consensus.cc:515] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:11.223393 20493 leader_election.cc:304] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:11.216022 20680 raft_consensus.cc:2468] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:11.227559 21494 raft_consensus.cc:515] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:11.236630 20753 raft_consensus.cc:1240] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260808 01:55:11.237330 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:11.239841 21499 leader_election.cc:290] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:11.241562 20493 leader_election.cc:304] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:11.241901 21494 leader_election.cc:290] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:11.225730 20756 pending_rounds.cc:107] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260808 01:55:11.242666 20756 raft_consensus.cc:2889] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:55:11.243472 20493 leader_election.cc:304] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:11.244143 21544 raft_consensus.cc:2749] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:11.244237 21534 raft_consensus.cc:2749] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:11.246076 21492 leader_election.cc:290] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:11.246111 21534 raft_consensus.cc:2804] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:11.246884 21534 raft_consensus.cc:493] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:11.247273 21534 raft_consensus.cc:3060] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:11.249284 21497 leader_election.cc:290] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:11.235226 20726 leader_election.cc:304] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:11.255018 20759 raft_consensus.cc:2368] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc for earlier term 2. Current term is 3.
I20260808 01:55:11.258656 21497 raft_consensus.cc:2804] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:11.267937 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:11.267817 21240 consensus_queue.cc:1059] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:55:11.270970 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:11.271090 20686 raft_consensus.cc:1275] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:11.273212 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:55:11.296983 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 53679us
I20260808 01:55:11.345054 20683 raft_consensus.cc:2468] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:11.347113 21336 raft_consensus.cc:493] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:11.347738 21336 raft_consensus.cc:515] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:11.350245 21336 leader_election.cc:290] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:11.350167 20753 raft_consensus.cc:1275] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260808 01:55:11.346148 21271 leader_election.cc:290] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:11.354300 20753 pending_rounds.cc:85] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3: Aborting all ops after (but not including) 1
I20260808 01:55:11.346356 21497 raft_consensus.cc:697] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [term 3 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:11.355795 21497 consensus_queue.cc:237] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:11.357295 21534 raft_consensus.cc:515] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:11.354911 20753 pending_rounds.cc:107] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260808 01:55:11.358358 20753 raft_consensus.cc:2889] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:55:11.341465 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:11.342149 20648 leader_election.cc:304] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:11.343617 21493 raft_consensus.cc:493] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:11.261507 21492 raft_consensus.cc:493] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:11.358135 21594 consensus_queue.cc:1048] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260808 01:55:11.340422 20529 raft_consensus.cc:2468] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:11.376132 21595 consensus_queue.cc:1048] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:11.360306 21477 raft_consensus.cc:3055] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:11.379910 20570 leader_election.cc:304] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:11.377854 21492 raft_consensus.cc:515] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:11.391057 21477 raft_consensus.cc:740] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:11.410558 21477 consensus_queue.cc:260] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:11.412237 21477 raft_consensus.cc:3060] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:11.385517 21568 raft_consensus.cc:2749] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:11.419147 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:11.419732 21534 leader_election.cc:290] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:11.402297 21572 leader_election.cc:290] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:11.421505 21494 raft_consensus.cc:493] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:11.359617 21493 raft_consensus.cc:515] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:11.422333 21494 raft_consensus.cc:515] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
W20260808 01:55:11.424311 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 44519.9us
I20260808 01:55:11.425432 21336 raft_consensus.cc:2804] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:11.426002 21336 raft_consensus.cc:697] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [term 2 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:11.426718 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:11.427520 20680 raft_consensus.cc:3060] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:11.427099 21336 consensus_queue.cc:237] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
W20260808 01:55:11.412187 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 70153.1us
I20260808 01:55:11.429652 20726 leader_election.cc:304] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:11.439822 21492 leader_election.cc:290] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:11.437734 21499 raft_consensus.cc:2804] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:11.443480 21499 raft_consensus.cc:493] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:11.441143 21240 consensus_queue.cc:1059] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
W20260808 01:55:11.446992 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 183191us
W20260808 01:55:11.448061 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 73241.5us
I20260808 01:55:11.420137 20679 raft_consensus.cc:2468] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:11.446224 21499 raft_consensus.cc:3060] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:11.474409 21534 consensus_queue.cc:1048] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:55:11.496101 20725 leader_election.cc:304] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:55:11.488384 21494 leader_election.cc:290] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:11.513619 21240 raft_consensus.cc:493] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:11.505031 21492 raft_consensus.cc:2804] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:11.523700 21240 raft_consensus.cc:515] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:11.505640 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:11.508477 21594 raft_consensus.cc:493] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:11.508495 21493 leader_election.cc:290] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:11.528699 20726 leader_election.cc:304] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
W20260808 01:55:11.533114 20649 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:11.535854 20682 raft_consensus.cc:1275] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:11.536180 20758 raft_consensus.cc:1275] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:11.537065 20529 raft_consensus.cc:1240] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:11.539611 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a313d0a0979e4dc8a4205c4f16a24257" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:11.541492 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:11.530550 20650 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:11.543650 21494 raft_consensus.cc:2804] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:11.544159 21494 raft_consensus.cc:493] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:11.544570 21494 raft_consensus.cc:3060] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:11.532555 21594 raft_consensus.cc:515] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:11.547330 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:11.550185 21240 leader_election.cc:290] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:11.538538 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:11.551262 20529 raft_consensus.cc:2393] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 2: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:11.523944 21492 raft_consensus.cc:493] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:11.563248 21240 raft_consensus.cc:493] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:11.564656 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:11.567593 21240 raft_consensus.cc:515] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:11.569077 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:11.569862 21608 raft_consensus.cc:493] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:11.570464 21608 raft_consensus.cc:515] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:11.572330 21502 consensus_queue.cc:1048] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:11.573156 21608 leader_election.cc:290] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:11.573745 21608 raft_consensus.cc:493] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:11.574568 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:11.576327 20527 raft_consensus.cc:2368] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3.
I20260808 01:55:11.576548 20759 raft_consensus.cc:1275] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260808 01:55:11.579686 21598 log.cc:927] Time spent T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260808 01:55:11.548830 21493 raft_consensus.cc:493] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:11.581264 21493 raft_consensus.cc:515] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:11.582020 21500 raft_consensus.cc:493] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:11.582564 21500 raft_consensus.cc:515] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:11.583489 21493 leader_election.cc:290] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:11.584671 21500 leader_election.cc:290] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:11.533905 21572 raft_consensus.cc:493] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:11.592321 21602 consensus_queue.cc:1048] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260808 01:55:11.594304 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d8bf19c324463fb9903b9981068e9d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:11.595274 20522 raft_consensus.cc:2393] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:11.596460 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:11.597899 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:11.587756 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:11.598963 20525 raft_consensus.cc:3060] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:11.616250 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:11.617147 20682 raft_consensus.cc:2468] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:11.603438 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:11.617933 21459 consensus_queue.cc:1048] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 01:55:11.618775 20528 raft_consensus.cc:2393] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 2: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:11.604239 21568 raft_consensus.cc:3055] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:11.619517 21568 raft_consensus.cc:740] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:11.619952 20680 raft_consensus.cc:2468] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:11.620918 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:11.620708 21568 consensus_queue.cc:260] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:11.621709 20524 raft_consensus.cc:2468] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:11.625711 20524 raft_consensus.cc:1240] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 3. Current term is 4. Ops: []
I20260808 01:55:11.636857 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:11.591320 21572 raft_consensus.cc:515] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:11.638945 20757 raft_consensus.cc:1275] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Refusing update from remote peer 598db12c38d442c6988b4c4f991874fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:11.639750 21572 leader_election.cc:290] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:11.591460 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:11.609431 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:11.610744 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:11.622273 21568 raft_consensus.cc:3060] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:11.641613 20726 leader_election.cc:304] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:11.622535 20492 leader_election.cc:304] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:11.643937 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:11.605113 21594 leader_election.cc:290] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:11.642688 21572 raft_consensus.cc:2804] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:11.574265 21608 raft_consensus.cc:515] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
W20260808 01:55:11.620826 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 58083.3us
I20260808 01:55:11.565161 21492 raft_consensus.cc:3060] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:11.623962 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:11.647022 21605 consensus_queue.cc:1059] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260808 01:55:11.651139 20528 raft_consensus.cc:2468] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:11.626610 21240 leader_election.cc:290] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:11.629351 21500 raft_consensus.cc:493] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:11.630399 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:11.653424 20530 raft_consensus.cc:1240] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:11.654230 20529 raft_consensus.cc:1275] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Refusing update from remote peer 598db12c38d442c6988b4c4f991874fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:11.631789 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:11.649139 21594 raft_consensus.cc:2804] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:11.657012 21500 raft_consensus.cc:515] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:11.629043 20423 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48846 (request call id 81) took 5585 ms. Trace:
I20260808 01:55:11.634264 20648 leader_election.cc:304] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:11.636094 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:11.644618 20726 leader_election.cc:304] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:11.659631 21500 leader_election.cc:290] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:11.649010 20757 raft_consensus.cc:2468] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 3.
I20260808 01:55:11.660799 21493 raft_consensus.cc:2749] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:55:11.635419 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d8bf19c324463fb9903b9981068e9d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:11.660378 21500 raft_consensus.cc:2804] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:11.656620 20530 raft_consensus.cc:1275] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:11.658064 21594 raft_consensus.cc:697] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 2 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:11.625864 20522 raft_consensus.cc:1240] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:11.651279 20571 leader_election.cc:304] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:11.670135 21240 consensus_queue.cc:1059] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:11.661258 21398 raft_consensus.cc:493] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:11.679291 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:11.679606 20759 raft_consensus.cc:1240] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:11.680256 21283 raft_consensus.cc:2749] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:11.680222 20682 raft_consensus.cc:2468] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:11.657248 21595 raft_consensus.cc:3055] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:55:11.609213 21494 raft_consensus.cc:515] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:11.649241 21572 raft_consensus.cc:493] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:11.658069 21608 leader_election.cc:290] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:11.718091 21240 raft_consensus.cc:3055] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:11.718557 21240 raft_consensus.cc:740] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:11.642628 20425 catalog_manager.cc:5697] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 1 to 3, leader changed from a80df9f992014374aeef30979cfe393e (127.7.165.193) to 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194). New cstate: current_term: 3 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:11.668246 21500 raft_consensus.cc:697] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 3 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:11.714725 21572 raft_consensus.cc:3060] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:11.689487 20493 leader_election.cc:304] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
W20260808 01:55:11.732191 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 80983.3us
I20260808 01:55:11.729063 20726 leader_election.cc:304] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:11.671782 21499 raft_consensus.cc:515] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:11.687546 21595 raft_consensus.cc:740] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:11.720355 20526 raft_consensus.cc:2393] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:11.717614 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:11.681742 21398 raft_consensus.cc:515] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:11.744715 20425 catalog_manager.cc:5697] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } }
I20260808 01:55:11.713857 21544 consensus_queue.cc:1059] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260808 01:55:11.758541 20649 leader_election.cc:304] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:11.688215 21283 consensus_queue.cc:1048] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260808 01:55:11.763757 20754 raft_consensus.cc:2368] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e for earlier term 3. Current term is 4.
I20260808 01:55:11.763799 21602 raft_consensus.cc:2749] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:11.765647 21398 leader_election.cc:290] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:11.751322 20680 raft_consensus.cc:1275] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 4 index: 1. (index mismatch)
I20260808 01:55:11.767325 21594 consensus_queue.cc:237] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:11.768450 20571 leader_election.cc:304] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:11.769533 21544 raft_consensus.cc:3055] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:55:11.787313 21544 raft_consensus.cc:740] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:11.787755 20761 raft_consensus.cc:1275] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Refusing update from remote peer 598db12c38d442c6988b4c4f991874fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:11.788969 21500 consensus_queue.cc:237] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:11.777602 21493 raft_consensus.cc:2804] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:11.799327 21499 leader_election.cc:290] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:11.799710 21493 raft_consensus.cc:493] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:11.800160 21493 raft_consensus.cc:3060] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:11.755594 20757 raft_consensus.cc:2468] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:11.800290 20571 leader_election.cc:304] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:11.739944 21388 raft_consensus.cc:2804] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:11.801805 20571 leader_election.cc:304] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:11.802078 21388 raft_consensus.cc:493] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:11.802474 21388 raft_consensus.cc:3060] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:11.802644 21398 raft_consensus.cc:2749] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:55:11.803504 21398 raft_consensus.cc:2804] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:11.803746 20682 raft_consensus.cc:1275] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260808 01:55:11.803970 21398 raft_consensus.cc:697] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 2 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:11.766424 21595 consensus_queue.cc:260] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:11.805392 21595 raft_consensus.cc:3060] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:11.805903 21336 raft_consensus.cc:2804] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:11.806337 21336 raft_consensus.cc:493] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:11.806726 21336 raft_consensus.cc:3060] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:11.731823 21609 consensus_queue.cc:1048] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.095s
I20260808 01:55:11.809280 21459 raft_consensus.cc:493] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:11.769490 21240 consensus_queue.cc:260] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:11.809808 21459 raft_consensus.cc:515] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:11.810873 21240 raft_consensus.cc:3060] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:11.811723 20571 leader_election.cc:304] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:11.812724 21401 raft_consensus.cc:2749] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:11.814092 20571 leader_election.cc:304] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:11.658391 20423 rpcz_store.cc:276] 0808 01:55:06.064375 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:06.073503 (+  9128us) service_pool.cc:224] Handling call
0808 01:55:11.628956 (+5555453us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:55:10.849418 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:55:10.850253 (+   835us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:55:10.850316 (+    63us) write_op.cc:435] Acquired schema lock
0808 01:55:10.850362 (+    46us) tablet.cc:662] Decoding operations
0808 01:55:10.851201 (+   839us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:55:10.851393 (+   192us) write_op.cc:641] Partition lock acquired for write op
0808 01:55:10.851435 (+    42us) tablet.cc:685] Acquiring locks for 4 operations
0808 01:55:10.852041 (+   606us) tablet.cc:701] Row locks acquired
0808 01:55:10.852074 (+    33us) write_op.cc:260] PREPARE: finished
0808 01:55:10.852306 (+   232us) write_op.cc:270] Start()
0808 01:55:10.852461 (+   155us) write_op.cc:276] Timestamp: P: 1786154110852219 usec, L: 0
0808 01:55:10.864203 (+ 11742us) op_driver.cc:348] REPLICATION: starting
0808 01:55:10.899756 (+ 35553us) log.cc:844] Serialized 1705 byte log entry
0808 01:55:11.347578 (+447822us) op_driver.cc:464] REPLICATION: finished
0808 01:55:11.490572 (+142994us) write_op.cc:301] APPLY: starting
0808 01:55:11.491830 (+  1258us) tablet.cc:1367] starting BulkCheckPresence
0808 01:55:11.496643 (+  4813us) tablet.cc:1370] finished BulkCheckPresence
0808 01:55:11.508297 (+ 11654us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:55:11.537722 (+ 29425us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:55:11.538074 (+   352us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0808 01:55:11.538140 (+    66us) write_op.cc:312] APPLY: finished
0808 01:55:11.538481 (+   341us) log.cc:844] Serialized 40 byte log entry
0808 01:55:11.543739 (+  5258us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:55:11.543945 (+   206us) write_op.cc:454] Released schema lock
0808 01:55:11.544024 (+    79us) write_op.cc:341] FINISH: Updating metrics
0808 01:55:11.544545 (+   521us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:55:11.544578 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":122459,"apply.run_cpu_time_us":3731,"apply.run_wall_time_us":54011,"num_ops":4,"prepare.queue_time_us":74873,"prepare.run_cpu_time_us":59730,"prepare.run_wall_time_us":288855,"raft.queue_time_us":167470,"raft.run_cpu_time_us":37146,"raft.run_wall_time_us":57941,"replication_time_us":480380,"thread_start_us":400613,"threads_started":4,"wal-append.queue_time_us":277378}]]}
I20260808 01:55:11.776046 21477 consensus_queue.cc:1048] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260808 01:55:11.805024 21398 consensus_queue.cc:237] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:11.815649 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
W20260808 01:55:11.817266 20651 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:11.788805 21544 consensus_queue.cc:260] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
W20260808 01:55:11.779261 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 89840.8us
W20260808 01:55:11.778370 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 117179us
I20260808 01:55:11.776257 20425 catalog_manager.cc:5697] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 1 to 2, leader changed from 598db12c38d442c6988b4c4f991874fb (127.7.165.195) to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 2 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } }
I20260808 01:55:11.832223 20648 leader_election.cc:304] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:11.835512 20682 raft_consensus.cc:3060] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:11.824905 21401 raft_consensus.cc:2804] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:11.839322 21401 raft_consensus.cc:493] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:11.841509 21240 raft_consensus.cc:2749] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:11.837199 20425 catalog_manager.cc:5697] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 1 to 2, leader changed from a80df9f992014374aeef30979cfe393e (127.7.165.193) to 598db12c38d442c6988b4c4f991874fb (127.7.165.195). New cstate: current_term: 2 leader_uuid: "598db12c38d442c6988b4c4f991874fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } }
I20260808 01:55:11.831504 21609 raft_consensus.cc:493] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:11.851024 20425 catalog_manager.cc:5697] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 2 to 4, leader changed from a80df9f992014374aeef30979cfe393e (127.7.165.193) to 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194). New cstate: current_term: 4 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } }
I20260808 01:55:11.854981 21605 consensus_queue.cc:1048] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:11.856536 20571 leader_election.cc:304] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:11.857928 21526 raft_consensus.cc:2749] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:11.857498 21621 consensus_queue.cc:1048] T 20ca72baa7bc4572a51de6b57b4dc3e9 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:11.858762 21526 consensus_queue.cc:1048] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:55:11.828182 21502 consensus_queue.cc:1048] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260808 01:55:11.861011 20649 leader_election.cc:304] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:11.861948 21608 raft_consensus.cc:2804] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:11.862022 21602 consensus_queue.cc:1048] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:55:11.862471 21608 raft_consensus.cc:493] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:11.863260 21608 raft_consensus.cc:3060] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:11.865748 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:11.847879 21401 raft_consensus.cc:3060] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:11.854398 20425 catalog_manager.cc:5697] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 1 to 2, leader changed from df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196) to 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194). New cstate: current_term: 2 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } }
I20260808 01:55:11.828366 21499 consensus_queue.cc:1048] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260808 01:55:11.875140 21609 raft_consensus.cc:515] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:11.883733 21526 consensus_queue.cc:1048] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:11.884305 21609 leader_election.cc:290] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:11.884330 20761 raft_consensus.cc:1275] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:11.886696 21544 raft_consensus.cc:3060] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:11.888847 21526 consensus_queue.cc:1048] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:11.882612 20425 catalog_manager.cc:5697] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:11.885146 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:11.827692 21568 consensus_queue.cc:1059] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260808 01:55:11.899993 20425 catalog_manager.cc:5697] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 1 to 3, leader changed from a80df9f992014374aeef30979cfe393e (127.7.165.193) to 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194). New cstate: current_term: 3 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:11.901932 20425 catalog_manager.cc:5697] T 28e98a7c32694002b806eddd7007e550 P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 1 to 3, leader changed from a80df9f992014374aeef30979cfe393e (127.7.165.193) to 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194). New cstate: current_term: 3 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:11.899355 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc85af5215440be884412c1d240c7af" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:11.905972 20686 raft_consensus.cc:1275] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:11.906288 20683 raft_consensus.cc:1275] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:11.907028 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d8bf19c324463fb9903b9981068e9d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:11.907212 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:11.907895 20680 raft_consensus.cc:3060] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:11.906359 20685 raft_consensus.cc:1275] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:11.908224 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:11.907930 20530 raft_consensus.cc:2393] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 1: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
W20260808 01:55:11.896009 20650 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:11.909821 20425 catalog_manager.cc:5697] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 1 to 2, leader changed from 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194) to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 2 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } }
I20260808 01:55:11.920521 20529 raft_consensus.cc:2468] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:11.921733 20425 catalog_manager.cc:5697] T ecc85af5215440be884412c1d240c7af P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 0 to 2, leader changed from <none> to 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194). New cstate: current_term: 2 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } }
I20260808 01:55:11.923456 20425 catalog_manager.cc:5697] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 1 to 3, leader changed from a80df9f992014374aeef30979cfe393e (127.7.165.193) to 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194). New cstate: current_term: 3 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:11.926014 20425 catalog_manager.cc:5697] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:11.929359 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:11.934950 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:11.942371 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:11.943333 20684 raft_consensus.cc:2468] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:11.925202 21336 raft_consensus.cc:515] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:11.930471 21430 raft_consensus.cc:493] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:11.946954 21336 leader_election.cc:290] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:11.947487 21430 raft_consensus.cc:515] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:11.953027 21609 raft_consensus.cc:3055] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:11.854387 21572 raft_consensus.cc:515] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:11.976560 21609 raft_consensus.cc:740] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:11.958556 20529 raft_consensus.cc:2468] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:12.008014 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:12.009406 20529 raft_consensus.cc:2393] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 2: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:11.959097 20682 raft_consensus.cc:2468] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 4.
I20260808 01:55:11.950331 21430 leader_election.cc:290] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:11.930317 21497 raft_consensus.cc:493] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:11.904470 20525 raft_consensus.cc:2468] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:11.968112 20683 raft_consensus.cc:2393] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 3: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
W20260808 01:55:12.004402 20649 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:12.047178 20761 raft_consensus.cc:3055] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 LEADER]: Stepping down as leader of term 3
W20260808 01:55:12.021045 20493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:55:12.058573 21618 log.cc:927] Time spent T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260808 01:55:12.058970 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:12.059005 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:12.059917 20529 raft_consensus.cc:2468] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:12.061072 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:12.061753 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a313d0a0979e4dc8a4205c4f16a24257" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:12.062909 21609 consensus_queue.cc:260] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:12.064770 21609 raft_consensus.cc:3060] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:12.064932 20600 raft_consensus.cc:1275] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:12.066408 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:12.066871 21497 raft_consensus.cc:515] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:12.068594 20600 raft_consensus.cc:1240] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260808 01:55:12.069525 21497 leader_election.cc:290] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:12.070204 20607 raft_consensus.cc:1240] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:12.061857 20529 raft_consensus.cc:2393] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 3: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:12.047849 20761 raft_consensus.cc:740] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
W20260808 01:55:12.048646 20493 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.195:44325 timed out after 1.533s (SENT) [suppressed 25 similar messages]
I20260808 01:55:12.080294 20528 raft_consensus.cc:1240] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:12.088361 20529 raft_consensus.cc:1275] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:12.101212 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e71aab2ce9b45369b7eff65e1645f9b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:12.102223 20682 raft_consensus.cc:2468] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:11.999966 21494 leader_election.cc:290] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:12.079900 21602 raft_consensus.cc:493] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:12.127678 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:12.130656 20682 raft_consensus.cc:2393] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 3: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:12.134692 21398 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:12.135495 21398 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:12.138033 20525 raft_consensus.cc:1275] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 1. (term mismatch)
I20260808 01:55:12.138654 20525 pending_rounds.cc:85] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e: Aborting all ops after (but not including) 0
I20260808 01:55:12.138361 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:12.111766 21477 raft_consensus.cc:493] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:12.139424 20756 raft_consensus.cc:2468] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:12.139629 21477 raft_consensus.cc:515] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:12.141646 20756 raft_consensus.cc:1275] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:12.085208 20761 consensus_queue.cc:260] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:12.145907 20761 raft_consensus.cc:3060] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:12.139019 20525 pending_rounds.cc:107] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260808 01:55:12.151311 20525 raft_consensus.cc:2889] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:55:12.151760 20525 pending_rounds.cc:107] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260808 01:55:12.152101 20525 raft_consensus.cc:2889] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:55:12.156211 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d8bf19c324463fb9903b9981068e9d" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:12.158067 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e71aab2ce9b45369b7eff65e1645f9b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:12.159215 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:12.163192 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:12.165555 20683 raft_consensus.cc:1240] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 3. Current term is 4. Ops: [1.1-3.2]
I20260808 01:55:12.162557 21617 raft_consensus.cc:493] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:12.167109 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:12.142338 21477 leader_election.cc:290] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:12.168051 20685 raft_consensus.cc:2468] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3.
I20260808 01:55:12.170027 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:12.098466 21492 raft_consensus.cc:515] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:12.241156 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:12.246344 20726 leader_election.cc:304] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:12.252718 20686 raft_consensus.cc:2376] T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3. Re-sending same reply.
I20260808 01:55:12.273725 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:55:12.275743 20651 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:55:12.282858 20494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:12.283185 20756 raft_consensus.cc:1275] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260808 01:55:12.283951 20475 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:55:12.284548 21477 raft_consensus.cc:493] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:55:12.286916 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 162829us
I20260808 01:55:12.287429 21502 consensus_queue.cc:1059] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:12.287977 20725 leader_election.cc:304] T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:12.288812 21502 raft_consensus.cc:3055] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:12.289283 21502 raft_consensus.cc:740] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:12.290892 21499 raft_consensus.cc:2749] T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260808 01:55:12.292443 20728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:12.285265 21477 raft_consensus.cc:515] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:12.295334 21477 leader_election.cc:290] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:12.290261 21502 consensus_queue.cc:260] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:12.296725 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:12.297497 21502 raft_consensus.cc:3060] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:12.297873 20678 raft_consensus.cc:2393] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 3: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:12.173146 21602 raft_consensus.cc:515] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:12.290470 20725 leader_election.cc:400] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Vote denied by peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531) with higher term. Message: Invalid argument: T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3.
I20260808 01:55:12.303898 20686 raft_consensus.cc:2468] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
W20260808 01:55:12.316371 20727 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:12.126132 21401 raft_consensus.cc:515] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
W20260808 01:55:12.335830 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 220794us
I20260808 01:55:12.337311 20682 raft_consensus.cc:2468] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:12.277033 21494 raft_consensus.cc:2804] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:55:12.329329 21628 raft_consensus.cc:493] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:12.307565 20726 leader_election.cc:304] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:12.249747 21497 raft_consensus.cc:493] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
W20260808 01:55:12.329895 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 281270us
W20260808 01:55:12.360105 20493 leader_election.cc:336] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325): Timed out: RequestConsensusVote RPC to 127.7.165.195:44325 timed out after 1.557s (SENT)
I20260808 01:55:12.360651 20493 leader_election.cc:304] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:12.167431 21617 raft_consensus.cc:515] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:12.368023 21617 leader_election.cc:290] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:12.351444 20686 raft_consensus.cc:1275] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260808 01:55:12.371297 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 63854.1us
I20260808 01:55:12.302232 21602 leader_election.cc:290] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:12.363085 21595 raft_consensus.cc:3060] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:12.363420 21622 raft_consensus.cc:493] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:12.347121 21534 raft_consensus.cc:493] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:55:12.364243 21630 log.cc:927] Time spent T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260808 01:55:12.364560 20605 raft_consensus.cc:1240] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:12.347465 20753 raft_consensus.cc:1240] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:12.383721 20493 leader_election.cc:400] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Vote denied by peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117) with higher term. Message: Invalid argument: T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e for earlier term 3. Current term is 4.
I20260808 01:55:12.362915 21628 raft_consensus.cc:515] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:12.387043 20571 leader_election.cc:400] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Vote denied by peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117) with higher term. Message: Invalid argument: T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc for earlier term 2. Current term is 3.
I20260808 01:55:12.404363 20571 leader_election.cc:403] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260808 01:55:12.358268 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:12.362008 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:12.399832 20493 leader_election.cc:403] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260808 01:55:12.383042 21499 raft_consensus.cc:2804] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260808 01:55:12.408212 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 117955us
I20260808 01:55:12.368716 20677 raft_consensus.cc:2468] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:12.439172 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:12.440696 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:12.440824 21459 leader_election.cc:290] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:12.407686 21388 consensus_meta.cc:287] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3: Time spent flushing consensus metadata: real 0.605s	user 0.011s	sys 0.003s
W20260808 01:55:12.396653 20728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:12.355973 21283 consensus_queue.cc:1048] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260808 01:55:12.378626 21494 raft_consensus.cc:697] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 4 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
W20260808 01:55:12.413897 21493 consensus_meta.cc:287] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb: Time spent flushing consensus metadata: real 0.613s	user 0.011s	sys 0.004s
I20260808 01:55:12.413700 21602 raft_consensus.cc:3060] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:12.414254 20571 leader_election.cc:304] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:12.459039 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:12.459676 20682 raft_consensus.cc:3060] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:55:12.459704 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 45538.2us
I20260808 01:55:12.414489 21544 consensus_queue.cc:1048] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:12.418785 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:12.419564 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1753378f13594e89aee6641e2cb9e6da" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:12.421070 21572 leader_election.cc:290] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:12.421537 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:12.397361 21534 raft_consensus.cc:515] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:12.425196 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:12.421602 21336 raft_consensus.cc:3060] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:55:12.427258 20650 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:12.422657 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:12.429705 21628 leader_election.cc:290] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:12.391201 21617 raft_consensus.cc:493] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:55:12.431339 21625 log.cc:927] Time spent T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260808 01:55:12.372500 21608 raft_consensus.cc:515] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:12.437798 20725 leader_election.cc:304] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:12.395426 21538 raft_consensus.cc:493] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:12.441304 21243 raft_consensus.cc:493] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:12.441414 20677 raft_consensus.cc:2468] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:12.454854 21388 raft_consensus.cc:515] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:12.375931 21497 raft_consensus.cc:515] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:12.456324 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:12.412886 21499 raft_consensus.cc:493] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:12.463884 20571 leader_election.cc:304] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:12.422638 21401 leader_election.cc:290] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:12.464313 21430 consensus_queue.cc:1048] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260808 01:55:12.466511 21620 consensus_queue.cc:1048] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260808 01:55:12.467705 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:12.484861 20682 raft_consensus.cc:2468] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 4.
I20260808 01:55:12.493108 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:12.547065 20530 raft_consensus.cc:2468] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 3.
I20260808 01:55:12.463574 21526 raft_consensus.cc:2749] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 01:55:12.496336 20649 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:55:12.495132 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 57409.7us
I20260808 01:55:12.497785 20761 raft_consensus.cc:1275] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260808 01:55:12.500654 20648 leader_election.cc:304] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:12.499929 20529 raft_consensus.cc:2468] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:12.462088 21493 raft_consensus.cc:515] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:12.391609 21622 raft_consensus.cc:515] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:12.554632 20725 leader_election.cc:304] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
W20260808 01:55:12.513547 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 49713us
I20260808 01:55:12.494561 21492 leader_election.cc:290] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:12.549695 20761 pending_rounds.cc:85] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3: Aborting all ops after (but not including) 1
I20260808 01:55:12.563198 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:12.563562 20761 pending_rounds.cc:107] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260808 01:55:12.563935 20761 raft_consensus.cc:2889] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:55:12.551790 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:12.553424 21388 leader_election.cc:290] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:12.498431 21628 raft_consensus.cc:2804] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:12.563961 20528 raft_consensus.cc:2468] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
W20260808 01:55:12.570808 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 160051us
I20260808 01:55:12.509920 21499 raft_consensus.cc:3060] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:55:12.572628 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 196476us
I20260808 01:55:12.574718 21502 consensus_queue.cc:1048] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:12.577556 20726 leader_election.cc:304] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:12.544742 21283 raft_consensus.cc:2749] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:12.498622 21617 raft_consensus.cc:515] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:12.579838 21628 raft_consensus.cc:493] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:12.597908 20762 raft_consensus.cc:2468] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:12.585194 21502 raft_consensus.cc:2804] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader election won for term 4
W20260808 01:55:12.609304 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 108778us
I20260808 01:55:12.610145 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac71e66cb4042eb8fe8aafd139af68a" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:12.503003 21538 raft_consensus.cc:515] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:12.568333 21459 raft_consensus.cc:2749] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260808 01:55:12.581933 20680 consensus_meta.cc:287] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb: Time spent flushing consensus metadata: real 0.673s	user 0.014s	sys 0.000s
I20260808 01:55:12.581070 21401 raft_consensus.cc:493] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:12.500190 21243 raft_consensus.cc:515] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:12.620124 20528 raft_consensus.cc:1275] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260808 01:55:12.584224 21609 raft_consensus.cc:2749] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:55:12.629941 20648 leader_election.cc:304] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:12.606801 21628 raft_consensus.cc:3060] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:12.615487 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:55:12.628223 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 48256.6us
I20260808 01:55:12.632295 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:12.646945 20524 raft_consensus.cc:2468] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 4.
I20260808 01:55:12.647699 20757 raft_consensus.cc:2468] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3.
I20260808 01:55:12.650959 20755 raft_consensus.cc:3060] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:12.651783 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:12.653529 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:12.654245 20683 raft_consensus.cc:3060] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:12.633675 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:12.626838 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:12.623765 20680 raft_consensus.cc:2468] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 4.
I20260808 01:55:12.636598 21502 raft_consensus.cc:697] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 4 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:12.639856 20758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:12.512898 20685 raft_consensus.cc:1240] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
W20260808 01:55:12.674494 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 118280us
W20260808 01:55:12.681921 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 109938us
I20260808 01:55:12.674944 20725 leader_election.cc:304] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:55:12.684410 20571 leader_election.cc:304] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:12.600641 21494 consensus_queue.cc:237] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:12.686174 20493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:12.688833 20648 leader_election.cc:304] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:12.666462 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:12.664831 21477 raft_consensus.cc:2749] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:55:12.647576 21622 leader_election.cc:290] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:12.632735 21401 raft_consensus.cc:515] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:12.702234 20493 leader_election.cc:304] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:12.708599 21544 raft_consensus.cc:2749] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:12.709095 20686 raft_consensus.cc:1240] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 3. Current term is 4. Ops: []
I20260808 01:55:12.702992 20726 leader_election.cc:304] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:12.690959 21500 raft_consensus.cc:2804] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:12.725543 20493 leader_election.cc:304] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:12.728317 20725 leader_election.cc:304] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
W20260808 01:55:12.721190 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 82043.3us
I20260808 01:55:12.691519 21398 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:12.732278 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 43588.1us
W20260808 01:55:12.744654 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 60423.7us
I20260808 01:55:12.696673 21492 consensus_queue.cc:1059] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260808 01:55:12.750290 20648 leader_election.cc:304] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:12.706756 20679 raft_consensus.cc:2468] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:12.731621 20526 raft_consensus.cc:1275] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:12.729883 21568 raft_consensus.cc:2749] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:55:12.702387 21283 consensus_queue.cc:1048] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20260808 01:55:12.726203 21500 raft_consensus.cc:493] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:12.760844 20648 leader_election.cc:304] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:12.725647 21534 leader_election.cc:290] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:12.763265 20683 raft_consensus.cc:2468] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 4.
I20260808 01:55:12.764102 21656 raft_consensus.cc:2749] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:12.767117 20493 leader_election.cc:304] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:12.743671 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:12.761353 21398 raft_consensus.cc:2804] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 01:55:12.779644 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 51422.6us
I20260808 01:55:12.783334 20725 leader_election.cc:304] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:55:12.760025 21477 raft_consensus.cc:493] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:12.788398 21477 raft_consensus.cc:515] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:12.790855 21477 leader_election.cc:290] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:12.791460 21477 raft_consensus.cc:2804] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:12.791860 21477 raft_consensus.cc:493] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:12.792207 21477 raft_consensus.cc:3060] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:12.772230 21500 raft_consensus.cc:3060] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:12.769634 21493 leader_election.cc:290] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:12.787150 20570 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:12.804478 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:12.781975 21645 raft_consensus.cc:2804] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:12.828938 21645 raft_consensus.cc:493] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:12.829360 21645 raft_consensus.cc:3060] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:12.829541 21608 leader_election.cc:290] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:12.830222 21608 raft_consensus.cc:493] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:12.782290 21398 raft_consensus.cc:493] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:12.830856 21398 raft_consensus.cc:3060] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:12.830729 21608 raft_consensus.cc:515] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:12.832556 21665 consensus_queue.cc:1048] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:12.834825 21617 leader_election.cc:290] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:55:12.805858 21649 log.cc:927] Time spent T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260808 01:55:12.783949 20753 raft_consensus.cc:2468] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 4.
I20260808 01:55:12.828635 21667 raft_consensus.cc:493] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
W20260808 01:55:12.810460 21640 log.cc:927] Time spent T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
W20260808 01:55:12.853394 20728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:12.854529 21631 raft_consensus.cc:3055] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:55:12.767874 21502 consensus_queue.cc:237] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:12.855029 21631 raft_consensus.cc:740] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:12.855901 21283 consensus_queue.cc:1048] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:12.774539 21497 leader_election.cc:290] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:12.826323 20681 raft_consensus.cc:2468] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:12.827561 21668 raft_consensus.cc:493] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:12.795476 20680 raft_consensus.cc:1275] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260808 01:55:12.806689 21647 log.cc:927] Time spent T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260808 01:55:12.856016 21655 consensus_queue.cc:1048] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260808 01:55:12.810561 21388 raft_consensus.cc:2804] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:55:12.863056 21538 leader_election.cc:290] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:12.871690 21668 raft_consensus.cc:515] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:12.882808 20650 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:55:12.869343 20494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:12.875000 21497 raft_consensus.cc:2804] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:12.882104 21388 raft_consensus.cc:697] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:12.884553 21243 leader_election.cc:290] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:12.884761 21497 raft_consensus.cc:697] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:12.885278 21388 consensus_queue.cc:237] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:12.885900 21497 consensus_queue.cc:237] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:12.868574 21667 raft_consensus.cc:515] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:12.828019 21492 raft_consensus.cc:3060] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:12.848206 21568 raft_consensus.cc:2804] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:12.893604 21568 raft_consensus.cc:493] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:12.894066 21568 raft_consensus.cc:3060] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:12.899513 21526 raft_consensus.cc:493] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:12.900082 21526 raft_consensus.cc:515] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:12.897966 21631 consensus_queue.cc:260] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:12.913136 21631 raft_consensus.cc:3060] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:12.962358 21401 leader_election.cc:290] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:12.965094 21526 leader_election.cc:290] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:12.968921 21668 leader_election.cc:290] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:12.970223 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 186286us
I20260808 01:55:12.974196 21667 leader_election.cc:290] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:12.977797 20755 raft_consensus.cc:1275] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:12.985510 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:12.986225 20530 raft_consensus.cc:2468] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:12.978148 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:12.986830 20728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:12.987005 21608 leader_election.cc:290] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:12.969172 21336 raft_consensus.cc:2749] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117) with higher term. Message: Invalid argument: T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc for earlier term 2. Current term is 3.
I20260808 01:55:12.978902 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:12.980775 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 197184us
I20260808 01:55:12.991359 20493 leader_election.cc:304] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:12.992270 21538 raft_consensus.cc:3060] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:12.994326 20493 leader_election.cc:304] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:12.982633 21628 raft_consensus.cc:515] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:12.996930 21628 leader_election.cc:290] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:12.984069 21526 raft_consensus.cc:493] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:55:12.984702 21595 consensus_meta.cc:287] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e: Time spent flushing consensus metadata: real 0.602s	user 0.010s	sys 0.011s
I20260808 01:55:12.983563 21401 raft_consensus.cc:2804] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:55:13.002226 20726 leader_election.cc:304] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
W20260808 01:55:13.015328 20649 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:13.038414 20570 leader_election.cc:304] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:13.000560 20493 leader_election.cc:304] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
W20260808 01:55:13.030769 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 247507us
W20260808 01:55:13.042645 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 281874us
W20260808 01:55:13.049467 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 296560us
W20260808 01:55:13.050324 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 48216.4us
I20260808 01:55:13.050534 20571 leader_election.cc:304] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:13.046927 20648 leader_election.cc:304] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:13.056226 20726 leader_election.cc:304] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:13.037979 21401 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:13.041983 21595 raft_consensus.cc:2749] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:12.969182 21499 raft_consensus.cc:515] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:13.068832 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:13.079604 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:13.053957 21336 raft_consensus.cc:2804] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:13.013713 21243 consensus_queue.cc:1048] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20260808 01:55:13.058543 21667 raft_consensus.cc:2804] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:55:13.069077 21608 raft_consensus.cc:2804] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:13.085626 20726 leader_election.cc:304] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:13.064570 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:13.039698 21526 raft_consensus.cc:515] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:13.099139 21602 consensus_meta.cc:287] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e: Time spent flushing consensus metadata: real 0.626s	user 0.002s	sys 0.021s
I20260808 01:55:13.077445 21401 raft_consensus.cc:3060] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:55:13.074438 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:13.065623 21628 consensus_queue.cc:1048] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.049s
I20260808 01:55:13.091461 21336 raft_consensus.cc:493] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:13.110608 21544 raft_consensus.cc:2804] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:55:13.112654 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63fb8c2f494a462686500ce10025a39f" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:13.067593 21668 consensus_queue.cc:1048] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.051s
I20260808 01:55:13.114156 21608 raft_consensus.cc:493] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:13.087533 21617 consensus_queue.cc:1059] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.053s
I20260808 01:55:13.136399 21602 raft_consensus.cc:2749] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117) with higher term. Message: Invalid argument: T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e for earlier term 3. Current term is 4.
I20260808 01:55:13.125540 21667 raft_consensus.cc:697] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
W20260808 01:55:13.188771 20726 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:13.187678 21669 raft_consensus.cc:2804] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:13.160882 21336 raft_consensus.cc:3060] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:13.168721 21608 raft_consensus.cc:3060] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:13.165069 21609 raft_consensus.cc:493] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:13.168109 21497 raft_consensus.cc:2804] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:13.164506 21673 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:13.161296 20684 raft_consensus.cc:2468] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:13.168184 21430 raft_consensus.cc:493] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:13.201339 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:13.204981 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:13.207110 20493 leader_election.cc:304] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:13.264936 20682 raft_consensus.cc:2376] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2. Re-sending same reply.
I20260808 01:55:13.265340 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d8bf19c324463fb9903b9981068e9d" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:13.210031 21526 leader_election.cc:290] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:13.190198 21669 raft_consensus.cc:493] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:13.231392 21492 raft_consensus.cc:2749] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:13.238163 21594 raft_consensus.cc:493] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:13.169160 21477 raft_consensus.cc:515] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:13.213857 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:13.248243 21538 raft_consensus.cc:2749] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:55:13.245751 20758 consensus_meta.cc:287] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3: Time spent flushing consensus metadata: real 0.541s	user 0.015s	sys 0.002s
I20260808 01:55:13.207788 21497 raft_consensus.cc:493] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:55:13.260789 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 198359us
I20260808 01:55:13.238026 21667 consensus_queue.cc:237] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:13.210888 21662 raft_consensus.cc:2749] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:13.206171 21609 raft_consensus.cc:515] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:13.212374 21698 raft_consensus.cc:493] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:13.202797 21604 consensus_queue.cc:1048] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.199s
I20260808 01:55:13.351445 21674 raft_consensus.cc:493] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:13.352648 21620 consensus_queue.cc:1048] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:13.330165 20758 raft_consensus.cc:2468] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
W20260808 01:55:13.362118 21398 consensus_meta.cc:287] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc: Time spent flushing consensus metadata: real 0.527s	user 0.019s	sys 0.000s
I20260808 01:55:13.348928 21497 raft_consensus.cc:3060] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:13.355986 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:13.354882 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:13.301658 21669 raft_consensus.cc:3060] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:13.379431 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:13.358871 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:13.290922 21568 raft_consensus.cc:515] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:13.302284 21430 raft_consensus.cc:515] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:13.394902 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:13.396654 21568 leader_election.cc:290] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:13.401216 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:13.401741 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:55:13.402909 21686 log.cc:927] Time spent T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
I20260808 01:55:13.270479 21673 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:13.382714 20678 raft_consensus.cc:2393] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 3: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
W20260808 01:55:13.386358 20758 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53322 (request call id 206) took 1469 ms (client timeout 1868 ms). Trace:
I20260808 01:55:13.408682 20759 raft_consensus.cc:2468] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:13.307852 21594 raft_consensus.cc:515] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:13.405072 21609 leader_election.cc:290] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:13.377249 21336 raft_consensus.cc:515] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
W20260808 01:55:13.445991 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 238974us
W20260808 01:55:13.446642 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 61788.2us
W20260808 01:55:13.469514 20649 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:13.476132 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:13.477427 20728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:13.477640 21608 raft_consensus.cc:515] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:13.478564 20493 leader_election.cc:304] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:13.479574 21710 raft_consensus.cc:2749] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:55:13.481438 21608 leader_election.cc:290] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:13.481758 20493 leader_election.cc:304] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:13.484303 20601 raft_consensus.cc:2393] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 4: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:13.448061 21656 raft_consensus.cc:2804] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:13.484613 21691 consensus_queue.cc:1059] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:13.485082 21656 raft_consensus.cc:697] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [term 2 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:13.485877 21707 raft_consensus.cc:3055] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:55:13.486340 21707 raft_consensus.cc:740] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:13.486128 21656 consensus_queue.cc:237] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:13.487264 20602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:13.487392 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:13.488029 20602 raft_consensus.cc:3060] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:13.488687 20600 raft_consensus.cc:1240] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 4 LEADER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 3. Current term is 4. Ops: []
I20260808 01:55:13.488858 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.489306 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.489828 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.489957 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:13.490747 20603 raft_consensus.cc:2393] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 2: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:13.491788 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:55:13.492899 21631 consensus_meta.cc:287] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3: Time spent flushing consensus metadata: real 0.579s	user 0.009s	sys 0.005s
I20260808 01:55:13.493013 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.474865 21430 leader_election.cc:290] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:13.493737 20606 raft_consensus.cc:2468] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:13.495829 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc85af5215440be884412c1d240c7af" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.496744 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:13.497421 20526 raft_consensus.cc:3060] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:55:13.447510 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 72600us
I20260808 01:55:13.463907 21663 raft_consensus.cc:493] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:13.499404 21538 consensus_queue.cc:1059] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:13.383843 21674 raft_consensus.cc:515] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:13.465253 21679 raft_consensus.cc:493] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:13.501240 21679 raft_consensus.cc:515] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:13.502753 21645 consensus_meta.cc:287] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e: Time spent flushing consensus metadata: real 0.673s	user 0.006s	sys 0.016s
I20260808 01:55:13.503226 21645 raft_consensus.cc:515] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:13.503693 21679 leader_election.cc:290] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:13.504302 20530 raft_consensus.cc:1240] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 3. Current term is 4. Ops: []
I20260808 01:55:13.504979 21538 raft_consensus.cc:3055] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:13.505398 21538 raft_consensus.cc:740] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:13.465472 21696 raft_consensus.cc:493] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:13.507858 21696 raft_consensus.cc:515] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:13.387048 21398 raft_consensus.cc:515] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:13.377086 21698 raft_consensus.cc:515] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:13.510509 21662 raft_consensus.cc:493] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:13.511096 21662 raft_consensus.cc:515] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:13.512431 21698 leader_election.cc:290] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:13.467013 21700 raft_consensus.cc:493] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:55:13.474841 20650 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:13.513211 21673 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:13.499925 21663 raft_consensus.cc:515] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:13.513263 21700 raft_consensus.cc:515] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
W20260808 01:55:13.515175 20728 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:13.515614 21663 leader_election.cc:290] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:13.516299 21700 leader_election.cc:290] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:13.516453 21674 leader_election.cc:290] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:13.516744 21398 leader_election.cc:290] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:13.413060 21401 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:13.517540 21534 consensus_queue.cc:1048] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:13.519179 21401 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 5 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:13.464318 21703 raft_consensus.cc:3055] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:13.519979 21703 raft_consensus.cc:740] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:13.484633 21336 leader_election.cc:290] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:13.520957 21544 raft_consensus.cc:493] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:13.521045 21703 consensus_queue.cc:260] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:13.521586 21538 consensus_queue.cc:260] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:13.524003 20607 raft_consensus.cc:2393] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 2: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:13.488066 20608 raft_consensus.cc:3060] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:13.393399 21499 leader_election.cc:290] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:13.498356 20602 raft_consensus.cc:2468] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:13.458118 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:55:13.528295 20494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:13.513422 21662 leader_election.cc:290] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:13.455626 20679 raft_consensus.cc:2468] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:13.463657 21477 leader_election.cc:290] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:13.530313 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:13.527935 20609 raft_consensus.cc:2376] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3. Re-sending same reply.
I20260808 01:55:13.510393 21696 leader_election.cc:290] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:13.530191 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:13.535907 20649 leader_election.cc:304] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:13.465186 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:55:13.455049 20758 rpcz_store.cc:269] 0808 01:55:11.916992 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:12.589486 (+672494us) service_pool.cc:224] Handling call
0808 01:55:13.383097 (+793611us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:13.540695 20528 raft_consensus.cc:2468] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:13.522567 21703 raft_consensus.cc:3060] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:13.542754 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.521617 21544 raft_consensus.cc:3060] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:55:13.545127 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:13.555444 20529 raft_consensus.cc:3060] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:13.540454 21538 raft_consensus.cc:3060] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:13.490025 20604 raft_consensus.cc:2468] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:13.549391 20649 leader_election.cc:304] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
W20260808 01:55:13.537978 21500 consensus_meta.cc:287] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3: Time spent flushing consensus metadata: real 0.690s	user 0.011s	sys 0.007s
I20260808 01:55:13.538408 20680 raft_consensus.cc:1275] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:13.544507 20530 raft_consensus.cc:2468] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:13.543548 21497 raft_consensus.cc:515] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:13.539968 21605 raft_consensus.cc:2804] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:13.560250 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:13.562729 20649 leader_election.cc:304] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
W20260808 01:55:13.563997 20725 leader_election.cc:336] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.726s (SENT)
I20260808 01:55:13.567433 20609 raft_consensus.cc:2393] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 3: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:13.552079 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:13.555441 21477 raft_consensus.cc:2749] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260808 01:55:13.569953 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 63365.5us
I20260808 01:55:13.572345 20493 leader_election.cc:304] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:13.568816 20605 raft_consensus.cc:2368] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3.
I20260808 01:55:13.562861 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:13.562096 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:13.568819 21700 raft_consensus.cc:2804] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:13.565943 21617 raft_consensus.cc:2749] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:13.563711 21500 raft_consensus.cc:515] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:13.584088 21700 raft_consensus.cc:493] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:13.592166 21700 raft_consensus.cc:3060] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:13.592249 20522 raft_consensus.cc:3060] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:13.586257 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:13.591149 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.588428 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce51fcffe2ae4bcba55cd864c973bf0d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.591571 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:13.591320 20725 leader_election.cc:336] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.883s (SENT)
I20260808 01:55:13.573666 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:13.599608 20725 leader_election.cc:304] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:13.588840 20600 raft_consensus.cc:1240] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [term 2 LEADER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:13.605891 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.606187 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:13.606794 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.607177 20525 raft_consensus.cc:2468] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 4.
I20260808 01:55:13.607278 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.608088 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.581507 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:13.609170 21659 raft_consensus.cc:493] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:13.609386 20754 raft_consensus.cc:2468] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:13.609777 21659 raft_consensus.cc:515] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:13.608472 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.590818 21713 raft_consensus.cc:2804] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:13.612107 21659 leader_election.cc:290] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:13.612160 20524 raft_consensus.cc:2468] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:13.614894 21674 raft_consensus.cc:2749] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:55:13.621392 21713 raft_consensus.cc:697] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 2 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:13.619845 21594 leader_election.cc:290] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:13.622280 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:13.622387 21713 consensus_queue.cc:237] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:13.623591 20524 raft_consensus.cc:2468] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:13.620641 20609 raft_consensus.cc:2468] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:13.622843 20649 leader_election.cc:304] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:13.598742 21497 leader_election.cc:290] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:13.629681 21499 raft_consensus.cc:493] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:13.581460 21653 consensus_queue.cc:1059] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.088s
I20260808 01:55:13.630725 21659 raft_consensus.cc:3055] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:55:13.631198 21659 raft_consensus.cc:740] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:13.631829 21673 raft_consensus.cc:493] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:13.632153 21659 consensus_queue.cc:260] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:13.632398 21673 raft_consensus.cc:515] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:13.633715 21681 log.cc:927] Time spent T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.425s	user 0.001s	sys 0.001s
I20260808 01:55:13.634877 21673 leader_election.cc:290] T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:13.635676 21645 leader_election.cc:290] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:13.642256 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 70009.2us
I20260808 01:55:13.647511 20608 raft_consensus.cc:2468] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 4.
I20260808 01:55:13.648351 20753 raft_consensus.cc:1240] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer 598db12c38d442c6988b4c4f991874fb for earlier term 1. Current term is 2. Ops: []
W20260808 01:55:13.650701 20726 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:13.646631 20493 leader_election.cc:304] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:13.645126 20529 raft_consensus.cc:2468] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 3.
I20260808 01:55:13.664554 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.633658 21477 raft_consensus.cc:2804] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:13.668318 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc85af5215440be884412c1d240c7af" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.668893 20493 leader_election.cc:304] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:13.675282 20602 raft_consensus.cc:1275] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260808 01:55:13.676051 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.677047 20600 raft_consensus.cc:2393] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 2: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:13.677397 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:13.677793 21645 raft_consensus.cc:2749] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260808 01:55:13.678040 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:55:13.678598 21715 log.cc:927] Time spent T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260808 01:55:13.680332 21500 leader_election.cc:290] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:13.681481 20522 raft_consensus.cc:2468] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 4.
I20260808 01:55:13.681658 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce51fcffe2ae4bcba55cd864c973bf0d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:13.630303 21499 raft_consensus.cc:515] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:13.682705 21716 raft_consensus.cc:2749] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:13.665967 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:13.684237 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:13.684824 21499 leader_election.cc:290] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:13.684955 20524 raft_consensus.cc:3060] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:13.682715 21669 raft_consensus.cc:515] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:13.688016 21669 leader_election.cc:290] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:13.697618 20529 raft_consensus.cc:2468] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 4.
I20260808 01:55:13.699471 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:13.701247 20648 leader_election.cc:304] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:13.706080 20760 raft_consensus.cc:2393] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 3: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:13.663295 21710 raft_consensus.cc:2804] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:13.709625 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.710501 20609 raft_consensus.cc:2376] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3. Re-sending same reply.
I20260808 01:55:13.712384 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:13.717079 20648 leader_election.cc:304] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:13.669214 21477 raft_consensus.cc:493] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:13.719554 20761 raft_consensus.cc:1240] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [term 4 LEADER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 01:55:13.671927 21659 raft_consensus.cc:3060] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:13.722059 20686 raft_consensus.cc:1240] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 01:55:13.671020 21679 raft_consensus.cc:493] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:13.728493 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:13.772069 20527 raft_consensus.cc:3060] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:13.771874 21679 raft_consensus.cc:515] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:13.719038 21673 raft_consensus.cc:2749] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:13.737160 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:13.775831 20685 raft_consensus.cc:2468] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 4.
I20260808 01:55:13.776962 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:13.777530 20685 raft_consensus.cc:3060] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:13.739467 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac71e66cb4042eb8fe8aafd139af68a" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:13.779996 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:13.740417 21716 consensus_queue.cc:1048] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260808 01:55:13.745599 20682 raft_consensus.cc:1240] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:13.782181 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:13.783694 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:13.784344 20682 raft_consensus.cc:3060] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:55:13.792758 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 58325.7us
I20260808 01:55:13.721692 21645 consensus_queue.cc:1048] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.072s
I20260808 01:55:13.797097 21534 consensus_queue.cc:1059] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 01:55:13.797467 20680 raft_consensus.cc:1275] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:13.799050 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:13.799233 21691 raft_consensus.cc:3055] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:13.799841 20680 raft_consensus.cc:2468] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:13.799870 21691 raft_consensus.cc:740] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:13.803820 21691 consensus_queue.cc:260] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:13.805338 21691 raft_consensus.cc:3060] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:13.807528 20759 raft_consensus.cc:1275] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:13.809113 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:13.809849 20759 raft_consensus.cc:2468] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
W20260808 01:55:13.811136 21721 log.cc:927] Time spent T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260808 01:55:13.711779 20600 raft_consensus.cc:2468] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:13.815096 20685 raft_consensus.cc:2468] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:13.747330 21710 raft_consensus.cc:493] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:13.747903 21702 raft_consensus.cc:2749] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:55:13.746477 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:13.755115 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:13.706215 21653 raft_consensus.cc:493] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:13.711957 21655 raft_consensus.cc:493] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:13.769183 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:13.835212 20682 raft_consensus.cc:2468] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 5.
W20260808 01:55:13.838477 20493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:13.847730 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:13.867697 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:13.845543 20725 leader_election.cc:304] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
W20260808 01:55:13.886272 20493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:13.888804 20686 raft_consensus.cc:2368] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e for earlier term 3. Current term is 4.
W20260808 01:55:13.892191 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 46762.1us
I20260808 01:55:13.892396 20493 leader_election.cc:304] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:13.734915 21477 raft_consensus.cc:3060] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:13.902988 20725 leader_election.cc:304] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:13.904881 20725 leader_election.cc:304] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:13.907220 20725 leader_election.cc:304] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:13.852550 21710 raft_consensus.cc:3060] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:13.839136 21668 raft_consensus.cc:493] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
W20260808 01:55:13.873757 21595 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.681s	user 0.040s	sys 0.051s
I20260808 01:55:13.924793 20725 leader_election.cc:304] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:13.918637 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.928659 20527 raft_consensus.cc:2468] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 4.
I20260808 01:55:13.911335 21669 raft_consensus.cc:2749] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:55:13.901039 21604 raft_consensus.cc:493] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:55:13.944542 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 52237.8us
I20260808 01:55:13.924918 20677 raft_consensus.cc:2368] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc for earlier term 3. Current term is 4.
W20260808 01:55:13.928510 21595 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.040s	sys 0.052s
I20260808 01:55:13.919772 21538 raft_consensus.cc:2749] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:55:13.914803 21500 raft_consensus.cc:2749] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:13.946993 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.924265 21497 raft_consensus.cc:2749] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:13.935684 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:13.922300 20522 raft_consensus.cc:2468] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:13.961158 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.738211 21707 consensus_queue.cc:260] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:13.979769 20609 raft_consensus.cc:2368] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3.
I20260808 01:55:13.984108 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:13.929626 21668 raft_consensus.cc:515] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:14.009936 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:13.995555 20762 raft_consensus.cc:1240] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 01:55:13.980901 20759 raft_consensus.cc:3060] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 3
I20260808 01:55:13.979245 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:14.037886 21694 raft_consensus.cc:2749] T 611b7203dac24e2e909dee3e0fc5fda7 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:14.038367 21642 raft_consensus.cc:2749] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:14.039680 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd11a6553c444d488651367f3f0143f" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:13.898376 21653 raft_consensus.cc:515] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:13.953241 21604 raft_consensus.cc:515] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:13.897272 21655 raft_consensus.cc:515] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:14.044832 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:14.035360 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:14.058095 20609 raft_consensus.cc:2376] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3. Re-sending same reply.
I20260808 01:55:14.063387 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:14.064361 20762 raft_consensus.cc:2468] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
W20260808 01:55:14.049181 20526 consensus_meta.cc:287] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e: Time spent flushing consensus metadata: real 0.551s	user 0.014s	sys 0.000s
I20260808 01:55:14.071785 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:14.072718 20762 raft_consensus.cc:2468] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3.
W20260808 01:55:14.074599 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 128803us
I20260808 01:55:14.075075 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:14.082535 20493 leader_election.cc:400] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Vote denied by peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325) with higher term. Message: Invalid argument: T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e for earlier term 3. Current term is 4.
I20260808 01:55:14.085129 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:14.043126 21544 raft_consensus.cc:515] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:14.087457 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:14.063989 20523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41410 (request call id 190) took 1600 ms. Trace:
I20260808 01:55:14.091732 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:14.094852 20609 raft_consensus.cc:3060] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:14.095489 20762 raft_consensus.cc:2393] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 3: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:14.084276 20493 leader_election.cc:304] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:14.097095 20493 leader_election.cc:304] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:14.071036 20526 raft_consensus.cc:2468] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 4.
I20260808 01:55:14.103454 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:14.105027 20762 raft_consensus.cc:2393] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 4: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
W20260808 01:55:14.099887 21700 consensus_meta.cc:287] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb: Time spent flushing consensus metadata: real 0.507s	user 0.014s	sys 0.001s
I20260808 01:55:14.100852 21716 raft_consensus.cc:2804] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:14.115187 21730 raft_consensus.cc:2804] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:14.116292 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:14.119587 20522 raft_consensus.cc:1275] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260808 01:55:14.128646 21716 raft_consensus.cc:493] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:14.131482 21730 raft_consensus.cc:493] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:14.145006 21716 raft_consensus.cc:3060] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:14.145478 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:14.133080 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:14.131415 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:14.149641 20762 raft_consensus.cc:2468] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:14.148674 21730 raft_consensus.cc:3060] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:14.154547 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:14.155308 20608 raft_consensus.cc:2376] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 4. Re-sending same reply.
I20260808 01:55:14.156201 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:14.156755 20608 raft_consensus.cc:3060] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:14.159008 20757 raft_consensus.cc:3060] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:14.161559 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:14.162380 20762 raft_consensus.cc:2393] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 4: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:14.171926 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:14.175935 20762 raft_consensus.cc:2410] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260808 01:55:14.181612 20608 raft_consensus.cc:2468] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 4.
W20260808 01:55:14.183631 20726 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:14.187851 20608 raft_consensus.cc:3055] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:55:14.188719 20608 raft_consensus.cc:740] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
W20260808 01:55:14.165359 21725 log.cc:927] Time spent T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260808 01:55:14.192368 20570 leader_election.cc:304] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:14.189750 20608 consensus_queue.cc:260] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:14.133102 21700 raft_consensus.cc:515] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:14.206893 20608 raft_consensus.cc:3060] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:14.173461 21673 raft_consensus.cc:493] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
W20260808 01:55:14.209758 21726 log.cc:927] Time spent T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260808 01:55:14.209646 21673 raft_consensus.cc:515] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:14.210469 21430 raft_consensus.cc:2804] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:14.210955 21430 raft_consensus.cc:493] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:14.211346 21430 raft_consensus.cc:3060] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:14.212080 21673 leader_election.cc:290] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:14.205972 21668 leader_election.cc:290] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:14.199910 21492 raft_consensus.cc:493] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:14.192269 21500 raft_consensus.cc:493] T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:14.206355 21493 raft_consensus.cc:493] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
W20260808 01:55:14.231874 21728 log.cc:927] Time spent T f681a9e4ae5a4909afeec36dfb20b490 P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260808 01:55:14.235366 21604 leader_election.cc:290] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:14.240058 21700 leader_election.cc:290] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:14.241480 21605 raft_consensus.cc:493] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:14.242033 21605 raft_consensus.cc:3060] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:14.225093 21492 raft_consensus.cc:515] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:14.249629 21492 leader_election.cc:290] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:14.229957 20526 raft_consensus.cc:1275] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:55:14.252450 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e71aab2ce9b45369b7eff65e1645f9b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:14.253135 21723 log.cc:927] Time spent T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
W20260808 01:55:14.260001 21659 consensus_meta.cc:287] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc: Time spent flushing consensus metadata: real 0.504s	user 0.011s	sys 0.003s
I20260808 01:55:14.261586 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a313d0a0979e4dc8a4205c4f16a24257" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:14.093327 20523 rpcz_store.cc:276] 0808 01:55:12.454328 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:12.456069 (+  1741us) service_pool.cc:224] Handling call
0808 01:55:12.468616 (+ 12547us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:12.690419 (+221803us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:12.698247 (+  7828us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:12.752530 (+ 54283us) log.cc:844] Serialized 26 byte log entry
0808 01:55:13.082886 (+330356us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:13.082955 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:13.633032 (+550077us) spinlock_profiling.cc:243] Waited 6.19 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:55:14.030898 (+397866us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b5400631854. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:14.034627 (+  3729us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:14.039328 (+  4701us) raft_consensus.cc:1685] finished
0808 01:55:14.042027 (+  2699us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:14.048916 (+  6889us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26160,"raft.queue_time_us":35697,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":13587,"spinlock_wait_cycles":406101248,"thread_start_us":253192,"threads_started":1,"wal-append.queue_time_us":394734}
I20260808 01:55:14.224994 21500 raft_consensus.cc:515] T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:14.265417 21653 leader_election.cc:290] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:14.266264 21655 leader_election.cc:290] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:14.287909 21730 raft_consensus.cc:515] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
W20260808 01:55:14.289594 20573 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:14.250116 21477 raft_consensus.cc:515] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:14.271373 21493 raft_consensus.cc:515] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:14.266273 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:55:14.225560 21727 log.cc:927] Time spent T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
I20260808 01:55:14.284778 20684 raft_consensus.cc:2393] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 3: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:14.232208 20525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:14.285606 21500 leader_election.cc:290] T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:14.291811 20754 raft_consensus.cc:2410] T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260808 01:55:14.290665 21605 raft_consensus.cc:515] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:14.306385 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:14.307302 20603 raft_consensus.cc:2393] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 4: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:14.308223 21605 leader_election.cc:290] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:14.313784 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:14.330096 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:14.330700 21707 raft_consensus.cc:3060] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:14.330971 20681 raft_consensus.cc:3060] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:14.331357 20726 leader_election.cc:304] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:14.332532 20678 raft_consensus.cc:1275] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:55:14.333119 20726 leader_election.cc:304] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:14.334803 21499 raft_consensus.cc:2804] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:14.335273 21499 raft_consensus.cc:493] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:14.335666 21499 raft_consensus.cc:3060] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:14.292964 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 5 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:14.302273 20608 raft_consensus.cc:1275] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260808 01:55:14.338835 20608 pending_rounds.cc:85] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc: Aborting all ops after (but not including) 1
I20260808 01:55:14.339064 20608 pending_rounds.cc:107] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260808 01:55:14.339268 20608 raft_consensus.cc:2889] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:55:14.341889 20609 raft_consensus.cc:2410] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260808 01:55:14.303298 20524 consensus_meta.cc:287] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e: Time spent flushing consensus metadata: real 0.618s	user 0.019s	sys 0.000s
I20260808 01:55:14.298224 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:14.303313 20604 raft_consensus.cc:1275] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:14.304039 20756 raft_consensus.cc:1240] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:14.297320 21477 leader_election.cc:290] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:14.292840 21493 leader_election.cc:290] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:14.383499 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d32821d9c17456394a662fcfd05f6c6" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:14.391916 20685 raft_consensus.cc:1275] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260808 01:55:14.394114 21653 raft_consensus.cc:493] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:14.302202 21710 raft_consensus.cc:515] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
W20260808 01:55:14.395802 20571 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:14.368506 20524 raft_consensus.cc:2468] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 4.
I20260808 01:55:14.405021 20528 raft_consensus.cc:1275] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:55:14.407181 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8fc615366d544af38cf664c9ad47ab0c" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:14.408205 20685 raft_consensus.cc:3060] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:14.349937 21674 consensus_queue.cc:1048] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
W20260808 01:55:14.411888 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 432932us
I20260808 01:55:14.415689 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a313d0a0979e4dc8a4205c4f16a24257" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:14.420399 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 5 candidate_status { last_received { term: 4 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:14.394769 20680 raft_consensus.cc:1275] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:55:14.394135 21730 leader_election.cc:290] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:14.394963 21659 raft_consensus.cc:493] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:14.429243 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac71e66cb4042eb8fe8aafd139af68a" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:14.429384 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:14.431324 20686 raft_consensus.cc:1275] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260808 01:55:14.434532 20686 pending_rounds.cc:85] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb: Aborting all ops after (but not including) 0
I20260808 01:55:14.434942 20686 pending_rounds.cc:107] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260808 01:55:14.435242 20686 raft_consensus.cc:2889] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:55:14.438297 20725 leader_election.cc:304] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:14.444748 21631 raft_consensus.cc:2804] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:14.445345 21631 raft_consensus.cc:493] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:14.439211 20528 raft_consensus.cc:2468] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:14.402207 21477 consensus_queue.cc:1059] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260808 01:55:14.448450 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:14.427770 20527 raft_consensus.cc:1275] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:55:14.421981 20759 raft_consensus.cc:2468] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:14.440282 20605 raft_consensus.cc:2393] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 2: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:14.464717 20683 raft_consensus.cc:2393] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 2: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:14.435971 21493 raft_consensus.cc:493] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:14.468833 20685 raft_consensus.cc:2468] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 4.
I20260808 01:55:14.445757 21631 raft_consensus.cc:3060] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:14.403939 21653 raft_consensus.cc:515] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
W20260808 01:55:14.470969 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 253237us
I20260808 01:55:14.469726 20648 leader_election.cc:304] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:14.480036 20570 leader_election.cc:304] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:14.488838 20648 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:14.493965 21544 leader_election.cc:290] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 5 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:14.495512 20683 raft_consensus.cc:1275] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:55:14.496156 21622 raft_consensus.cc:493] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:14.496872 21622 raft_consensus.cc:515] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:14.497124 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:14.460626 21700 raft_consensus.cc:493] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:14.499367 21700 raft_consensus.cc:515] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:14.500916 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 5 candidate_status { last_received { term: 4 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:14.503167 21702 raft_consensus.cc:2804] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:14.503715 21702 raft_consensus.cc:493] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:14.504068 21702 raft_consensus.cc:3060] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:14.480216 21703 raft_consensus.cc:493] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:14.480664 21655 consensus_queue.cc:1048] T e9201fed5ac044119ff493c519d289a7 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260808 01:55:14.464103 21667 consensus_queue.cc:1048] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20260808 01:55:14.481395 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:14.488198 21662 raft_consensus.cc:2764] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260808 01:55:14.510027 21664 raft_consensus.cc:2764] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260808 01:55:14.511323 21703 raft_consensus.cc:515] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:14.527382 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:14.528088 20762 raft_consensus.cc:3060] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:14.514860 20726 leader_election.cc:304] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:14.441296 21659 raft_consensus.cc:515] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
W20260808 01:55:14.553352 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 142923us
I20260808 01:55:14.463747 21668 consensus_queue.cc:1048] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
I20260808 01:55:14.476643 21401 raft_consensus.cc:493] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
W20260808 01:55:14.558514 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 54232.7us
I20260808 01:55:14.486330 21663 raft_consensus.cc:2804] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:14.547585 20726 leader_election.cc:304] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:14.530112 21703 leader_election.cc:290] T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:14.512547 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:14.523068 20526 raft_consensus.cc:3060] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 3
I20260808 01:55:14.517751 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:14.514261 21679 leader_election.cc:290] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:14.523540 21710 leader_election.cc:290] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:14.513474 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:14.502449 21700 leader_election.cc:290] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:14.506683 20681 raft_consensus.cc:2468] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 4.
I20260808 01:55:14.470716 21733 consensus_queue.cc:1048] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.105s
I20260808 01:55:14.552500 21642 raft_consensus.cc:2749] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:14.517876 21667 consensus_queue.cc:1048] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20260808 01:55:14.477123 21493 raft_consensus.cc:515] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:14.511739 20757 raft_consensus.cc:2468] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 3.
I20260808 01:55:14.515060 21694 consensus_queue.cc:1048] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20260808 01:55:14.570006 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:14.627761 20759 raft_consensus.cc:3060] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 3
I20260808 01:55:14.606745 21674 consensus_queue.cc:1048] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.027s
I20260808 01:55:14.607668 20754 raft_consensus.cc:2468] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 3.
W20260808 01:55:14.634541 20494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:14.636255 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:55:14.638089 21737 log.cc:927] Time spent T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260808 01:55:14.617419 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53306 (request call id 204) took 1074 ms. Trace:
W20260808 01:55:14.645988 20493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:14.590341 21477 raft_consensus.cc:493] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:14.605990 20520 raft_consensus.cc:2468] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:14.647586 21477 raft_consensus.cc:515] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:14.642516 20757 rpcz_store.cc:276] 0808 01:55:13.543116 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:13.544903 (+  1787us) service_pool.cc:224] Handling call
0808 01:55:14.122656 (+577753us) spinlock_profiling.cc:243] Waited 4.37 ms on lock 0x7b5800184e98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d4185589e 000000000038d209 00007f5d49914940 00007f5d499dced4 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:14.614369 (+491713us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10059008}
I20260808 01:55:14.652274 21659 leader_election.cc:290] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:14.655467 21653 leader_election.cc:290] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:14.656680 21622 leader_election.cc:290] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
W20260808 01:55:14.659406 21749 log.cc:927] Time spent T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
W20260808 01:55:14.608655 20572 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:55:14.666792 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 178028us
I20260808 01:55:14.600625 21500 raft_consensus.cc:2804] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:55:14.568351 21401 raft_consensus.cc:515] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:14.650403 21477 leader_election.cc:290] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:14.674247 20648 leader_election.cc:304] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:14.609310 20678 raft_consensus.cc:3060] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:14.672847 21430 raft_consensus.cc:515] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:14.681726 20648 leader_election.cc:304] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:14.609890 21716 raft_consensus.cc:515] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:14.607969 21679 consensus_queue.cc:1048] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260808 01:55:14.684639 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:14.704680 20648 leader_election.cc:304] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:14.707712 20525 raft_consensus.cc:2393] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 3: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:14.679701 21500 raft_consensus.cc:697] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [term 4 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:14.685246 20526 raft_consensus.cc:2468] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 3.
I20260808 01:55:14.710424 20648 leader_election.cc:304] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:14.700088 21703 raft_consensus.cc:2804] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:55:14.698052 21477 raft_consensus.cc:2804] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:14.721299 20758 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53322 (request call id 216) took 1008 ms. Trace:
W20260808 01:55:14.721635 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 185553us
I20260808 01:55:14.731679 20762 raft_consensus.cc:2468] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 3.
W20260808 01:55:14.718433 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 170966us
I20260808 01:55:14.700764 21702 raft_consensus.cc:515] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:14.737515 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:14.746997 20761 raft_consensus.cc:3060] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:14.751303 20570 leader_election.cc:304] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:14.756402 20726 leader_election.cc:304] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:14.714367 21499 raft_consensus.cc:515] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:14.764417 20685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:14.765599 20685 raft_consensus.cc:2393] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 4: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:14.762331 21622 consensus_queue.cc:1048] T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.004s
I20260808 01:55:14.764411 21642 raft_consensus.cc:2804] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:55:14.767797 21642 raft_consensus.cc:697] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
W20260808 01:55:14.756762 20573 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:14.744789 21747 raft_consensus.cc:493] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:14.791330 21747 raft_consensus.cc:515] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:14.794093 21747 leader_election.cc:290] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:14.795035 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:14.769053 21642 consensus_queue.cc:237] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:14.803310 20759 raft_consensus.cc:2468] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 3.
I20260808 01:55:14.804924 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d32821d9c17456394a662fcfd05f6c6" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:14.806555 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:14.807926 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1753378f13594e89aee6641e2cb9e6da" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:14.811844 20528 raft_consensus.cc:1240] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 3. Current term is 4. Ops: []
I20260808 01:55:14.813647 20530 raft_consensus.cc:1240] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:14.746070 21401 leader_election.cc:290] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:14.764529 21700 raft_consensus.cc:2804] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:14.762648 21653 raft_consensus.cc:2804] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:55:14.781953 21702 leader_election.cc:290] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:14.759940 21667 consensus_queue.cc:1048] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260808 01:55:14.784279 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:14.787209 21499 leader_election.cc:290] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:14.786912 21500 consensus_queue.cc:237] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:14.788671 21716 leader_election.cc:290] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:14.790066 21430 leader_election.cc:290] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:14.803587 20492 leader_election.cc:336] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.560s (SENT)
I20260808 01:55:14.805202 20527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41392 (request call id 181) took 1063 ms. Trace:
W20260808 01:55:14.814579 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 260076us
I20260808 01:55:14.901031 20527 rpcz_store.cc:276] 0808 01:55:13.741499 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:14.107723 (+366224us) service_pool.cc:224] Handling call
0808 01:55:14.408930 (+301207us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:14.463345 (+ 54415us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:14.804956 (+341611us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b540016e9d0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:14.805131 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":691746688}
I20260808 01:55:14.811844 20759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:14.870407 21673 raft_consensus.cc:3055] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:14.904950 20571 leader_election.cc:304] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:14.905591 21673 raft_consensus.cc:740] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
W20260808 01:55:14.778142 21742 log.cc:927] Time spent T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.188s	user 0.000s	sys 0.004s
I20260808 01:55:14.779927 21493 leader_election.cc:290] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:14.908847 21746 consensus_queue.cc:1048] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:55:14.909022 20760 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53322 (request call id 225) took 1168 ms. Trace:
I20260808 01:55:14.910980 21430 raft_consensus.cc:2749] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260808 01:55:14.909720 20760 rpcz_store.cc:276] 0808 01:55:13.740584 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:14.300031 (+559447us) service_pool.cc:224] Handling call
0808 01:55:14.300427 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:14.301149 (+   722us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:14.301181 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:14.301751 (+   570us) log.cc:844] Serialized 26 byte log entry
0808 01:55:14.482575 (+180824us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:14.503918 (+ 21343us) log.cc:844] Serialized 14 byte log entry
0808 01:55:14.504782 (+   864us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:14.504923 (+   141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:14.713567 (+208644us) spinlock_profiling.cc:243] Waited 8.58 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:55:14.908570 (+195003us) raft_consensus.cc:1685] finished
0808 01:55:14.908740 (+   170us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:14.908939 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20382,"prepare.queue_time_us":20686,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":3749,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":223,"spinlock_wait_cycles":19726208,"thread_start_us":180389,"threads_started":1,"wal-append.queue_time_us":287407}
I20260808 01:55:14.919719 20571 leader_election.cc:304] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: 
I20260808 01:55:14.921579 21663 raft_consensus.cc:493] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:14.922209 21663 raft_consensus.cc:3060] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:14.923024 20571 leader_election.cc:304] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:14.924373 20525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41392 (request call id 183) took 1182 ms. Trace:
I20260808 01:55:14.925026 21679 raft_consensus.cc:2804] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:55:14.924932 20571 leader_election.cc:304] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:14.921661 21746 raft_consensus.cc:2804] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:14.935521 20522 raft_consensus.cc:1240] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:14.935827 21746 raft_consensus.cc:697] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:14.937434 21746 consensus_queue.cc:237] T 38b9406154bf4f86bf631ea9b73424df P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:14.738721 20758 rpcz_store.cc:276] 0808 01:55:13.713216 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:13.715751 (+  2535us) service_pool.cc:224] Handling call
0808 01:55:14.506996 (+791245us) spinlock_profiling.cc:243] Waited 85 ms on lock 0x7b5800184e98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:55:14.507274 (+   278us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:14.507789 (+   515us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:14.507830 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:14.508548 (+   718us) log.cc:844] Serialized 26 byte log entry
0808 01:55:14.560180 (+ 51632us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:55:14.572813 (+ 12633us) log.cc:844] Serialized 14 byte log entry
0808 01:55:14.717394 (+144581us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:14.720807 (+  3413us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:14.720893 (+    86us) raft_consensus.cc:1685] finished
0808 01:55:14.721028 (+   135us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:14.721227 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12126,"prepare.queue_time_us":12671,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":50622,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":208,"spinlock_wait_cycles":195464448,"thread_start_us":191414,"threads_started":2,"wal-append.queue_time_us":51517}
I20260808 01:55:14.942056 20756 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53322 (request call id 215) took 1234 ms. Trace:
I20260808 01:55:14.942700 20756 rpcz_store.cc:276] 0808 01:55:13.707342 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:13.708509 (+  1167us) service_pool.cc:224] Handling call
0808 01:55:14.303801 (+595292us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:14.389194 (+ 85393us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:14.941700 (+552506us) spinlock_profiling.cc:243] Waited 511 ms on lock 0x7b54006d9cd0. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:14.941969 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1175291264}
I20260808 01:55:14.928357 21679 raft_consensus.cc:697] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [term 5 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:14.944818 21679 consensus_queue.cc:237] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:14.894227 20678 raft_consensus.cc:2468] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:14.953605 20571 leader_election.cc:400] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Vote denied by peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325) with higher term. Message: Invalid argument: T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc for earlier term 3. Current term is 4.
I20260808 01:55:14.954258 20571 leader_election.cc:403] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260808 01:55:14.934082 21534 raft_consensus.cc:493] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:14.960567 21430 raft_consensus.cc:3060] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:14.960740 21534 raft_consensus.cc:515] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
W20260808 01:55:14.819805 20475 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:55:14.840337 20425 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48838 (request call id 86) took 3573 ms. Trace:
I20260808 01:55:14.845949 20755 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53306 (request call id 203) took 1303 ms. Trace:
W20260808 01:55:14.845840 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 239161us
W20260808 01:55:14.849323 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 138985us
I20260808 01:55:14.848397 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
W20260808 01:55:14.877978 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 74424.5us
I20260808 01:55:14.881543 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd11a6553c444d488651367f3f0143f" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:14.883975 21662 consensus_queue.cc:1059] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:14.838735 21700 raft_consensus.cc:493] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:14.895419 21398 consensus_queue.cc:1059] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:55:14.898097 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:14.847354 21653 raft_consensus.cc:697] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [term 4 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:14.986313 20725 leader_election.cc:304] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:55:14.931267 21659 consensus_queue.cc:1048] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260808 01:55:14.958335 21655 consensus_queue.cc:1059] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260808 01:55:14.959270 21631 raft_consensus.cc:515] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:14.906795 21673 consensus_queue.cc:260] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:14.965763 20726 leader_election.cc:304] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:14.971318 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:14.972160 20649 leader_election.cc:304] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:14.949662 21401 raft_consensus.cc:2749] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:14.924854 20525 rpcz_store.cc:276] 0808 01:55:13.742256 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:13.801128 (+ 58872us) service_pool.cc:224] Handling call
0808 01:55:14.231966 (+430838us) spinlock_profiling.cc:243] Waited 37.8 ms on lock 0x7b58000d9e98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:14.924108 (+692142us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x7b5400915fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:14.924287 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1035004928}
I20260808 01:55:14.988277 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:14.968999 21702 raft_consensus.cc:493] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:14.963603 20425 rpcz_store.cc:276] 0808 01:55:11.247408 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:11.247719 (+   311us) service_pool.cc:224] Handling call
0808 01:55:14.820941 (+3573222us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:55:12.408649 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:55:12.421693 (+ 13044us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:55:12.421728 (+    35us) write_op.cc:435] Acquired schema lock
0808 01:55:12.421741 (+    13us) tablet.cc:662] Decoding operations
0808 01:55:12.423100 (+  1359us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:55:12.423167 (+    67us) write_op.cc:641] Partition lock acquired for write op
0808 01:55:12.423198 (+    31us) tablet.cc:685] Acquiring locks for 13 operations
0808 01:55:12.424568 (+  1370us) tablet.cc:701] Row locks acquired
0808 01:55:12.424582 (+    14us) write_op.cc:260] PREPARE: finished
0808 01:55:12.424739 (+   157us) write_op.cc:270] Start()
0808 01:55:12.424864 (+   125us) write_op.cc:276] Timestamp: P: 1786154112424688 usec, L: 0
0808 01:55:12.424879 (+    15us) op_driver.cc:348] REPLICATION: starting
0808 01:55:12.425704 (+   825us) log.cc:844] Serialized 5197 byte log entry
0808 01:55:12.759900 (+334196us) op_driver.cc:464] REPLICATION: finished
0808 01:55:13.230989 (+471089us) write_op.cc:301] APPLY: starting
0808 01:55:13.239302 (+  8313us) tablet.cc:1367] starting BulkCheckPresence
0808 01:55:13.239616 (+   314us) tablet.cc:1370] finished BulkCheckPresence
0808 01:55:13.239827 (+   211us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:55:13.241492 (+  1665us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:55:13.241748 (+   256us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=13
0808 01:55:13.241798 (+    50us) write_op.cc:312] APPLY: finished
0808 01:55:13.242144 (+   346us) log.cc:844] Serialized 94 byte log entry
0808 01:55:13.243091 (+   947us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:55:13.243295 (+   204us) write_op.cc:454] Released schema lock
0808 01:55:13.243352 (+    57us) write_op.cc:341] FINISH: Updating metrics
0808 01:55:13.243770 (+   418us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:55:13.243806 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":430329,"apply.run_cpu_time_us":5010,"apply.run_wall_time_us":38781,"num_ops":13,"prepare.queue_time_us":435121,"prepare.run_cpu_time_us":5081,"prepare.run_wall_time_us":35026,"replication_time_us":323360,"thread_start_us":537318,"threads_started":2}]]}
I20260808 01:55:14.994297 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:14.995965 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:14.993237 20648 leader_election.cc:304] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:14.996459 20762 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53302 (request call id 251) took 1149 ms. Trace:
I20260808 01:55:15.000299 20682 raft_consensus.cc:2393] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 3: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:15.003804 20528 raft_consensus.cc:2468] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 4.
I20260808 01:55:14.983246 21700 raft_consensus.cc:3060] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:15.010176 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:15.011302 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:15.015585 20528 raft_consensus.cc:2468] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 4.
I20260808 01:55:15.018980 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:14.987236 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:14.987754 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 5 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:14.999899 21703 raft_consensus.cc:697] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 4 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:15.035666 20726 leader_election.cc:304] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:15.009626 20648 leader_election.cc:304] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:14.993614 21499 raft_consensus.cc:2804] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:15.038439 20648 leader_election.cc:304] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:14.817068 20761 raft_consensus.cc:2468] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 4.
I20260808 01:55:15.046979 20493 leader_election.cc:304] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:15.049769 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:15.050798 20682 raft_consensus.cc:2468] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 4.
I20260808 01:55:14.996803 21492 consensus_queue.cc:1048] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260808 01:55:15.010076 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e71aab2ce9b45369b7eff65e1645f9b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:15.012120 21493 raft_consensus.cc:2804] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:15.008481 21762 raft_consensus.cc:2804] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:15.060665 20757 raft_consensus.cc:1240] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:15.012252 21756 raft_consensus.cc:2804] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:15.010490 20753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:15.063493 20608 raft_consensus.cc:2468] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 4.
I20260808 01:55:14.997825 21751 consensus_queue.cc:1059] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20260808 01:55:15.081043 21673 raft_consensus.cc:3060] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:15.111040 21674 raft_consensus.cc:2749] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:15.018105 21702 raft_consensus.cc:515] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:15.033296 20602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:55:15.119395 21758 log.cc:927] Time spent T fdef2db6e2e04aa795d92ad39538d90b P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260808 01:55:15.105595 21534 leader_election.cc:290] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:55:15.123775 20725 leader_election.cc:336] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.557s (SENT)
I20260808 01:55:15.121759 20426 catalog_manager.cc:5697] T 51bbd1ceb9fb4045b31d153ff959c663 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:15.040726 21499 raft_consensus.cc:493] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:15.125245 21499 raft_consensus.cc:3060] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:15.100713 20754 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53302 (request call id 247) took 1245 ms. Trace:
W20260808 01:55:15.107018 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 71474.6us
I20260808 01:55:15.107733 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:15.121815 20684 raft_consensus.cc:1275] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:15.084306 20493 leader_election.cc:304] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
W20260808 01:55:15.142608 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 58436.1us
I20260808 01:55:15.055831 21696 raft_consensus.cc:2804] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader election won for term 4
W20260808 01:55:15.158900 20725 leader_election.cc:336] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.635s (SENT)
I20260808 01:55:15.127708 20426 catalog_manager.cc:5697] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 1 to 2, leader changed from a80df9f992014374aeef30979cfe393e (127.7.165.193) to df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196). New cstate: current_term: 2 leader_uuid: "df82e7f402364d369ed1ec3c5a8772b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } }
W20260808 01:55:15.168886 20494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:55:15.161141 20493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:15.104915 21747 raft_consensus.cc:2804] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:55:15.200230 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:15.101516 21733 raft_consensus.cc:493] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:15.201331 20761 raft_consensus.cc:2468] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3.
I20260808 01:55:15.101963 21762 raft_consensus.cc:493] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:15.103744 21493 raft_consensus.cc:697] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [term 3 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:15.200026 20426 catalog_manager.cc:5697] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:15.107821 21401 raft_consensus.cc:3055] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:55:15.108584 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:15.163501 21767 raft_consensus.cc:2804] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:55:15.217229 20426 catalog_manager.cc:5697] T a313d0a0979e4dc8a4205c4f16a24257 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 1, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 1 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:15.166086 21534 raft_consensus.cc:2804] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:55:15.164090 21477 raft_consensus.cc:697] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [term 3 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
W20260808 01:55:15.213696 20725 leader_election.cc:336] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.687s (SENT)
I20260808 01:55:15.232566 20426 catalog_manager.cc:5697] T 0add41ce95b74822ab26d48b72ccafe0 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: HEALTHY } } }
I20260808 01:55:15.201804 21696 raft_consensus.cc:697] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [term 4 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:15.241866 20725 leader_election.cc:304] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:15.224182 21401 raft_consensus.cc:740] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:15.247655 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:15.258001 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:15.263863 20682 raft_consensus.cc:2393] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 2: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:15.264001 20426 catalog_manager.cc:5697] T f681a9e4ae5a4909afeec36dfb20b490 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:15.212265 21659 raft_consensus.cc:493] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:15.285898 20426 catalog_manager.cc:5697] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 0 to 3, leader changed from <none> to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 3 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } }
W20260808 01:55:15.293366 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 79705.3us
I20260808 01:55:15.284422 20571 leader_election.cc:304] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:15.218509 21762 raft_consensus.cc:3060] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:55:15.299922 20725 leader_election.cc:336] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.653s (SENT)
I20260808 01:55:15.301196 20426 catalog_manager.cc:5697] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 1 to 2, leader changed from 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194) to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 2 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:15.123838 21703 consensus_queue.cc:237] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:15.234086 21534 raft_consensus.cc:493] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260808 01:55:15.317355 20648 leader_election.cc:336] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.824s (SENT)
I20260808 01:55:15.323469 20648 leader_election.cc:304] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:15.217458 21733 raft_consensus.cc:515] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:15.304850 21669 raft_consensus.cc:2749] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:55:15.313369 20426 catalog_manager.cc:5697] T 711b7cabb802484094416a0e94a14803 P a80df9f992014374aeef30979cfe393e reported cstate change: term changed from 1 to 2, leader changed from 598db12c38d442c6988b4c4f991874fb (127.7.165.195) to a80df9f992014374aeef30979cfe393e (127.7.165.193). New cstate: current_term: 2 leader_uuid: "a80df9f992014374aeef30979cfe393e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:15.302628 20571 leader_election.cc:304] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:55:15.311599 21774 raft_consensus.cc:493] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:15.355556 20682 raft_consensus.cc:1275] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:15.205984 21653 consensus_queue.cc:237] T 0590088b74d04c6b96146266db7a6266 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:15.322330 21763 raft_consensus.cc:493] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:15.328944 21534 raft_consensus.cc:3060] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:55:15.325660 21664 raft_consensus.cc:2749] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:55:15.325426 21756 raft_consensus.cc:697] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [term 3 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:15.314188 21659 raft_consensus.cc:515] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
W20260808 01:55:15.371263 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 68729.9us
I20260808 01:55:15.392035 20682 raft_consensus.cc:1240] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 2. Current term is 3. Ops: [2.1-2.1]
I20260808 01:55:15.367733 21777 raft_consensus.cc:2804] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:15.399950 20571 leader_election.cc:304] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:15.395162 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63fb8c2f494a462686500ce10025a39f" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:15.386433 21770 raft_consensus.cc:493] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:15.370158 21679 consensus_queue.cc:1059] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.103s
I20260808 01:55:15.357453 21776 consensus_queue.cc:1048] T 4b877573d5d6413da0a0af3fe2ecb25b P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.120s
I20260808 01:55:15.407424 21738 raft_consensus.cc:2804] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:15.407478 21777 raft_consensus.cc:493] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:15.366828 21696 consensus_queue.cc:237] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
W20260808 01:55:15.496712 20725 leader_election.cc:336] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.965s (SENT)
I20260808 01:55:15.506585 20725 leader_election.cc:304] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:15.126230 20754 rpcz_store.cc:276] 0808 01:55:13.836098 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:14.640233 (+804135us) service_pool.cc:224] Handling call
0808 01:55:14.661557 (+ 21324us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:14.661792 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:55:14.662712 (+   920us) log.cc:844] Serialized 14 byte log entry
0808 01:55:15.020083 (+357371us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:15.032345 (+ 12262us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:15.046938 (+ 14593us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:15.057645 (+ 10707us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:15.081765 (+ 24120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18337,"prepare.queue_time_us":179,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":6325,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":20867,"thread_start_us":336858,"threads_started":1,"wal-append.queue_time_us":441332}
I20260808 01:55:15.352051 21493 consensus_queue.cc:237] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:15.358238 21401 consensus_queue.cc:260] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:15.365091 21774 raft_consensus.cc:515] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
W20260808 01:55:15.495416 21752 log.cc:927] Time spent T 13ab4fe0e17342379d043078e08ca66b P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 0.385s	user 0.000s	sys 0.002s
I20260808 01:55:14.966639 20755 rpcz_store.cc:276] 0808 01:55:13.542761 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:13.551631 (+  8870us) service_pool.cc:224] Handling call
0808 01:55:14.131729 (+580098us) spinlock_profiling.cc:243] Waited 2.96 ms on lock 0x7b5800184e98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:55:14.135826 (+  4097us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:14.145000 (+  9174us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:55:14.466019 (+321019us) log.cc:844] Serialized 14 byte log entry
0808 01:55:14.845464 (+379445us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:14.845526 (+    62us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:14.845588 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:14.845669 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:14.845836 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":317595,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":73,"raft.queue_time_us":107,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":97,"spinlock_wait_cycles":6809984,"thread_start_us":607873,"threads_started":2,"wal-append.queue_time_us":416719}
I20260808 01:55:15.514178 21492 raft_consensus.cc:2749] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:15.502547 21738 raft_consensus.cc:493] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:15.385452 21763 raft_consensus.cc:515] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:15.367890 21477 consensus_queue.cc:237] T 56010b7f5e3c48ed97bfe024f84ad2ae P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:15.507464 21777 raft_consensus.cc:3060] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:55:15.514040 20725 leader_election.cc:336] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.968s (SENT)
I20260808 01:55:15.505342 21789 raft_consensus.cc:2749] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:55:15.527410 21765 log.cc:927] Time spent T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.312s	user 0.002s	sys 0.000s
I20260808 01:55:15.561638 20725 leader_election.cc:304] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:15.494712 21770 raft_consensus.cc:515] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
W20260808 01:55:15.567855 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 53850.9us
W20260808 01:55:15.573729 20725 leader_election.cc:336] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.698s (SENT)
I20260808 01:55:15.553900 21738 raft_consensus.cc:3060] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:15.553834 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:15.546855 21790 raft_consensus.cc:2749] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 01:55:15.588367 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 178809us
I20260808 01:55:15.589277 20571 leader_election.cc:304] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:15.553174 21401 raft_consensus.cc:3060] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:55:15.591423 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 590319us
W20260808 01:55:15.575601 21764 log.cc:927] Time spent T a313d0a0979e4dc8a4205c4f16a24257 P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.243s	user 0.002s	sys 0.000s
I20260808 01:55:15.594543 20649 leader_election.cc:304] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:15.576363 21787 raft_consensus.cc:2749] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:15.581923 21776 raft_consensus.cc:3055] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:55:15.582708 21789 raft_consensus.cc:493] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:55:15.596630 21769 log.cc:927] Time spent T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.244s	user 0.001s	sys 0.001s
I20260808 01:55:15.544339 21756 consensus_queue.cc:237] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:15.620919 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 491897us
I20260808 01:55:15.623286 20571 leader_election.cc:304] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:15.594424 21544 raft_consensus.cc:493] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
W20260808 01:55:15.610188 21772 log.cc:927] Time spent T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.223s	user 0.001s	sys 0.001s
I20260808 01:55:15.595405 21733 leader_election.cc:290] T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:15.627408 20682 raft_consensus.cc:1275] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260808 01:55:15.603502 21776 raft_consensus.cc:740] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:15.629741 21398 raft_consensus.cc:493] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:15.630132 21763 leader_election.cc:290] T 4f6c2d3c4da449688d1fca55d391ead2 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:15.630841 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:15.641693 21544 raft_consensus.cc:515] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:15.650456 21796 raft_consensus.cc:493] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:15.679806 21796 raft_consensus.cc:515] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
W20260808 01:55:15.631615 21788 log.cc:927] Time spent T aecd7bfdd8a746239184356052321a48 P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260808 01:55:15.633926 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53302 (request call id 248) took 1797 ms. Trace:
I20260808 01:55:15.621479 21679 raft_consensus.cc:2749] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260808 01:55:15.635933 21781 log.cc:927] Time spent T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260808 01:55:15.596401 21786 raft_consensus.cc:493] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
W20260808 01:55:15.640928 21430 consensus_meta.cc:287] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc: Time spent flushing consensus metadata: real 0.679s	user 0.013s	sys 0.002s
W20260808 01:55:15.639212 21783 log.cc:927] Time spent T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
W20260808 01:55:15.684857 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 90398.6us
I20260808 01:55:15.622280 21710 raft_consensus.cc:493] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:15.678098 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:55:15.686617 20725 leader_election.cc:336] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 2.065s (SENT)
W20260808 01:55:15.717582 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 94368.9us
W20260808 01:55:15.785626 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 99051.1us
W20260808 01:55:15.786350 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 99312.9us
W20260808 01:55:15.648417 21663 consensus_meta.cc:287] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3: Time spent flushing consensus metadata: real 0.718s	user 0.016s	sys 0.000s
W20260808 01:55:15.786437 20725 leader_election.cc:336] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 2.055s (SENT)
W20260808 01:55:15.817373 20725 leader_election.cc:336] T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 2.114s (SENT)
W20260808 01:55:15.818638 20725 leader_election.cc:336] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 2.131s (SENT)
I20260808 01:55:15.647914 21659 leader_election.cc:290] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
W20260808 01:55:15.670253 21673 consensus_meta.cc:287] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb: Time spent flushing consensus metadata: real 0.559s	user 0.009s	sys 0.004s
W20260808 01:55:15.919684 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 132476us
I20260808 01:55:15.670598 21763 raft_consensus.cc:2804] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:15.819427 20725 leader_election.cc:304] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
W20260808 01:55:15.783756 20493 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:55:15.676017 21773 log.cc:927] Time spent T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 0.306s	user 0.002s	sys 0.003s
W20260808 01:55:15.852001 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 66907.2us
I20260808 01:55:15.673293 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:15.668090 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:15.679978 21544 leader_election.cc:290] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:55:15.993707 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 206532us
I20260808 01:55:15.006888 20762 rpcz_store.cc:276] 0808 01:55:13.847228 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:14.752413 (+905185us) service_pool.cc:224] Handling call
0808 01:55:14.752862 (+   449us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:14.753354 (+   492us) raft_consensus.cc:1534] Early marking committed up to index 1
0808 01:55:14.761626 (+  8272us) log.cc:844] Serialized 14 byte log entry
0808 01:55:14.996087 (+234461us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:14.996156 (+    69us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:14.996200 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:14.996266 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:14.996400 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":7982,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":2947,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":8640,"thread_start_us":232369,"threads_started":1}
I20260808 01:55:15.784634 21786 raft_consensus.cc:515] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:15.785825 21787 consensus_queue.cc:1059] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.164s
I20260808 01:55:15.818670 21801 raft_consensus.cc:493] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:15.950763 21754 consensus_queue.cc:1048] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:55:15.682555 21796 leader_election.cc:290] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:15.775134 21430 raft_consensus.cc:2749] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325) with higher term. Message: Invalid argument: T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc for earlier term 3. Current term is 4.
I20260808 01:55:15.684270 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:15.932493 20682 raft_consensus.cc:2468] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:15.786437 21492 raft_consensus.cc:493] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:15.950299 20677 raft_consensus.cc:1275] T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260808 01:55:15.621320 21789 raft_consensus.cc:515] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:16.053339 21789 leader_election.cc:290] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
W20260808 01:55:16.019994 20475 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260808 01:55:15.678859 21398 raft_consensus.cc:515] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:15.818732 21710 raft_consensus.cc:515] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:15.985311 21763 raft_consensus.cc:697] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [term 3 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:16.055883 21398 leader_election.cc:290] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:16.055577 21763 consensus_queue.cc:237] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:16.056948 20755 raft_consensus.cc:2393] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 2: Already voted for candidate df82e7f402364d369ed1ec3c5a8772b3 in this term.
I20260808 01:55:15.821457 21663 raft_consensus.cc:515] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:16.063269 21770 leader_election.cc:290] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:16.032509 21800 consensus_queue.cc:1059] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260808 01:55:16.040329 20522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41392 (request call id 194) took 1069 ms. Trace:
W20260808 01:55:16.041570 21793 log.cc:927] Time spent T fdef2db6e2e04aa795d92ad39538d90b P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.420s	user 0.001s	sys 0.000s
I20260808 01:55:16.043308 21786 leader_election.cc:290] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:16.047086 21803 raft_consensus.cc:493] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:16.016863 21544 raft_consensus.cc:493] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:15.732889 20757 rpcz_store.cc:276] 0808 01:55:13.836361 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:14.649386 (+813025us) service_pool.cc:224] Handling call
0808 01:55:14.649830 (+   444us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:15.106112 (+456282us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:15.633623 (+527511us) spinlock_profiling.cc:243] Waited 380 ms on lock 0x7b54008372d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:15.633855 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":873684224}
I20260808 01:55:16.022118 20603 raft_consensus.cc:2468] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3.
I20260808 01:55:16.034963 21500 raft_consensus.cc:493] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
W20260808 01:55:16.103277 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 284645us
I20260808 01:55:16.080116 21669 raft_consensus.cc:2749] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260808 01:55:16.117774 20725 leader_election.cc:336] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 2.173s (SENT)
I20260808 01:55:16.078485 21663 leader_election.cc:290] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:16.047142 21790 raft_consensus.cc:493] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:16.101926 21774 leader_election.cc:290] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:16.111014 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 249) took 1116 ms. Trace:
I20260808 01:55:16.079756 21492 raft_consensus.cc:515] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:16.153074 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "611b7203dac24e2e909dee3e0fc5fda7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:16.154285 20761 raft_consensus.cc:1240] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 3. Current term is 4. Ops: []
W20260808 01:55:16.157747 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 198913us
I20260808 01:55:16.023777 21801 raft_consensus.cc:515] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
W20260808 01:55:16.152878 20725 leader_election.cc:336] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 2.216s (SENT)
I20260808 01:55:16.101866 21544 raft_consensus.cc:515] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:16.118247 21500 raft_consensus.cc:515] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:16.155339 21669 raft_consensus.cc:493] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:16.161933 21811 raft_consensus.cc:3055] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:55:16.162036 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:16.102345 21803 raft_consensus.cc:515] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:16.183583 21767 raft_consensus.cc:493] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting leader election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
W20260808 01:55:16.184185 21806 log.cc:927] Time spent T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260808 01:55:16.182132 21663 consensus_queue.cc:1048] T b9a194d9a9eb40d397c76e44235f7aed P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260808 01:55:16.182466 21811 raft_consensus.cc:740] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:16.184847 21747 raft_consensus.cc:493] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting leader election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:16.185057 21500 leader_election.cc:290] T 022d6236053d4519bff37cbef43a6170 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:16.185536 21710 leader_election.cc:290] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:16.186219 21818 consensus_queue.cc:1048] T 8fc615366d544af38cf664c9ad47ab0c P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260808 01:55:16.189536 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:16.228164 20677 raft_consensus.cc:3060] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:55:16.189929 21401 consensus_meta.cc:287] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc: Time spent flushing consensus metadata: real 0.593s	user 0.009s	sys 0.001s
I20260808 01:55:16.192525 21813 consensus_queue.cc:1048] T d3997511f7d7419985d678e8fa59e5d0 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260808 01:55:16.230974 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 190) took 1143 ms. Trace:
I20260808 01:55:16.136755 21790 raft_consensus.cc:515] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:16.192564 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:16.231472 20601 rpcz_store.cc:276] 0808 01:55:15.087233 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.120390 (+1033157us) service_pool.cc:224] Handling call
0808 01:55:16.230902 (+110512us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.232542 20608 raft_consensus.cc:2410] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:55:16.232793 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:16.136426 20603 rpcz_store.cc:276] 0808 01:55:14.994503 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:14.997043 (+  2540us) service_pool.cc:224] Handling call
0808 01:55:15.673124 (+676081us) spinlock_profiling.cc:243] Waited 6.4 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d4185589e 000000000038d209 00007f5d49914940 00007f5d499dced4 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:16.105823 (+432699us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14715776}
W20260808 01:55:16.180879 21762 consensus_meta.cc:287] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb: Time spent flushing consensus metadata: real 0.854s	user 0.014s	sys 0.000s
I20260808 01:55:16.233870 21762 raft_consensus.cc:515] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:16.234241 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:16.212538 21767 raft_consensus.cc:3060] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:55:16.235159 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 196) took 1143 ms. Trace:
I20260808 01:55:16.235562 20603 rpcz_store.cc:276] 0808 01:55:15.092264 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.233895 (+1141631us) service_pool.cc:224] Handling call
0808 01:55:16.235092 (+  1197us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.236131 21762 leader_election.cc:290] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:55:16.180025 21534 consensus_meta.cc:287] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e: Time spent flushing consensus metadata: real 0.794s	user 0.014s	sys 0.000s
W20260808 01:55:16.194598 21777 consensus_meta.cc:287] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc: Time spent flushing consensus metadata: real 0.616s	user 0.005s	sys 0.003s
I20260808 01:55:16.237047 21534 raft_consensus.cc:515] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:16.179466 21733 raft_consensus.cc:493] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:16.078265 20522 rpcz_store.cc:276] 0808 01:55:14.970960 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:14.975058 (+  4098us) service_pool.cc:224] Handling call
0808 01:55:15.056522 (+ 81464us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:15.103601 (+ 47079us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:15.103640 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:15.119552 (+ 15912us) log.cc:844] Serialized 26 byte log entry
0808 01:55:15.304093 (+184541us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:15.320690 (+ 16597us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:15.332994 (+ 12304us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:16.035461 (+702467us) raft_consensus.cc:1685] finished
0808 01:55:16.035505 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:16.040228 (+  4723us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":876,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":6225,"thread_start_us":165395,"threads_started":1,"wal-append.queue_time_us":288416}
W20260808 01:55:16.224123 21738 consensus_meta.cc:287] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc: Time spent flushing consensus metadata: real 0.626s	user 0.008s	sys 0.005s
I20260808 01:55:16.239431 21534 leader_election.cc:290] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 5 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:16.239287 21738 raft_consensus.cc:515] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:16.233075 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 187) took 1189 ms. Trace:
I20260808 01:55:16.207823 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:16.238173 21733 raft_consensus.cc:515] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
W20260808 01:55:16.243654 20607 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 266) took 1228 ms (client timeout 1503 ms). Trace:
I20260808 01:55:16.237136 21777 raft_consensus.cc:515] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:16.196208 21801 leader_election.cc:290] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:16.236749 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:16.194500 21492 leader_election.cc:290] T 27e69fc628bc49cd8d6c653f82ef8c1d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:16.242563 20608 rpcz_store.cc:276] 0808 01:55:15.043741 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.192222 (+1148481us) service_pool.cc:224] Handling call
0808 01:55:16.233007 (+ 40785us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.251724 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:16.252249 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44770 (request call id 243) took 1271 ms. Trace:
W20260808 01:55:16.244068 20607 rpcz_store.cc:269] 0808 01:55:15.015932 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:15.028824 (+ 12892us) service_pool.cc:224] Handling call
0808 01:55:16.207559 (+1178735us) spinlock_profiling.cc:243] Waited 6.38 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:16.243583 (+ 36024us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14671232}
I20260808 01:55:16.240593 21747 raft_consensus.cc:3060] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:55:16.250548 21544 leader_election.cc:290] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:16.252436 20608 raft_consensus.cc:2468] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 3.
I20260808 01:55:16.253564 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:16.259374 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 202) took 1041 ms. Trace:
I20260808 01:55:16.259891 20608 rpcz_store.cc:276] 0808 01:55:15.218078 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.251416 (+1033338us) service_pool.cc:224] Handling call
0808 01:55:16.259277 (+  7861us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.261206 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:16.261909 20608 raft_consensus.cc:3060] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:16.248864 21824 raft_consensus.cc:493] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:16.249178 21803 leader_election.cc:290] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:16.267158 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:16.267401 21824 raft_consensus.cc:515] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:16.268079 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 265) took 1253 ms. Trace:
I20260808 01:55:16.268256 20677 raft_consensus.cc:2468] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 4.
I20260808 01:55:16.268539 20607 rpcz_store.cc:276] 0808 01:55:15.015249 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.266848 (+1251599us) service_pool.cc:224] Handling call
0808 01:55:16.268007 (+  1159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.269688 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:16.248915 21777 leader_election.cc:290] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:16.266573 20601 raft_consensus.cc:2393] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 3: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
W20260808 01:55:16.240494 21700 consensus_meta.cc:287] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb: Time spent flushing consensus metadata: real 1.189s	user 0.015s	sys 0.000s
W20260808 01:55:16.272989 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 91182.2us
I20260808 01:55:16.274829 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 200) took 1108 ms. Trace:
I20260808 01:55:16.240300 21814 consensus_queue.cc:1048] T 13ab4fe0e17342379d043078e08ca66b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260808 01:55:16.275424 20607 rpcz_store.cc:276] 0808 01:55:15.166106 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.269415 (+1103309us) service_pool.cc:224] Handling call
0808 01:55:16.274748 (+  5333us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.275807 21824 leader_election.cc:290] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:16.252444 21763 raft_consensus.cc:493] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:16.276693 21763 raft_consensus.cc:515] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:16.276818 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:16.277995 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 195) took 1186 ms. Trace:
I20260808 01:55:16.254172 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 264) took 1243 ms. Trace:
I20260808 01:55:16.278909 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:16.279506 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e71aab2ce9b45369b7eff65e1645f9b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:16.279847 20520 raft_consensus.cc:2368] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for earlier term 4. Current term is 5.
I20260808 01:55:16.280330 20684 raft_consensus.cc:2393] T 6e71aab2ce9b45369b7eff65e1645f9b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 2: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:16.280383 20520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41392 (request call id 195) took 1309 ms. Trace:
I20260808 01:55:16.280865 20520 rpcz_store.cc:276] 0808 01:55:14.971288 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:14.975234 (+  3946us) service_pool.cc:224] Handling call
0808 01:55:16.278712 (+1303478us) spinlock_profiling.cc:243] Waited 3.48 ms on lock 0x7b58000d9e98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:16.280317 (+  1605us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8007552}
I20260808 01:55:16.281544 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 194) took 1192 ms. Trace:
I20260808 01:55:16.253198 21811 consensus_queue.cc:260] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:16.279238 20603 rpcz_store.cc:276] 0808 01:55:15.014667 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.236421 (+1221754us) service_pool.cc:224] Handling call
0808 01:55:16.254090 (+ 17669us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.252723 20600 rpcz_store.cc:276] 0808 01:55:14.986654 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:15.004512 (+ 17858us) service_pool.cc:224] Handling call
0808 01:55:15.067399 (+ 62887us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:15.103529 (+ 36130us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:15.103582 (+    53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:15.103640 (+    58us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:15.107233 (+  3593us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:16.251948 (+1144715us) spinlock_profiling.cc:243] Waited 35.8 ms on lock 0x7b54007ba7d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:16.252012 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:16.252160 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":82424704}
I20260808 01:55:16.285063 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e71aab2ce9b45369b7eff65e1645f9b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:16.286068 20600 raft_consensus.cc:2468] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:16.286887 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 204) took 1040 ms. Trace:
I20260808 01:55:16.287403 20600 rpcz_store.cc:276] 0808 01:55:15.246558 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.284668 (+1038110us) service_pool.cc:224] Handling call
0808 01:55:16.286784 (+  2116us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.288251 21733 leader_election.cc:290] T 56010b7f5e3c48ed97bfe024f84ad2ae P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:16.206331 21669 raft_consensus.cc:515] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:16.278463 20607 rpcz_store.cc:276] 0808 01:55:15.091917 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.276439 (+1184522us) service_pool.cc:224] Handling call
0808 01:55:16.277916 (+  1477us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.281478 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1753378f13594e89aee6641e2cb9e6da" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:16.296669 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:16.283957 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:16.299324 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac71e66cb4042eb8fe8aafd139af68a" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:16.331614 20607 raft_consensus.cc:2393] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 2: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:16.300451 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:16.301331 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d8bf19c324463fb9903b9981068e9d" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:55:16.301566 21499 consensus_meta.cc:287] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3: Time spent flushing consensus metadata: real 1.176s	user 0.015s	sys 0.000s
I20260808 01:55:16.300473 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:16.302070 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:16.257304 21738 leader_election.cc:290] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:16.398502 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 201) took 1228 ms. Trace:
W20260808 01:55:16.398312 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 88433us
I20260808 01:55:16.401208 20607 rpcz_store.cc:276] 0808 01:55:15.170316 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.290152 (+1119836us) service_pool.cc:224] Handling call
0808 01:55:16.398420 (+108268us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.404570 20608 raft_consensus.cc:2468] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 4.
I20260808 01:55:16.431180 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 262) took 1420 ms. Trace:
I20260808 01:55:16.431927 20608 rpcz_store.cc:276] 0808 01:55:15.010257 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.260833 (+1250576us) service_pool.cc:224] Handling call
0808 01:55:16.431099 (+170266us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.288715 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:16.408804 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:16.435637 20762 raft_consensus.cc:2468] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:16.282164 20601 rpcz_store.cc:276] 0808 01:55:15.091445 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.232454 (+1141009us) service_pool.cc:224] Handling call
0808 01:55:16.281252 (+ 48798us) spinlock_profiling.cc:243] Waited 20.7 ms on lock 0x7b5400afc154. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:16.281467 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":47679744}
W20260808 01:55:16.437649 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 1.21176e+06us
I20260808 01:55:16.438905 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:55:16.440438 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 232797us
I20260808 01:55:16.289002 21700 raft_consensus.cc:515] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:16.439961 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:16.431136 21811 raft_consensus.cc:3060] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
W20260808 01:55:16.443259 20492 connection.cc:256] RPC call timeout handler was delayed by -1.0554980599845294s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20850000000000002s for a response
I20260808 01:55:16.444569 20607 raft_consensus.cc:2468] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 3.
I20260808 01:55:16.447268 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 199) took 1301 ms. Trace:
I20260808 01:55:16.430243 21716 raft_consensus.cc:493] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:16.447670 20607 rpcz_store.cc:276] 0808 01:55:15.146259 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.402368 (+1256109us) service_pool.cc:224] Handling call
0808 01:55:16.447207 (+ 44839us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.434152 21733 raft_consensus.cc:493] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:16.445838 20608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 188) took 1376 ms. Trace:
W20260808 01:55:16.449985 20726 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:16.450222 20608 rpcz_store.cc:276] 0808 01:55:15.069660 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.438510 (+1368850us) service_pool.cc:224] Handling call
0808 01:55:16.445748 (+  7238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.452260 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:16.435339 21814 raft_consensus.cc:493] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:16.451041 20726 leader_election.cc:304] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:16.433774 20755 raft_consensus.cc:2468] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:16.446648 21669 leader_election.cc:290] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:16.453892 21716 raft_consensus.cc:515] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:16.461270 21716 leader_election.cc:290] T 4b877573d5d6413da0a0af3fe2ecb25b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:16.453796 20492 leader_election.cc:304] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb
I20260808 01:55:16.463021 21655 raft_consensus.cc:3055] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:55:16.463511 21655 raft_consensus.cc:740] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:16.464372 21655 consensus_queue.cc:260] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:16.465888 21655 raft_consensus.cc:3060] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:55:16.465965 21779 log.cc:927] Time spent T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc: Append to log took a long time: real 0.981s	user 0.008s	sys 0.012s
I20260808 01:55:16.444018 21500 raft_consensus.cc:493] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:16.455859 20686 raft_consensus.cc:1240] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:16.452174 21790 leader_election.cc:290] T b9a194d9a9eb40d397c76e44235f7aed P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:16.454787 21667 raft_consensus.cc:493] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:16.457579 20605 raft_consensus.cc:1240] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:16.452033 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 261) took 1442 ms. Trace:
I20260808 01:55:16.446386 21763 leader_election.cc:290] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:16.433367 21499 raft_consensus.cc:515] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:16.479223 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44770 (request call id 245) took 1486 ms. Trace:
I20260808 01:55:16.480082 21499 leader_election.cc:290] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:16.479832 20605 rpcz_store.cc:276] 0808 01:55:14.993250 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:15.007805 (+ 14555us) service_pool.cc:224] Handling call
0808 01:55:16.457062 (+1449257us) spinlock_profiling.cc:243] Waited 6.89 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:55:16.457329 (+   267us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:16.478831 (+ 21502us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:16.479128 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15847296}
I20260808 01:55:16.481520 20761 raft_consensus.cc:2368] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for earlier term 3. Current term is 4.
I20260808 01:55:16.481773 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:16.451267 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:16.454284 21710 raft_consensus.cc:493] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:16.475369 20492 leader_election.cc:304] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:16.467944 21500 raft_consensus.cc:515] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:16.471215 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:16.454576 21786 raft_consensus.cc:493] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:16.477766 20600 rpcz_store.cc:276] 0808 01:55:15.010006 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.288348 (+1278342us) service_pool.cc:224] Handling call
0808 01:55:16.449502 (+161154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.494658 20605 raft_consensus.cc:2410] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:55:16.484033 21716 raft_consensus.cc:2804] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:55:16.505098 21716 raft_consensus.cc:697] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [term 4 LEADER]: Becoming Leader. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:16.506273 21716 consensus_queue.cc:237] T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:16.497989 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:16.510521 20684 raft_consensus.cc:3060] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:16.499138 21826 raft_consensus.cc:2804] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:55:16.484021 21667 raft_consensus.cc:515] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:16.490522 21544 raft_consensus.cc:2749] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:55:16.456118 21733 raft_consensus.cc:515] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:16.495486 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:16.499963 21700 leader_election.cc:290] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:16.500437 21679 raft_consensus.cc:3055] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:55:16.521588 21679 raft_consensus.cc:740] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:16.522900 20679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46954 (request call id 287) took 1553 ms. Trace:
I20260808 01:55:16.522769 21679 consensus_queue.cc:260] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:16.526332 21631 leader_election.cc:290] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:16.501439 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:16.503268 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 198) took 1393 ms. Trace:
I20260808 01:55:16.503763 21786 raft_consensus.cc:515] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:16.472340 21814 raft_consensus.cc:515] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:16.519047 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 186) took 1502 ms. Trace:
I20260808 01:55:16.524219 21679 raft_consensus.cc:3060] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:16.534663 20649 leader_election.cc:304] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:16.520589 21767 raft_consensus.cc:515] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:16.492239 21710 raft_consensus.cc:515] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:16.537863 21826 raft_consensus.cc:697] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 4 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:16.562546 21826 consensus_queue.cc:237] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:16.563657 21710 leader_election.cc:290] T e9201fed5ac044119ff493c519d289a7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:16.564435 20609 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44770 (request call id 252) took 1561 ms. Trace:
I20260808 01:55:16.564857 20609 rpcz_store.cc:276] 0808 01:55:15.002566 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:15.002923 (+   357us) service_pool.cc:224] Handling call
0808 01:55:15.003233 (+   310us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:15.292482 (+289249us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:15.292516 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:15.347301 (+ 54785us) log.cc:844] Serialized 26 byte log entry
0808 01:55:15.530816 (+183515us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:15.535466 (+  4650us) log.cc:844] Serialized 14 byte log entry
0808 01:55:16.285344 (+749878us) spinlock_profiling.cc:243] Waited 1.12 ms on lock 0x7b18003a6e40. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4188cd33 00007f5d4188f121 00007f5d4188f0d9 00007f5d4188f069 00007f5d4188f031 00007f5d4188e32d 00000000003aa554 00000000003aa399 00007f5d418701ea
0808 01:55:16.563928 (+278584us) spinlock_profiling.cc:243] Waited 3.19 ms on lock 0x7b18003a6e40. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4453a6ac 00007f5d4186fe3b 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d41889e05 00007f5d4498b440 00007f5d4499daf6 00007f5d449ae667
0808 01:55:16.564076 (+   148us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:16.564170 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:16.564211 (+    41us) raft_consensus.cc:1685] finished
0808 01:55:16.564226 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:16.564364 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11065,"prepare.queue_time_us":4767,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":204197,"raft.run_cpu_time_us":443,"raft.run_wall_time_us":37789,"spinlock_wait_cycles":9927424,"thread_start_us":481450,"threads_started":3,"wal-append.queue_time_us":261508}
I20260808 01:55:16.537003 20601 rpcz_store.cc:276] 0808 01:55:15.109487 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.443892 (+1334405us) service_pool.cc:224] Handling call
0808 01:55:16.503175 (+ 59283us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.539916 20605 rpcz_store.cc:276] 0808 01:55:15.016844 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.481437 (+1464593us) service_pool.cc:224] Handling call
0808 01:55:16.518968 (+ 37531us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.523522 20679 rpcz_store.cc:276] 0808 01:55:14.969129 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:15.233805 (+264676us) service_pool.cc:224] Handling call
0808 01:55:15.321699 (+ 87894us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:15.350239 (+ 28540us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:55:15.624699 (+274460us) log.cc:844] Serialized 14 byte log entry
0808 01:55:16.508819 (+884120us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b1800471e20. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4453a6ac 00007f5d4186fe3b 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d4453242a
0808 01:55:16.522269 (+ 13450us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:16.522373 (+   104us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:16.522449 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:16.522558 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:16.522785 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":426133,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":72,"raft.queue_time_us":12050,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":3792,"spinlock_wait_cycles":27723008,"thread_start_us":548867,"threads_started":2,"wal-append.queue_time_us":458449}
W20260808 01:55:16.547149 21775 log.cc:927] Time spent T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.997s	user 0.009s	sys 0.023s
I20260808 01:55:16.547936 21667 leader_election.cc:290] T ecc85af5215440be884412c1d240c7af P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
W20260808 01:55:16.589474 20492 leader_election.cc:336] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.503s (SENT)
I20260808 01:55:16.549220 21814 leader_election.cc:290] T 0add41ce95b74822ab26d48b72ccafe0 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:16.551859 20756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53306 (request call id 216) took 1569 ms (client timeout 1615 ms). Trace:
I20260808 01:55:16.553861 21786 leader_election.cc:290] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:16.554970 20604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 246) took 1561 ms (client timeout 1503 ms). Trace:
I20260808 01:55:16.557372 21733 leader_election.cc:290] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:16.558118 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:16.558751 21700 raft_consensus.cc:2804] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260808 01:55:16.559931 21782 log.cc:927] Time spent T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 0.911s	user 0.004s	sys 0.011s
I20260808 01:55:16.550225 21776 consensus_queue.cc:260] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:16.542338 20649 leader_election.cc:304] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:16.546393 21500 leader_election.cc:290] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
W20260808 01:55:16.599808 20725 leader_election.cc:336] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.528s (SENT)
W20260808 01:55:16.607813 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 65560.7us
I20260808 01:55:16.609009 20725 leader_election.cc:304] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
I20260808 01:55:16.612519 21500 raft_consensus.cc:2749] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260808 01:55:16.600554 20604 rpcz_store.cc:269] 0808 01:55:14.993614 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:14.996882 (+  3268us) service_pool.cc:224] Handling call
0808 01:55:16.492599 (+1495717us) spinlock_profiling.cc:243] Waited 7.38 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d4185589e 000000000038d209 00007f5d49914940 00007f5d499dced4 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:16.554843 (+ 62244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16972544}
I20260808 01:55:16.613853 20758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:16.614712 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44770 (request call id 257) took 1606 ms. Trace:
I20260808 01:55:16.615221 20604 rpcz_store.cc:276] 0808 01:55:15.008812 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.613515 (+1604703us) service_pool.cc:224] Handling call
0808 01:55:16.614007 (+   492us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:16.614262 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:16.614295 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:16.614347 (+    52us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:16.614396 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:16.614464 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:16.614619 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.618235 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44770 (request call id 259) took 1608 ms. Trace:
I20260808 01:55:16.618341 20680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 212) took 1826 ms. Trace:
I20260808 01:55:16.618853 20680 rpcz_store.cc:276] 0808 01:55:14.791962 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:14.792182 (+   220us) service_pool.cc:224] Handling call
0808 01:55:14.792720 (+   538us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:55:15.011650 (+218930us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:15.029076 (+ 17426us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:55:15.119274 (+ 90198us) log.cc:844] Serialized 52 byte log entry
0808 01:55:15.361132 (+241858us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:15.367627 (+  6495us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:15.372726 (+  5099us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:16.559211 (+1186485us) spinlock_profiling.cc:243] Waited 3.16 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:55:16.617997 (+ 58786us) raft_consensus.cc:1685] finished
0808 01:55:16.618021 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:16.618259 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":112776,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":98,"spinlock_wait_cycles":7261440,"thread_start_us":424646,"threads_started":2,"wal-append.queue_time_us":479287}
W20260808 01:55:16.596386 20756 rpcz_store.cc:269] 0808 01:55:14.982814 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:14.986634 (+  3820us) service_pool.cc:224] Handling call
0808 01:55:16.501220 (+1514586us) spinlock_profiling.cc:243] Waited 6.91 ms on lock 0x7b5800184e98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d4185584c 00007f5d4185540b 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:16.551767 (+ 50547us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15882624}
I20260808 01:55:16.621153 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:16.622277 20680 raft_consensus.cc:2393] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 4: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:16.622867 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:16.621153 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:16.623682 21776 raft_consensus.cc:3060] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:16.624300 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:16.626562 20603 raft_consensus.cc:2468] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3.
W20260808 01:55:16.627285 20603 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 267) took 1611 ms (client timeout 2044 ms). Trace:
I20260808 01:55:16.612284 21798 raft_consensus.cc:2749] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:16.610471 21824 raft_consensus.cc:2804] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:16.628396 21824 raft_consensus.cc:493] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:16.628785 21824 raft_consensus.cc:3060] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:16.627156 20604 rpcz_store.cc:276] 0808 01:55:15.009408 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.616316 (+1606908us) service_pool.cc:224] Handling call
0808 01:55:16.616682 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:16.617796 (+  1114us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:16.617849 (+    53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:16.617922 (+    73us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:16.617960 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:16.618040 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:16.618169 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.631027 21767 leader_election.cc:290] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:16.606287 20677 raft_consensus.cc:3060] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 4
W20260808 01:55:16.642928 20492 leader_election.cc:336] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 2.021s (SENT)
I20260808 01:55:16.604343 21700 raft_consensus.cc:493] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:16.609833 20605 raft_consensus.cc:3060] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:16.654378 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:16.656214 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:16.657176 20520 raft_consensus.cc:2468] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:16.658195 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:16.652323 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:16.667716 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d8bf19c324463fb9903b9981068e9d" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:16.668820 20529 raft_consensus.cc:2468] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
I20260808 01:55:16.669932 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:16.671525 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:16.677104 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:16.677943 20529 raft_consensus.cc:2468] T 28e98a7c32694002b806eddd7007e550 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 3.
W20260808 01:55:16.652299 20572 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:16.682307 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:16.652663 21700 raft_consensus.cc:3060] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:16.651638 21816 raft_consensus.cc:493] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
W20260808 01:55:16.627878 20603 rpcz_store.cc:269] 0808 01:55:15.016212 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.283612 (+1267400us) service_pool.cc:224] Handling call
0808 01:55:16.627216 (+343604us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.689756 20528 raft_consensus.cc:2393] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 3: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:16.660535 20604 raft_consensus.cc:1240] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:16.687991 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac71e66cb4042eb8fe8aafd139af68a" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:16.689034 21774 raft_consensus.cc:493] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:16.696014 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:16.701558 20759 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53302 (request call id 253) took 2853 ms. Trace:
I20260808 01:55:16.738595 20529 raft_consensus.cc:2468] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:16.711138 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349d2dad09a3492ca13f8784fbdb961b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:16.695052 21816 raft_consensus.cc:515] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:16.736236 20648 leader_election.cc:304] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:16.702868 21747 raft_consensus.cc:515] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:16.751566 20648 leader_election.cc:304] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:16.726832 20604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44770 (request call id 260) took 1717 ms. Trace:
I20260808 01:55:16.755856 20648 leader_election.cc:304] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:16.755160 21763 raft_consensus.cc:2804] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:16.760310 21763 raft_consensus.cc:493] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting leader election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:16.760361 20755 raft_consensus.cc:1275] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Refusing update from remote peer 598db12c38d442c6988b4c4f991874fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:16.760398 21767 raft_consensus.cc:2804] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:16.761003 21763 raft_consensus.cc:3060] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:16.761319 21767 raft_consensus.cc:493] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:16.761809 21767 raft_consensus.cc:3060] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:16.764587 20602 raft_consensus.cc:1275] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Refusing update from remote peer 598db12c38d442c6988b4c4f991874fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:16.765470 20602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44780 (request call id 191) took 1677 ms. Trace:
I20260808 01:55:16.765822 20602 rpcz_store.cc:276] 0808 01:55:15.087619 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.763726 (+1676107us) service_pool.cc:224] Handling call
0808 01:55:16.764155 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:16.765122 (+   967us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:16.765399 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.766309 20605 raft_consensus.cc:1275] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:16.786468 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:16.782248 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d8bf19c324463fb9903b9981068e9d" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:16.781318 20725 leader_election.cc:400] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Vote denied by peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531) with higher term. Message: Invalid argument: T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for earlier term 4. Current term is 5.
I20260808 01:55:16.789104 20725 leader_election.cc:403] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:55:16.787582 20760 raft_consensus.cc:2468] T 566d023f94bb449d8cf248744cd5b6ba P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 2.
I20260808 01:55:16.781004 20648 leader_election.cc:304] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:16.772226 21816 leader_election.cc:290] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:16.796657 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:16.750756 20758 raft_consensus.cc:2410] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for term 3 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260808 01:55:16.757567 20522 raft_consensus.cc:1275] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Refusing update from remote peer 598db12c38d442c6988b4c4f991874fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:55:16.784895 20685 raft_consensus.cc:1240] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 01:55:16.752125 20524 raft_consensus.cc:3060] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:55:16.792474 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:16.728977 21774 raft_consensus.cc:515] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:16.793781 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:16.771086 20649 leader_election.cc:304] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
W20260808 01:55:16.760658 21834 log.cc:927] Time spent T 4b877573d5d6413da0a0af3fe2ecb25b P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260808 01:55:16.797955 20684 raft_consensus.cc:2468] T e3b0638baa0f4f58b2d6146e98b8648b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:16.800429 20760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:16.792917 21787 raft_consensus.cc:3060] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Advancing to term 5
I20260808 01:55:16.783883 20603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44770 (request call id 263) took 1773 ms. Trace:
I20260808 01:55:16.785224 21664 raft_consensus.cc:2749] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:55:16.810644 20762 raft_consensus.cc:2468] T 054af38e4814474c92c93f3f09db43a3 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 4.
I20260808 01:55:16.811597 20528 raft_consensus.cc:1240] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 2. Current term is 3. Ops: []
I20260808 01:55:16.784902 21702 leader_election.cc:290] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:55:16.819955 20494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:16.834218 20493 leader_election.cc:304] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:16.834915 21534 raft_consensus.cc:493] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:16.835186 21747 leader_election.cc:290] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:16.819495 21786 raft_consensus.cc:493] T ce51fcffe2ae4bcba55cd864c973bf0d P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:16.835294 20756 raft_consensus.cc:1275] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:16.836210 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:16.755484 20604 rpcz_store.cc:276] 0808 01:55:15.009664 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.630875 (+1621211us) service_pool.cc:224] Handling call
0808 01:55:16.633237 (+  2362us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:16.700866 (+ 67629us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:16.715353 (+ 14487us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.804258 21801 raft_consensus.cc:493] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:16.795197 21696 raft_consensus.cc:493] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:16.851718 20682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46954 (request call id 293) took 1174 ms. Trace:
I20260808 01:55:16.839699 21664 raft_consensus.cc:2804] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader election won for term 2
I20260808 01:55:16.841825 21774 leader_election.cc:290] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:16.843703 21534 raft_consensus.cc:515] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:16.925432 21801 raft_consensus.cc:515] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:16.926506 21774 raft_consensus.cc:493] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:16.927238 21774 raft_consensus.cc:515] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:16.927884 21534 leader_election.cc:290] T 0bd11a6553c444d488651367f3f0143f P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:16.928501 21801 leader_election.cc:290] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:16.854411 21430 raft_consensus.cc:493] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:16.932437 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:16.852208 21761 raft_consensus.cc:493] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:16.933157 21430 raft_consensus.cc:515] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
W20260808 01:55:16.842577 21771 log.cc:927] Time spent T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 1.502s	user 0.012s	sys 0.010s
I20260808 01:55:16.855944 20609 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44770 (request call id 254) took 1852 ms. Trace:
I20260808 01:55:16.936440 20683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 215) took 2046 ms. Trace:
I20260808 01:55:16.936767 21430 leader_election.cc:290] T d3997511f7d7419985d678e8fa59e5d0 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:16.863246 20529 raft_consensus.cc:1240] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 5 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 4. Current term is 5. Ops: []
I20260808 01:55:16.845288 20602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:16.937772 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:16.844712 20678 raft_consensus.cc:1240] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260808 01:55:16.937043 20683 rpcz_store.cc:276] 0808 01:55:14.890623 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:14.890883 (+   260us) service_pool.cc:224] Handling call
0808 01:55:14.891408 (+   525us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:14.891947 (+   539us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:14.891990 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:14.892869 (+   879us) log.cc:844] Serialized 26 byte log entry
0808 01:55:15.081816 (+188947us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:15.081905 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:15.082003 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:16.841843 (+1759840us) spinlock_profiling.cc:243] Waited 3.83 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:55:16.936137 (+ 94294us) raft_consensus.cc:1685] finished
0808 01:55:16.936162 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:16.936378 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":15810,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"spinlock_wait_cycles":8810880,"thread_start_us":188378,"threads_started":1,"wal-append.queue_time_us":321880}
I20260808 01:55:16.809365 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44770 (request call id 256) took 1801 ms. Trace:
I20260808 01:55:16.935518 20609 rpcz_store.cc:276] 0808 01:55:15.003096 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.568254 (+1565158us) service_pool.cc:224] Handling call
0808 01:55:16.568599 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:16.568868 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:55:16.700884 (+132016us) log.cc:844] Serialized 14 byte log entry
0808 01:55:16.855563 (+154679us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:16.855623 (+    60us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:16.855668 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:16.855740 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:16.855879 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":250830,"prepare.run_wall_time_us":60,"raft.queue_time_us":1326,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":118,"spinlock_wait_cycles":12017408,"thread_start_us":238296,"threads_started":2}
W20260808 01:55:16.943287 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 162367us
I20260808 01:55:16.867432 21786 raft_consensus.cc:515] T ce51fcffe2ae4bcba55cd864c973bf0d P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:16.844023 21807 consensus_queue.cc:1048] T 6e71aab2ce9b45369b7eff65e1645f9b P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.087s
I20260808 01:55:16.846947 20606 raft_consensus.cc:2468] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 4.
I20260808 01:55:16.843863 21710 raft_consensus.cc:2749] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260808 01:55:16.846364 20686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46954 (request call id 281) took 1879 ms. Trace:
I20260808 01:55:16.838615 20677 raft_consensus.cc:2468] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 4.
I20260808 01:55:16.933501 21761 raft_consensus.cc:515] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:16.859759 20526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41410 (request call id 212) took 1875 ms. Trace:
I20260808 01:55:16.828528 20603 rpcz_store.cc:276] 0808 01:55:15.010512 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.710162 (+1699650us) service_pool.cc:224] Handling call
0808 01:55:16.744199 (+ 34037us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:16.754924 (+ 10725us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:16.759211 (+  4287us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:16.783504 (+ 24293us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:16.783563 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:16.783653 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:16.783813 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:16.848663 21738 raft_consensus.cc:493] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:16.840756 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:16.850025 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:16.840937 21702 raft_consensus.cc:2749] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:55:16.946216 20529 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:16.947640 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:16.938761 20602 raft_consensus.cc:2410] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260808 01:55:16.946141 20648 leader_election.cc:336] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.720s (SENT)
I20260808 01:55:16.931460 21767 raft_consensus.cc:515] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
W20260808 01:55:16.997273 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 51162us
W20260808 01:55:16.997991 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 226993us
I20260808 01:55:16.923467 20682 rpcz_store.cc:276] 0808 01:55:15.677574 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:15.677770 (+   196us) service_pool.cc:224] Handling call
0808 01:55:16.851445 (+1173675us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b54000ac8d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:16.851609 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":339949440}
I20260808 01:55:16.999454 20649 leader_election.cc:400] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Vote denied by peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117) with higher term. Message: Invalid argument: T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for earlier term 3. Current term is 4.
I20260808 01:55:16.999934 20649 leader_election.cc:403] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260808 01:55:16.971355 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21214e47c3d34850ae5c4202d6761065" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:16.962399 20686 rpcz_store.cc:276] 0808 01:55:14.967384 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:14.967578 (+   194us) service_pool.cc:224] Handling call
0808 01:55:14.967895 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:16.468743 (+1500848us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:16.846092 (+377349us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54006e91d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:16.846295 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3421592960}
I20260808 01:55:16.940583 21755 raft_consensus.cc:493] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:16.940937 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:16.933180 21774 leader_election.cc:290] T bf6d6021e1384c4eafe94f910b25d517 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:16.981726 20756 raft_consensus.cc:1240] T 48b36b5d19af469885eddc3fa63ce410 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 LEADER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 01:55:16.953822 21696 raft_consensus.cc:515] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:16.989835 20726 leader_election.cc:304] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:17.019909 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d681d5f24f674af0a5024867f8a1fb27" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:16.971020 21534 consensus_queue.cc:1059] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260808 01:55:16.992931 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:17.039252 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:17.040836 20607 raft_consensus.cc:2410] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:55:17.030028 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdef2db6e2e04aa795d92ad39538d90b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:16.952316 20605 rpcz_store.cc:276] 0808 01:55:15.007171 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.597916 (+1590745us) service_pool.cc:224] Handling call
0808 01:55:16.605919 (+  8003us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:16.791167 (+185248us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:16.805290 (+ 14123us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:55:17.037034 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 47341.1us
I20260808 01:55:17.031764 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.032804 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63fb8c2f494a462686500ce10025a39f" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:55:17.046742 20571 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:44768 dropped due to backpressure: service queue is full with 50 items [suppressed 3 similar messages]
I20260808 01:55:17.044317 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3"
I20260808 01:55:17.057204 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:16.738896 20759 rpcz_store.cc:276] 0808 01:55:13.847861 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:14.903459 (+1055598us) service_pool.cc:224] Handling call
0808 01:55:14.903914 (+   455us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:14.921929 (+ 18015us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:14.921965 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:14.922623 (+   658us) log.cc:844] Serialized 26 byte log entry
0808 01:55:15.297677 (+375054us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:15.317554 (+ 19877us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:15.329808 (+ 12254us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:16.546505 (+1216697us) spinlock_profiling.cc:243] Waited 3.32 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:55:16.664138 (+117633us) raft_consensus.cc:1685] finished
0808 01:55:16.664256 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:16.698964 (+ 34708us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15392,"raft.queue_time_us":19117,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":125,"spinlock_wait_cycles":7643520,"thread_start_us":310444,"threads_started":1,"wal-append.queue_time_us":622808}
I20260808 01:55:17.067811 20761 raft_consensus.cc:2393] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 3: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:16.999531 21738 raft_consensus.cc:515] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:17.076761 20762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:17.033138 21702 raft_consensus.cc:493] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:17.039238 20753 raft_consensus.cc:1240] T 21214e47c3d34850ae5c4202d6761065 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:17.058534 20604 raft_consensus.cc:2393] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 2: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:17.079473 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:17.033825 20755 raft_consensus.cc:2468] T 20ca72baa7bc4572a51de6b57b4dc3e9 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3.
I20260808 01:55:17.082619 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:17.050927 21534 raft_consensus.cc:3055] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 2 LEADER]: Stepping down as leader of term 2
I20260808 01:55:17.058192 21642 raft_consensus.cc:2804] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader election won for term 4
I20260808 01:55:17.025137 21755 raft_consensus.cc:515] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:17.050765 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44770 (request call id 253) took 2047 ms. Trace:
I20260808 01:55:17.052788 21767 leader_election.cc:290] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:17.084040 20762 raft_consensus.cc:2410] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for term 4 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260808 01:55:17.052347 21786 leader_election.cc:290] T ce51fcffe2ae4bcba55cd864c973bf0d P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:17.049569 21774 raft_consensus.cc:2749] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117) with higher term. Message: Invalid argument: T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for earlier term 3. Current term is 4.
I20260808 01:55:17.072922 21849 consensus_queue.cc:1048] T 62f797fc263e40c0a0380b65f9430154 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260808 01:55:17.056782 20571 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:44768 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 228)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 242)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 269)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 218)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 213)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 217)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 208)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 232)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 230)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 218)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 206)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 237)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 219)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 210)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 229)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 240)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 236)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 238)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 227)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 211)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 209)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 231)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 228)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 239)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 207)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 219)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 212)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 227)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44768 (request call id 217)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44768 (request call id 220)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44768 (request call id 224)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44768 (request call id 226)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44768 (request call id 233)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44768 (request call id 234)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44768 (request call id 235)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44768 (request call id 241)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44780 (request call id 215)
W20260808 01:55:17.203684 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 234999us
W20260808 01:55:17.180233 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 160123us
I20260808 01:55:17.082978 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 210) took 2306 ms. Trace:
I20260808 01:55:17.202591 20685 raft_consensus.cc:1240] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260808 01:55:17.188192 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1753378f13594e89aee6641e2cb9e6da" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.243161 20608 raft_consensus.cc:2468] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
W20260808 01:55:17.243976 20608 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 185) took 2227 ms (client timeout 2966 ms). Trace:
I20260808 01:55:17.244055 20529 raft_consensus.cc:3060] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Advancing to term 5
W20260808 01:55:17.244432 20608 rpcz_store.cc:269] 0808 01:55:15.016524 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.452875 (+1436351us) service_pool.cc:224] Handling call
0808 01:55:17.187961 (+735086us) spinlock_profiling.cc:243] Waited 3.32 ms on lock 0x7b58001b0398. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:17.243803 (+ 55842us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7628032}
I20260808 01:55:17.246348 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac71e66cb4042eb8fe8aafd139af68a" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:17.249732 20608 raft_consensus.cc:2393] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 2: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
W20260808 01:55:17.163146 21846 log.cc:927] Time spent T 4f6c2d3c4da449688d1fca55d391ead2 P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.173s	user 0.001s	sys 0.000s
F20260808 01:55:17.156406  7831 test_workload.cc:382] Timed out: Timed out waiting for Table Creation
I20260808 01:55:17.154639 21738 leader_election.cc:290] T bf6d6021e1384c4eafe94f910b25d517 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:17.385046 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:55:17.194069 20572 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:17.522629 20608 raft_consensus.cc:2410] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260808 01:55:17.154630 21824 consensus_meta.cc:287] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb: Time spent flushing consensus metadata: real 0.525s	user 0.009s	sys 0.004s
I20260808 01:55:17.523667 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f6c2d3c4da449688d1fca55d391ead2" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.524423 20608 raft_consensus.cc:2393] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 4: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
W20260808 01:55:17.151170 21841 log.cc:927] Time spent T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260808 01:55:17.102720 21664 raft_consensus.cc:697] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [term 2 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
*** Check failure stack trace: ***
I20260808 01:55:17.403710 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:17.150126 21754 raft_consensus.cc:493] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:55:17.419499 20494 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:17.151835 21761 leader_election.cc:290] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
*** Aborted at 1786154117 (unix time) try "date -d @1786154117" if you are using GNU date ***
I20260808 01:55:17.224575 21853 raft_consensus.cc:493] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:17.104817 20602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.095175 21702 raft_consensus.cc:515] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:17.449093 20529 raft_consensus.cc:1275] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 5 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260808 01:55:17.099164 21534 raft_consensus.cc:740] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
I20260808 01:55:17.153462 21755 leader_election.cc:290] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:17.107139 21642 raft_consensus.cc:697] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 LEADER]: Becoming Leader. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:17.193789 21811 raft_consensus.cc:493] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:17.201354 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:17.201973 20679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 224) took 1040 ms. Trace:
W20260808 01:55:17.453186 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 53388.4us
I20260808 01:55:17.237139 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d32821d9c17456394a662fcfd05f6c6" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.156184 21696 leader_election.cc:290] T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:17.251611 21801 raft_consensus.cc:493] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:17.397615 21756 raft_consensus.cc:493] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:17.411809 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d32821d9c17456394a662fcfd05f6c6" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:17.142518 21850 consensus_queue.cc:1048] T 27e69fc628bc49cd8d6c653f82ef8c1d P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.321s
I20260808 01:55:17.531102 20528 raft_consensus.cc:1275] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Refusing update from remote peer 598db12c38d442c6988b4c4f991874fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:17.538968 20757 raft_consensus.cc:2468] T 6d32821d9c17456394a662fcfd05f6c6 P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
W20260808 01:55:17.531931 20493 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.689s	user 0.034s	sys 0.046s
W20260808 01:55:17.464869 21763 consensus_meta.cc:287] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb: Time spent flushing consensus metadata: real 0.703s	user 0.008s	sys 0.007s
I20260808 01:55:17.446504 21659 consensus_queue.cc:1059] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260808 01:55:17.412082 21852 consensus_queue.cc:1059] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.217s
I20260808 01:55:17.448410 21774 consensus_queue.cc:1048] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.027s
I20260808 01:55:17.250451 20681 rpcz_store.cc:276] 0808 01:55:14.769640 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:14.769908 (+   268us) service_pool.cc:224] Handling call
0808 01:55:14.770318 (+   410us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:14.846586 (+ 76268us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:14.846699 (+   113us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:14.912220 (+ 65521us) log.cc:844] Serialized 26 byte log entry
0808 01:55:15.101687 (+189467us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:15.104881 (+  3194us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:15.675350 (+570469us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:55:17.035693 (+1360343us) spinlock_profiling.cc:243] Waited 44.8 ms on lock 0x7b540082cb54. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:17.037438 (+  1745us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:17.039603 (+  2165us) raft_consensus.cc:1685] finished
0808 01:55:17.043979 (+  4376us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:17.072039 (+ 28060us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":65341,"raft.run_cpu_time_us":53,"raft.run_wall_time_us":54,"spinlock_wait_cycles":134635136,"thread_start_us":148562,"threads_started":1,"wal-append.queue_time_us":414466,"wal-append.run_cpu_time_us":11195,"wal-append.run_wall_time_us":1618373}
I20260808 01:55:17.102069 20600 rpcz_store.cc:276] 0808 01:55:15.002838 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.516666 (+1513828us) service_pool.cc:224] Handling call
0808 01:55:16.519422 (+  2756us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:55:16.520080 (+   658us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:16.520138 (+    58us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:55:16.521061 (+   923us) log.cc:844] Serialized 52 byte log entry
0808 01:55:16.692005 (+170944us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:17.024729 (+332724us) log.cc:844] Serialized 14 byte log entry
0808 01:55:17.049952 (+ 25223us) log.cc:844] Serialized 14 byte log entry
0808 01:55:17.050228 (+   276us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:17.050315 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:17.050385 (+    70us) raft_consensus.cc:1685] finished
0808 01:55:17.050399 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:17.050638 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4285,"prepare.queue_time_us":519390,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":1870,"raft.queue_time_us":12401,"raft.run_cpu_time_us":401,"raft.run_wall_time_us":9719,"thread_start_us":420112,"threads_started":2,"wal-append.queue_time_us":212453}
I20260808 01:55:17.487474 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:17.497581 20571 leader_election.cc:304] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:17.502202 21778 raft_consensus.cc:3055] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Stepping down as leader of term 3
W20260808 01:55:17.508867 21787 consensus_meta.cc:287] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3: Time spent flushing consensus metadata: real 0.653s	user 0.011s	sys 0.003s
I20260808 01:55:17.509461 21747 raft_consensus.cc:493] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
W20260808 01:55:17.509390 20524 consensus_meta.cc:287] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e: Time spent flushing consensus metadata: real 0.681s	user 0.013s	sys 0.001s
W20260808 01:55:17.515534 21700 consensus_meta.cc:287] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb: Time spent flushing consensus metadata: real 0.816s	user 0.012s	sys 0.002s
I20260808 01:55:17.521611 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8135c0503a8480998454f58362956e5" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.525745 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.525566 20678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 229) took 1080 ms. Trace:
W20260808 01:55:17.540866 20493 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.698s	user 0.034s	sys 0.046s
I20260808 01:55:16.980595 20526 rpcz_store.cc:276] 0808 01:55:14.983989 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:14.985381 (+  1392us) service_pool.cc:224] Handling call
0808 01:55:16.467335 (+1481954us) spinlock_profiling.cc:243] Waited 6.73 ms on lock 0x7b58000d9e98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:55:16.467603 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:16.467852 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:55:16.597616 (+129764us) log.cc:844] Serialized 14 byte log entry
0808 01:55:16.859174 (+261558us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:16.859254 (+    80us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:16.859310 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:16.859530 (+   220us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:16.859689 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6346,"prepare.queue_time_us":223765,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":2245,"raft.queue_time_us":31727,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":4707,"spinlock_wait_cycles":15479680,"thread_start_us":285429,"threads_started":2,"wal-append.queue_time_us":427336}
I20260808 01:55:17.539812 20757 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53322 (request call id 233) took 1388 ms. Trace:
W20260808 01:55:17.551824 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 54303us
I20260808 01:55:17.555622 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb4dbfa7c99f46d2a70153ce86f409fb" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
I20260808 01:55:17.535730 21664 consensus_queue.cc:237] T 8ac71e66cb4042eb8fe8aafd139af68a P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:17.551638 21778 raft_consensus.cc:740] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:17.582923 20681 raft_consensus.cc:3060] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:55:17.577864 21544 raft_consensus.cc:493] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:17.583811 21544 raft_consensus.cc:515] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:17.586158 21544 leader_election.cc:290] T d681d5f24f674af0a5024867f8a1fb27 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:17.552987 21696 raft_consensus.cc:3055] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 2 LEADER]: Stepping down as leader of term 2
W20260808 01:55:17.558847 20754 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53322 (request call id 237) took 1405 ms (client timeout 1777 ms). Trace:
I20260808 01:55:17.586939 21696 raft_consensus.cc:740] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a80df9f992014374aeef30979cfe393e, State: Running, Role: LEADER
W20260808 01:55:17.587106 20754 rpcz_store.cc:269] 0808 01:55:16.153241 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.155139 (+  1898us) service_pool.cc:224] Handling call
0808 01:55:17.201212 (+1046073us) spinlock_profiling.cc:243] Waited 9.24 ms on lock 0x7b5800184e98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d4185589e 000000000038d209 00007f5d49914940 00007f5d499dced4 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:17.558762 (+357550us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21262080}
I20260808 01:55:17.556375 21747 raft_consensus.cc:515] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:17.588043 21696 consensus_queue.cc:260] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:17.589286 21696 raft_consensus.cc:3060] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:17.589987 20680 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d8bf19c324463fb9903b9981068e9d" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:17.547695 21853 raft_consensus.cc:515] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:17.591050 20571 leader_election.cc:304] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:17.555912 21430 consensus_queue.cc:1048] T aade7cc941de4b5781fb940a6625beb3 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260808 01:55:17.550976 21659 raft_consensus.cc:3055] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:55:17.591861 21659 raft_consensus.cc:740] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:17.592940 21853 leader_election.cc:290] T aecd7bfdd8a746239184356052321a48 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:17.592813 21659 consensus_queue.cc:260] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:17.547783 21763 raft_consensus.cc:515] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:17.543355 20679 rpcz_store.cc:276] 0808 01:55:16.161038 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.625375 (+464337us) service_pool.cc:224] Handling call
0808 01:55:16.625751 (+   376us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:16.647147 (+ 21396us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:16.648409 (+  1262us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:16.704676 (+ 56267us) log.cc:844] Serialized 26 byte log entry
0808 01:55:16.997038 (+292362us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:17.024796 (+ 27758us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:17.030252 (+  5456us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:17.201595 (+171343us) raft_consensus.cc:1685] finished
0808 01:55:17.201629 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:17.201882 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":18820,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":127,"thread_start_us":232991,"threads_started":1,"wal-append.queue_time_us":407836}
I20260808 01:55:17.594924 21798 raft_consensus.cc:2749] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:17.595088 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:17.595221 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44770 (request call id 255) took 2591 ms. Trace:
I20260808 01:55:17.554693 21776 raft_consensus.cc:2749] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:17.562770 20608 raft_consensus.cc:2393] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 3: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:17.547452 21756 raft_consensus.cc:515] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:17.595804 20601 rpcz_store.cc:276] 0808 01:55:15.003368 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.594142 (+2590774us) service_pool.cc:224] Handling call
0808 01:55:17.594551 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:17.594829 (+   278us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:17.594864 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:17.594917 (+    53us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:17.594957 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:17.595019 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:17.595154 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:17.597482 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.597663 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "711b7cabb802484094416a0e94a14803" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.598230 20608 raft_consensus.cc:2393] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 3: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
I20260808 01:55:17.545457 21811 raft_consensus.cc:515] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:17.535041 21754 raft_consensus.cc:515] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:17.558954 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:17.527207 21824 raft_consensus.cc:515] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
W20260808 01:55:17.568228 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 725034us
I20260808 01:55:17.545310 20602 raft_consensus.cc:2376] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a80df9f992014374aeef30979cfe393e in term 4. Re-sending same reply.
I20260808 01:55:17.554602 20524 raft_consensus.cc:2468] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 5.
I20260808 01:55:17.550863 21642 consensus_queue.cc:237] T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:17.557860 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27e69fc628bc49cd8d6c653f82ef8c1d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.596545 20571 leader_election.cc:304] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:17.616087 20605 raft_consensus.cc:2468] T 27e69fc628bc49cd8d6c653f82ef8c1d P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:17.616174 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 5 candidate_status { last_received { term: 4 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:17.616871 20571 leader_election.cc:304] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e
I20260808 01:55:17.617074 21430 raft_consensus.cc:2749] T 711b7cabb802484094416a0e94a14803 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:17.617165 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:17.617792 21430 raft_consensus.cc:2749] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:17.618283 20605 raft_consensus.cc:2410] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:55:17.548414 21850 consensus_queue.cc:1048] T 73a8186fa08042f1ae22f9b47c77bd46 P 598db12c38d442c6988b4c4f991874fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260808 01:55:17.618522 21430 consensus_queue.cc:1059] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:55:17.562637 20609 raft_consensus.cc:2468] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 1.
I20260808 01:55:17.618549 20571 leader_election.cc:304] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:17.595815 21763 leader_election.cc:290] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:17.595983 20679 raft_consensus.cc:2393] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 4: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:17.547976 21801 raft_consensus.cc:515] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:17.598748 21756 leader_election.cc:290] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:17.599131 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d32821d9c17456394a662fcfd05f6c6" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.606592 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6d8bf19c324463fb9903b9981068e9d" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:17.608260 21778 consensus_queue.cc:260] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:17.568773 21852 raft_consensus.cc:493] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:17.619910 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc85af5215440be884412c1d240c7af" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.556856 21787 raft_consensus.cc:2749] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer a80df9f992014374aeef30979cfe393e (127.7.165.193:34531) with higher term. Message: Invalid argument: T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for earlier term 4. Current term is 5.
W20260808 01:55:17.678571 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 46900.5us
I20260808 01:55:17.620991 20648 leader_election.cc:304] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:17.570206 21700 raft_consensus.cc:515] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:17.570745 20757 rpcz_store.cc:276] 0808 01:55:16.151776 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.155169 (+  3393us) service_pool.cc:224] Handling call
0808 01:55:17.251975 (+1096806us) spinlock_profiling.cc:243] Waited 980 us on lock 0x7b5800184e98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:17.539746 (+287771us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2253696}
I20260808 01:55:17.678570 21694 raft_consensus.cc:493] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:17.622960 20571 leader_election.cc:304] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:17.623878 21776 raft_consensus.cc:3055] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Stepping down as leader of term 3
I20260808 01:55:17.616917 20527 raft_consensus.cc:2468] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 4.
I20260808 01:55:17.608271 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.569667 21534 consensus_queue.cc:260] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:17.627816 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:55:17.680650 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 59724.7us
I20260808 01:55:17.679607 20493 leader_election.cc:304] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:17.682552 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a313d0a0979e4dc8a4205c4f16a24257" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.628640 20680 raft_consensus.cc:2393] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 2: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
W20260808 01:55:17.688180 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 65288.7us
I20260808 01:55:17.643778 21430 raft_consensus.cc:2804] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election won for term 3
I20260808 01:55:17.630363 20528 raft_consensus.cc:3060] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:17.684841 21763 raft_consensus.cc:2804] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 5 FOLLOWER]: Leader election won for term 5
I20260808 01:55:17.692163 21798 raft_consensus.cc:2749] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:17.677242 21852 raft_consensus.cc:515] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:17.683061 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:17.684303 21778 raft_consensus.cc:3060] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:17.705562 20679 raft_consensus.cc:2468] T 38b9406154bf4f86bf631ea9b73424df P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:17.699681 21747 leader_election.cc:290] T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:17.703131 20602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.706444 21801 leader_election.cc:290] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:17.707506 21754 leader_election.cc:290] T 73a8186fa08042f1ae22f9b47c77bd46 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:17.707944 21534 raft_consensus.cc:3060] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:17.685844 21674 raft_consensus.cc:493] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:17.755223 21674 raft_consensus.cc:515] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:17.684233 21776 raft_consensus.cc:740] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:17.715817 20648 leader_election.cc:304] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:17.700163 20522 raft_consensus.cc:1275] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Refusing update from remote peer 598db12c38d442c6988b4c4f991874fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:17.716672 21763 raft_consensus.cc:697] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 5 LEADER]: Becoming Leader. State: Replica: 598db12c38d442c6988b4c4f991874fb, State: Running, Role: LEADER
I20260808 01:55:17.760376 21763 consensus_queue.cc:237] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:17.717630 21790 consensus_queue.cc:1059] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:55:17.690373 21694 raft_consensus.cc:515] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:17.721302 20609 raft_consensus.cc:2393] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 3: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
W20260808 01:55:17.763339 21863 log.cc:927] Time spent T 0590088b74d04c6b96146266db7a6266 P a80df9f992014374aeef30979cfe393e: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
I20260808 01:55:17.763911 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:17.764868 21694 leader_election.cc:290] T 07cea569c9ad4d18a8d2f97764e15cc9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:17.765254 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:17.765398 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce51fcffe2ae4bcba55cd864c973bf0d" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.725896 20682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 235) took 1103 ms. Trace:
I20260808 01:55:17.767115 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:17.768282 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:17.769797 21700 leader_election.cc:290] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:17.770012 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:17.567690 20678 rpcz_store.cc:276] 0808 01:55:16.442715 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.021882 (+579167us) service_pool.cc:224] Handling call
0808 01:55:17.522225 (+500343us) spinlock_profiling.cc:243] Waited 7.45 ms on lock 0x7b5800775598. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d4185589e 000000000038d209 00007f5d49914940 00007f5d499dced4 00007f5d499dcf41 00007f5d4999edd0 00007f5d49989739 00007f5d499891e2 00007f5d42eced34
0808 01:55:17.522450 (+   225us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:17.522722 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:17.522765 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:17.522821 (+    56us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:17.522857 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:17.522930 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:17.523088 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17133312}
I20260808 01:55:17.705842 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:17.714908 20601 raft_consensus.cc:2468] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:17.704679 21860 raft_consensus.cc:2804] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:17.747773 20681 raft_consensus.cc:2468] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 5.
I20260808 01:55:17.726902 21430 raft_consensus.cc:697] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [term 3 LEADER]: Becoming Leader. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:17.697826 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a313d0a0979e4dc8a4205c4f16a24257" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.766878 20682 rpcz_store.cc:276] 0808 01:55:16.617391 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.059704 (+442313us) service_pool.cc:224] Handling call
0808 01:55:17.060150 (+   446us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:17.553357 (+493207us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:17.553388 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:17.553436 (+    48us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:17.553465 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:17.719389 (+165924us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b540051b654. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:17.719460 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:17.720403 (+   943us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1133791104}
I20260808 01:55:17.772750 21852 leader_election.cc:290] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:17.707551 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.776162 20726 leader_election.cc:304] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
W20260808 01:55:17.777550 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 61828.7us
I20260808 01:55:17.777500 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3b0638baa0f4f58b2d6146e98b8648b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.783049 20686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 231) took 1310 ms. Trace:
I20260808 01:55:17.748908 20683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46968 (request call id 209) took 1482 ms. Trace:
I20260808 01:55:17.775041 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:17.758442 21674 leader_election.cc:290] T ce51fcffe2ae4bcba55cd864c973bf0d P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:17.782325 21747 raft_consensus.cc:2804] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:55:17.815534 21700 raft_consensus.cc:493] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
PC: @                0x0 (unknown)
I20260808 01:55:17.819485 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.831709 20683 rpcz_store.cc:276] 0808 01:55:16.273919 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.747843 (+1473924us) service_pool.cc:224] Handling call
0808 01:55:17.748197 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:17.748469 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:17.748505 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:17.748565 (+    60us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:17.748610 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:17.748684 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:17.748828 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:17.830955 21747 raft_consensus.cc:493] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting leader election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:18.976049 21747 raft_consensus.cc:3060] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:55:18.976678 20683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9a194d9a9eb40d397c76e44235f7aed" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:17.831229 21754 raft_consensus.cc:2804] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:18.977608 21754 raft_consensus.cc:493] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:18.978051 21754 raft_consensus.cc:3060] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:17.839815 21430 consensus_queue.cc:237] T e3b0638baa0f4f58b2d6146e98b8648b P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:17.843201 20678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:18.980901 21544 raft_consensus.cc:493] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:18.981575 21544 raft_consensus.cc:515] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:18.984285 21544 leader_election.cc:290] T 63fb8c2f494a462686500ce10025a39f P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:17.829787 20686 rpcz_store.cc:276] 0808 01:55:16.472911 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.781913 (+1309002us) service_pool.cc:224] Handling call
0808 01:55:17.782328 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:17.782567 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:17.782604 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:17.782665 (+    61us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:17.782730 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:17.782806 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:17.782947 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:18.985531 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb"
W20260808 01:55:18.985895 20603 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 228) took 1991 ms (client timeout 1645 ms). Trace:
W20260808 01:55:18.986366 20603 rpcz_store.cc:269] 0808 01:55:16.994071 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:18.985656 (+1991585us) service_pool.cc:211] Skipping call since client already timed out
0808 01:55:18.985790 (+   134us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260808 01:55:18.987114 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:18.987851 20686 raft_consensus.cc:2468] T 2a9d1eec1a5840328c553a2b78d08f2a P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
I20260808 01:55:18.987867 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:17.833865 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56010b7f5e3c48ed97bfe024f84ad2ae" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:17.838987 20523 raft_consensus.cc:2393] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 4: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:17.815253 21860 raft_consensus.cc:493] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:18.992057 21860 raft_consensus.cc:3060] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:18.992162 20523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41392 (request call id 203) took 1291 ms. Trace:
I20260808 01:55:18.992654 20523 rpcz_store.cc:276] 0808 01:55:17.700626 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.769664 (+ 69038us) service_pool.cc:224] Handling call
0808 01:55:18.992081 (+1222417us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:18.994133 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:17.829075 20605 raft_consensus.cc:2468] T a313d0a0979e4dc8a4205c4f16a24257 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
W20260808 01:55:18.998493 20605 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 238) took 2064 ms (client timeout 1934 ms). Trace:
W20260808 01:55:18.998986 20605 rpcz_store.cc:269] 0808 01:55:16.934517 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.680539 (+746022us) service_pool.cc:224] Handling call
0808 01:55:18.998396 (+1317857us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.000216 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:55:19.001132 20605 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 239) took 2066 ms (client timeout 2208 ms). Trace:
W20260808 01:55:19.001535 20605 rpcz_store.cc:269] 0808 01:55:16.934897 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:18.999883 (+2064986us) service_pool.cc:224] Handling call
0808 01:55:19.001063 (+  1180us) inbound_call.cc:177] Queueing success response
Metrics: {}
*** SIGABRT (@0x3e800001e97) received by PID 7831 (TID 0x7f5d3d4b7880) from PID 7831; stack trace: ***
    @           0x2ba310 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
    @           0x2bc7f3 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
I20260808 01:55:17.842861 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:19.205751 20609 raft_consensus.cc:2376] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a80df9f992014374aeef30979cfe393e in term 3. Re-sending same reply.
W20260808 01:55:19.206506 20609 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 272) took 1749 ms (client timeout 1603 ms). Trace:
W20260808 01:55:19.206997 20609 rpcz_store.cc:269] 0808 01:55:17.457581 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.842507 (+384926us) service_pool.cc:224] Handling call
0808 01:55:19.206412 (+1363905us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.208185 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b877573d5d6413da0a0af3fe2ecb25b" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:55:19.209213 20609 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 271) took 1756 ms (client timeout 1935 ms). Trace:
W20260808 01:55:19.209606 20609 rpcz_store.cc:269] 0808 01:55:17.453044 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.207845 (+1754801us) service_pool.cc:224] Handling call
0808 01:55:19.209151 (+  1306us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:17.841814 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
    @     0x7f5d44d9b980 (unknown) at ??:0
    @     0x7f5d3e177e87 gsignal at ??:0
W20260808 01:55:17.818150 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 103758us
W20260808 01:55:17.881800 21865 log.cc:927] Time spent T a1af0d1fa74847a390b02d2313464926 P df82e7f402364d369ed1ec3c5a8772b3: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20260808 01:55:17.904160 20606 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 189) took 2821 ms (client timeout 1561 ms). Trace:
I20260808 01:55:18.845460 21881 raft_consensus.cc:493] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:55:18.888724 20492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:18.916355 21879 raft_consensus.cc:493] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:19.357522 20571 leader_election.cc:304] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
W20260808 01:55:19.358366 20492 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:18.917038 21877 raft_consensus.cc:493] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:19.359010 21877 raft_consensus.cc:515] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:19.359748 21738 raft_consensus.cc:2749] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:55:19.357611 21881 raft_consensus.cc:515] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:19.361745 21877 leader_election.cc:290] T 48b36b5d19af469885eddc3fa63ce410 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:19.361900 21881 leader_election.cc:290] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:18.960526 20493 leader_election.cc:304] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
W20260808 01:55:18.961972 20520 consensus_meta.cc:287] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e: Time spent flushing consensus metadata: real 1.138s	user 0.022s	sys 0.005s
I20260808 01:55:18.963003 21883 raft_consensus.cc:493] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:18.967990 20522 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41410 (request call id 231) took 1439 ms. Trace:
I20260808 01:55:18.976044 20600 raft_consensus.cc:2468] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:18.990801 20527 raft_consensus.cc:2468] T 56010b7f5e3c48ed97bfe024f84ad2ae P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:19.368078 20527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41392 (request call id 205) took 1653 ms. Trace:
I20260808 01:55:19.368494 20527 rpcz_store.cc:276] 0808 01:55:17.715542 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.833579 (+118037us) service_pool.cc:224] Handling call
0808 01:55:19.367985 (+1534406us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.369705 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73a8186fa08042f1ae22f9b47c77bd46" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:19.003602 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac71e66cb4042eb8fe8aafd139af68a" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:19.004282 20523 raft_consensus.cc:2468] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
W20260808 01:55:19.011934 20525 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.178s	user 0.014s	sys 0.018s
I20260808 01:55:17.854254 20601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:17.849519 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "566d023f94bb449d8cf248744cd5b6ba" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:19.358106 21879 raft_consensus.cc:515] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:19.382236 20524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41410 (request call id 238) took 1540 ms. Trace:
I20260808 01:55:19.382714 20524 rpcz_store.cc:276] 0808 01:55:17.841932 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.849175 (+  7243us) service_pool.cc:224] Handling call
0808 01:55:19.382159 (+1532984us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.384006 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:19.384378 21879 leader_election.cc:290] T 21214e47c3d34850ae5c4202d6761065 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:19.384790 20524 raft_consensus.cc:2468] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:19.386176 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce51fcffe2ae4bcba55cd864c973bf0d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:19.083549 20424 catalog_manager.cc:5697] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 598db12c38d442c6988b4c4f991874fb (127.7.165.195). New cstate: current_term: 1 leader_uuid: "598db12c38d442c6988b4c4f991874fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } }
I20260808 01:55:19.387123 20605 raft_consensus.cc:2393] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb in current term 2: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
W20260808 01:55:19.098989 20475 auto_rebalancer.cc:300] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260808 01:55:18.813175 20648 leader_election.cc:336] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.580s (SENT)
    @     0x7f5d3e1797f1 abort at ??:0
W20260808 01:55:18.792778 20651 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.013s	sys 0.020s
W20260808 01:55:19.430075 20651 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.161s	user 0.015s	sys 0.021s
    @           0x2c0306 __interceptor_abort at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
I20260808 01:55:18.899572 20685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46968 (request call id 205) took 2634 ms. Trace:
I20260808 01:55:19.433748 20685 rpcz_store.cc:276] 0808 01:55:16.265211 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.808885 (+543674us) service_pool.cc:224] Handling call
0808 01:55:17.202183 (+393298us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:17.242825 (+ 40642us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:18.899271 (+1656446us) spinlock_profiling.cc:243] Waited 393 ms on lock 0x7b5400ae9050. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:18.899494 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":903426304}
I20260808 01:55:18.962352 20649 leader_election.cc:304] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:17.903259 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 237) took 1272 ms. Trace:
I20260808 01:55:19.443434 20681 rpcz_store.cc:276] 0808 01:55:16.631427 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.867407 (+1235980us) service_pool.cc:224] Handling call
0808 01:55:17.878090 (+ 10683us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:17.880984 (+  2894us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:55:17.881443 (+   459us) log.cc:844] Serialized 14 byte log entry
0808 01:55:17.902588 (+ 21145us) log.cc:844] Serialized 14 byte log entry
0808 01:55:17.902862 (+   274us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:17.902927 (+    65us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:55:17.902984 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:17.903073 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:17.903203 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5,"prepare.queue_time_us":987,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":84,"raft.queue_time_us":233,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":465,"thread_start_us":20391,"threads_started":1,"wal-append.queue_time_us":210340,"wal-append.run_cpu_time_us":1971,"wal-append.run_wall_time_us":1004789}
W20260808 01:55:19.446668 20649 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 484373us
I20260808 01:55:19.447024 20681 raft_consensus.cc:1240] T b6d8bf19c324463fb9903b9981068e9d P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:19.447965 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 239) took 2599 ms. Trace:
I20260808 01:55:19.448354 20681 rpcz_store.cc:276] 0808 01:55:16.848391 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.446350 (+2597959us) service_pool.cc:224] Handling call
0808 01:55:19.446789 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:19.447644 (+   855us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.447898 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.450098 20681 raft_consensus.cc:1275] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:17.831887 20679 raft_consensus.cc:2393] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a80df9f992014374aeef30979cfe393e in current term 3: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
W20260808 01:55:19.452322 20679 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46954 (request call id 296) took 2059 ms (client timeout 2191 ms). Trace:
W20260808 01:55:19.452790 20679 rpcz_store.cc:269] 0808 01:55:17.392480 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.766758 (+374278us) service_pool.cc:224] Handling call
0808 01:55:19.452244 (+1685486us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.453691 21861 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:19.454311 21861 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:17.821519 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20ca72baa7bc4572a51de6b57b4dc3e9" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:55:18.263662 20725 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.7.165.193:34531 timed out after 1.681s (SENT) [suppressed 56 similar messages]
W20260808 01:55:17.826828 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 50780.5us
I20260808 01:55:17.866641 20760 raft_consensus.cc:2468] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a80df9f992014374aeef30979cfe393e in term 5.
W20260808 01:55:19.468793 20760 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53302 (request call id 270) took 2676 ms (client timeout 1719 ms). Trace:
I20260808 01:55:19.468869 20426 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48828 (request call id 79) took 8373 ms. Trace:
W20260808 01:55:19.469228 20760 rpcz_store.cc:269] 0808 01:55:16.792151 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.796852 (+  4701us) service_pool.cc:224] Handling call
0808 01:55:19.468580 (+2671728us) spinlock_profiling.cc:243] Waited 878 ms on lock 0x7b5400732654. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:19.468714 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2019646720}
W20260808 01:55:17.902454 20758 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53306 (request call id 217) took 2919 ms (client timeout 1797 ms). Trace:
I20260808 01:55:19.090206 21653 raft_consensus.cc:493] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:19.474917 21653 raft_consensus.cc:515] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:19.469391 20426 rpcz_store.cc:276] 0808 01:55:11.095295 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:11.095984 (+   689us) service_pool.cc:224] Handling call
0808 01:55:19.468783 (+8372799us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:55:15.595792 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:55:16.013894 (+418102us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:55:16.022075 (+  8181us) write_op.cc:435] Acquired schema lock
0808 01:55:16.026955 (+  4880us) tablet.cc:662] Decoding operations
0808 01:55:16.193954 (+166999us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:55:16.194026 (+    72us) write_op.cc:641] Partition lock acquired for write op
0808 01:55:16.194051 (+    25us) tablet.cc:685] Acquiring locks for 9 operations
0808 01:55:16.244619 (+ 50568us) tablet.cc:701] Row locks acquired
0808 01:55:16.245694 (+  1075us) write_op.cc:260] PREPARE: finished
0808 01:55:16.252005 (+  6311us) write_op.cc:270] Start()
0808 01:55:16.252136 (+   131us) write_op.cc:276] Timestamp: P: 1786154116251956 usec, L: 0
0808 01:55:16.252156 (+    20us) op_driver.cc:348] REPLICATION: starting
0808 01:55:16.252902 (+   746us) log.cc:844] Serialized 3641 byte log entry
0808 01:55:16.688695 (+435793us) op_driver.cc:464] REPLICATION: finished
0808 01:55:16.819138 (+130443us) write_op.cc:301] APPLY: starting
0808 01:55:16.831599 (+ 12461us) tablet.cc:1367] starting BulkCheckPresence
0808 01:55:16.843401 (+ 11802us) tablet.cc:1370] finished BulkCheckPresence
0808 01:55:16.966222 (+122821us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:55:17.158902 (+192680us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:55:17.162396 (+  3494us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0808 01:55:17.162476 (+    80us) write_op.cc:312] APPLY: finished
0808 01:55:17.194891 (+ 32415us) log.cc:844] Serialized 70 byte log entry
0808 01:55:17.218629 (+ 23738us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:55:17.222545 (+  3916us) write_op.cc:454] Released schema lock
0808 01:55:17.226243 (+  3698us) write_op.cc:341] FINISH: Updating metrics
0808 01:55:17.386945 (+160702us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:55:17.397559 (+ 10614us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":150,"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":106667,"apply.run_cpu_time_us":6064,"apply.run_wall_time_us":432925,"num_ops":9,"prepare.queue_time_us":104369,"prepare.run_cpu_time_us":77641,"prepare.run_wall_time_us":856143,"raft.queue_time_us":106420,"raft.run_cpu_time_us":11158,"raft.run_wall_time_us":110149,"replication_time_us":412520,"thread_start_us":325232,"threads_started":4,"wal-append.queue_time_us":218779,"wal-append.run_cpu_time_us":13463,"wal-append.run_wall_time_us":1724805}]]}
I20260808 01:55:19.477536 21653 leader_election.cc:290] T 70e7899e1afa4d37a1a8bd26dfa9d803 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:17.931664 21663 raft_consensus.cc:493] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:19.479009 21663 raft_consensus.cc:515] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:19.481774 21663 leader_election.cc:290] T f275a817cc374438aa23f63da2a627b9 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:17.867098 21674 raft_consensus.cc:493] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:19.483502 21674 raft_consensus.cc:515] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:17.857758 21700 raft_consensus.cc:515] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
    @     0x7f5d40f4dd74 google::LogMessage::SendToLog() at ??:0
I20260808 01:55:19.489758 21700 leader_election.cc:290] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:18.937053 21659 raft_consensus.cc:3060] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:55:19.103709 21702 leader_election.cc:290] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:18.989207 21754 raft_consensus.cc:515] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:18.091231 21763 raft_consensus.cc:493] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:19.499727 21763 raft_consensus.cc:515] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:18.843906 21798 raft_consensus.cc:493] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:19.503118 21798 raft_consensus.cc:515] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:19.505962 20683 raft_consensus.cc:2410] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:55:19.508747 20683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 243) took 2028 ms. Trace:
I20260808 01:55:19.509258 20683 rpcz_store.cc:276] 0808 01:55:17.480191 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.507811 (+2027620us) service_pool.cc:224] Handling call
0808 01:55:19.508204 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:19.508411 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:19.508441 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:19.508489 (+    48us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:19.508522 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.508578 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:19.508688 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:18.184749 21802 raft_consensus.cc:493] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:19.512511 21802 raft_consensus.cc:515] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:18.870234 21811 leader_election.cc:290] T 62f797fc263e40c0a0380b65f9430154 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:17.826623 21824 leader_election.cc:290] T 566d023f94bb449d8cf248744cd5b6ba P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:19.066087 21850 raft_consensus.cc:493] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:19.517148 21850 raft_consensus.cc:515] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
W20260808 01:55:18.870965 21858 log.cc:927] Time spent T e9201fed5ac044119ff493c519d289a7 P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 1.385s	user 0.006s	sys 0.016s
W20260808 01:55:18.801978 21859 log.cc:927] Time spent T 8fc615366d544af38cf664c9ad47ab0c P 598db12c38d442c6988b4c4f991874fb: Append to log took a long time: real 1.315s	user 0.004s	sys 0.011s
I20260808 01:55:19.024955 21747 raft_consensus.cc:515] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
    @     0x7f5d40f4e920 google::LogMessage::Flush() at ??:0
I20260808 01:55:19.092387 21761 raft_consensus.cc:493] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:19.527598 21761 raft_consensus.cc:515] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:19.529943 21761 leader_election.cc:290] T de62f661f1484ca9af89dc7df85f8b9b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:19.026795 21860 raft_consensus.cc:515] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:19.533779 21860 leader_election.cc:290] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:19.152833 20526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41396 (request call id 292) took 1550 ms. Trace:
W20260808 01:55:19.155228 20528 consensus_meta.cc:287] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e: Time spent flushing consensus metadata: real 1.436s	user 0.015s	sys 0.019s
I20260808 01:55:19.137956 21889 raft_consensus.cc:493] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:19.534935 20528 raft_consensus.cc:2468] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
W20260808 01:55:19.535526 20528 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41410 (request call id 235) took 1930 ms (client timeout 1734 ms). Trace:
W20260808 01:55:19.535908 20528 rpcz_store.cc:269] 0808 01:55:17.605401 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.605589 (+   188us) service_pool.cc:224] Handling call
0808 01:55:19.535444 (+1929855us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.536247 20530 raft_consensus.cc:1240] T b6d8bf19c324463fb9903b9981068e9d P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 1. Current term is 2. Ops: []
I20260808 01:55:19.537154 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcae3a1b04b44ebfa3776d88ca63cbdb" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:19.538023 20528 raft_consensus.cc:2393] T bcae3a1b04b44ebfa3776d88ca63cbdb P a80df9f992014374aeef30979cfe393e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 5: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
I20260808 01:55:19.538067 20530 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41392 (request call id 200) took 1838 ms. Trace:
W20260808 01:55:19.168691 20607 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 242) took 2232 ms (client timeout 1713 ms). Trace:
I20260808 01:55:19.538595 20530 rpcz_store.cc:276] 0808 01:55:17.699845 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.699978 (+   133us) service_pool.cc:224] Handling call
0808 01:55:19.536028 (+1836050us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:19.536780 (+   752us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.537845 (+  1065us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b5400329450. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:19.538004 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4207113472}
W20260808 01:55:19.187173 20570 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:19.185007 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2b22456199b4e5f95a1eb974dd194fb" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:55:19.195711 20680 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46954 (request call id 295) took 1803 ms (client timeout 1802 ms). Trace:
I20260808 01:55:19.540426 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:19.540464 20680 rpcz_store.cc:269] 0808 01:55:17.392109 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.589647 (+197538us) service_pool.cc:224] Handling call
0808 01:55:19.195347 (+1605700us) spinlock_profiling.cc:243] Waited 8.92 ms on lock 0x7b540055d2d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:19.195599 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20521344}
I20260808 01:55:19.188421 21895 raft_consensus.cc:493] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:19.541885 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
W20260808 01:55:19.216724 20604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 274) took 1739 ms (client timeout 1576 ms). Trace:
I20260808 01:55:19.542436 21895 raft_consensus.cc:515] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
W20260808 01:55:19.542878 20604 rpcz_store.cc:269] 0808 01:55:17.477931 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.841489 (+363558us) service_pool.cc:224] Handling call
0808 01:55:19.216620 (+1375131us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:55:19.544196 20604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 275) took 2065 ms (client timeout 1952 ms). Trace:
W20260808 01:55:19.544551 20604 rpcz_store.cc:269] 0808 01:55:17.479227 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.543986 (+2064759us) service_pool.cc:211] Skipping call since client already timed out
0808 01:55:19.544128 (+   142us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260808 01:55:19.545061 21895 leader_election.cc:290] T 45432e629d3b49abab9715ce5c0124d1 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:19.545459 20604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 276) took 2056 ms (client timeout 1996 ms). Trace:
W20260808 01:55:19.545858 20604 rpcz_store.cc:269] 0808 01:55:17.489545 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.545265 (+2055720us) service_pool.cc:211] Skipping call since client already timed out
0808 01:55:19.545402 (+   137us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260808 01:55:19.217543 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0590088b74d04c6b96146266db7a6266" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:55:19.547230 20609 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 281) took 2050 ms (client timeout 1910 ms). Trace:
W20260808 01:55:19.547623 20609 rpcz_store.cc:269] 0808 01:55:17.497240 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.210370 (+1713130us) service_pool.cc:224] Handling call
0808 01:55:19.547136 (+336766us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:55:19.548457 20609 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 279) took 2058 ms (client timeout 2044 ms). Trace:
W20260808 01:55:19.548790 20609 rpcz_store.cc:269] 0808 01:55:17.490452 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.548290 (+2057838us) service_pool.cc:211] Skipping call since client already timed out
0808 01:55:19.548401 (+   111us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260808 01:55:19.549734 20609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:19.218303 21894 raft_consensus.cc:493] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:19.551007 21894 raft_consensus.cc:515] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:19.553380 21894 leader_election.cc:290] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:19.226150 21544 raft_consensus.cc:493] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:19.554171 21544 raft_consensus.cc:515] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:19.556643 21544 leader_election.cc:290] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:55:19.247771 20573 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260808 01:55:19.558202 20573 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:19.559219 21430 raft_consensus.cc:493] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:19.258841 21884 raft_consensus.cc:493] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
I20260808 01:55:19.559738 21430 raft_consensus.cc:515] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:19.560338 21884 raft_consensus.cc:515] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:19.287189 21776 consensus_queue.cc:260] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:19.301596 21716 raft_consensus.cc:493] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:19.563331 21716 raft_consensus.cc:515] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:19.564234 21738 raft_consensus.cc:493] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:19.564795 21738 raft_consensus.cc:515] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:19.565444 21716 leader_election.cc:290] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:55:19.321857 20529 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.568s	user 0.021s	sys 0.023s
W20260808 01:55:19.356662 20606 rpcz_store.cc:269] 0808 01:55:15.083624 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:15.215260 (+131636us) service_pool.cc:224] Handling call
0808 01:55:17.903933 (+2688673us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b5400498ed4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:17.904057 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3648422144}
W20260808 01:55:19.566776 20529 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.813s	user 0.021s	sys 0.025s
I20260808 01:55:19.566820 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:19.566999 21738 leader_election.cc:290] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:19.364588 20603 raft_consensus.cc:2410] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:55:19.567771 20530 raft_consensus.cc:3060] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Advancing to term 4
W20260808 01:55:19.569576 20603 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 228) took 2693 ms (client timeout 2187 ms). Trace:
W20260808 01:55:19.569978 20603 rpcz_store.cc:269] 0808 01:55:16.875986 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:18.987569 (+2111583us) service_pool.cc:224] Handling call
0808 01:55:19.569358 (+581789us) spinlock_profiling.cc:243] Waited 374 ms on lock 0x7b540077d8d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:19.569502 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":859714304}
I20260808 01:55:19.363696 20520 raft_consensus.cc:2468] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
W20260808 01:55:19.387938 20605 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 226) took 2394 ms (client timeout 2213 ms). Trace:
I20260808 01:55:19.570891 20678 raft_consensus.cc:2468] T 8c4d9d89ee1743829635451925a89c9e P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
W20260808 01:55:19.571125 20605 rpcz_store.cc:269] 0808 01:55:16.993518 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.002348 (+2008830us) service_pool.cc:224] Handling call
0808 01:55:19.387826 (+385478us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:55:19.572176 20678 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46972 (request call id 242) took 2416 ms (client timeout 3054 ms). Trace:
I20260808 01:55:19.572314 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a9d1eec1a5840328c553a2b78d08f2a" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:55:19.572587 20678 rpcz_store.cc:269] 0808 01:55:17.155439 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.824318 (+668879us) service_pool.cc:224] Handling call
0808 01:55:19.572010 (+1747692us) spinlock_profiling.cc:243] Waited 590 ms on lock 0x7b54000ac8d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:19.572117 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1357493632}
I20260808 01:55:19.573088 20605 raft_consensus.cc:2468] T 2a9d1eec1a5840328c553a2b78d08f2a P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
W20260808 01:55:19.573642 20605 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 219) took 2584 ms (client timeout 2762 ms). Trace:
W20260808 01:55:19.573987 20605 rpcz_store.cc:269] 0808 01:55:16.990621 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.572005 (+2581384us) service_pool.cc:224] Handling call
0808 01:55:19.573586 (+  1581us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:55:19.392243 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 431755us
I20260808 01:55:19.574880 21899 raft_consensus.cc:2804] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:19.575063 20684 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46968 (request call id 206) took 3309 ms. Trace:
I20260808 01:55:19.576462 21899 raft_consensus.cc:493] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:19.576959 21899 raft_consensus.cc:3060] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:19.575145 20493 leader_election.cc:304] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:19.381264 20527 raft_consensus.cc:2410] T 73a8186fa08042f1ae22f9b47c77bd46 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260808 01:55:19.389258 20525 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.555s	user 0.015s	sys 0.019s
I20260808 01:55:19.405540 20424 catalog_manager.cc:5697] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194). New cstate: current_term: 1 leader_uuid: "9bfb29168c8947e1b5816f1572d040cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } }
W20260808 01:55:19.379557 20600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 211) took 2400 ms (client timeout 1966 ms). Trace:
I20260808 01:55:19.397239 21879 raft_consensus.cc:493] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:19.371806 21883 raft_consensus.cc:515] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:19.367264 20522 rpcz_store.cc:276] 0808 01:55:17.528858 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.530589 (+  1731us) service_pool.cc:224] Handling call
0808 01:55:17.530942 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:17.759355 (+228413us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:18.967600 (+1208245us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b54003fa7d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:18.967894 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":340786432}
I20260808 01:55:19.416081 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ecc85af5215440be884412c1d240c7af" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:19.393715 20524 raft_consensus.cc:2468] T ce51fcffe2ae4bcba55cd864c973bf0d P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 1.
I20260808 01:55:19.411010 21791 consensus_queue.cc:1059] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20260808 01:55:19.428611 21900 raft_consensus.cc:493] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:19.428661 21885 raft_consensus.cc:493] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:19.450626 21877 raft_consensus.cc:2749] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260808 01:55:19.460057 20682 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46954 (request call id 294) took 2068 ms. Trace:
I20260808 01:55:19.466852 20681 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 240) took 2477 ms. Trace:
W20260808 01:55:19.482275 20492 leader_election.cc:336] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.603s (SENT)
I20260808 01:55:19.415570 20648 leader_election.cc:304] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:19.583720 20682 rpcz_store.cc:276] 0808 01:55:17.391656 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.821251 (+429595us) service_pool.cc:224] Handling call
0808 01:55:19.459987 (+1638736us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.534503 20526 rpcz_store.cc:276] 0808 01:55:17.602154 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.602756 (+   602us) service_pool.cc:224] Handling call
0808 01:55:17.619532 (+ 16776us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:17.683973 (+ 64441us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:17.686436 (+  2463us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:17.738967 (+ 52531us) log.cc:844] Serialized 26 byte log entry
0808 01:55:19.147993 (+1409026us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:19.148046 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.148149 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:19.152485 (+  4336us) raft_consensus.cc:1685] finished
0808 01:55:19.152519 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:19.152742 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5010,"raft.queue_time_us":5356,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":7841,"thread_start_us":40065,"threads_started":1,"wal-append.queue_time_us":1332906}
I20260808 01:55:19.585364 20682 raft_consensus.cc:1240] T 28e98a7c32694002b806eddd7007e550 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 3. Current term is 4. Ops: []
I20260808 01:55:19.585299 20527 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054af38e4814474c92c93f3f09db43a3" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:19.585873 20526 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:19.570963 20520 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41392 (request call id 198) took 1871 ms (client timeout 2054 ms). Trace:
I20260808 01:55:19.535101 21889 raft_consensus.cc:515] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:19.578596 20686 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 238) took 2903 ms. Trace:
I20260808 01:55:19.581302 20677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 228) took 3139 ms. Trace:
I20260808 01:55:19.587505 20424 catalog_manager.cc:5697] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196). New cstate: current_term: 1 leader_uuid: "df82e7f402364d369ed1ec3c5a8772b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:19.164768 21893 raft_consensus.cc:493] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260808 01:55:19.538883 20607 rpcz_store.cc:269] 0808 01:55:16.935802 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.168362 (+2232560us) service_pool.cc:211] Skipping call since client already timed out
0808 01:55:19.168560 (+   198us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260808 01:55:19.586594 20520 rpcz_store.cc:269] 0808 01:55:17.699145 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.702659 (+  3514us) service_pool.cc:224] Handling call
0808 01:55:19.570893 (+1868234us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.567816 20529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41410 (request call id 233) took 1999 ms. Trace:
W20260808 01:55:19.589224 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 106990us
I20260808 01:55:19.589625 21885 raft_consensus.cc:515] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
    @     0x7f5d40f53a5b google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7f5d4af9de5e kudu::TestWorkload::Setup() at ??:0
    @           0x37c998 kudu::master::LeaderRebalancerTest::CreateWorkloadTable() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_leader_rebalancer-test.cc:115
I20260808 01:55:19.365788 20571 leader_election.cc:304] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:19.579042 21903 raft_consensus.cc:2804] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260808 01:55:19.734690 21903 raft_consensus.cc:493] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 9bfb29168c8947e1b5816f1572d040cc)
W20260808 01:55:19.567742 20606 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 277) took 2077 ms (client timeout 2212 ms). Trace:
I20260808 01:55:19.735133 21659 raft_consensus.cc:2749] T b8135c0503a8480998454f58362956e5 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:55:19.472028 20758 rpcz_store.cc:269] 0808 01:55:14.983137 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:14.985366 (+  2229us) service_pool.cc:224] Handling call
0808 01:55:16.613674 (+1628308us) spinlock_profiling.cc:243] Waited 4.88 ms on lock 0x7b5800184e98. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d41855521 00007f5d418553d2 00007f5d4a7c1219 00007f5d49914640 00007f5d499dcebb 00007f5d499dcf41 00007f5d4998aeb0 00007f5d4998a794 00007f5d42ed3834 00007f5d42ed37c2
0808 01:55:17.902130 (+1288456us) spinlock_profiling.cc:243] Waited 90.3 ms on lock 0x7b5400158bd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:17.902334 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":218996352}
W20260808 01:55:19.735406 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 369713us
I20260808 01:55:19.581106 21776 raft_consensus.cc:3060] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:19.587380 20686 rpcz_store.cc:276] 0808 01:55:16.675008 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:18.988771 (+2313763us) service_pool.cc:224] Handling call
0808 01:55:19.577580 (+588809us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:19.577764 (+   184us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:19.577790 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:19.577831 (+    41us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:19.577870 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.577925 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:19.578436 (+   511us) spinlock_profiling.cc:243] Waited 588 ms on lock 0x7b5400126950. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d44992c8d 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:19.578538 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1353290496}
I20260808 01:55:19.591351 20681 rpcz_store.cc:276] 0808 01:55:16.989153 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.449272 (+2460119us) service_pool.cc:224] Handling call
0808 01:55:19.449665 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:19.450468 (+   803us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.466742 (+ 16274us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.736524 20571 leader_election.cc:304] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3; no voters: a80df9f992014374aeef30979cfe393e
I20260808 01:55:19.592926 20679 raft_consensus.cc:1240] T d681d5f24f674af0a5024867f8a1fb27 P 598db12c38d442c6988b4c4f991874fb [term 5 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 4. Current term is 5. Ops: []
W20260808 01:55:19.592478 20600 rpcz_store.cc:269] 0808 01:55:16.979458 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.739436 (+759978us) service_pool.cc:224] Handling call
0808 01:55:19.375265 (+1635829us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.593667 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:19.735232 21903 raft_consensus.cc:3060] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:55:19.739938 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:19.740633 20679 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 241) took 2639 ms. Trace:
I20260808 01:55:19.589651 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:19.741899 20520 raft_consensus.cc:2468] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:19.741137 20679 rpcz_store.cc:276] 0808 01:55:17.101331 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.453732 (+2352401us) service_pool.cc:224] Handling call
0808 01:55:19.454106 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:19.737864 (+283758us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.740317 (+  2453us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5400013bd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:19.740566 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":318671232}
I20260808 01:55:19.575603 20684 rpcz_store.cc:276] 0808 01:55:16.265516 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:16.813860 (+548344us) service_pool.cc:224] Handling call
0808 01:55:16.814322 (+   462us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:17.036349 (+222027us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:17.036382 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:17.047503 (+ 11121us) log.cc:844] Serialized 26 byte log entry
0808 01:55:17.445041 (+397538us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:17.508644 (+ 63603us) log.cc:844] Serialized 14 byte log entry
0808 01:55:17.509098 (+   454us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:17.509704 (+   606us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:18.870178 (+1360474us) spinlock_profiling.cc:243] Waited 8.97 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:55:19.574755 (+704577us) raft_consensus.cc:1685] finished
0808 01:55:19.574877 (+   122us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:19.575000 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":75028,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":445,"raft.queue_time_us":30338,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":1512,"spinlock_wait_cycles":20620032,"thread_start_us":607160,"threads_started":3,"wal-append.queue_time_us":405076}
I20260808 01:55:19.594942 20524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:19.595337 21884 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:19.526401 21747 leader_election.cc:290] T 0c59923b4b8949b0ae2f90ac80346e42 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:19.595686 20523 raft_consensus.cc:2468] T ecc85af5215440be884412c1d240c7af P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
    @           0x36ac16 kudu::master::LeaderRebalancerTest_RestartTserver_Test::TestBody() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_leader_rebalancer-test.cc:?
I20260808 01:55:19.745215 20523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:19.746608 20606 rpcz_store.cc:269] 0808 01:55:17.489907 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.566475 (+2076568us) service_pool.cc:224] Handling call
0808 01:55:19.567680 (+  1205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.596421 21879 raft_consensus.cc:515] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:19.810220 21879 leader_election.cc:290] T aecd7bfdd8a746239184356052321a48 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:19.597059 21716 consensus_queue.cc:1048] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
W20260808 01:55:19.598127 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 133095us
I20260808 01:55:19.813578 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:19.813834 20726 leader_election.cc:304] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:19.598740 21883 leader_election.cc:290] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:19.816246 21755 consensus_queue.cc:1059] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260808 01:55:19.599421 21900 raft_consensus.cc:515] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:19.817371 21755 raft_consensus.cc:2804] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260808 01:55:19.817755 21755 raft_consensus.cc:493] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:19.818075 21755 raft_consensus.cc:3060] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:19.818965 21900 leader_election.cc:290] T f275a817cc374438aa23f63da2a627b9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:19.589248 21893 raft_consensus.cc:515] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:19.819473 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:19.820286 20761 raft_consensus.cc:2468] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 4.
W20260808 01:55:19.590389 20492 leader_election.cc:336] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.628s (SENT)
I20260808 01:55:19.588500 21889 leader_election.cc:290] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:19.600535 20424 catalog_manager.cc:5697] T bf6d6021e1384c4eafe94f910b25d517 P df82e7f402364d369ed1ec3c5a8772b3 reported cstate change: term changed from 2 to 3, leader changed from 598db12c38d442c6988b4c4f991874fb (127.7.165.195) to df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196). New cstate: current_term: 3 leader_uuid: "df82e7f402364d369ed1ec3c5a8772b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: HEALTHY } } }
I20260808 01:55:19.822913 20424 catalog_manager.cc:5697] T bcae3a1b04b44ebfa3776d88ca63cbdb P df82e7f402364d369ed1ec3c5a8772b3 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "df82e7f402364d369ed1ec3c5a8772b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:19.824848 20424 catalog_manager.cc:5697] T 0c59923b4b8949b0ae2f90ac80346e42 P df82e7f402364d369ed1ec3c5a8772b3 reported cstate change: term changed from 3 to 4, leader changed from a80df9f992014374aeef30979cfe393e (127.7.165.193) to df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196). New cstate: current_term: 4 leader_uuid: "df82e7f402364d369ed1ec3c5a8772b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } health_report { overall_health: UNKNOWN } } }
I20260808 01:55:19.826027 20601 raft_consensus.cc:2468] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 4.
I20260808 01:55:19.601795 21877 raft_consensus.cc:2749] T 63fb8c2f494a462686500ce10025a39f P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 01:55:19.818962 21811 raft_consensus.cc:3055] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Stepping down as leader of term 1
W20260808 01:55:19.601935 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 788748us
I20260808 01:55:19.827174 21811 raft_consensus.cc:740] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
W20260808 01:55:19.827641 20648 leader_election.cc:336] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.597s (SENT)
I20260808 01:55:19.828126 20648 leader_election.cc:304] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:19.829012 21877 raft_consensus.cc:2749] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260808 01:55:19.829538 20601 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 231) took 2952 ms (client timeout 2131 ms). Trace:
I20260808 01:55:19.562291 21430 leader_election.cc:290] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:19.588229 20677 rpcz_store.cc:276] 0808 01:55:16.442331 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.020116 (+577785us) service_pool.cc:224] Handling call
0808 01:55:17.055618 (+ 35502us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:17.099563 (+ 43945us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:17.100524 (+   961us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:17.135253 (+ 34729us) log.cc:844] Serialized 26 byte log entry
0808 01:55:17.397152 (+261899us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:17.446876 (+ 49724us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:17.455045 (+  8169us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:18.801232 (+1346187us) spinlock_profiling.cc:243] Waited 1.25 ms on lock 0x7b440002250c. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d41715981 00007f5d417b3855 00007f5d417b4272 00007f5d4453e3bb 00007f5d44539c9f 00007f5d4452da5e 00007f5d4452cda6 00007f5d4452c9c8 00007f5d44536411 00007f5d445363c9 00007f5d44536359
0808 01:55:19.570395 (+769163us) raft_consensus.cc:1685] finished
0808 01:55:19.570425 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:19.570661 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":89272,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":16114,"spinlock_wait_cycles":2871680,"thread_start_us":98552,"threads_started":1,"wal-append.queue_time_us":342520}
I20260808 01:55:19.831548 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:19.832261 20677 raft_consensus.cc:2468] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:19.833334 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:19.834007 20677 raft_consensus.cc:2468] T 7170feed84a946ba9e383d9bfc614e61 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:19.604005 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:19.605067 21802 leader_election.cc:290] T 0bd11a6553c444d488651367f3f0143f P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:19.606236 21798 leader_election.cc:290] T 51bbd1ceb9fb4045b31d153ff959c663 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:19.606731 20607 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1753378f13594e89aee6641e2cb9e6da" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
W20260808 01:55:19.835253 20605 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 224) took 2842 ms (client timeout 2764 ms). Trace:
W20260808 01:55:19.835747 20605 rpcz_store.cc:269] 0808 01:55:16.992867 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.603702 (+2610835us) service_pool.cc:224] Handling call
0808 01:55:19.835164 (+231462us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.835855 20607 raft_consensus.cc:2468] T 1753378f13594e89aee6641e2cb9e6da P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 1.
W20260808 01:55:19.836966 20607 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 220) took 2846 ms (client timeout 2966 ms). Trace:
W20260808 01:55:19.837308 20607 rpcz_store.cc:269] 0808 01:55:16.991049 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.606424 (+2615375us) service_pool.cc:224] Handling call
0808 01:55:19.836884 (+230460us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.607465 21791 raft_consensus.cc:3055] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Stepping down as leader of term 1
I20260808 01:55:19.607713 21763 leader_election.cc:290] T b9a194d9a9eb40d397c76e44235f7aed P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:19.594952 21850 leader_election.cc:290] T a1af0d1fa74847a390b02d2313464926 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:19.589376 20529 rpcz_store.cc:276] 0808 01:55:17.568000 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.568337 (+   337us) service_pool.cc:224] Handling call
0808 01:55:17.568784 (+   447us) raft_consensus.cc:1471] Updating replica for 2 ops
0808 01:55:17.569312 (+   528us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:17.569366 (+    54us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0808 01:55:17.570695 (+  1329us) log.cc:844] Serialized 52 byte log entry
0808 01:55:17.753417 (+182722us) spinlock_profiling.cc:243] Waited 4.85 ms on lock 0x7b1800365f20. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d4a7a8f31 00007f5d4a7b56d1 00007f5d4186f7b7 00007f5d4a717fc9 00007f5d4188bfdb 00007f5d4188a2db 00007f5d4188c542 00007f5d4452c3fb 00007f5d44531f13 00007f5d445322c1 00007f5d4494edef
0808 01:55:19.567382 (+1813965us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:19.567424 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.567490 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:19.567528 (+    38us) raft_consensus.cc:1685] finished
0808 01:55:19.567539 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:19.567755 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":504,"raft.queue_time_us":1732,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":9564,"spinlock_wait_cycles":11166464,"thread_start_us":1813743,"threads_started":1,"wal-append.queue_time_us":321695,"wal-append.run_cpu_time_us":2244,"wal-append.run_wall_time_us":1002261}
    @     0x7f5d41abfe00 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
I20260808 01:55:19.613682 21754 leader_election.cc:290] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Requested vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:19.614172 20604 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 223) took 2622 ms (client timeout 2520 ms). Trace:
I20260808 01:55:19.614990 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7170feed84a946ba9e383d9bfc614e61" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:19.582441 20609 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44770 (request call id 273) took 2120 ms (client timeout 2143 ms). Trace:
I20260808 01:55:19.615643 20530 raft_consensus.cc:2468] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 4.
I20260808 01:55:19.616235 20685 raft_consensus.cc:1240] T 0add41ce95b74822ab26d48b72ccafe0 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Rejecting Update request from peer a80df9f992014374aeef30979cfe393e for earlier term 3. Current term is 4. Ops: []
I20260808 01:55:19.486380 21674 leader_election.cc:290] T 35a1e6afbae44a1db1e2a0e1f743b290 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:19.574460 20678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 248) took 1841 ms. Trace:
I20260808 01:55:19.616981 20526 raft_consensus.cc:2410] T 70e7899e1afa4d37a1a8bd26dfa9d803 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260808 01:55:19.618338 21824 raft_consensus.cc:493] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:19.619958 21894 raft_consensus.cc:493] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:19.621968 20683 raft_consensus.cc:1240] T 349d2dad09a3492ca13f8784fbdb961b P 598db12c38d442c6988b4c4f991874fb [term 5 LEADER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 4. Current term is 5. Ops: []
I20260808 01:55:19.624277 21899 raft_consensus.cc:515] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:19.627612 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f681a9e4ae5a4909afeec36dfb20b490" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:19.459295 21861 leader_election.cc:290] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:19.629848 20682 raft_consensus.cc:1275] T aade7cc941de4b5781fb940a6625beb3 P 598db12c38d442c6988b4c4f991874fb [term 5 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260808 01:55:19.592234 21885 leader_election.cc:290] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:19.631017 21886 raft_consensus.cc:493] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df82e7f402364d369ed1ec3c5a8772b3)
I20260808 01:55:19.631235 20525 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41392 (request call id 199) took 1931 ms. Trace:
I20260808 01:55:19.641073 20602 raft_consensus.cc:2468] T bcae3a1b04b44ebfa3776d88ca63cbdb P 9bfb29168c8947e1b5816f1572d040cc [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 5.
I20260808 01:55:19.647579 20680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 247) took 1915 ms. Trace:
W20260808 01:55:19.649425 20725 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 1.38573e+06us
W20260808 01:55:19.649394 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 70408.3us
I20260808 01:55:19.737464 21659 raft_consensus.cc:2804] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 01:55:19.747462 20523 raft_consensus.cc:1240] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 5 FOLLOWER]: Rejecting Update request from peer df82e7f402364d369ed1ec3c5a8772b3 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260808 01:55:19.738463 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:19.747905 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:19.744122 20524 raft_consensus.cc:2468] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 2.
I20260808 01:55:19.753782 20570 leader_election.cc:304] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:19.741525 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1753378f13594e89aee6641e2cb9e6da" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:19.760385 20684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:55:19.762823 20572 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:19.762308 20758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3997511f7d7419985d678e8fa59e5d0" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:19.763916 20760 raft_consensus.cc:1240] T 38b9406154bf4f86bf631ea9b73424df P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 3. Current term is 4. Ops: []
I20260808 01:55:19.765504 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:19.766253 20608 raft_consensus.cc:2410] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:55:19.767132 21903 raft_consensus.cc:515] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:19.767861 20757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:19.768014 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:19.770455 20754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf6d6021e1384c4eafe94f910b25d517" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:19.787329 20520 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41392 (request call id 204) took 2083 ms. Trace:
I20260808 01:55:19.789851 20762 raft_consensus.cc:1275] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260808 01:55:19.791718 20753 raft_consensus.cc:1240] T f2b22456199b4e5f95a1eb974dd194fb P df82e7f402364d369ed1ec3c5a8772b3 [term 4 LEADER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 3. Current term is 4. Ops: []
I20260808 01:55:19.792332 20686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b36b5d19af469885eddc3fa63ce410" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:19.821357 20761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e336b4e46ba6416e941f51812c6c38e7" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
W20260808 01:55:19.829938 20601 rpcz_store.cc:269] 0808 01:55:16.876898 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.853541 (+976643us) service_pool.cc:224] Handling call
0808 01:55:19.829328 (+1975787us) spinlock_profiling.cc:243] Waited 438 ms on lock 0x7b5400afcdd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:19.829460 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1006566272}
I20260808 01:55:19.838769 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44768 (request call id 220) took 2992 ms. Trace:
I20260808 01:55:19.838877 21791 raft_consensus.cc:740] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:19.884300 20607 rpcz_store.cc:276] 0808 01:55:16.846425 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.837953 (+2991528us) service_pool.cc:224] Handling call
0808 01:55:19.838247 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:19.838447 (+   200us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:19.838470 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:19.838509 (+    39us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:19.838539 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.838591 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:19.838715 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.885347 21791 consensus_queue.cc:260] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:19.886744 21791 raft_consensus.cc:3060] T 21214e47c3d34850ae5c4202d6761065 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:19.840112 20756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de62f661f1484ca9af89dc7df85f8b9b" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:19.840611 21755 raft_consensus.cc:515] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:19.841359 20528 raft_consensus.cc:2468] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
W20260808 01:55:19.841576 20609 rpcz_store.cc:269] 0808 01:55:17.461833 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.549437 (+2087604us) service_pool.cc:224] Handling call
0808 01:55:19.582360 (+ 32923us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.891590 21755 leader_election.cc:290] T 8c4d9d89ee1743829635451925a89c9e P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:19.839737 20755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51bbd1ceb9fb4045b31d153ff959c663" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df82e7f402364d369ed1ec3c5a8772b3" is_pre_election: true
I20260808 01:55:19.892468 20755 raft_consensus.cc:2468] T 51bbd1ceb9fb4045b31d153ff959c663 P df82e7f402364d369ed1ec3c5a8772b3 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 3.
I20260808 01:55:19.843389 20685 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46954 (request call id 298) took 2444 ms. Trace:
I20260808 01:55:19.893724 20685 rpcz_store.cc:276] 0808 01:55:17.398871 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.456728 (+2057857us) service_pool.cc:224] Handling call
0808 01:55:19.457071 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:19.842492 (+385421us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.843169 (+   677us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b5400316fd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:19.843331 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":365586176}
I20260808 01:55:19.843550 20530 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41396 (request call id 294) took 2099 ms. Trace:
W20260808 01:55:19.845505 20571 connection.cc:256] RPC call timeout handler was delayed by -1.314204045018414s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1676s for a response
W20260808 01:55:19.896005 20571 connection.cc:256] RPC call timeout handler was delayed by -1.2008040450164117s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18020000000000003s for a response
I20260808 01:55:19.895648 20530 rpcz_store.cc:276] 0808 01:55:17.744110 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.842767 (+2098657us) service_pool.cc:224] Handling call
0808 01:55:19.843049 (+   282us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:19.843241 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:19.843264 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:19.843305 (+    41us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:19.843342 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.843398 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:19.843489 (+    91us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.847497 20683 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46972 (request call id 244) took 2285 ms. Trace:
I20260808 01:55:19.897274 20683 rpcz_store.cc:276] 0808 01:55:17.561874 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.510541 (+1948667us) service_pool.cc:224] Handling call
0808 01:55:19.621778 (+111237us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:19.847221 (+225443us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.847434 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.852468 20682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1753378f13594e89aee6641e2cb9e6da" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:19.898382 20682 raft_consensus.cc:2468] T 1753378f13594e89aee6641e2cb9e6da P 598db12c38d442c6988b4c4f991874fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 1.
W20260808 01:55:19.858403 20725 connection.cc:256] RPC call timeout handler was delayed by -1.591742564982269s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19290000000000002s for a response
W20260808 01:55:19.899746 20725 connection.cc:256] RPC call timeout handler was delayed by -1.5872425650013611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19340000000000002s for a response
W20260808 01:55:19.900326 20725 connection.cc:256] RPC call timeout handler was delayed by -1.5733095599862281s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1713s for a response
W20260808 01:55:19.901038 20725 leader_election.cc:336] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.757s (SENT)
I20260808 01:55:19.901638 20725 leader_election.cc:304] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:19.843852 20678 rpcz_store.cc:276] 0808 01:55:17.732921 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.573537 (+1840616us) service_pool.cc:224] Handling call
0808 01:55:19.573897 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:19.574106 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:19.574143 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:19.574193 (+    50us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:55:19.574227 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.574294 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:19.574403 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:55:19.860546 20600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 212) took 2880 ms (client timeout 3047 ms). Trace:
W20260808 01:55:19.854270 20602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44768 (request call id 236) took 2920 ms (client timeout 1909 ms). Trace:
I20260808 01:55:19.853148 21886 raft_consensus.cc:515] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
W20260808 01:55:19.903123 20600 rpcz_store.cc:269] 0808 01:55:16.979787 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.738093 (+2758306us) service_pool.cc:224] Handling call
0808 01:55:19.860452 (+122359us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.904424 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce51fcffe2ae4bcba55cd864c973bf0d" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:19.861192 20606 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 227) took 2867 ms. Trace:
I20260808 01:55:19.849898 20603 raft_consensus.cc:2410] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260808 01:55:19.865659 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 111981us
W20260808 01:55:19.907289 20603 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 221) took 2915 ms (client timeout 3006 ms). Trace:
W20260808 01:55:19.907738 20603 rpcz_store.cc:269] 0808 01:55:16.991450 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.627293 (+2635843us) service_pool.cc:224] Handling call
0808 01:55:19.907218 (+279925us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:55:19.869813 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 56067us
I20260808 01:55:19.909168 20570 leader_election.cc:304] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:19.821363 21893 leader_election.cc:290] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:19.910188 21893 raft_consensus.cc:2804] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:19.867813 20605 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44768 (request call id 217) took 3022 ms. Trace:
I20260808 01:55:19.910699 21893 raft_consensus.cc:493] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:19.911237 21893 raft_consensus.cc:3060] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:19.871378 20684 raft_consensus.cc:2410] T d3997511f7d7419985d678e8fa59e5d0 P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20260808 01:55:19.874528 21798 raft_consensus.cc:2804] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:19.914024 21798 raft_consensus.cc:493] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:19.914371 21798 raft_consensus.cc:3060] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:19.858997 21659 raft_consensus.cc:493] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:19.906402 20606 rpcz_store.cc:276] 0808 01:55:16.993790 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.747590 (+2753800us) service_pool.cc:224] Handling call
0808 01:55:19.861105 (+113515us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.919397 20530 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022d6236053d4519bff37cbef43a6170" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:19.920116 20530 raft_consensus.cc:2468] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 4.
W20260808 01:55:19.879753 20608 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44780 (request call id 207) took 2902 ms (client timeout 2402 ms). Trace:
W20260808 01:55:19.923391 20608 rpcz_store.cc:269] 0808 01:55:16.977856 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.174514 (+2196658us) service_pool.cc:224] Handling call
0808 01:55:19.879594 (+705080us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b540027e7d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:19.879696 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":518764288}
W20260808 01:55:19.839323 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 248941us
I20260808 01:55:19.925297 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:19.840466 20529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41396 (request call id 293) took 2120 ms. Trace:
    @     0x7f5d41a9da02 testing::Test::Run() at ??:0
    @     0x7f5d41a9ed19 testing::TestInfo::Run() at ??:0
    @     0x7f5d41a9fcf5 testing::TestSuite::Run() at ??:0
W20260808 01:55:19.868613 20648 leader_election.cc:336] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.713s (SENT)
I20260808 01:55:20.134804 20648 leader_election.cc:304] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
W20260808 01:55:20.136062 20648 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 267472us
W20260808 01:55:20.136770 20648 leader_election.cc:336] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.775s (SENT)
I20260808 01:55:20.137177 20648 leader_election.cc:304] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 598db12c38d442c6988b4c4f991874fb; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
I20260808 01:55:19.877079 20757 raft_consensus.cc:2410] T de206cb1608f4bd1b20ae902c82f849f P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260808 01:55:19.878523 20762 pending_rounds.cc:85] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3: Aborting all ops after (but not including) 1
I20260808 01:55:20.138151 20762 pending_rounds.cc:107] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3: Aborting uncommitted NO_OP operation due to leader change: 2.2
W20260808 01:55:19.925385 20492 leader_election.cc:336] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.693s (SENT)
I20260808 01:55:20.138942 20492 leader_election.cc:304] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:20.139210 21861 raft_consensus.cc:2749] T de62f661f1484ca9af89dc7df85f8b9b P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260808 01:55:20.140153 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 214793us
I20260808 01:55:20.140844 21850 raft_consensus.cc:493] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:19.875810 20493 leader_election.cc:304] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:20.141649 21900 raft_consensus.cc:2749] T b8135c0503a8480998454f58362956e5 P a80df9f992014374aeef30979cfe393e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260808 01:55:19.924547 20725 connection.cc:256] RPC call timeout handler was delayed by -1.5661462959833443s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1897s for a response
I20260808 01:55:20.138481 20762 raft_consensus.cc:2889] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:55:20.142618 21885 raft_consensus.cc:2804] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260808 01:55:20.143177 21885 raft_consensus.cc:493] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 598db12c38d442c6988b4c4f991874fb)
I20260808 01:55:20.143581 21885 raft_consensus.cc:3060] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Advancing to term 4
I20260808 01:55:20.141656 21850 raft_consensus.cc:515] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:19.926750 20529 rpcz_store.cc:276] 0808 01:55:17.720493 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.839653 (+2119160us) service_pool.cc:224] Handling call
0808 01:55:19.839947 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:19.840122 (+   175us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:19.840145 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:19.840194 (+    49us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:19.840235 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.840300 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:19.840406 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:20.146330 21850 leader_election.cc:290] T 07cea569c9ad4d18a8d2f97764e15cc9 P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:20.147065 21921 raft_consensus.cc:2749] T 0590088b74d04c6b96146266db7a6266 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260808 01:55:20.147557 20492 leader_election.cc:336] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.952s (SENT)
    @     0x7f5d41ab47e5 testing::internal::UnitTestImpl::RunAllTests() at ??:0
    @     0x7f5d41ac0cc0 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
    @     0x7f5d41ab3d9d testing::UnitTest::Run() at ??:0
    @           0x3aa2bc RUN_ALL_TESTS() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:?
I20260808 01:55:20.148165 20492 leader_election.cc:304] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 598db12c38d442c6988b4c4f991874fb, 9bfb29168c8947e1b5816f1572d040cc
    @           0x3a8ebd main at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:?
I20260808 01:55:20.338332 21900 raft_consensus.cc:2749] T 7170feed84a946ba9e383d9bfc614e61 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260808 01:55:20.338562 20492 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.194:41191, user_credentials={real_user=slave}} blocked reactor thread for 191032us
W20260808 01:55:20.339366 20492 leader_election.cc:336] T 35a1e6afbae44a1db1e2a0e1f743b290 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.996s (SENT)
I20260808 01:55:19.879055 20527 raft_consensus.cc:2393] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 5: Already voted for candidate a80df9f992014374aeef30979cfe393e in this term.
W20260808 01:55:20.340903 20492 leader_election.cc:336] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 2.212s (SENT)
I20260808 01:55:20.341449 20492 leader_election.cc:304] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:19.828147 21811 consensus_queue.cc:260] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:20.342583 20528 raft_consensus.cc:1240] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [term 5 FOLLOWER]: Rejecting Update request from peer 9bfb29168c8947e1b5816f1572d040cc for earlier term 4. Current term is 5. Ops: []
I20260808 01:55:19.840600 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e7899e1afa4d37a1a8bd26dfa9d803" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:19.907281 21653 raft_consensus.cc:493] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a80df9f992014374aeef30979cfe393e)
I20260808 01:55:19.910831 20605 rpcz_store.cc:276] 0808 01:55:16.845529 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.836884 (+2991355us) service_pool.cc:224] Handling call
0808 01:55:19.837289 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:19.837561 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:19.837597 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:19.838194 (+   597us) log.cc:844] Serialized 26 byte log entry
0808 01:55:19.867385 (+ 29191us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:19.867447 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.867533 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:19.867590 (+    57us) raft_consensus.cc:1685] finished
0808 01:55:19.867608 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:19.867744 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":28751,"threads_started":1,"wal-append.queue_time_us":25337}
I20260808 01:55:20.344818 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0add41ce95b74822ab26d48b72ccafe0" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:20.344596 21653 raft_consensus.cc:515] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:20.345645 20605 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "13ab4fe0e17342379d043078e08ca66b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:20.345992 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4d9d89ee1743829635451925a89c9e" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
W20260808 01:55:19.840994 20604 rpcz_store.cc:269] 0808 01:55:16.992580 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.546577 (+2553997us) service_pool.cc:211] Skipping call since client already timed out
0808 01:55:19.614088 (+ 67511us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260808 01:55:20.346603 20528 raft_consensus.cc:2468] T 8c4d9d89ee1743829635451925a89c9e P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:20.342152 20527 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41392 (request call id 208) took 1396 ms. Trace:
I20260808 01:55:20.347496 20604 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e71aab2ce9b45369b7eff65e1645f9b" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:19.950625 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35a1e6afbae44a1db1e2a0e1f743b290" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:19.846071 21824 raft_consensus.cc:515] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:20.348912 20603 raft_consensus.cc:2410] T 35a1e6afbae44a1db1e2a0e1f743b290 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260808 01:55:20.350097 20603 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de206cb1608f4bd1b20ae902c82f849f" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:20.350769 20603 raft_consensus.cc:3060] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:19.847033 21894 raft_consensus.cc:515] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:20.354512 21894 leader_election.cc:290] T 611b7203dac24e2e909dee3e0fc5fda7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:19.882107 20601 raft_consensus.cc:1275] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260808 01:55:20.355465 20601 pending_rounds.cc:85] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc: Aborting all ops after (but not including) 1
I20260808 01:55:20.355849 20601 pending_rounds.cc:107] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260808 01:55:20.347496 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aecd7bfdd8a746239184356052321a48" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:20.356168 20601 raft_consensus.cc:2889] T 48b36b5d19af469885eddc3fa63ce410 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:55:20.357558 20601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44768 (request call id 224) took 3483 ms. Trace:
I20260808 01:55:19.849218 21899 leader_election.cc:290] T 566d023f94bb449d8cf248744cd5b6ba P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:19.878372 20520 rpcz_store.cc:276] 0808 01:55:17.703589 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.751021 (+2047432us) service_pool.cc:224] Handling call
0808 01:55:19.751510 (+   489us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:19.751800 (+   290us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:55:19.782622 (+ 30822us) log.cc:844] Serialized 14 byte log entry
0808 01:55:19.786845 (+  4223us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:19.786916 (+    71us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:55:19.786974 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.787058 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:19.787244 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":30147,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":127,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"thread_start_us":33696,"threads_started":2,"wal-append.queue_time_us":2353}
I20260808 01:55:19.952536 21802 consensus_queue.cc:1048] T de62f661f1484ca9af89dc7df85f8b9b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:20.358919 21900 raft_consensus.cc:2749] T 022d6236053d4519bff37cbef43a6170 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260808 01:55:20.359565 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aade7cc941de4b5781fb940a6625beb3" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 6 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:20.360908 20520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1af0d1fa74847a390b02d2313464926" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:20.361510 20520 raft_consensus.cc:2468] T a1af0d1fa74847a390b02d2313464926 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 4.
I20260808 01:55:19.845085 21776 raft_consensus.cc:493] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:19.954371 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38b9406154bf4f86bf631ea9b73424df" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:20.362433 21776 raft_consensus.cc:515] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } }
I20260808 01:55:20.363217 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63fb8c2f494a462686500ce10025a39f" candidate_uuid: "a80df9f992014374aeef30979cfe393e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:20.364493 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9201fed5ac044119ff493c519d289a7" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:20.364847 21776 leader_election.cc:290] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:20.365787 20607 raft_consensus.cc:1275] T 8fc615366d544af38cf664c9ad47ab0c P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260808 01:55:20.367261 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44768 (request call id 226) took 3492 ms. Trace:
I20260808 01:55:20.367600 20607 rpcz_store.cc:276] 0808 01:55:16.875201 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.885636 (+3010435us) service_pool.cc:224] Handling call
0808 01:55:19.885968 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:20.366362 (+480394us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:20.367016 (+   654us) spinlock_profiling.cc:243] Waited 479 ms on lock 0x7b54009d9254. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449941e1 00007f5d44992c1e 00007f5d49989229 00007f5d42eced34 00007f5d42ececc2 00007f5d42ecebf1 00007f5d42eceb6c 00007f5d42ecde22 00007f5d42ccd07c 00007f5d42ccc4b6
0808 01:55:20.367201 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1102615936}
I20260808 01:55:19.925736 21659 raft_consensus.cc:3060] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:19.955415 20600 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:19.955783 20524 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41392 (request call id 213) took 1008 ms. Trace:
W20260808 01:55:19.903307 20602 rpcz_store.cc:269] 0808 01:55:16.933727 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.677252 (+743525us) service_pool.cc:224] Handling call
0808 01:55:19.854045 (+2176793us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b54007bb1d4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:19.854182 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4289080192}
I20260808 01:55:20.376791 20600 raft_consensus.cc:2468] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df82e7f402364d369ed1ec3c5a8772b3 in term 3.
I20260808 01:55:19.859764 20523 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41392 (request call id 201) took 2159 ms. Trace:
I20260808 01:55:20.378108 20523 rpcz_store.cc:276] 0808 01:55:17.700167 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.746645 (+2046478us) service_pool.cc:224] Handling call
0808 01:55:19.747121 (+   476us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:19.859315 (+112194us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.859667 (+   352us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:19.878737 21903 leader_election.cc:290] T 054af38e4814474c92c93f3f09db43a3 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 5 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191)
I20260808 01:55:19.877542 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45432e629d3b49abab9715ce5c0124d1" candidate_uuid: "9bfb29168c8947e1b5816f1572d040cc" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:19.966923 20606 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f275a817cc374438aa23f63da2a627b9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc" is_pre_election: true
I20260808 01:55:19.843856 21889 raft_consensus.cc:493] T 13ab4fe0e17342379d043078e08ca66b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:19.845595 21738 raft_consensus.cc:493] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:19.845629 20526 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41396 (request call id 295) took 1996 ms. Trace:
I20260808 01:55:19.954716 21755 consensus_queue.cc:1059] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.020s
I20260808 01:55:19.974052 21886 leader_election.cc:290] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
I20260808 01:55:20.001832 20571 leader_election.cc:304] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc; no voters: 598db12c38d442c6988b4c4f991874fb, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:19.853293 20525 rpcz_store.cc:276] 0808 01:55:17.699488 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:17.702883 (+  3395us) service_pool.cc:224] Handling call
0808 01:55:17.718428 (+ 15545us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:17.718752 (+   324us) raft_consensus.cc:1534] Early marking committed up to index 1
0808 01:55:17.833801 (+115049us) log.cc:844] Serialized 14 byte log entry
0808 01:55:19.623726 (+1789925us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:19.630757 (+  7031us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:19.630835 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.630960 (+   125us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:19.631148 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":153643,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":1044,"raft.queue_time_us":23283,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":103,"thread_start_us":1901134,"threads_started":3,"wal-append.queue_time_us":73242,"wal-append.run_cpu_time_us":1336,"wal-append.run_wall_time_us":1001344}
W20260808 01:55:19.995720 20573 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:19.854429 20680 rpcz_store.cc:276] 0808 01:55:17.732560 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.541684 (+1809124us) service_pool.cc:224] Handling call
0808 01:55:19.542080 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:19.542318 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 2
0808 01:55:19.542661 (+   343us) log.cc:844] Serialized 14 byte log entry
0808 01:55:19.647074 (+104413us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:19.647156 (+    82us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:19.647217 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.647312 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:19.647502 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":8098,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":149,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":127,"thread_start_us":54151,"threads_started":1,"wal-append.queue_time_us":52744}
I20260808 01:55:20.008867 21893 raft_consensus.cc:515] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:19.995759 20609 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44768 (request call id 233) took 3118 ms. Trace:
I20260808 01:55:20.017096 21798 raft_consensus.cc:515] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
W20260808 01:55:20.027937 20726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 96933.6us
W20260808 01:55:20.030352 20570 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.165.193:34531, user_credentials={real_user=slave}} blocked reactor thread for 69046.9us
I20260808 01:55:20.122371 20424 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48860 (request call id 75) took 8326 ms. Trace:
I20260808 01:55:19.881011 20761 raft_consensus.cc:2468] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:20.140144 21861 raft_consensus.cc:2749] T 711b7cabb802484094416a0e94a14803 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260808 01:55:19.879034 20679 raft_consensus.cc:2393] T de206cb1608f4bd1b20ae902c82f849f P 598db12c38d442c6988b4c4f991874fb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9bfb29168c8947e1b5816f1572d040cc in current term 3: Already voted for candidate 598db12c38d442c6988b4c4f991874fb in this term.
I20260808 01:55:20.378583 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44768 (request call id 241) took 3443 ms. Trace:
W20260808 01:55:20.142313 20725 connection.cc:256] RPC call timeout handler was delayed by -1.5661462959833443s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1897s for a response
W20260808 01:55:20.387982 20725 connection.cc:256] RPC call timeout handler was delayed by -1.5103462959814351s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19590000000000002s for a response
W20260808 01:55:20.388602 20725 leader_election.cc:336] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 1.838s (SENT)
I20260808 01:55:20.388356 21659 raft_consensus.cc:515] T 6d32821d9c17456394a662fcfd05f6c6 P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:20.389209 20725 leader_election.cc:304] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df82e7f402364d369ed1ec3c5a8772b3; no voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e
W20260808 01:55:20.151717 20631 debug-util.cc:398] Leaking SignalData structure 0x7b08002e9680 after lost signal to thread 7831
I20260808 01:55:20.347992 20527 rpcz_store.cc:276] 0808 01:55:18.945379 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.584980 (+639601us) service_pool.cc:224] Handling call
0808 01:55:20.341843 (+756863us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x7b54003f1bd4. stack: 00007f5d4185acf3 00007f5d4185abf0 00007f5d411e5574 00007f5d4af7b6c4 00007f5d4af7b659 00007f5d448fc951 00007f5d449976da 00007f5d4998a927 00007f5d42ed3834 00007f5d42ed37c2 00007f5d42ed36f1 00007f5d42ed366c 00007f5d42ed2922 00007f5d42ccd07c 00007f5d42ccc4b6 00007f5d42ccbe5f
0808 01:55:20.342075 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":673321728}
W20260808 01:55:20.391070 20631 debug-util.cc:398] Leaking SignalData structure 0x7b08002e96a0 after lost signal to thread 7834
I20260808 01:55:20.348222 21653 leader_election.cc:290] T f681a9e4ae5a4909afeec36dfb20b490 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
W20260808 01:55:20.392419 20631 debug-util.cc:398] Leaking SignalData structure 0x7b08002e6d80 after lost signal to thread 20461
I20260808 01:55:20.379711 20606 raft_consensus.cc:2393] T f275a817cc374438aa23f63da2a627b9 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df82e7f402364d369ed1ec3c5a8772b3 in current term 3: Already voted for candidate 9bfb29168c8947e1b5816f1572d040cc in this term.
W20260808 01:55:20.396695 20492 leader_election.cc:336] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9bfb29168c8947e1b5816f1572d040cc (127.7.165.194:41191): Timed out: RequestConsensusVote RPC to 127.7.165.194:41191 timed out after 2.044s (SENT)
I20260808 01:55:20.397305 20492 leader_election.cc:304] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a80df9f992014374aeef30979cfe393e; no voters: 9bfb29168c8947e1b5816f1572d040cc, df82e7f402364d369ed1ec3c5a8772b3
I20260808 01:55:20.164702 20530 raft_consensus.cc:1275] T 349d2dad09a3492ca13f8784fbdb961b P a80df9f992014374aeef30979cfe393e [term 5 FOLLOWER]: Refusing update from remote peer 598db12c38d442c6988b4c4f991874fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260808 01:55:20.398227 21886 raft_consensus.cc:2749] T e336b4e46ba6416e941f51812c6c38e7 P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260808 01:55:20.198238 20522 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c59923b4b8949b0ae2f90ac80346e42" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 5 candidate_status { last_received { term: 4 index: 1 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e"
I20260808 01:55:20.385914 20570 leader_election.cc:304] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bfb29168c8947e1b5816f1572d040cc, a80df9f992014374aeef30979cfe393e; no voters: 
I20260808 01:55:20.399494 20522 raft_consensus.cc:3060] T 0c59923b4b8949b0ae2f90ac80346e42 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:55:20.375041 20524 rpcz_store.cc:276] 0808 01:55:18.947641 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.863983 (+916342us) service_pool.cc:224] Handling call
0808 01:55:19.864650 (+   667us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:19.864989 (+   339us) raft_consensus.cc:1534] Early marking committed up to index 3
0808 01:55:19.887921 (+ 22932us) log.cc:844] Serialized 14 byte log entry
0808 01:55:19.955378 (+ 67457us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:19.955447 (+    69us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:55:19.955492 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.955570 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:19.955714 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":21133,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":7641,"raft.run_cpu_time_us":65,"raft.run_wall_time_us":64,"thread_start_us":57893,"threads_started":1,"wal-append.queue_time_us":83463}
I20260808 01:55:20.387399 20600 rpcz_store.cc:276] 0808 01:55:16.935486 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:20.377645 (+3442159us) service_pool.cc:224] Handling call
0808 01:55:20.377985 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:20.378190 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:20.378223 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:20.378277 (+    54us) raft_consensus.cc:1645] Marking committed up to 3
0808 01:55:20.378312 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:20.378374 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:20.378526 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:55:20.201644 20648 connection.cc:256] RPC call timeout handler was delayed by -1.4710702749725897s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22130000000000002s for a response
I20260808 01:55:20.192427 21885 raft_consensus.cc:515] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:20.402983 20600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44770 (request call id 278) took 2912 ms. Trace:
I20260808 01:55:20.403489 20600 rpcz_store.cc:276] 0808 01:55:17.490196 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:20.401944 (+2911748us) service_pool.cc:224] Handling call
0808 01:55:20.402323 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:20.402541 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:20.402588 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:20.402640 (+    52us) raft_consensus.cc:1645] Marking committed up to 1
0808 01:55:20.402702 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:20.402772 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:20.402919 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:20.404343 21895 consensus_queue.cc:1059] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260808 01:55:20.404724 21885 leader_election.cc:290] T 20ca72baa7bc4572a51de6b57b4dc3e9 P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 4 election: Requested vote from peers 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325), df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117)
W20260808 01:55:20.250716 20572 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:20.405484 21933 raft_consensus.cc:3055] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:55:20.336319 21921 raft_consensus.cc:493] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:20.379522 20681 raft_consensus.cc:2468] T 45432e629d3b49abab9715ce5c0124d1 P 598db12c38d442c6988b4c4f991874fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bfb29168c8947e1b5816f1572d040cc in term 2.
I20260808 01:55:20.406056 21933 raft_consensus.cc:740] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9bfb29168c8947e1b5816f1572d040cc, State: Running, Role: LEADER
I20260808 01:55:20.406330 21921 raft_consensus.cc:515] T fb4dbfa7c99f46d2a70153ce86f409fb P df82e7f402364d369ed1ec3c5a8772b3 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:20.402892 21931 raft_consensus.cc:2804] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260808 01:55:20.381443 20571 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.195:44325, user_credentials={real_user=slave}} blocked reactor thread for 379695us
I20260808 01:55:20.407402 21931 raft_consensus.cc:493] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 01:55:20.407825 21931 raft_consensus.cc:3060] T 022d6236053d4519bff37cbef43a6170 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:55:20.348286 20604 raft_consensus.cc:2468] T 6e71aab2ce9b45369b7eff65e1645f9b P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 2.
I20260808 01:55:20.380105 21738 raft_consensus.cc:515] T 8ac71e66cb4042eb8fe8aafd139af68a P 9bfb29168c8947e1b5816f1572d040cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:20.391067 21802 raft_consensus.cc:3060] T e336b4e46ba6416e941f51812c6c38e7 P df82e7f402364d369ed1ec3c5a8772b3 [term 2 FOLLOWER]: Advancing to term 3
I20260808 01:55:20.358330 20528 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80df9f992014374aeef30979cfe393e" is_pre_election: true
I20260808 01:55:20.370265 20603 raft_consensus.cc:2468] T de206cb1608f4bd1b20ae902c82f849f P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598db12c38d442c6988b4c4f991874fb in term 3.
I20260808 01:55:20.385361 21893 leader_election.cc:290] T 7170feed84a946ba9e383d9bfc614e61 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 election: Requested vote from peers a80df9f992014374aeef30979cfe393e (127.7.165.193:34531), 598db12c38d442c6988b4c4f991874fb (127.7.165.195:44325)
I20260808 01:55:20.411296 20528 raft_consensus.cc:2410] T 07cea569c9ad4d18a8d2f97764e15cc9 P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 598db12c38d442c6988b4c4f991874fb for term 4 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260808 01:55:19.953383 21756 consensus_queue.cc:1059] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20260808 01:55:20.386343 21798 leader_election.cc:290] T e336b4e46ba6416e941f51812c6c38e7 P 9bfb29168c8947e1b5816f1572d040cc [CANDIDATE]: Term 4 election: Requested vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:20.365749 20608 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "28e98a7c32694002b806eddd7007e550" candidate_uuid: "598db12c38d442c6988b4c4f991874fb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "9bfb29168c8947e1b5816f1572d040cc"
I20260808 01:55:20.412535 21929 raft_consensus.cc:3055] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 LEADER]: Stepping down as leader of term 4
I20260808 01:55:19.843705 21430 raft_consensus.cc:493] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 01:55:20.380249 20526 rpcz_store.cc:276] 0808 01:55:17.848936 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.844605 (+1995669us) service_pool.cc:224] Handling call
0808 01:55:19.844947 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:19.845196 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:19.845243 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0808 01:55:19.845308 (+    65us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:19.845356 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.845426 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:19.845550 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260808 01:55:20.357962 20601 rpcz_store.cc:276] 0808 01:55:16.874377 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.881193 (+3006816us) service_pool.cc:224] Handling call
0808 01:55:19.881675 (+   482us) raft_consensus.cc:1471] Updating replica for 0 ops
0808 01:55:20.357279 (+475604us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:20.357501 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260808 01:55:20.142050 20493 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.7.165.196:41117, user_credentials={real_user=slave}} blocked reactor thread for 266399us
I20260808 01:55:20.419523 20607 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44768 (request call id 234) took 3541 ms. Trace:
I20260808 01:55:20.157832 20529 raft_consensus.cc:1275] T 8ac71e66cb4042eb8fe8aafd139af68a P a80df9f992014374aeef30979cfe393e [term 2 FOLLOWER]: Refusing update from remote peer 598db12c38d442c6988b4c4f991874fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260808 01:55:20.382122 21811 raft_consensus.cc:3060] T b6d8bf19c324463fb9903b9981068e9d P df82e7f402364d369ed1ec3c5a8772b3 [term 1 FOLLOWER]: Advancing to term 2
I20260808 01:55:20.383445 21894 raft_consensus.cc:2749] T fdef2db6e2e04aa795d92ad39538d90b P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260808 01:55:20.380561 21755 consensus_queue.cc:1243] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [LEADER]: Peer 598db12c38d442c6988b4c4f991874fb log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260808 01:55:20.420315 20607 rpcz_store.cc:276] 0808 01:55:16.877777 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:20.368580 (+3490803us) service_pool.cc:224] Handling call
0808 01:55:20.368952 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:20.369371 (+   419us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:20.369401 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:20.369956 (+   555us) log.cc:844] Serialized 26 byte log entry
0808 01:55:20.418381 (+ 48425us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:20.418910 (+   529us) log.cc:844] Serialized 14 byte log entry
0808 01:55:20.419173 (+   263us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:20.419242 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:20.419285 (+    43us) raft_consensus.cc:1685] finished
0808 01:55:20.419297 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:20.419437 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":249,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":3089,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":16251,"thread_start_us":47959,"threads_started":1,"wal-append.queue_time_us":42810}
I20260808 01:55:20.422369 21430 raft_consensus.cc:515] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } } peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } }
I20260808 01:55:20.405514 20600 raft_consensus.cc:1275] T f2b22456199b4e5f95a1eb974dd194fb P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:55:20.413007 21929 raft_consensus.cc:740] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df82e7f402364d369ed1ec3c5a8772b3, State: Running, Role: LEADER
I20260808 01:55:20.433069 21929 consensus_queue.cc:260] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598db12c38d442c6988b4c4f991874fb" member_type: VOTER last_known_addr { host: "127.7.165.195" port: 44325 } } peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } }
I20260808 01:55:20.398118 20606 raft_consensus.cc:1275] T 62f797fc263e40c0a0380b65f9430154 P 9bfb29168c8947e1b5816f1572d040cc [term 3 FOLLOWER]: Refusing update from remote peer 598db12c38d442c6988b4c4f991874fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260808 01:55:20.434616 21929 raft_consensus.cc:3060] T d681d5f24f674af0a5024867f8a1fb27 P df82e7f402364d369ed1ec3c5a8772b3 [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:55:20.384529 20609 rpcz_store.cc:276] 0808 01:55:16.877484 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:19.891584 (+3014100us) service_pool.cc:224] Handling call
0808 01:55:19.891947 (+   363us) raft_consensus.cc:1471] Updating replica for 1 ops
0808 01:55:19.892356 (+   409us) raft_consensus.cc:1534] Early marking committed up to index 0
0808 01:55:19.892386 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0808 01:55:19.892923 (+   537us) log.cc:844] Serialized 26 byte log entry
0808 01:55:19.969560 (+ 76637us) raft_consensus.cc:1645] Marking committed up to 2
0808 01:55:19.970654 (+  1094us) log.cc:844] Serialized 14 byte log entry
0808 01:55:19.995023 (+ 24369us) raft_consensus.cc:1697] Filling consensus response to leader.
0808 01:55:19.995151 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0808 01:55:19.995220 (+    69us) raft_consensus.cc:1685] finished
0808 01:55:19.995239 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0808 01:55:19.995669 (+   430us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2741,"prepare.queue_time_us":207,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":62,"raft.queue_time_us":20929,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":245,"thread_start_us":97210,"threads_started":2,"wal-append.queue_time_us":76610}
W20260808 01:55:20.402894 20648 connection.cc:256] RPC call timeout handler was delayed by -1.3466234059887938s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24020000000000002s for a response
I20260808 01:55:20.386059 20424 rpcz_store.cc:276] 0808 01:55:11.796306 (+     0us) service_pool.cc:167] Inserting onto call queue
0808 01:55:11.797880 (+  1574us) service_pool.cc:224] Handling call
0808 01:55:20.122288 (+8324408us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0808 01:55:19.889701 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0808 01:55:19.983223 (+ 93522us) write_op.cc:432] Acquiring schema lock in shared mode
0808 01:55:19.987386 (+  4163us) write_op.cc:435] Acquired schema lock
0808 01:55:19.987411 (+    25us) tablet.cc:662] Decoding operations
0808 01:55:19.988694 (+  1283us) write_op.cc:620] Acquiring the partition lock for write op
0808 01:55:19.988770 (+    76us) write_op.cc:641] Partition lock acquired for write op
0808 01:55:19.988799 (+    29us) tablet.cc:685] Acquiring locks for 6 operations
0808 01:55:19.989985 (+  1186us) tablet.cc:701] Row locks acquired
0808 01:55:19.990070 (+    85us) write_op.cc:260] PREPARE: finished
0808 01:55:19.990253 (+   183us) write_op.cc:270] Start()
0808 01:55:19.995985 (+  5732us) write_op.cc:276] Timestamp: P: 1786154119990198 usec, L: 0
0808 01:55:19.996026 (+    41us) op_driver.cc:348] REPLICATION: starting
0808 01:55:20.016002 (+ 19976us) log.cc:844] Serialized 2477 byte log entry
0808 01:55:20.065883 (+ 49881us) op_driver.cc:464] REPLICATION: finished
0808 01:55:20.079654 (+ 13771us) write_op.cc:301] APPLY: starting
0808 01:55:20.079749 (+    95us) tablet.cc:1367] starting BulkCheckPresence
0808 01:55:20.080029 (+   280us) tablet.cc:1370] finished BulkCheckPresence
0808 01:55:20.080166 (+   137us) tablet.cc:1372] starting ApplyRowOperation cycle
0808 01:55:20.081022 (+   856us) tablet.cc:1383] finished ApplyRowOperation cycle
0808 01:55:20.081295 (+   273us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0808 01:55:20.081340 (+    45us) write_op.cc:312] APPLY: finished
0808 01:55:20.081626 (+   286us) log.cc:844] Serialized 52 byte log entry
0808 01:55:20.090504 (+  8878us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:55:20.090631 (+   127us) write_op.cc:454] Released schema lock
0808 01:55:20.090720 (+    89us) write_op.cc:341] FINISH: Updating metrics
0808 01:55:20.091208 (+   488us) write_op.cc:489] Releasing partition, row and schema locks
0808 01:55:20.091243 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":240011,"reported_tablets":22,"child_traces":[["op",{"apply.queue_time_us":13489,"apply.run_cpu_time_us":2895,"apply.run_wall_time_us":11519,"num_ops":6,"prepare.queue_time_us":41877,"prepare.run_cpu_time_us":32239,"prepare.run_wall_time_us":157580,"raft.queue_time_us":11143,"raft.run_cpu_time_us":4150,"raft.run_wall_time_us":13560,"replication_time_us":50543,"spinlock_wait_cycles":27520,"thread_start_us":77686,"threads_started":4,"wal-append.queue_time_us":31088}]]}
I20260808 01:55:20.407142 21933 consensus_queue.cc:260] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df82e7f402364d369ed1ec3c5a8772b3" member_type: VOTER last_known_addr { host: "127.7.165.196" port: 41117 } } peers { permanent_uuid: "a80df9f992014374aeef30979cfe393e" member_type: VOTER last_known_addr { host: "127.7.165.193" port: 34531 } } peers { permanent_uuid: "9bfb29168c8947e1b5816f1572d040cc" member_type: VOTER last_known_addr { host: "127.7.165.194" port: 41191 } }
I20260808 01:55:20.425297 20677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0bd11a6553c444d488651367f3f0143f" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:20.437438 21933 raft_consensus.cc:3060] T 054af38e4814474c92c93f3f09db43a3 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Advancing to term 5
I20260808 01:55:20.419782 20524 raft_consensus.cc:1275] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: Refusing update from remote peer 9bfb29168c8947e1b5816f1572d040cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260808 01:55:20.438915 20524 pending_rounds.cc:85] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e: Aborting all ops after (but not including) 1
I20260808 01:55:20.439312 20524 pending_rounds.cc:107] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260808 01:55:20.439632 20524 raft_consensus.cc:2889] T e3b0638baa0f4f58b2d6146e98b8648b P a80df9f992014374aeef30979cfe393e [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260808 01:55:20.426579 20680 raft_consensus.cc:1275] T f2b22456199b4e5f95a1eb974dd194fb P 598db12c38d442c6988b4c4f991874fb [term 4 FOLLOWER]: Refusing update from remote peer df82e7f402364d369ed1ec3c5a8772b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260808 01:55:20.405794 20572 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:20.427075 20681 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f797fc263e40c0a0380b65f9430154" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
W20260808 01:55:20.441630 20572 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260808 01:55:20.428263 20679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07cea569c9ad4d18a8d2f97764e15cc9" candidate_uuid: "df82e7f402364d369ed1ec3c5a8772b3" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "598db12c38d442c6988b4c4f991874fb" is_pre_election: true
I20260808 01:55:20.420208 20493 leader_election.cc:304] T fb4dbfa7c99f46d2a70153ce86f409fb P a80df9f992014374aeef30979cfe393e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a80df9f992014374aeef30979cfe393e, df82e7f402364d369ed1ec3c5a8772b3; no voters: 
I20260808 01:55:20.350332 21824 leader_election.cc:290] T fb4dbfa7c99f46d2a70153ce86f409fb P 598db12c38d442c6988b4c4f991874fb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df82e7f402364d369ed1ec3c5a8772b3 (127.7.165.196:41117), a80df9f992014374aeef30979cfe393e (127.7.165.193:34531)
I20260808 01:55:20.432000 20608 raft_consensus.cc:1275] T 4f6c2d3c4da449688d1fca55d391ead2 P 9bfb29168c8947e1b5816f1572d040cc [term 4 FOLLOWER]: Refusing update from remote peer a80df9f992014374aeef30979cfe393e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260808 01:55:20.433434 21674 raft_consensus.cc:3055] T 349d2dad09a3492ca13f8784fbdb961b P df82e7f402364d369ed1ec3c5a8772b3 [term 4 LEADER]: Stepping down as leader of term 4
    @     0x7f5d3e15ac87 __libc_start_main at ??:0
    @           0x2b13aa _start at ??:?